From 80e4bbe415b6e65a41277b749ee4f32c3202ad6c Mon Sep 17 00:00:00 2001 From: Johannes Date: Thu, 2 Jul 2026 10:58:00 -0400 Subject: [PATCH] docs(guide): ClamAV remediation status + VM-sizing OOM gotcha devops.security: #16 remediated (freshclam + clamscan, no resident clamd); 5/6 VMs done incl CDE; proxy EXCLUDED (957MB, install OOM d it -> hard reboot). platform.gotchas: check VM RAM before installing memory-heavy tools. Co-Authored-By: Claude Opus 4.8 --- .../content/devops/security/overview.md | 33 ++++++++++++------- .../content/platform/gotchas/overview.md | 5 +++ 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/backend/src/main/resources/content/devops/security/overview.md b/backend/src/main/resources/content/devops/security/overview.md index 629dedd..7960056 100644 --- a/backend/src/main/resources/content/devops/security/overview.md +++ b/backend/src/main/resources/content/devops/security/overview.md @@ -21,20 +21,31 @@ audience: dev So iptables auto-blocking and AIDE FIM are live — the box is **not** wide open. -## The one genuine gap: anti-virus (ClamAV) +## Anti-virus (ClamAV) — remediated 2026-07-02 (#16) -This is the real finding and it still stands (**hiveiq-devops#16, Critical**). Audit 2026-07-02: +Audit 2026-07-02 found AV broken/missing on most hosts (PCI Req 5 gap, incl. the CDE) — someone had +disabled `clamav-freshclam` + `clamd` on 2026-06-09 (resource reasons). **Remediated the same day.** -| VM | ClamAV | freshclam | DB age | -|----|--------|-----------|--------| -| services `.250` | installed | active | current — but **`clamd` failed** | -| **database `.188` (PCI CDE)** | installed | **stopped** | **stale 06-09** | -| cdn `.252` | installed | **stopped** | **stale 06-09** | -| proxy `.253` / share `.254` / ollama `.7` | **not installed** | — | — | +**Approach (script: `scripts/ops/security/clamav-remediate.sh`):** `freshclam` (signature updates, +cheap) **+ a daily 04:30 `clamscan`** of writable/ingress dirs. **Do NOT enable the resident `clamd` +daemon** — it holds the ~2 GB signature DB in RAM permanently (that's why it was disabled); `clamscan` +loads it only transiently per run. -PCI Req 5 (anti-malware) is not met on most hosts, including the CDE. Remediation: install/enable -ClamAV + freshclam everywhere, fix `clamd` on `.250`. The daily health check (platform.production-checks -§4) flags this per-VM. +| VM | ClamAV | Notes | +|----|--------|-------| +| services `.250`, cdn `.252`, database `.188` (CDE) | ✅ freshclam + clamscan | CDE PCI gap closed | +| share `.254`, ollama `.7` | ✅ installed + freshclam + clamscan | ample RAM (3.9 / 16 GB) | +| **proxy `.253`** | ❌ **EXCLUDED** | **957 MB VM — too small; installing it OOM'd the VM** (see gotcha below). Agentless scan if AV ever required. | + +Daily health check (platform.production-checks §4) flags per-VM state; also reads +`/var/log/clamav/scan-status.log` for detections. Alerting on detection currently uses local `mail` +(unreliable — see #21). + +> ⚠️ **Gotcha — check VM RAM before installing heavy tools.** `bcos-proxy` is only **957 MB**. +> Installing ClamAV there (freshclam + a 2 GB clamscan) **OOM'd the guest** — it hung mid-`apt` and +> needed a **hard reboot** (`openstack server reboot --hard`) to recover; soft reboot doesn't work on +> a hung guest. Core services were unaffected. Always `free -m` a VM before adding memory-heavy agents. +> Tiny VMs (proxy) → agentless/remote scanning, not on-host clamscan. ## Residual (re-scoped from #17) diff --git a/backend/src/main/resources/content/platform/gotchas/overview.md b/backend/src/main/resources/content/platform/gotchas/overview.md index 2dcfbf1..5789fbe 100644 --- a/backend/src/main/resources/content/platform/gotchas/overview.md +++ b/backend/src/main/resources/content/platform/gotchas/overview.md @@ -18,6 +18,11 @@ audience: dev password/token fails, fetch the current value — never retry stale plaintext from memory. - **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod). - **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat. +- **Check VM RAM before installing memory-heavy tools.** VMs are sized very differently + (`bcos-proxy` = **957 MB**; CDE = 32 GB). Installing ClamAV (freshclam + a ~2 GB clamscan) on the + tiny proxy VM **OOM'd the guest** — it hung mid-`apt` and needed a **hard** reboot + (`openstack server reboot --hard`; soft doesn't work on a hung guest). Always `free -m` first; + tiny VMs → agentless/remote, not on-host agents. → **devops.security** - **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once: