docs(guide): ClamAV remediation status + VM-sizing OOM gotcha

devops.security: #16 remediated (freshclam + clamscan, no resident clamd); 5/6 VMs done incl CDE; proxy EXCLUDED (957MB, install OOM d it -> hard reboot). platform.gotchas: check VM RAM before installing memory-heavy tools. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 10:58:00 -04:00
parent 03f6c16eab
commit 80e4bbe415
2 changed files with 27 additions and 11 deletions
@@ -21,20 +21,31 @@ audience: dev

 So iptables auto-blocking and AIDE FIM are live — the box is **not** wide open.

-## The one genuine gap: anti-virus (ClamAV)
+## Anti-virus (ClamAV) — remediated 2026-07-02 (#16)

-This is the real finding and it still stands (**hiveiq-devops#16, Critical**). Audit 2026-07-02:
+Audit 2026-07-02 found AV broken/missing on most hosts (PCI Req 5 gap, incl. the CDE) — someone had
+disabled `clamav-freshclam` + `clamd` on 2026-06-09 (resource reasons). **Remediated the same day.**

-| VM | ClamAV | freshclam | DB age |
-|----|--------|-----------|--------|
-| services `.250` | installed | active | current — but **`clamd` failed** |
-| **database `.188` (PCI CDE)** | installed | **stopped** | **stale 06-09** |
-| cdn `.252` | installed | **stopped** | **stale 06-09** |
-| proxy `.253` / share `.254` / ollama `.7` | **not installed** | — | — |
+**Approach (script: `scripts/ops/security/clamav-remediate.sh`):** `freshclam` (signature updates,
+cheap) **+ a daily 04:30 `clamscan`** of writable/ingress dirs. **Do NOT enable the resident `clamd`
+daemon** — it holds the ~2 GB signature DB in RAM permanently (that's why it was disabled); `clamscan`
+loads it only transiently per run.

-PCI Req 5 (anti-malware) is not met on most hosts, including the CDE. Remediation: install/enable
-ClamAV + freshclam everywhere, fix `clamd` on `.250`. The daily health check (platform.production-checks
-§4) flags this per-VM.
+| VM | ClamAV | Notes |
+|----|--------|-------|
+| services `.250`, cdn `.252`, database `.188` (CDE) | ✅ freshclam + clamscan | CDE PCI gap closed |
+| share `.254`, ollama `.7` | ✅ installed + freshclam + clamscan | ample RAM (3.9 / 16 GB) |
+| **proxy `.253`** | ❌ **EXCLUDED** | **957 MB VM — too small; installing it OOM'd the VM** (see gotcha below). Agentless scan if AV ever required. |
+
+Daily health check (platform.production-checks §4) flags per-VM state; also reads
+`/var/log/clamav/scan-status.log` for detections. Alerting on detection currently uses local `mail`
+(unreliable — see #21).
+
+> ⚠️ **Gotcha — check VM RAM before installing heavy tools.** `bcos-proxy` is only **957 MB**.
+> Installing ClamAV there (freshclam + a 2 GB clamscan) **OOM'd the guest** — it hung mid-`apt` and
+> needed a **hard reboot** (`openstack server reboot --hard`) to recover; soft reboot doesn't work on
+> a hung guest. Core services were unaffected. Always `free -m` a VM before adding memory-heavy agents.
+> Tiny VMs (proxy) → agentless/remote scanning, not on-host clamscan.

 ## Residual (re-scoped from #17)

@@ -18,6 +18,11 @@ audience: dev
  password/token fails, fetch the current value — never retry stale plaintext from memory.
 - **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod).
 - **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat.
+- **Check VM RAM before installing memory-heavy tools.** VMs are sized very differently
+  (`bcos-proxy` = **957 MB**; CDE = 32 GB). Installing ClamAV (freshclam + a ~2 GB clamscan) on the
+  tiny proxy VM **OOM'd the guest** — it hung mid-`apt` and needed a **hard** reboot
+  (`openstack server reboot --hard`; soft doesn't work on a hung guest). Always `free -m` first;
+  tiny VMs → agentless/remote, not on-host agents. → **devops.security**
 - **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard
  resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is
  down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once: