diff --git a/backend/src/main/resources/content/devops/backups/overview.md b/backend/src/main/resources/content/devops/backups/overview.md index 5dcb8f7..7c1228f 100644 --- a/backend/src/main/resources/content/devops/backups/overview.md +++ b/backend/src/main/resources/content/devops/backups/overview.md @@ -1,55 +1,58 @@ --- module: devops.backups -title: Database Backups +title: Database & Data Backups tab: Overview order: 10 audience: dev --- -> **DRAFT · internal.** Written 2026-07-02 from the live backup job on `hiveiq-database`. Verify -> against the script before relying on specifics. +> **DRAFT · internal.** Rewritten 2026-07-02 after a full audit of the prod VMs (the first pass only +> knew about the nightly `pg_dumpall` and mis-scoped the gaps). Scripts are version-controlled in +> `hiveops-devops/scripts/ops/backup/`. -## What runs +## The backup story — three independent layers -Production databases are backed up by **`/opt/hiveiq/backup-db-swift.sh`** on the **database VM** -(`10.10.10.188`, `hiveiq-database`), via **cron daily at 01:00** (logs to `/var/log/hiveiq-backup.log`). +Production data is protected by **three independent mechanisms**, not one: -- **Method:** `docker exec hiveiq-postgres pg_dumpall -U hiveiq | gzip` — one logical dump of **all** - databases (`hiveiq_auth`, `hiveiq_mgmt`, `hiveiq_incident`, `hiveiq_journal`, …). -- **Destination:** OpenStack **Swift**, container `hiveiq-backups`, object `db/db_.sql.gz`. -- **Retention:** **30 days** (Swift `X-Delete-After`, set at upload). -- **Auth:** Swift creds in `/opt/hiveiq/.backup-env` (sourced by the script). -- Postgres image: `pgvector/pgvector:pg16`, container `hiveiq-postgres`. +| Layer | Script (host) | Schedule | What / where | Retention | +|-------|---------------|----------|--------------|-----------| +| **Logical dumps → Swift** | `backup-db-swift.sh` (`.188`) | daily 01:00 | `pg_dumpall` all DBs → OpenStack Swift `hiveiq-backups/db/` | 30 days | +| | `backup-uploads-swift.sh` (`.250`) | daily 01:30 | tar of `/data/hiveiq` uploads → Swift | 14 days | +| | `backup-vaultwarden-swift.sh` (`.254`) | daily 01:15 | Vaultwarden data → Swift | — | +| **Encrypted borg → DLX (off-cluster)** | `borgmatic` (`.250`) | daily 03:00 | encrypted borg of upload dirs + Loki **+ all Postgres DBs** (borgmatic dumps them) → `ssh://dlxadmin@100.102.108.34` — a **DLX host, OFF the OpenMetal cluster** | 7d / 4w / 3m | +| **Volume snapshots** | `backup-cinder-snapshots.sh` (`.250`) | weekly Sun | OpenStack **Cinder snapshots** of db/services/cdn/ollama data volumes | keep 4 | -Verified healthy 2026-07-02: dump **1.4 GB**, uploaded OK, **31 backups** in Swift. +So the DB is captured **three ways** (Swift dump, borg dump, Cinder snapshot), one of them **encrypted and off-cluster** (borg → DLX over Tailscale `100.102.108.34`). ## Verify / operate ```bash -# on hiveiq-database (10.10.10.188, via ProxyJump) -sudo tail -20 /var/log/hiveiq-backup.log # last run — expect "=== DB backup done ===" -source /opt/hiveiq/.backup-env -swift --insecure list hiveiq-backups --prefix db/ # newest object should be today -sudo /opt/hiveiq/backup-db-swift.sh # run on-demand +# Swift dumps (on .188 or .250 — both have creds) +source /opt/hiveiq/.backup-env; swift --insecure list hiveiq-backups --prefix db/ # newest = today +sudo tail -20 /var/log/hiveiq-backup.log # all jobs log here +# borg (on .250) +sudo borgmatic list ; sudo borgmatic info # archives + integrity +# cinder +source /opt/hiveiq/.backup-env; openstack --insecure volume snapshot list ``` -The **daily production health check** (platform.production-checks §3) asserts: log done < 24h, -dump size ~1.3–1.4 GB, Swift count ≈ 30. +The **daily production health check** (platform.production-checks §3) asserts the Swift dump is +< 24h old and sanely sized; the borg **weekly integrity checks** (`checks: repository, archives`) +verify the off-cluster repo. ## Restore -```bash -source /opt/hiveiq/.backup-env -swift --insecure download hiveiq-backups db/db_.sql.gz -o restore.sql.gz -gunzip -c restore.sql.gz | docker exec -i psql -U hiveiq # pg_dumpall = full cluster -``` +`pg_dumpall` is a full cluster SQL restore; borg archives restore via `borgmatic restore`; Cinder +snapshots restore at the volume layer. Full pg restore: see the Swift download + `psql` steps in +`hiveops-devops/scripts/ops/backup/`. -## Known limitations (tracked) +## Remaining gaps (re-scoped after the audit) -- **No PITR / WAL archiving** — nightly logical snapshot only, so worst-case data loss is **up to - ~24h**. → **hiveiq-devops#13** -- **No restore test** — backups are never verified by an actual restore. → **hiveiq-devops#14** -- **DR boundary** — Swift is offsite from the VM but likely on the **same OpenMetal cluster**; a - full-cluster loss could take the backups too. → **hiveiq-devops#15** +- **No WAL-based PITR** — all three layers are point-in-time daily snapshots; there's still no + continuous archiving for sub-day recovery. RPO ≈ hours (multiple daily jobs), not the "24h" the + first pass claimed. → **hiveiq-devops#13** (re-scoped) +- **No full restore-to-scratch test** — borg does weekly *integrity* checks, but nothing does an + end-to-end restore-and-verify. Still worth adding. → **hiveiq-devops#14** (partially covered) +- ~~DR boundary / same-cluster~~ — **resolved:** borg → DLX is genuinely off-cluster. → #15 closed. See **platform.production-checks**, **devops.security**, **platform.topology**. diff --git a/backend/src/main/resources/content/devops/security/overview.md b/backend/src/main/resources/content/devops/security/overview.md index 629b47d..629dedd 100644 --- a/backend/src/main/resources/content/devops/security/overview.md +++ b/backend/src/main/resources/content/devops/security/overview.md @@ -6,44 +6,41 @@ order: 20 audience: dev --- -> **DRAFT · internal.** Written 2026-07-02 from a live sweep of all prod VMs. Security posture -> below was **failing** at time of writing — see the tracked issues. Re-verify before citing. +> **DRAFT · internal.** Rewritten 2026-07-02 after auditing the prod VMs. The first pass reported a +> bare posture ("no firewall/fail2ban") — that was mis-scoped; there IS custom IDS + FIM. Scripts: +> `hiveops-devops/scripts/ops/security/`. -## What the daily check verifies (platform.production-checks §4) +## Controls actually in place -Lightweight "are the controls up" across **all prod VMs** — services (`10.10.10.164`), database -(`10.10.10.188`, **PCI CDE**), cdn (`10.10.10.103`), proxy (`10.10.10.12`), share (`10.10.10.170`), -ollama (`10.10.10.7`). Per VM: +| Control | How | Hosts | PCI | +|---------|-----|-------|-----| +| **Intrusion detection + auto-block** | `security-monitor.sh` (every 5 min): scans nginx logs for attack patterns (`.env`, `.git`, `id_rsa`, `/actuator/env`, `/actuator/heapdump`, webshells…) → **`iptables -I INPUT -s -j DROP`**; Slack alerts; daily/weekly reports | `.250`, `.252` | Req 1/10/11 (compensating) | +| **File-integrity monitoring** | `aide-check.sh` (daily 03:15): AIDE `--check` → email `alerts@bcos.cloud` on any change | `.250`, `.188` | **Req 11.5** | +| **Firewall** | `iptables` (managed via security-monitor blocks) + almost certainly **OpenStack security groups** at the network layer | all | Req 1 | +| **TLS** | `acme.sh` auto-renew | `.250`, `.252` | — | -```bash -dpkg -l | grep -c clamav # ClamAV installed? -systemctl is-active clamav-freshclam clamav-daemon # updater + daemon running? -ls -l --time-style=+%F /var/lib/clamav/daily.c[vl]d # signature DB age (< 2 days) -sudo ufw status | head -1 ; systemctl is-active fail2ban # host firewall / brute-force guard -``` +So iptables auto-blocking and AIDE FIM are live — the box is **not** wide open. -**RED** if AV is missing/stale on any in-scope VM — loudest on the **database VM (PCI CDE)**. +## The one genuine gap: anti-virus (ClamAV) -## Posture as of 2026-07-02 (FAILING — being remediated) +This is the real finding and it still stands (**hiveiq-devops#16, Critical**). Audit 2026-07-02: -| VM | ClamAV | freshclam | DB age | clamd | ufw | fail2ban | -|----|--------|-----------|--------|-------|-----|----------| -| services `.250` | installed | active | current | **failed** | off | off | -| **database `.188` (CDE)** | installed | **stopped** | **stale 06-09** | — | off | off | -| cdn `.252` | installed | **stopped** | **stale 06-09** | off | off | off | -| proxy `.253` | **none** | — | — | — | off | off | -| share `.254` | **none** | — | — | — | off | off | -| ollama `.7` | **none** | — | — | — | off | off | +| VM | ClamAV | freshclam | DB age | +|----|--------|-----------|--------| +| services `.250` | installed | active | current — but **`clamd` failed** | +| **database `.188` (PCI CDE)** | installed | **stopped** | **stale 06-09** | +| cdn `.252` | installed | **stopped** | **stale 06-09** | +| proxy `.253` / share `.254` / ollama `.7` | **not installed** | — | — | -- **Anti-virus (PCI Req 5):** 3 VMs have no AV; db + cdn have a ~3-week-stale signature DB with the - updater stopped; services' `clamd` daemon is failed. Gap on the **CDE** host. → **hiveiq-devops#16 (Critical)** -- **Firewall / brute-force (PCI Req 1):** no host `ufw` and no `fail2ban` anywhere. **Caveat:** - network access is very likely gated by **OpenStack security groups** at the cloud layer — confirm - the SG rules before concluding "no firewall." → **hiveiq-devops#17 (High)** +PCI Req 5 (anti-malware) is not met on most hosts, including the CDE. Remediation: install/enable +ClamAV + freshclam everywhere, fix `clamd` on `.250`. The daily health check (platform.production-checks +§4) flags this per-VM. -## PCI context +## Residual (re-scoped from #17) -The A-LIGN engagement puts the prod stack (esp. the CDE DB VM) in scope. Anti-malware on the CDE is -a hard requirement (Req 5); this must be closed and evidenced. See the PCI notes in project memory. +- No **`ufw`** (iptables is used directly via security-monitor; confirm OpenStack SG rules are the + network firewall of record and document them). +- No **`fail2ban`** — but security-monitor covers *web* attack auto-blocking; **SSH brute-force** + specifically is the residual gap (SSH is key-only, which mitigates). → **hiveiq-devops#17** (re-scoped) See **platform.production-checks**, **devops.backups**, **platform.topology**.