hiveiq-src/hiveops-guide
4
0
Fork 0
Code Issues 6 Pull Requests Actions 1 Packages Projects Releases Wiki Activity

docs(guide): correct devops.backups + devops.security after prod audit

Browse Source 
First pass had a partial view. Backups are 3 layers (Swift dumps + encrypted
off-cluster borg to DLX + Cinder snapshots), not just pg_dumpall — #15 (DR)
resolved, #13/#14 re-scoped. Security has iptables auto-block (security-monitor)
+ AIDE FIM — #17 re-scoped; ClamAV (#16) remains the one real gap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Johannes
2026-07-02 08:27:18 -04:00
parent 5d23388ef5
commit 03f6c16eab
2 changed files with 61 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/main/resources/content/devops/backups/overview.md
+34 -31
View File
@@ -1,55 +1,58 @@
--- ---
module: devops.backups module: devops.backups
title: Database Backups title: Database & Data Backups
tab: Overview tab: Overview
order: 10 order: 10
audience: dev audience: dev
--- ---
> **DRAFT · internal.** Written 2026-07-02 from the live backup job on `hiveiq-database`. Verify > **DRAFT · internal.** Rewritten 2026-07-02 after a full audit of the prod VMs (the first pass only
> against the script before relying on specifics. > knew about the nightly `pg_dumpall` and mis-scoped the gaps). Scripts are version-controlled in
> `hiveops-devops/scripts/ops/backup/`.
## What runs ## The backup story — three independent layers
Production databases are backed up by **`/opt/hiveiq/backup-db-swift.sh`** on the **database VM** Production data is protected by **three independent mechanisms**, not one:
(`10.10.10.188`, `hiveiq-database`), via **cron daily at 01:00** (logs to `/var/log/hiveiq-backup.log`).
- **Method:** `docker exec hiveiq-postgres pg_dumpall -U hiveiq | gzip` — one logical dump of **all** | Layer | Script (host) | Schedule | What / where | Retention |
databases (`hiveiq_auth`, `hiveiq_mgmt`, `hiveiq_incident`, `hiveiq_journal`, …). |-------|---------------|----------|--------------|-----------|
- **Destination:** OpenStack **Swift**, container `hiveiq-backups`, object `db/db_<timestamp>.sql.gz`. | **Logical dumps → Swift** | `backup-db-swift.sh` (`.188`) | daily 01:00 | `pg_dumpall` all DBs → OpenStack Swift `hiveiq-backups/db/` | 30 days |
- **Retention:** **30 days** (Swift `X-Delete-After`, set at upload). | | `backup-uploads-swift.sh` (`.250`) | daily 01:30 | tar of `/data/hiveiq` uploads → Swift | 14 days |
- **Auth:** Swift creds in `/opt/hiveiq/.backup-env` (sourced by the script). | | `backup-vaultwarden-swift.sh` (`.254`) | daily 01:15 | Vaultwarden data → Swift | — |
- Postgres image: `pgvector/pgvector:pg16`, container `hiveiq-postgres`. | **Encrypted borg → DLX (off-cluster)** | `borgmatic` (`.250`) | daily 03:00 | encrypted borg of upload dirs + Loki **+ all Postgres DBs** (borgmatic dumps them) → `ssh://dlxadmin@100.102.108.34` — a **DLX host, OFF the OpenMetal cluster** | 7d / 4w / 3m |
| **Volume snapshots** | `backup-cinder-snapshots.sh` (`.250`) | weekly Sun | OpenStack **Cinder snapshots** of db/services/cdn/ollama data volumes | keep 4 |
Verified healthy 2026-07-02: dump **1.4 GB**, uploaded OK, **31 backups** in Swift. So the DB is captured **three ways** (Swift dump, borg dump, Cinder snapshot), one of them **encrypted and off-cluster** (borg → DLX over Tailscale `100.102.108.34`).
## Verify / operate ## Verify / operate
```bash ```bash
# on hiveiq-database (10.10.10.188, via ProxyJump) # Swift dumps (on .188 or .250 — both have creds)
sudo tail -20 /var/log/hiveiq-backup.log # last run — expect "=== DB backup done ===" source /opt/hiveiq/.backup-env; swift --insecure list hiveiq-backups --prefix db/ # newest = today
source /opt/hiveiq/.backup-env sudo tail -20 /var/log/hiveiq-backup.log # all jobs log here
swift --insecure list hiveiq-backups --prefix db/ # newest object should be today # borg (on .250)
sudo /opt/hiveiq/backup-db-swift.sh # run on-demand sudo borgmatic list ; sudo borgmatic info # archives + integrity
# cinder
source /opt/hiveiq/.backup-env; openstack --insecure volume snapshot list
``` ```
The **daily production health check** (platform.production-checks §3) asserts: log done < 24h, The **daily production health check** (platform.production-checks §3) asserts the Swift dump is
dump size ~1.3–1.4 GB, Swift count ≈ 30. < 24h old and sanely sized; the borg **weekly integrity checks** (`checks: repository, archives`)
verify the off-cluster repo.
## Restore ## Restore
```bash `pg_dumpall` is a full cluster SQL restore; borg archives restore via `borgmatic restore`; Cinder
source /opt/hiveiq/.backup-env snapshots restore at the volume layer. Full pg restore: see the Swift download + `psql` steps in
swift --insecure download hiveiq-backups db/db_<timestamp>.sql.gz -o restore.sql.gz `hiveops-devops/scripts/ops/backup/`.
gunzip -c restore.sql.gz | docker exec -i <target-postgres> psql -U hiveiq # pg_dumpall = full cluster
```
## Known limitations (tracked) ## Remaining gaps (re-scoped after the audit)
- **No PITR / WAL archiving** — nightly logical snapshot only, so worst-case data loss is **up to - **No WAL-based PITR** — all three layers are point-in-time daily snapshots; there's still no
~24h**. → **hiveiq-devops#13** continuous archiving for sub-day recovery. RPO ≈ hours (multiple daily jobs), not the "24h" the
- **No restore test** — backups are never verified by an actual restore. → **hiveiq-devops#14** first pass claimed. → **hiveiq-devops#13** (re-scoped)
- **DR boundary** — Swift is offsite from the VM but likely on the **same OpenMetal cluster**; a - **No full restore-to-scratch test** — borg does weekly *integrity* checks, but nothing does an
full-cluster loss could take the backups too. → **hiveiq-devops#15** end-to-end restore-and-verify. Still worth adding. → **hiveiq-devops#14** (partially covered)
- ~~DR boundary / same-cluster~~ — **resolved:** borg → DLX is genuinely off-cluster. → #15 closed.
See **platform.production-checks**, **devops.security**, **platform.topology**. See **platform.production-checks**, **devops.security**, **platform.topology**.
backend/src/main/resources/content/devops/security/overview.md
+27 -30
View File
@@ -6,44 +6,41 @@ order: 20
audience: dev audience: dev
--- ---
> **DRAFT · internal.** Written 2026-07-02 from a live sweep of all prod VMs. Security posture > **DRAFT · internal.** Rewritten 2026-07-02 after auditing the prod VMs. The first pass reported a
> below was **failing** at time of writing — see the tracked issues. Re-verify before citing. > bare posture ("no firewall/fail2ban") — that was mis-scoped; there IS custom IDS + FIM. Scripts:
> `hiveops-devops/scripts/ops/security/`.
## What the daily check verifies (platform.production-checks §4) ## Controls actually in place
Lightweight "are the controls up" across **all prod VMs** — services (`10.10.10.164`), database | Control | How | Hosts | PCI |
(`10.10.10.188`, **PCI CDE**), cdn (`10.10.10.103`), proxy (`10.10.10.12`), share (`10.10.10.170`), |---------|-----|-------|-----|
ollama (`10.10.10.7`). Per VM: | **Intrusion detection + auto-block** | `security-monitor.sh` (every 5 min): scans nginx logs for attack patterns (`.env`, `.git`, `id_rsa`, `/actuator/env`, `/actuator/heapdump`, webshells…) → **`iptables -I INPUT -s <ip> -j DROP`**; Slack alerts; daily/weekly reports | `.250`, `.252` | Req 1/10/11 (compensating) |
| **File-integrity monitoring** | `aide-check.sh` (daily 03:15): AIDE `--check` → email `alerts@bcos.cloud` on any change | `.250`, `.188` | **Req 11.5** |
| **Firewall** | `iptables` (managed via security-monitor blocks) + almost certainly **OpenStack security groups** at the network layer | all | Req 1 |
| **TLS** | `acme.sh` auto-renew | `.250`, `.252` | — |
```bash So iptables auto-blocking and AIDE FIM are live — the box is **not** wide open.
dpkg -l | grep -c clamav # ClamAV installed?
systemctl is-active clamav-freshclam clamav-daemon # updater + daemon running?
ls -l --time-style=+%F /var/lib/clamav/daily.c[vl]d # signature DB age (< 2 days)
sudo ufw status | head -1 ; systemctl is-active fail2ban # host firewall / brute-force guard
```
**RED** if AV is missing/stale on any in-scope VM — loudest on the **database VM (PCI CDE)**. ## The one genuine gap: anti-virus (ClamAV)
## Posture as of 2026-07-02 (FAILING — being remediated) This is the real finding and it still stands (**hiveiq-devops#16, Critical**). Audit 2026-07-02:
| VM | ClamAV | freshclam | DB age | clamd | ufw | fail2ban | | VM | ClamAV | freshclam | DB age |
|----|--------|-----------|--------|-------|-----|----------| |----|--------|-----------|--------|
| services `.250` | installed | active | current | **failed** | off | off | | services `.250` | installed | active | current — but **`clamd` failed** |
| **database `.188` (CDE)** | installed | **stopped** | **stale 06-09** | — | off | off | | **database `.188` (PCI CDE)** | installed | **stopped** | **stale 06-09** |
| cdn `.252` | installed | **stopped** | **stale 06-09** | off | off | off | | cdn `.252` | installed | **stopped** | **stale 06-09** |
| proxy `.253` | **none** | — | — | — | off | off | | proxy `.253` / share `.254` / ollama `.7` | **not installed** | — | — |
| share `.254` | **none** | — | — | — | off | off |
| ollama `.7` | **none** | — | — | — | off | off |
- **Anti-virus (PCI Req 5):** 3 VMs have no AV; db + cdn have a ~3-week-stale signature DB with the PCI Req 5 (anti-malware) is not met on most hosts, including the CDE. Remediation: install/enable
updater stopped; services' `clamd` daemon is failed. Gap on the **CDE** host. → **hiveiq-devops#16 (Critical)** ClamAV + freshclam everywhere, fix `clamd` on `.250`. The daily health check (platform.production-checks
- **Firewall / brute-force (PCI Req 1):** no host `ufw` and no `fail2ban` anywhere. **Caveat:** §4) flags this per-VM.
network access is very likely gated by **OpenStack security groups** at the cloud layer — confirm
the SG rules before concluding "no firewall." → **hiveiq-devops#17 (High)**
## PCI context ## Residual (re-scoped from #17)
The A-LIGN engagement puts the prod stack (esp. the CDE DB VM) in scope. Anti-malware on the CDE is - No **`ufw`** (iptables is used directly via security-monitor; confirm OpenStack SG rules are the
a hard requirement (Req 5); this must be closed and evidenced. See the PCI notes in project memory. network firewall of record and document them).
- No **`fail2ban`** — but security-monitor covers *web* attack auto-blocking; **SSH brute-force**
specifically is the residual gap (SSH is key-only, which mitigates). → **hiveiq-devops#17** (re-scoped)
See **platform.production-checks**, **devops.backups**, **platform.topology**. See **platform.production-checks**, **devops.backups**, **platform.topology**.