docs(guide): correct devops.backups + devops.security after prod audit
First pass had a partial view. Backups are 3 layers (Swift dumps + encrypted off-cluster borg to DLX + Cinder snapshots), not just pg_dumpall — #15 (DR) resolved, #13/#14 re-scoped. Security has iptables auto-block (security-monitor) + AIDE FIM — #17 re-scoped; ClamAV (#16) remains the one real gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,55 +1,58 @@
|
||||
---
|
||||
module: devops.backups
|
||||
title: Database Backups
|
||||
title: Database & Data Backups
|
||||
tab: Overview
|
||||
order: 10
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **DRAFT · internal.** Written 2026-07-02 from the live backup job on `hiveiq-database`. Verify
|
||||
> against the script before relying on specifics.
|
||||
> **DRAFT · internal.** Rewritten 2026-07-02 after a full audit of the prod VMs (the first pass only
|
||||
> knew about the nightly `pg_dumpall` and mis-scoped the gaps). Scripts are version-controlled in
|
||||
> `hiveops-devops/scripts/ops/backup/`.
|
||||
|
||||
## What runs
|
||||
## The backup story — three independent layers
|
||||
|
||||
Production databases are backed up by **`/opt/hiveiq/backup-db-swift.sh`** on the **database VM**
|
||||
(`10.10.10.188`, `hiveiq-database`), via **cron daily at 01:00** (logs to `/var/log/hiveiq-backup.log`).
|
||||
Production data is protected by **three independent mechanisms**, not one:
|
||||
|
||||
- **Method:** `docker exec hiveiq-postgres pg_dumpall -U hiveiq | gzip` — one logical dump of **all**
|
||||
databases (`hiveiq_auth`, `hiveiq_mgmt`, `hiveiq_incident`, `hiveiq_journal`, …).
|
||||
- **Destination:** OpenStack **Swift**, container `hiveiq-backups`, object `db/db_<timestamp>.sql.gz`.
|
||||
- **Retention:** **30 days** (Swift `X-Delete-After`, set at upload).
|
||||
- **Auth:** Swift creds in `/opt/hiveiq/.backup-env` (sourced by the script).
|
||||
- Postgres image: `pgvector/pgvector:pg16`, container `hiveiq-postgres`.
|
||||
| Layer | Script (host) | Schedule | What / where | Retention |
|
||||
|-------|---------------|----------|--------------|-----------|
|
||||
| **Logical dumps → Swift** | `backup-db-swift.sh` (`.188`) | daily 01:00 | `pg_dumpall` all DBs → OpenStack Swift `hiveiq-backups/db/` | 30 days |
|
||||
| | `backup-uploads-swift.sh` (`.250`) | daily 01:30 | tar of `/data/hiveiq` uploads → Swift | 14 days |
|
||||
| | `backup-vaultwarden-swift.sh` (`.254`) | daily 01:15 | Vaultwarden data → Swift | — |
|
||||
| **Encrypted borg → DLX (off-cluster)** | `borgmatic` (`.250`) | daily 03:00 | encrypted borg of upload dirs + Loki **+ all Postgres DBs** (borgmatic dumps them) → `ssh://dlxadmin@100.102.108.34` — a **DLX host, OFF the OpenMetal cluster** | 7d / 4w / 3m |
|
||||
| **Volume snapshots** | `backup-cinder-snapshots.sh` (`.250`) | weekly Sun | OpenStack **Cinder snapshots** of db/services/cdn/ollama data volumes | keep 4 |
|
||||
|
||||
Verified healthy 2026-07-02: dump **1.4 GB**, uploaded OK, **31 backups** in Swift.
|
||||
So the DB is captured **three ways** (Swift dump, borg dump, Cinder snapshot), one of them **encrypted and off-cluster** (borg → DLX over Tailscale `100.102.108.34`).
|
||||
|
||||
## Verify / operate
|
||||
|
||||
```bash
|
||||
# on hiveiq-database (10.10.10.188, via ProxyJump)
|
||||
sudo tail -20 /var/log/hiveiq-backup.log # last run — expect "=== DB backup done ==="
|
||||
source /opt/hiveiq/.backup-env
|
||||
swift --insecure list hiveiq-backups --prefix db/ # newest object should be today
|
||||
sudo /opt/hiveiq/backup-db-swift.sh # run on-demand
|
||||
# Swift dumps (on .188 or .250 — both have creds)
|
||||
source /opt/hiveiq/.backup-env; swift --insecure list hiveiq-backups --prefix db/ # newest = today
|
||||
sudo tail -20 /var/log/hiveiq-backup.log # all jobs log here
|
||||
# borg (on .250)
|
||||
sudo borgmatic list ; sudo borgmatic info # archives + integrity
|
||||
# cinder
|
||||
source /opt/hiveiq/.backup-env; openstack --insecure volume snapshot list
|
||||
```
|
||||
|
||||
The **daily production health check** (platform.production-checks §3) asserts: log done < 24h,
|
||||
dump size ~1.3–1.4 GB, Swift count ≈ 30.
|
||||
The **daily production health check** (platform.production-checks §3) asserts the Swift dump is
|
||||
< 24h old and sanely sized; the borg **weekly integrity checks** (`checks: repository, archives`)
|
||||
verify the off-cluster repo.
|
||||
|
||||
## Restore
|
||||
|
||||
```bash
|
||||
source /opt/hiveiq/.backup-env
|
||||
swift --insecure download hiveiq-backups db/db_<timestamp>.sql.gz -o restore.sql.gz
|
||||
gunzip -c restore.sql.gz | docker exec -i <target-postgres> psql -U hiveiq # pg_dumpall = full cluster
|
||||
```
|
||||
`pg_dumpall` is a full cluster SQL restore; borg archives restore via `borgmatic restore`; Cinder
|
||||
snapshots restore at the volume layer. Full pg restore: see the Swift download + `psql` steps in
|
||||
`hiveops-devops/scripts/ops/backup/`.
|
||||
|
||||
## Known limitations (tracked)
|
||||
## Remaining gaps (re-scoped after the audit)
|
||||
|
||||
- **No PITR / WAL archiving** — nightly logical snapshot only, so worst-case data loss is **up to
|
||||
~24h**. → **hiveiq-devops#13**
|
||||
- **No restore test** — backups are never verified by an actual restore. → **hiveiq-devops#14**
|
||||
- **DR boundary** — Swift is offsite from the VM but likely on the **same OpenMetal cluster**; a
|
||||
full-cluster loss could take the backups too. → **hiveiq-devops#15**
|
||||
- **No WAL-based PITR** — all three layers are point-in-time daily snapshots; there's still no
|
||||
continuous archiving for sub-day recovery. RPO ≈ hours (multiple daily jobs), not the "24h" the
|
||||
first pass claimed. → **hiveiq-devops#13** (re-scoped)
|
||||
- **No full restore-to-scratch test** — borg does weekly *integrity* checks, but nothing does an
|
||||
end-to-end restore-and-verify. Still worth adding. → **hiveiq-devops#14** (partially covered)
|
||||
- ~~DR boundary / same-cluster~~ — **resolved:** borg → DLX is genuinely off-cluster. → #15 closed.
|
||||
|
||||
See **platform.production-checks**, **devops.security**, **platform.topology**.
|
||||
|
||||
@@ -6,44 +6,41 @@ order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **DRAFT · internal.** Written 2026-07-02 from a live sweep of all prod VMs. Security posture
|
||||
> below was **failing** at time of writing — see the tracked issues. Re-verify before citing.
|
||||
> **DRAFT · internal.** Rewritten 2026-07-02 after auditing the prod VMs. The first pass reported a
|
||||
> bare posture ("no firewall/fail2ban") — that was mis-scoped; there IS custom IDS + FIM. Scripts:
|
||||
> `hiveops-devops/scripts/ops/security/`.
|
||||
|
||||
## What the daily check verifies (platform.production-checks §4)
|
||||
## Controls actually in place
|
||||
|
||||
Lightweight "are the controls up" across **all prod VMs** — services (`10.10.10.164`), database
|
||||
(`10.10.10.188`, **PCI CDE**), cdn (`10.10.10.103`), proxy (`10.10.10.12`), share (`10.10.10.170`),
|
||||
ollama (`10.10.10.7`). Per VM:
|
||||
| Control | How | Hosts | PCI |
|
||||
|---------|-----|-------|-----|
|
||||
| **Intrusion detection + auto-block** | `security-monitor.sh` (every 5 min): scans nginx logs for attack patterns (`.env`, `.git`, `id_rsa`, `/actuator/env`, `/actuator/heapdump`, webshells…) → **`iptables -I INPUT -s <ip> -j DROP`**; Slack alerts; daily/weekly reports | `.250`, `.252` | Req 1/10/11 (compensating) |
|
||||
| **File-integrity monitoring** | `aide-check.sh` (daily 03:15): AIDE `--check` → email `alerts@bcos.cloud` on any change | `.250`, `.188` | **Req 11.5** |
|
||||
| **Firewall** | `iptables` (managed via security-monitor blocks) + almost certainly **OpenStack security groups** at the network layer | all | Req 1 |
|
||||
| **TLS** | `acme.sh` auto-renew | `.250`, `.252` | — |
|
||||
|
||||
```bash
|
||||
dpkg -l | grep -c clamav # ClamAV installed?
|
||||
systemctl is-active clamav-freshclam clamav-daemon # updater + daemon running?
|
||||
ls -l --time-style=+%F /var/lib/clamav/daily.c[vl]d # signature DB age (< 2 days)
|
||||
sudo ufw status | head -1 ; systemctl is-active fail2ban # host firewall / brute-force guard
|
||||
```
|
||||
So iptables auto-blocking and AIDE FIM are live — the box is **not** wide open.
|
||||
|
||||
**RED** if AV is missing/stale on any in-scope VM — loudest on the **database VM (PCI CDE)**.
|
||||
## The one genuine gap: anti-virus (ClamAV)
|
||||
|
||||
## Posture as of 2026-07-02 (FAILING — being remediated)
|
||||
This is the real finding and it still stands (**hiveiq-devops#16, Critical**). Audit 2026-07-02:
|
||||
|
||||
| VM | ClamAV | freshclam | DB age | clamd | ufw | fail2ban |
|
||||
|----|--------|-----------|--------|-------|-----|----------|
|
||||
| services `.250` | installed | active | current | **failed** | off | off |
|
||||
| **database `.188` (CDE)** | installed | **stopped** | **stale 06-09** | — | off | off |
|
||||
| cdn `.252` | installed | **stopped** | **stale 06-09** | off | off | off |
|
||||
| proxy `.253` | **none** | — | — | — | off | off |
|
||||
| share `.254` | **none** | — | — | — | off | off |
|
||||
| ollama `.7` | **none** | — | — | — | off | off |
|
||||
| VM | ClamAV | freshclam | DB age |
|
||||
|----|--------|-----------|--------|
|
||||
| services `.250` | installed | active | current — but **`clamd` failed** |
|
||||
| **database `.188` (PCI CDE)** | installed | **stopped** | **stale 06-09** |
|
||||
| cdn `.252` | installed | **stopped** | **stale 06-09** |
|
||||
| proxy `.253` / share `.254` / ollama `.7` | **not installed** | — | — |
|
||||
|
||||
- **Anti-virus (PCI Req 5):** 3 VMs have no AV; db + cdn have a ~3-week-stale signature DB with the
|
||||
updater stopped; services' `clamd` daemon is failed. Gap on the **CDE** host. → **hiveiq-devops#16 (Critical)**
|
||||
- **Firewall / brute-force (PCI Req 1):** no host `ufw` and no `fail2ban` anywhere. **Caveat:**
|
||||
network access is very likely gated by **OpenStack security groups** at the cloud layer — confirm
|
||||
the SG rules before concluding "no firewall." → **hiveiq-devops#17 (High)**
|
||||
PCI Req 5 (anti-malware) is not met on most hosts, including the CDE. Remediation: install/enable
|
||||
ClamAV + freshclam everywhere, fix `clamd` on `.250`. The daily health check (platform.production-checks
|
||||
§4) flags this per-VM.
|
||||
|
||||
## PCI context
|
||||
## Residual (re-scoped from #17)
|
||||
|
||||
The A-LIGN engagement puts the prod stack (esp. the CDE DB VM) in scope. Anti-malware on the CDE is
|
||||
a hard requirement (Req 5); this must be closed and evidenced. See the PCI notes in project memory.
|
||||
- No **`ufw`** (iptables is used directly via security-monitor; confirm OpenStack SG rules are the
|
||||
network firewall of record and document them).
|
||||
- No **`fail2ban`** — but security-monitor covers *web* attack auto-blocking; **SSH brute-force**
|
||||
specifically is the residual gap (SSH is key-only, which mitigates). → **hiveiq-devops#17** (re-scoped)
|
||||
|
||||
See **platform.production-checks**, **devops.backups**, **platform.topology**.
|
||||
|
||||
Reference in New Issue
Block a user