---
module: devops.backups
title: Database & Data Backups
tab: Overview
order: 10
audience: dev
---

> **DRAFT · internal.** Rewritten 2026-07-02 after a full audit of the prod VMs (the first pass only
> knew about the nightly `pg_dumpall` and mis-scoped the gaps). Scripts are version-controlled in
> `hiveops-devops/scripts/ops/backup/`.

## The backup story — three independent layers

Production data is protected by **three independent mechanisms**, not one:

| Layer | Script (host) | Schedule | What / where | Retention |
|-------|---------------|----------|--------------|-----------|
| **Logical dumps → Swift** | `backup-db-swift.sh` (`.188`) | daily 01:00 | `pg_dumpall` all DBs → OpenStack Swift `hiveiq-backups/db/` | 30 days |
| | `backup-uploads-swift.sh` (`.250`) | daily 01:30 | tar of `/data/hiveiq` uploads → Swift | 14 days |
| | `backup-vaultwarden-swift.sh` (`.254`) | daily 01:15 | Vaultwarden data → Swift | — |
| **Encrypted borg → DLX (off-cluster)** | `borgmatic` (`.250`) | daily 03:00 | encrypted borg of upload dirs + Loki **+ all Postgres DBs** (borgmatic dumps them) → `ssh://dlxadmin@100.102.108.34` — a **DLX host, OFF the OpenMetal cluster** | 7d / 4w / 3m |
| **Volume snapshots** | `backup-cinder-snapshots.sh` (`.250`) | weekly Sun | OpenStack **Cinder snapshots** of db/services/cdn/ollama data volumes | keep 4 |

So the DB is captured **three ways** (Swift dump, borg dump, Cinder snapshot), one of them **encrypted and off-cluster** (borg → DLX over Tailscale `100.102.108.34`).

## Verify / operate

```bash
# Swift dumps (on .188 or .250 — both have creds)
source /opt/hiveiq/.backup-env; swift --insecure list hiveiq-backups --prefix db/   # newest = today
sudo tail -20 /var/log/hiveiq-backup.log                                            # all jobs log here
# borg (on .250)
sudo borgmatic list ; sudo borgmatic info                                           # archives + integrity
# cinder
source /opt/hiveiq/.backup-env; openstack --insecure volume snapshot list
```

The **daily production health check** (platform.production-checks §3) asserts the Swift dump is
< 24h old and sanely sized; the borg **weekly integrity checks** (`checks: repository, archives`)
verify the off-cluster repo.

## Restore

`pg_dumpall` is a full cluster SQL restore; borg archives restore via `borgmatic restore`; Cinder
snapshots restore at the volume layer. Full pg restore: see the Swift download + `psql` steps in
`hiveops-devops/scripts/ops/backup/`.

## Remaining gaps (re-scoped after the audit)

- **No WAL-based PITR** — all three layers are point-in-time daily snapshots; there's still no
  continuous archiving for sub-day recovery. RPO ≈ hours (multiple daily jobs), not the "24h" the
  first pass claimed. → **hiveiq-devops#13** (re-scoped)
- **No full restore-to-scratch test** — borg does weekly *integrity* checks, but nothing does an
  end-to-end restore-and-verify. Still worth adding. → **hiveiq-devops#14** (partially covered)
- ~~DR boundary / same-cluster~~ — **resolved:** borg → DLX is genuinely off-cluster. → #15 closed.

See **platform.production-checks**, **devops.security**, **platform.topology**.
