docs(guide): add devops.prod + devops.dev (BCOS-only), fix stale ops script paths
CD - Develop (guide → bcos.dev) / build-and-deploy (push) Failing after 35m7s
CD - Develop (guide → bcos.dev) / build-and-deploy (push) Failing after 35m7s
Two new BCOS-only modules documenting the two environments and the rule that
separates them:
devops.prod Production Environment (OpenMetal) — hosts, repo layout, health
report + config drift, and the KNOWN DRIFT warning: main's
00-backends.conf names different active blue/green slots than prod
actually serves (devices/transactions/agent-proxy), so deploying it
as-is flips three services to the wrong slot. Do NOT bulk-copy main
over prod.
devops.dev Dev Environment (bcos.dev) — single 16G all-in-one VM at ~93-97%
RAM (recreate JVM services ONE at a time), its own postgres and
registry, the new 06:00 health report (johannes only), and the
health-check gotchas that must not be "fixed" back: SPA frontends
return 200 for /actuator/health so the body must be checked, the
probe must retry, and exited *-green/*-blue are standby slots.
Both audience: bcos.
The rule, now stated in both:
everything bcos.dev -> hiveiq-openmetal-dev
everything prod -> hiveiq-openmetal-prod
shared, env-agnostic -> hiveiq-devops
Also corrects devops.backups and devops.security, which still told readers the
backup/security scripts live in `hiveops-devops/scripts/ops/`. They were moved to
`hiveiq-openmetal-prod/scripts/ops/` today — the Guide is the source of truth, so
a stale path in it is worse than none.
Refs: hiveiq-ops/hiveiq-devops#39, #38
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -8,7 +8,7 @@ audience: dev
|
||||
|
||||
> **DRAFT · internal.** Rewritten 2026-07-02 after a full audit of the prod VMs (the first pass only
|
||||
> knew about the nightly `pg_dumpall` and mis-scoped the gaps). Scripts are version-controlled in
|
||||
> `hiveops-devops/scripts/ops/backup/`.
|
||||
> `hiveiq-openmetal-prod/scripts/ops/backup/`.
|
||||
|
||||
## The backup story — three independent layers
|
||||
|
||||
@@ -44,7 +44,7 @@ verify the off-cluster repo.
|
||||
|
||||
`pg_dumpall` is a full cluster SQL restore; borg archives restore via `borgmatic restore`; Cinder
|
||||
snapshots restore at the volume layer. Full pg restore: see the Swift download + `psql` steps in
|
||||
`hiveops-devops/scripts/ops/backup/`.
|
||||
`hiveiq-openmetal-prod/scripts/ops/backup/`.
|
||||
|
||||
## Remaining gaps (re-scoped after the audit)
|
||||
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
module: devops.dev
|
||||
title: Dev Environment (bcos.dev)
|
||||
tab: Overview
|
||||
order: 40
|
||||
audience: bcos
|
||||
---
|
||||
|
||||
> **BCOS only.** Written 2026-07-13, when bcos.dev had **no health monitoring at all** — empty
|
||||
> crontab, no `/opt/hiveiq`. It is the environment that is supposed to catch prod problems first.
|
||||
|
||||
## The rule
|
||||
|
||||
**Everything bcos.dev lives in `hiveiq-ops/hiveiq-openmetal-dev`. Everything production lives in
|
||||
`hiveiq-ops/hiveiq-openmetal-prod`** (see **devops.prod**). Tooling lives in the same repo as the
|
||||
config it watches, so dev tooling can never be pointed at prod boxes.
|
||||
|
||||
This rule was broken in *both* directions — prod ops scripts sat in the shared `hiveiq-devops` repo,
|
||||
and `provision-dev-vm.sh` / `setup-dev-environment.sh` sat in the **prod** repo. Both corrected in
|
||||
**hiveiq-devops#39**.
|
||||
|
||||
## The host
|
||||
|
||||
`bcos.dev` is a **single all-in-one VM** — fully isolated from production, with its **own** PostgreSQL
|
||||
and its **own** registry.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| VM | `hiveiq-dev` — FIP `173.231.195.251`, internal `10.10.10.235` |
|
||||
| Registry | `registry.bcos.dev` — **never** mix with `registry.bcos.cloud` |
|
||||
| API | `api.bcos.dev` |
|
||||
| Deployed config | `~/hiveiq/hiveops-openmetal/hiveops/instances/dev/` |
|
||||
| Repo | `hiveiq-ops/hiveiq-openmetal-dev` → `hiveops/instances/dev/` |
|
||||
|
||||
**Postgres is local to `.251`.** A ProxyJump to `10.10.10.188` reaches **production** — do not reach
|
||||
for the prod DB pattern out of habit when you are working on dev.
|
||||
|
||||
## ⚠ Memory is the real constraint
|
||||
|
||||
The whole platform runs on **one 16 GB VM** and sits chronically at **~93–97% RAM**.
|
||||
|
||||
**On deploys, recreate JVM services ONE AT A TIME.** Recreating several at once causes an OOM cascade.
|
||||
This is the single most common way to break bcos.dev.
|
||||
|
||||
## Health report
|
||||
|
||||
`/opt/hiveiq/dev-health-report.sh --send`, cron **06:00** (30 min ahead of the prod report), log
|
||||
`/var/log/hiveiq-dev-health-report.log`. Run without `--send` for a dry-run to stdout.
|
||||
Source: `hiveiq-openmetal-dev/scripts/dev-health/`.
|
||||
|
||||
**Emails `johannes@bcos.cloud` only** — dev is engineering-only; the prod report goes wider.
|
||||
|
||||
Deliberately **not** a copy of the prod report. Dev has no backups, no ClamAV, no Loki and no certs of
|
||||
its own; checking for them would be noise.
|
||||
|
||||
| check | notes |
|
||||
|-------|-------|
|
||||
| Platform | actuator per Spring backend, discovered from the Docker port map |
|
||||
| Resources | RAM 🔴≥95% / 🟡≥90% · disk 🔴≥90% / 🟡≥85% |
|
||||
| Container state | restarting/unhealthy = 🔴 |
|
||||
| Config drift | deployed `compose/*.yml` vs `hiveiq-openmetal-dev@main` + stray `.bak` count |
|
||||
|
||||
Needs a **read-only** `GITEA_TOKEN` (`read:repository`) in `/opt/hiveiq/.health-report-env`; absent =
|
||||
yellow "skipped", never a failed run.
|
||||
|
||||
## Gotchas in the health check — do not "fix" these back
|
||||
|
||||
- **The SPA frontends return HTTP 200 with `index.html` for *any* path**, including
|
||||
`/actuator/health`. A status-code check reports every frontend as healthy **forever**. The probe
|
||||
must require `"status":"UP"` **in the body**.
|
||||
- **The probe retries 3×.** At ~93% RAM a single read blips — `feedback` once showed red while
|
||||
answering `{"status":"UP"}` in 14 ms. A report that cries wolf is a report nobody reads.
|
||||
- **Exited `*-green` / `*-blue` containers are blue/green standby slots, not failures.** Alerting on
|
||||
them would fire every single day.
|
||||
- **Cron runs as `bcosadmin`, not root.** Never test with `sudo` — `$HOME` becomes `/root`, every path
|
||||
resolves wrong, and everything reads as missing.
|
||||
|
||||
## Known drift (first run, 2026-07-13)
|
||||
|
||||
The deployed tree is **not a git checkout** here either. Six compose files differed from `main` —
|
||||
**exactly the six with `.bak` siblings** (`agent-proxy`, `core`, `guide`, `incident`, `nginx`,
|
||||
`simulators`), alongside 15 stray `.bak` files. Edited on the box, backed up, never committed.
|
||||
|
||||
The first run also caught **`hiveiq-fleet-blue` crash-looping** — nothing had been watching for it.
|
||||
|
||||
Tracked in **hiveiq-devops#39**.
|
||||
|
||||
## See also
|
||||
|
||||
**devops.prod** · **platform.ci-cd** · **platform.deployment**
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
module: devops.prod
|
||||
title: Production Environment (OpenMetal)
|
||||
tab: Overview
|
||||
order: 30
|
||||
audience: bcos
|
||||
---
|
||||
|
||||
> **BCOS only.** Written 2026-07-13 after an audit that found the repo was **not** describing
|
||||
> production. Read the "Known drift" section before you deploy anything from `main`.
|
||||
|
||||
## The rule
|
||||
|
||||
**Everything production lives in `hiveiq-ops/hiveiq-openmetal-prod`. Everything bcos.dev lives in
|
||||
`hiveiq-ops/hiveiq-openmetal-dev`** (see **devops.dev**). Tooling lives in the same repo as the config
|
||||
it watches, so dev tooling can never be pointed at prod boxes and a script cannot quietly fork into
|
||||
two copies that drift apart.
|
||||
|
||||
`hiveiq-devops` is now **shared, environment-agnostic utilities only**. If what you are adding names a
|
||||
specific VM, registry, domain, or cron on a specific box, it does not belong there.
|
||||
|
||||
This rule exists because it was broken: ops scripts lived in **both** repos and had already drifted —
|
||||
`openmetal-prod`'s `aide-check.sh` was missing a fix that was already live in production, so deploying
|
||||
from that repo would have silently reintroduced a bug we had already fixed (**hiveiq-devops#39**).
|
||||
|
||||
## Hosts
|
||||
|
||||
All workloads run in **Nova VMs** on the OpenStack cluster. The `.229`/`.230`/`.231` addresses are
|
||||
bare-metal hypervisors — **never SSH or deploy to them.**
|
||||
|
||||
| VM | Floating IP | Internal | Purpose |
|
||||
|----|-------------|----------|---------|
|
||||
| hiveiq-services | `173.231.195.250` | `10.10.10.164` | all microservices + NGINX |
|
||||
| hiveiq-database | *(internal only)* | `10.10.10.188` | PostgreSQL — reach via ProxyJump through `.250` |
|
||||
| hiveiq-cdn | `173.231.195.252` | `10.10.10.103` | browser/agent CDN + Docker registry |
|
||||
| hiveiq-ollama | *(internal only)* | `10.10.10.7` | LLM inference |
|
||||
| bcos-share | `173.231.195.254` | `10.10.10.170` | Pingvin Share + Vaultwarden |
|
||||
| bcos-proxy | `173.231.195.253` | `10.10.10.12` | Nginx Proxy Manager |
|
||||
| bcos-office | *(internal only)* | `10.10.10.222` | Nextcloud (`office.bcos.cloud`) |
|
||||
| bcos-mail | `173.231.195.249` | `10.10.10.41` | docker-mailserver |
|
||||
|
||||
Registry `registry.bcos.cloud` · domain `*.bcos.cloud` · SSH key `~/.ssh/id_ed25519_hiveiq`, user
|
||||
`bcosadmin`.
|
||||
|
||||
## What lives in the repo
|
||||
|
||||
```
|
||||
hiveiq-openmetal-prod/
|
||||
├── hiveops/instances/ # compose + nginx per instance (services, cdn, database, bcos-office…)
|
||||
└── scripts/
|
||||
├── prod-health/ # daily health report -> .250 /opt/hiveiq/
|
||||
├── ops/backup/ # Swift dumps, borgmatic, Cinder snapshots
|
||||
├── ops/security/ # security-monitor, aide-check, clamav-remediate
|
||||
├── ops/office-monitor.sh # office.bcos.cloud uptime probe
|
||||
└── software-registry.py # weekly PCI/SOC2 third-party inventory (.gitea workflow)
|
||||
```
|
||||
|
||||
Deployed config on `.250` lives at `~/hiveiq/hiveops-openmetal/hiveops/instances/services/`.
|
||||
|
||||
## Health & drift
|
||||
|
||||
The daily report (`/opt/hiveiq/prod-health-report.sh --send`, cron **06:30**, emails johannes +
|
||||
tristan) is documented in **platform.production-checks**. Since 2026-07-13 it has a **section 8 —
|
||||
config drift**, which compares the deployed files against `hiveiq-openmetal-prod@main` and checksums
|
||||
the patched Nextcloud core file on the office VM.
|
||||
|
||||
It needs a **read-only** `GITEA_TOKEN` (`read:repository`) in `/opt/hiveiq/.health-report-env`.
|
||||
Without it the section reports a yellow "skipped" rather than failing the run. Do **not** put an admin
|
||||
token there — `.250` is internet-facing.
|
||||
|
||||
## ⚠ Known drift — read before deploying from `main`
|
||||
|
||||
**The deployed tree on `.250` is not a git checkout.** It is loose files, hand-edited in place, so
|
||||
`main` being clean proves nothing about what is running. On the drift check's first run (2026-07-13):
|
||||
|
||||
**33 of 46 services config files differed from `main`.**
|
||||
|
||||
The dangerous one — `main`'s `nginx/conf.d/00-backends.conf` names **different active blue/green slots
|
||||
than production is actually serving**:
|
||||
|
||||
| backend | `main` says | prod actually runs |
|
||||
|---------|-------------|--------------------|
|
||||
| `devices_backend` | green | **blue** |
|
||||
| `transactions_backend` | blue | **green** |
|
||||
| `agent_proxy_backend` | blue | **green** |
|
||||
|
||||
Deploying that file believing the repo is authoritative flips three services to the wrong slot.
|
||||
**Do not bulk-copy `main` over prod.** The reconcile has to go file by file and decide which side is
|
||||
correct — and the answer is not always "the repo". Tracked in **hiveiq-devops#39**.
|
||||
|
||||
## Why this matters
|
||||
|
||||
Two failures in one day came through the same gap, and PRs caught neither, because a PR gates what
|
||||
enters `main` — not what is running on a box:
|
||||
|
||||
- A **core Nextcloud file was patched inside a running container**, existed in no repo, and broke every
|
||||
calendar invite for two weeks before anyone diagnosed it (**hiveiq-devops#38**).
|
||||
- An **abandoned blue/green cutover** sat as uncommitted working-tree edits for ten days — never a
|
||||
commit, so no PR ever saw it.
|
||||
|
||||
The drift check does not fix this; it makes it **impossible to hide**. The real fix is making the
|
||||
deployed tree an actual checkout, at which point drift detection collapses into `git status`.
|
||||
|
||||
## See also
|
||||
|
||||
**devops.dev** · **platform.production-checks** · **platform.deployment** · **platform.nginx-routing**
|
||||
· **devops.backups** · **devops.security**
|
||||
@@ -8,7 +8,7 @@ audience: dev
|
||||
|
||||
> **DRAFT · internal.** Rewritten 2026-07-02 after auditing the prod VMs. The first pass reported a
|
||||
> bare posture ("no firewall/fail2ban") — that was mis-scoped; there IS custom IDS + FIM. Scripts:
|
||||
> `hiveops-devops/scripts/ops/security/`.
|
||||
> `hiveiq-openmetal-prod/scripts/ops/security/`.
|
||||
|
||||
## Controls actually in place
|
||||
|
||||
@@ -26,7 +26,7 @@ So iptables auto-blocking and AIDE FIM are live — the box is **not** wide open
|
||||
Audit 2026-07-02 found AV broken/missing on most hosts (PCI Req 5 gap, incl. the CDE) — someone had
|
||||
disabled `clamav-freshclam` + `clamd` on 2026-06-09 (resource reasons). **Remediated the same day.**
|
||||
|
||||
**Approach (script: `scripts/ops/security/clamav-remediate.sh`):** `freshclam` (signature updates,
|
||||
**Approach (script: `hiveiq-openmetal-prod/scripts/ops/security/clamav-remediate.sh`):** `freshclam` (signature updates,
|
||||
cheap) **+ a daily 04:30 `clamscan`** of writable/ingress dirs. **Do NOT enable the resident `clamd`
|
||||
daemon** — it holds the ~2 GB signature DB in RAM permanently (that's why it was disabled); `clamscan`
|
||||
loads it only transiently per run.
|
||||
|
||||
Reference in New Issue
Block a user