docs(guide): CI builds images, not you — correct deployment + release + ci-cd

platform.deployment and processes.release both walked the reader through ./build.sh, never
mentioning that cd-develop/cd-main already build and scan. Following either one produces an
UNSCANNED image that races CI for the mutable :dev tag. That is what happened on 2026-07-13:
a full day of hand-built images reached bcos.dev without passing Trivy.

Verified against the workflow YAML and both runners on 2026-07-14:

- cd-develop does NOT deploy for 6 of 8 repos. Only incident and devices deploy to bcos.dev;
  everywhere else CI stops at push. The old 'a green run redeploys the service' claim was wrong.
- hiveops-fleet CI builds the FRONTEND ONLY — the fleet backend is built by neither cd-develop
  nor cd-main. hiveops-devices cd-main builds only the frontend too. Flagged as defects.
- runner-02 IS active (incident built on it today) — the 'not the active worker' note was wrong.
- CI now pushes an immutable :<sha8> alongside :dev/:latest, so deploys can pin a scanned image.

Also records the runner failure signatures ('lease does not exist', 'network ... not found'),
the daemon-restart + prune runbook, the standing warning never to garbage-collect or delete
/opt/docker-registry, and the fact that merging a batch of PRs at once overwhelms a runner's
4 shared job slots and breaks its Docker daemon.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

backend/src/main/resources/content/platform/ci-cd/overview.md

@@ -6,79 +6,141 @@ order: 35
 audience: dev
 ---
 
-> **DRAFT · internal.** Written 2026-07-01 from the runner + workflow files. CI **is enabled and
-> runs on every push** — the older "CI is disabled, builds are manual" note is stale. Verify
-> specifics against the workflow YAML before relying on them.
+> **CI IS ENABLED and runs on every push.** Any note claiming "CI is disabled, builds are manual"
+> is **stale — do not believe it.** Verified against the workflow YAML and the runners on
+> **2026-07-14**.
 
-## How CI is wired
+## Do not hand-build images
 
-Each `hiveops-*` repo has `.gitea/workflows/`. GitHub-Actions-compatible syntax, run on
-self-hosted Gitea Action runners. The common set:
+**Merge → let CI build → deploy the image CI produced.** Do **not** run `./build.sh` yourself.
+
+A hand-built image **skips the security gate entirely** (npm audit + Trivy) and **races CI for the
+mutable `:dev` tag**, so what ends up in the registry is nondeterministic. On 2026-07-13 a full
+day's worth of images were hand-built on the false belief that CI was off; every one of them
+reached bcos.dev **unscanned**.
+
+The narrow exception — **CI is genuinely broken AND the user explicitly asks.** Confirm it is
+broken with the actions API first, not from memory:
+`GET /repos/hiveiq-src/<repo>/actions/tasks?limit=3`.
+
+## What the workflows actually do
 
 | Workflow | Trigger | Does |
 |----------|---------|------|
-| `cd-develop.yml` | push to `develop` | build → **Trivy** scan → **OWASP** dep-check → build+push Docker image(s) → **deploy to bcos.dev** |
-| `cd-main.yml` | push to `main` | same, targeting production registry/host |
-| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube scan (`192.168.200.230:9000`) |
+| `cd-develop.yml` | push to `develop` | `npm audit --audit-level=critical` → **Trivy** (`--exit-code 1 --severity HIGH,CRITICAL`) → `docker build` → push `:dev` **and `:<sha8>`** |
+| `cd-main.yml` | push to `main` | same, against the production registry |
+| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube (`192.168.200.230:9000`) |
 | `version-check.yml` | PR | guards the version bump |
-| `ci.yml` / `build.yml` | varies | build-only (a few repos) |
 
-`cd-develop` is **build-and-deploy** — a green run redeploys the service to bcos.dev. Java build
-is validated in-pipeline and is rarely the failure point; the network-dependent steps (Trivy DB,
-Sonar host, registries, the deploy SSH) are where breakage shows.
+**Immutable tags (added 2026-07-14):** every build now pushes `:<sha8>` alongside `:dev`, so a
+deploy can pin a **scanned** artifact instead of chasing a mutable tag. Pin `:<sha8>` in the
+instance `.env` — never deploy from `:dev` or `:latest`.
 
-## The runner — `dlx-hiveiq-runner-01`
+### ⚠️ `cd-develop` does NOT deploy for most repos
 
-- **Host:** `192.168.200.227`, SSH `dlxadmin` (key `~/.ssh/id_ed25519_dlx`). A **Proxmox VM on
-  `dlx-proxmox-03`** (DLX network) — the same node that gets maintained, which matters (see below).
-- **Service:** `act_runner.service` (root, `WorkingDirectory=/opt/act_runner_hiveiq`,
-  config `config.yaml`, registration `.runner` id=18). Labels `ubuntu-latest`, `sonar`. 4 concurrent jobs.
-- **Runs ALL hiveops-src jobs.** runner-02 (`.228`) is not the active worker.
-- **The API `online` field reads `false` even when active** — a Gitea display quirk, ignore it;
-  check `journalctl` instead.
+The old claim that "a green `cd-develop` run redeploys the service to bcos.dev" is **wrong for 6 of
+8 repos.** Verified 2026-07-14:
 
-### No internet — everything is pre-baked or mirrored
+| Repo | `cd-develop` | Images CI builds |
+|------|--------------|------------------|
+| `hiveops-incident` | **builds + DEPLOYS to bcos.dev** | backend + frontend |
+| `hiveops-devices` | **builds + DEPLOYS to bcos.dev** | backend + frontend |
+| `hiveops-recon` | build + push only | backend + frontend |
+| `hiveops-reports` | build + push only | backend + frontend |
+| `hiveops-journal` | build + push only | backend |
+| `hiveops-auth` | build + push only | backend |
+| `hiveops-dashboard` | build + push only | frontend |
+| `hiveops-fleet` | build + push only | **frontend ONLY** |
 
-The runner VM has **no internet access**. Nothing is pulled from the public net at job time:
+For every repo except incident and devices, a green run means **the image exists in the registry —
+nothing has been deployed.** Someone still has to pull and restart on `.251`.
 
-- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7,
-  **Trivy** (DB baked in), Docker, openssh-client all pre-installed.
-- **OWASP data** at `/opt/owasp-data` (mounted read-only into jobs). The `cd-develop` OWASP step
-  **skips with a warning** if that dir is absent — it never hard-fails on it. NVD refresh:
-  `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`).
-- **Maven cache** at `/opt/maven-cache`, mounted into jobs.
-- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` live under the Gitea
-  `actions` org; `DEFAULT_ACTIONS_URL = self` so the runner never reaches out to github.com.
-- **Local registry** `192.168.200.227:5000`.
+### 🕳️ Known gap — some backends are never built by CI
 
-To update the job image: build on `dlx-proxmox-03` (has internet) → push to `227:5000`.
+- **`hiveops-fleet`: the backend (`hiveiq-fleet`) is built by neither `cd-develop` nor `cd-main`.**
+  Only the frontend is. Any fleet backend image therefore *has* to be hand-built, which means it
+  never passes Trivy.
+- **`hiveops-devices`: `cd-main` builds only the frontend** — the production devices backend image
+  is not produced by CI either.
 
-## Access & diagnosis
+Treat these as **defects, not as licence to hand-build**. Fix the workflow.
+
+## The runners
+
+Both are **Proxmox VMs on `dlx-proxmox-03`** (DLX network, no internet). Both are active — the
+older "runner-02 is not the active worker" note is wrong: `hiveops-incident` built on **runner-02**
+on 2026-07-14.
+
+| Runner | IP | Notes |
+|--------|-----|-------|
+| `dlx-hiveiq-runner-01` | `192.168.200.227` | 4 concurrent jobs. Also hosts the local registry on `:5000`. |
+| `dlx-hiveiq-runner-02` | `192.168.200.228` | 4 concurrent jobs. |
 
 ```bash
 ssh -i ~/.ssh/id_ed25519_dlx -o IdentitiesOnly=yes dlxadmin@192.168.200.227
-sudo journalctl -u act_runner -n 40 --no-pager     # runner activity / task fetch
-sudo docker ps                                      # running job containers (none = idle)
-sudo systemctl restart act_runner                   # if it's wedged
+sudo journalctl -u act_runner -n 40 --no-pager   # runner activity / task fetch
+sudo docker ps                                    # job containers (none = idle)
 ```
 
-## Failure signatures (read this before blaming the build)
+The Gitea API `online` field reads `false` even when a runner is active — a display quirk, ignore it.
 
-- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev on 127.0.0.53:53: server misbehaving`**
-  → **the runner's DNS is down**, not a build break. The runner's only resolver is
-  `127.0.0.53` → AdGuard (`192.168.200.100`) with **no fallback**, so a DLX/AdGuard outage
-  (e.g. Proxmox maintenance) makes CI go red across *every* repo at once. This is a known
-  single-point-of-failure → **hiveiq-devops#11** and **platform.gotchas**. Confirm the build is
-  actually fine by running `mvn clean package` locally on the same commit.
-- **Run shows "running" forever in Gitea but `docker ps` on the runner has no job container**
-  → an **orphaned/zombie run** — the runner lost its Gitea connection mid-job (DNS blip) and
-  couldn't report status. It's not really running; a fresh push on stable DNS supersedes it.
-- **Sonar job red, build green** → the Sonar host (`192.168.200.230:9000`, DLX side) was
-  unreachable — again infra, not code.
+### ⛔ The registry on runner-01 — never garbage-collect, never delete
+
+`192.168.200.227:5000` stores its data on a **bind mount at `/opt/docker-registry`** (~1.2G). It
+holds `ubuntu-latest`, the job base image **every CI job depends on**. `docker volume prune` cannot
+touch it (it is not a volume), but:
+
+- **Never** run `registry garbage-collect` against a live registry. On 2026-07-13 that reaped ~86GB
+  from the *dev* registry (88GB → 2.3GB); images not still in a local Docker cache were **lost
+  permanently**.
+- **Never** `rm -rf /opt/docker-registry`.
+
+### No internet — everything is pre-baked or mirrored
+
+- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7,
+  **Trivy** (DB baked in), Docker, openssh-client.
+- **OWASP data** `/opt/owasp-data` (ro mount). The step **skips with a warning** if absent.
+  Refresh: `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`).
+- **Maven cache** `/opt/maven-cache`.
+- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` under the `actions` org;
+  `DEFAULT_ACTIONS_URL = self`, so the runner never reaches github.com.
+
+## Failure signatures — read before blaming the build
+
+- **`network GITEA-ACTIONS-TASK-… not found`** / **`NotFound: lease does not exist`** mid-`docker
+  build` / `session healthcheck failed fatally`
+  → **the runner's Docker daemon is degraded, not your code.** Jobs share one
+  `/var/run/docker.sock`, so concurrent builds on a long-lived daemon race each other.
+  **Fix:** `sudo systemctl restart docker && sudo systemctl restart act_runner`, then reclaim the
+  stale state — orphaned `GITEA-ACTIONS-TASK-*` volumes, build cache, dangling images:
+  ```bash
+  sudo docker volume ls --format '{{.Name}}' | grep '^GITEA-ACTIONS-TASK-' | xargs -r sudo docker volume rm
+  sudo docker builder prune -af
+  sudo docker image prune -f          # dangling ONLY — never -a (it evicts ubuntu-latest)
+  ```
+  Keep the `act-toolcache` volume. The registry container is `restart=always` and comes back on its
+  own. (2026-07-14: runner-01 had 34 days of uptime, 497 orphaned volumes and 9.7GB of build cache;
+  four builds died on it.)
+
+- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev … server misbehaving`**
+  → **the runner's DNS is down**, not a build break. Its only resolver is AdGuard
+  (`192.168.200.100`) with **no fallback**, so DLX maintenance turns *every* repo red at once.
+  Tracked in **hiveiq-devops#11**.
+
+- **Run stuck "running" forever but `docker ps` shows no job container** → an orphaned run; the
+  runner lost Gitea mid-job and could not report status. A fresh push supersedes it.
+
+- **Sonar red, build green** → the Sonar host (`192.168.200.230:9000`) was unreachable. Infra, not code.
+
+## ⚠️ Do not merge a batch of PRs at once
+
+Each runner has **4 concurrent job slots against a single shared Docker daemon.** Merging six PRs
+together on 2026-07-13 fired six workflows simultaneously and **broke runner-01's daemon** — four
+builds failed with the lease/network errors above. **Stagger merges**, or accept that you will be
+repairing the runner afterwards.
 
 ## Root fragility
 
-The runner is a VM on the cluster it depends on, with a single non-HA resolver — so cluster
-maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11** (secondary
-resolver on the runner + AdGuard HA). Same DNS single-point-of-failure that hits the app boxes →
-**platform.gotchas**.
+The runners are VMs on the cluster they depend on, with a single non-HA resolver — so cluster
+maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11**. Same DNS
+single-point-of-failure that hits the app boxes → **platform.gotchas**.

backend/src/main/resources/content/platform/deployment/overview.md

@@ -24,14 +24,30 @@ audience: dev
 - Live VM edits must be **committed back to `hiveiq-openmetal-dev`** or they drift.
 
 ## Deploy a service to bcos.dev
+
+> ⛔ **CI builds the image. You do not.** Merging to `develop` runs `cd-develop.yml`
+> (npm audit → **Trivy** → build → push `:dev` **and `:<sha8>`**). **Do NOT run `./build.sh`** —
+> a hand-built image skips the security gate and races CI for the `:dev` tag. Full detail, the
+> per-repo table of what CI actually builds, and the runner runbook are in **[[platform.ci-cd]]**.
+
+**1. Merge to `develop` and let CI build.** Confirm the image exists before deploying:
+
+```bash
+curl -sk -u hiveiq:<pw> https://registry.bcos.dev/v2/hiveiq-<svc>/tags/list   # expect your :<sha8>
+```
+
+**2. Deploy — but check whether CI already did it.** `cd-develop` **only deploys for
+`hiveops-incident` and `hiveops-devices`.** For every other repo CI stops at push, and you deploy:
+
 ```bash
-cd /source/hiveops-src/hiveops-<svc>
-./build.sh --registry registry.bcos.dev --push          # tag :dev
 ssh -i ~/.ssh/id_ed25519_hiveiq bcosadmin@173.231.195.251
 cd ~/hiveiq/hiveops-openmetal/hiveops/instances/dev
+# pin the immutable, scanned tag in .env — never deploy from :dev or :latest
 docker compose pull hiveiq-<svc> && docker compose up -d hiveiq-<svc>
 ```
 
+**Then smoke-test it** (see the bottom of this page). A green container is not a deploy.
+
 ## Production blue/green cutover (services VM)
 Six prod services run blue/green (auth, incident, devices, fleet, transactions, agent-proxy). The
 routing mechanics — the `00-backends.conf` map, the targeted-flip gotcha, and how **internal

backend/src/main/resources/content/processes/release/overview.md

@@ -15,9 +15,16 @@ Milestones section in [[processes.gitea-issues]]. Roll up everything targeted at
 
 ## Microservice release
 1. Merge to `main` via the [[processes.main]] gate.
-2. Build + push: `./build.sh --registry registry.bcos.cloud --push` (image `hiveiq-<svc>`).
-3. Deploy on the production services VM (`173.231.195.250`): `docker compose pull hiveiq-<svc>`
-   → `docker compose up -d hiveiq-<svc>` — blue/green rules apply (see [[processes.main]]).
+2. **CI builds it — you do not.** `cd-main.yml` fires on the merge: npm audit → **Trivy** →
+   build → push `:latest` **and `:<sha8>`** to `registry.bcos.cloud`. **Never run `./build.sh`
+   yourself** — a hand-built image skips the security gate. See **[[platform.ci-cd]]**, which also
+   lists the repos whose backend CI does *not* build (a defect to fix, not a licence to hand-build).
+3. **Confirm the scanned image exists**, then deploy on the production services VM
+   (`173.231.195.250`) by **pinning `:<sha8>`** — never `:latest`, which is mutable.
+   Blue/green rules apply: deploy the **STANDBY** colour, then flip the nginx map
+   (`scripts/bluegreen-deploy.sh`). **Never `docker compose up -d` the ACTIVE colour** — that
+   recreates the live container and is an outage, not a cutover (see [[processes.main]] and
+   [[platform.deployment]]).
 4. Update **VERSIONS.md** and add a **What's New** entry (below).
 
 ## Agent release (`hiveops-agent`)