docs(guide): add platform.ci-cd + capture DNS single-point-of-failure gotcha

New platform.ci-cd page: runner topology (dlx-hiveiq-runner-01), no-internet
design, workflow set, access, and failure signatures. gotchas: correct the
stale \"CI is disabled\" line and add the AdGuard DNS SPOF (hits app boxes + CI
+ Claude sessions). Grounded in runner logs from the 2026-07-01 outage.

Refs hiveiq-ops/hiveiq-devops#11

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

backend/src/main/resources/content/platform/ci-cd/overview.md

@@ -0,0 +1,84 @@
+---
+module: platform.ci-cd
+title: CI/CD & Gitea Action Runners
+tab: Overview
+order: 35
+audience: dev
+---
+
+> **DRAFT · internal.** Written 2026-07-01 from the runner + workflow files. CI **is enabled and
+> runs on every push** — the older "CI is disabled, builds are manual" note is stale. Verify
+> specifics against the workflow YAML before relying on them.
+
+## How CI is wired
+
+Each `hiveops-*` repo has `.gitea/workflows/`. GitHub-Actions-compatible syntax, run on
+self-hosted Gitea Action runners. The common set:
+
+| Workflow | Trigger | Does |
+|----------|---------|------|
+| `cd-develop.yml` | push to `develop` | build → **Trivy** scan → **OWASP** dep-check → build+push Docker image(s) → **deploy to bcos.dev** |
+| `cd-main.yml` | push to `main` | same, targeting production registry/host |
+| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube scan (`192.168.200.230:9000`) |
+| `version-check.yml` | PR | guards the version bump |
+| `ci.yml` / `build.yml` | varies | build-only (a few repos) |
+
+`cd-develop` is **build-and-deploy** — a green run redeploys the service to bcos.dev. Java build
+is validated in-pipeline and is rarely the failure point; the network-dependent steps (Trivy DB,
+Sonar host, registries, the deploy SSH) are where breakage shows.
+
+## The runner — `dlx-hiveiq-runner-01`
+
+- **Host:** `192.168.200.227`, SSH `dlxadmin` (key `~/.ssh/id_ed25519_dlx`). A **Proxmox VM on
+  `dlx-proxmox-03`** (DLX network) — the same node that gets maintained, which matters (see below).
+- **Service:** `act_runner.service` (root, `WorkingDirectory=/opt/act_runner_hiveiq`,
+  config `config.yaml`, registration `.runner` id=18). Labels `ubuntu-latest`, `sonar`. 4 concurrent jobs.
+- **Runs ALL hiveops-src jobs.** runner-02 (`.228`) is not the active worker.
+- **The API `online` field reads `false` even when active** — a Gitea display quirk, ignore it;
+  check `journalctl` instead.
+
+### No internet — everything is pre-baked or mirrored
+
+The runner VM has **no internet access**. Nothing is pulled from the public net at job time:
+
+- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7,
+  **Trivy** (DB baked in), Docker, openssh-client all pre-installed.
+- **OWASP data** at `/opt/owasp-data` (mounted read-only into jobs). The `cd-develop` OWASP step
+  **skips with a warning** if that dir is absent — it never hard-fails on it. NVD refresh:
+  `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`).
+- **Maven cache** at `/opt/maven-cache`, mounted into jobs.
+- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` live under the Gitea
+  `actions` org; `DEFAULT_ACTIONS_URL = self` so the runner never reaches out to github.com.
+- **Local registry** `192.168.200.227:5000`.
+
+To update the job image: build on `dlx-proxmox-03` (has internet) → push to `227:5000`.
+
+## Access & diagnosis
+
+```bash
+ssh -i ~/.ssh/id_ed25519_dlx -o IdentitiesOnly=yes dlxadmin@192.168.200.227
+sudo journalctl -u act_runner -n 40 --no-pager     # runner activity / task fetch
+sudo docker ps                                      # running job containers (none = idle)
+sudo systemctl restart act_runner                   # if it's wedged
+```
+
+## Failure signatures (read this before blaming the build)
+
+- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev on 127.0.0.53:53: server misbehaving`**
+  → **the runner's DNS is down**, not a build break. The runner's only resolver is
+  `127.0.0.53` → AdGuard (`192.168.200.100`) with **no fallback**, so a DLX/AdGuard outage
+  (e.g. Proxmox maintenance) makes CI go red across *every* repo at once. This is a known
+  single-point-of-failure → **hiveiq-devops#11** and **platform.gotchas**. Confirm the build is
+  actually fine by running `mvn clean package` locally on the same commit.
+- **Run shows "running" forever in Gitea but `docker ps` on the runner has no job container**
+  → an **orphaned/zombie run** — the runner lost its Gitea connection mid-job (DNS blip) and
+  couldn't report status. It's not really running; a fresh push on stable DNS supersedes it.
+- **Sonar job red, build green** → the Sonar host (`192.168.200.230:9000`, DLX side) was
+  unreachable — again infra, not code.
+
+## Root fragility
+
+The runner is a VM on the cluster it depends on, with a single non-HA resolver — so cluster
+maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11** (secondary
+resolver on the runner + AdGuard HA). Same DNS single-point-of-failure that hits the app boxes →
+**platform.gotchas**.

backend/src/main/resources/content/platform/gotchas/overview.md

@@ -18,6 +18,14 @@ audience: dev
   password/token fails, fetch the current value — never retry stale plaintext from memory.
 - **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod).
 - **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat.
+- **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard
+  resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is
+  down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once:
+  Claude Code sessions get `ConnectionRefused` (can't resolve `api.anthropic.com`), app boxes lose
+  public DNS, and CI goes red (`failed to fetch task … server misbehaving`). Diagnose:
+  `getent hosts example.com` fails = DNS, not the app. AdGuard up but public names return empty =
+  its **upstream** is down (don't restart AdGuard). Stopgap: point the box at `.254`/`1.1.1.1`.
+  HA fix pending. → **hiveiq-devops#11**, **platform.ci-cd**
 
 ## Auth
 - **Login field is `email`, not `username`** — `POST /auth/api/login` with
@@ -56,7 +64,9 @@ audience: dev
   parseable body. A green container is not a smoke test.
 - **Production = blue/green on the standby slot only**, hotfix→PR main→approval→explicit go.
 - **Never bypass branch protection on `main`.**
-- **CI is disabled** — builds are manual.
+- **CI is enabled and runs on every push** (build → Trivy → OWASP → Docker → deploy to bcos.dev on
+  `develop`). A red pipeline is usually **infra** (runner DNS / Sonar host), not your build —
+  confirm with a local `mvn clean package`. → **platform.ci-cd**
 - **Work on `develop`**, never commit directly to `main`.
 
 ## Branding