From 340e131dc7924135997af4e37f4b76f2cacd8d3f Mon Sep 17 00:00:00 2001 From: Johannes Date: Thu, 2 Jul 2026 00:06:27 -0400 Subject: [PATCH] docs(guide): add platform.ci-cd + capture DNS single-point-of-failure gotcha New platform.ci-cd page: runner topology (dlx-hiveiq-runner-01), no-internet design, workflow set, access, and failure signatures. gotchas: correct the stale \"CI is disabled\" line and add the AdGuard DNS SPOF (hits app boxes + CI + Claude sessions). Grounded in runner logs from the 2026-07-01 outage. Refs hiveiq-ops/hiveiq-devops#11 Co-Authored-By: Claude Opus 4.8 --- .../content/platform/ci-cd/overview.md | 84 +++++++++++++++++++ .../content/platform/gotchas/overview.md | 12 ++- 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 backend/src/main/resources/content/platform/ci-cd/overview.md diff --git a/backend/src/main/resources/content/platform/ci-cd/overview.md b/backend/src/main/resources/content/platform/ci-cd/overview.md new file mode 100644 index 0000000..49655cf --- /dev/null +++ b/backend/src/main/resources/content/platform/ci-cd/overview.md @@ -0,0 +1,84 @@ +--- +module: platform.ci-cd +title: CI/CD & Gitea Action Runners +tab: Overview +order: 35 +audience: dev +--- + +> **DRAFT · internal.** Written 2026-07-01 from the runner + workflow files. CI **is enabled and +> runs on every push** — the older "CI is disabled, builds are manual" note is stale. Verify +> specifics against the workflow YAML before relying on them. + +## How CI is wired + +Each `hiveops-*` repo has `.gitea/workflows/`. GitHub-Actions-compatible syntax, run on +self-hosted Gitea Action runners. The common set: + +| Workflow | Trigger | Does | +|----------|---------|------| +| `cd-develop.yml` | push to `develop` | build → **Trivy** scan → **OWASP** dep-check → build+push Docker image(s) → **deploy to bcos.dev** | +| `cd-main.yml` | push to `main` | same, targeting production registry/host | +| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube scan (`192.168.200.230:9000`) | +| `version-check.yml` | PR | guards the version bump | +| `ci.yml` / `build.yml` | varies | build-only (a few repos) | + +`cd-develop` is **build-and-deploy** — a green run redeploys the service to bcos.dev. Java build +is validated in-pipeline and is rarely the failure point; the network-dependent steps (Trivy DB, +Sonar host, registries, the deploy SSH) are where breakage shows. + +## The runner — `dlx-hiveiq-runner-01` + +- **Host:** `192.168.200.227`, SSH `dlxadmin` (key `~/.ssh/id_ed25519_dlx`). A **Proxmox VM on + `dlx-proxmox-03`** (DLX network) — the same node that gets maintained, which matters (see below). +- **Service:** `act_runner.service` (root, `WorkingDirectory=/opt/act_runner_hiveiq`, + config `config.yaml`, registration `.runner` id=18). Labels `ubuntu-latest`, `sonar`. 4 concurrent jobs. +- **Runs ALL hiveops-src jobs.** runner-02 (`.228`) is not the active worker. +- **The API `online` field reads `false` even when active** — a Gitea display quirk, ignore it; + check `journalctl` instead. + +### No internet — everything is pre-baked or mirrored + +The runner VM has **no internet access**. Nothing is pulled from the public net at job time: + +- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7, + **Trivy** (DB baked in), Docker, openssh-client all pre-installed. +- **OWASP data** at `/opt/owasp-data` (mounted read-only into jobs). The `cd-develop` OWASP step + **skips with a warning** if that dir is absent — it never hard-fails on it. NVD refresh: + `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`). +- **Maven cache** at `/opt/maven-cache`, mounted into jobs. +- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` live under the Gitea + `actions` org; `DEFAULT_ACTIONS_URL = self` so the runner never reaches out to github.com. +- **Local registry** `192.168.200.227:5000`. + +To update the job image: build on `dlx-proxmox-03` (has internet) → push to `227:5000`. + +## Access & diagnosis + +```bash +ssh -i ~/.ssh/id_ed25519_dlx -o IdentitiesOnly=yes dlxadmin@192.168.200.227 +sudo journalctl -u act_runner -n 40 --no-pager # runner activity / task fetch +sudo docker ps # running job containers (none = idle) +sudo systemctl restart act_runner # if it's wedged +``` + +## Failure signatures (read this before blaming the build) + +- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev on 127.0.0.53:53: server misbehaving`** + → **the runner's DNS is down**, not a build break. The runner's only resolver is + `127.0.0.53` → AdGuard (`192.168.200.100`) with **no fallback**, so a DLX/AdGuard outage + (e.g. Proxmox maintenance) makes CI go red across *every* repo at once. This is a known + single-point-of-failure → **hiveiq-devops#11** and **platform.gotchas**. Confirm the build is + actually fine by running `mvn clean package` locally on the same commit. +- **Run shows "running" forever in Gitea but `docker ps` on the runner has no job container** + → an **orphaned/zombie run** — the runner lost its Gitea connection mid-job (DNS blip) and + couldn't report status. It's not really running; a fresh push on stable DNS supersedes it. +- **Sonar job red, build green** → the Sonar host (`192.168.200.230:9000`, DLX side) was + unreachable — again infra, not code. + +## Root fragility + +The runner is a VM on the cluster it depends on, with a single non-HA resolver — so cluster +maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11** (secondary +resolver on the runner + AdGuard HA). Same DNS single-point-of-failure that hits the app boxes → +**platform.gotchas**. diff --git a/backend/src/main/resources/content/platform/gotchas/overview.md b/backend/src/main/resources/content/platform/gotchas/overview.md index d807c3d..2dcfbf1 100644 --- a/backend/src/main/resources/content/platform/gotchas/overview.md +++ b/backend/src/main/resources/content/platform/gotchas/overview.md @@ -18,6 +18,14 @@ audience: dev password/token fails, fetch the current value — never retry stale plaintext from memory. - **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod). - **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat. +- **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard + resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is + down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once: + Claude Code sessions get `ConnectionRefused` (can't resolve `api.anthropic.com`), app boxes lose + public DNS, and CI goes red (`failed to fetch task … server misbehaving`). Diagnose: + `getent hosts example.com` fails = DNS, not the app. AdGuard up but public names return empty = + its **upstream** is down (don't restart AdGuard). Stopgap: point the box at `.254`/`1.1.1.1`. + HA fix pending. → **hiveiq-devops#11**, **platform.ci-cd** ## Auth - **Login field is `email`, not `username`** — `POST /auth/api/login` with @@ -56,7 +64,9 @@ audience: dev parseable body. A green container is not a smoke test. - **Production = blue/green on the standby slot only**, hotfix→PR main→approval→explicit go. - **Never bypass branch protection on `main`.** -- **CI is disabled** — builds are manual. +- **CI is enabled and runs on every push** (build → Trivy → OWASP → Docker → deploy to bcos.dev on + `develop`). A red pipeline is usually **infra** (runner DNS / Sonar host), not your build — + confirm with a local `mvn clean package`. → **platform.ci-cd** - **Work on `develop`**, never commit directly to `main`. ## Branding