docs(guide): add platform.ci-cd + capture DNS single-point-of-failure gotcha
New platform.ci-cd page: runner topology (dlx-hiveiq-runner-01), no-internet design, workflow set, access, and failure signatures. gotchas: correct the stale \"CI is disabled\" line and add the AdGuard DNS SPOF (hits app boxes + CI + Claude sessions). Grounded in runner logs from the 2026-07-01 outage. Refs hiveiq-ops/hiveiq-devops#11 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
---
|
||||
module: platform.ci-cd
|
||||
title: CI/CD & Gitea Action Runners
|
||||
tab: Overview
|
||||
order: 35
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **DRAFT · internal.** Written 2026-07-01 from the runner + workflow files. CI **is enabled and
|
||||
> runs on every push** — the older "CI is disabled, builds are manual" note is stale. Verify
|
||||
> specifics against the workflow YAML before relying on them.
|
||||
|
||||
## How CI is wired
|
||||
|
||||
Each `hiveops-*` repo has `.gitea/workflows/`. GitHub-Actions-compatible syntax, run on
|
||||
self-hosted Gitea Action runners. The common set:
|
||||
|
||||
| Workflow | Trigger | Does |
|
||||
|----------|---------|------|
|
||||
| `cd-develop.yml` | push to `develop` | build → **Trivy** scan → **OWASP** dep-check → build+push Docker image(s) → **deploy to bcos.dev** |
|
||||
| `cd-main.yml` | push to `main` | same, targeting production registry/host |
|
||||
| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube scan (`192.168.200.230:9000`) |
|
||||
| `version-check.yml` | PR | guards the version bump |
|
||||
| `ci.yml` / `build.yml` | varies | build-only (a few repos) |
|
||||
|
||||
`cd-develop` is **build-and-deploy** — a green run redeploys the service to bcos.dev. Java build
|
||||
is validated in-pipeline and is rarely the failure point; the network-dependent steps (Trivy DB,
|
||||
Sonar host, registries, the deploy SSH) are where breakage shows.
|
||||
|
||||
## The runner — `dlx-hiveiq-runner-01`
|
||||
|
||||
- **Host:** `192.168.200.227`, SSH `dlxadmin` (key `~/.ssh/id_ed25519_dlx`). A **Proxmox VM on
|
||||
`dlx-proxmox-03`** (DLX network) — the same node that gets maintained, which matters (see below).
|
||||
- **Service:** `act_runner.service` (root, `WorkingDirectory=/opt/act_runner_hiveiq`,
|
||||
config `config.yaml`, registration `.runner` id=18). Labels `ubuntu-latest`, `sonar`. 4 concurrent jobs.
|
||||
- **Runs ALL hiveops-src jobs.** runner-02 (`.228`) is not the active worker.
|
||||
- **The API `online` field reads `false` even when active** — a Gitea display quirk, ignore it;
|
||||
check `journalctl` instead.
|
||||
|
||||
### No internet — everything is pre-baked or mirrored
|
||||
|
||||
The runner VM has **no internet access**. Nothing is pulled from the public net at job time:
|
||||
|
||||
- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7,
|
||||
**Trivy** (DB baked in), Docker, openssh-client all pre-installed.
|
||||
- **OWASP data** at `/opt/owasp-data` (mounted read-only into jobs). The `cd-develop` OWASP step
|
||||
**skips with a warning** if that dir is absent — it never hard-fails on it. NVD refresh:
|
||||
`sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`).
|
||||
- **Maven cache** at `/opt/maven-cache`, mounted into jobs.
|
||||
- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` live under the Gitea
|
||||
`actions` org; `DEFAULT_ACTIONS_URL = self` so the runner never reaches out to github.com.
|
||||
- **Local registry** `192.168.200.227:5000`.
|
||||
|
||||
To update the job image: build on `dlx-proxmox-03` (has internet) → push to `227:5000`.
|
||||
|
||||
## Access & diagnosis
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/id_ed25519_dlx -o IdentitiesOnly=yes dlxadmin@192.168.200.227
|
||||
sudo journalctl -u act_runner -n 40 --no-pager # runner activity / task fetch
|
||||
sudo docker ps # running job containers (none = idle)
|
||||
sudo systemctl restart act_runner # if it's wedged
|
||||
```
|
||||
|
||||
## Failure signatures (read this before blaming the build)
|
||||
|
||||
- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev on 127.0.0.53:53: server misbehaving`**
|
||||
→ **the runner's DNS is down**, not a build break. The runner's only resolver is
|
||||
`127.0.0.53` → AdGuard (`192.168.200.100`) with **no fallback**, so a DLX/AdGuard outage
|
||||
(e.g. Proxmox maintenance) makes CI go red across *every* repo at once. This is a known
|
||||
single-point-of-failure → **hiveiq-devops#11** and **platform.gotchas**. Confirm the build is
|
||||
actually fine by running `mvn clean package` locally on the same commit.
|
||||
- **Run shows "running" forever in Gitea but `docker ps` on the runner has no job container**
|
||||
→ an **orphaned/zombie run** — the runner lost its Gitea connection mid-job (DNS blip) and
|
||||
couldn't report status. It's not really running; a fresh push on stable DNS supersedes it.
|
||||
- **Sonar job red, build green** → the Sonar host (`192.168.200.230:9000`, DLX side) was
|
||||
unreachable — again infra, not code.
|
||||
|
||||
## Root fragility
|
||||
|
||||
The runner is a VM on the cluster it depends on, with a single non-HA resolver — so cluster
|
||||
maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11** (secondary
|
||||
resolver on the runner + AdGuard HA). Same DNS single-point-of-failure that hits the app boxes →
|
||||
**platform.gotchas**.
|
||||
@@ -18,6 +18,14 @@ audience: dev
|
||||
password/token fails, fetch the current value — never retry stale plaintext from memory.
|
||||
- **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod).
|
||||
- **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat.
|
||||
- **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard
|
||||
resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is
|
||||
down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once:
|
||||
Claude Code sessions get `ConnectionRefused` (can't resolve `api.anthropic.com`), app boxes lose
|
||||
public DNS, and CI goes red (`failed to fetch task … server misbehaving`). Diagnose:
|
||||
`getent hosts example.com` fails = DNS, not the app. AdGuard up but public names return empty =
|
||||
its **upstream** is down (don't restart AdGuard). Stopgap: point the box at `.254`/`1.1.1.1`.
|
||||
HA fix pending. → **hiveiq-devops#11**, **platform.ci-cd**
|
||||
|
||||
## Auth
|
||||
- **Login field is `email`, not `username`** — `POST /auth/api/login` with
|
||||
@@ -56,7 +64,9 @@ audience: dev
|
||||
parseable body. A green container is not a smoke test.
|
||||
- **Production = blue/green on the standby slot only**, hotfix→PR main→approval→explicit go.
|
||||
- **Never bypass branch protection on `main`.**
|
||||
- **CI is disabled** — builds are manual.
|
||||
- **CI is enabled and runs on every push** (build → Trivy → OWASP → Docker → deploy to bcos.dev on
|
||||
`develop`). A red pipeline is usually **infra** (runner DNS / Sonar host), not your build —
|
||||
confirm with a local `mvn clean package`. → **platform.ci-cd**
|
||||
- **Work on `develop`**, never commit directly to `main`.
|
||||
|
||||
## Branding
|
||||
|
||||
Reference in New Issue
Block a user