diff --git a/backend/src/main/resources/content/platform/ci-cd/overview.md b/backend/src/main/resources/content/platform/ci-cd/overview.md index 49655cf..30b9361 100644 --- a/backend/src/main/resources/content/platform/ci-cd/overview.md +++ b/backend/src/main/resources/content/platform/ci-cd/overview.md @@ -6,79 +6,141 @@ order: 35 audience: dev --- -> **DRAFT · internal.** Written 2026-07-01 from the runner + workflow files. CI **is enabled and -> runs on every push** — the older "CI is disabled, builds are manual" note is stale. Verify -> specifics against the workflow YAML before relying on them. +> **CI IS ENABLED and runs on every push.** Any note claiming "CI is disabled, builds are manual" +> is **stale — do not believe it.** Verified against the workflow YAML and the runners on +> **2026-07-14**. -## How CI is wired +## Do not hand-build images -Each `hiveops-*` repo has `.gitea/workflows/`. GitHub-Actions-compatible syntax, run on -self-hosted Gitea Action runners. The common set: +**Merge → let CI build → deploy the image CI produced.** Do **not** run `./build.sh` yourself. + +A hand-built image **skips the security gate entirely** (npm audit + Trivy) and **races CI for the +mutable `:dev` tag**, so what ends up in the registry is nondeterministic. On 2026-07-13 a full +day's worth of images were hand-built on the false belief that CI was off; every one of them +reached bcos.dev **unscanned**. + +The narrow exception — **CI is genuinely broken AND the user explicitly asks.** Confirm it is +broken with the actions API first, not from memory: +`GET /repos/hiveiq-src//actions/tasks?limit=3`. + +## What the workflows actually do | Workflow | Trigger | Does | |----------|---------|------| -| `cd-develop.yml` | push to `develop` | build → **Trivy** scan → **OWASP** dep-check → build+push Docker image(s) → **deploy to bcos.dev** | -| `cd-main.yml` | push to `main` | same, targeting production registry/host | -| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube scan (`192.168.200.230:9000`) | +| `cd-develop.yml` | push to `develop` | `npm audit --audit-level=critical` → **Trivy** (`--exit-code 1 --severity HIGH,CRITICAL`) → `docker build` → push `:dev` **and `:`** | +| `cd-main.yml` | push to `main` | same, against the production registry | +| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube (`192.168.200.230:9000`) | | `version-check.yml` | PR | guards the version bump | -| `ci.yml` / `build.yml` | varies | build-only (a few repos) | -`cd-develop` is **build-and-deploy** — a green run redeploys the service to bcos.dev. Java build -is validated in-pipeline and is rarely the failure point; the network-dependent steps (Trivy DB, -Sonar host, registries, the deploy SSH) are where breakage shows. +**Immutable tags (added 2026-07-14):** every build now pushes `:` alongside `:dev`, so a +deploy can pin a **scanned** artifact instead of chasing a mutable tag. Pin `:` in the +instance `.env` — never deploy from `:dev` or `:latest`. -## The runner — `dlx-hiveiq-runner-01` +### ⚠️ `cd-develop` does NOT deploy for most repos -- **Host:** `192.168.200.227`, SSH `dlxadmin` (key `~/.ssh/id_ed25519_dlx`). A **Proxmox VM on - `dlx-proxmox-03`** (DLX network) — the same node that gets maintained, which matters (see below). -- **Service:** `act_runner.service` (root, `WorkingDirectory=/opt/act_runner_hiveiq`, - config `config.yaml`, registration `.runner` id=18). Labels `ubuntu-latest`, `sonar`. 4 concurrent jobs. -- **Runs ALL hiveops-src jobs.** runner-02 (`.228`) is not the active worker. -- **The API `online` field reads `false` even when active** — a Gitea display quirk, ignore it; - check `journalctl` instead. +The old claim that "a green `cd-develop` run redeploys the service to bcos.dev" is **wrong for 6 of +8 repos.** Verified 2026-07-14: -### No internet — everything is pre-baked or mirrored +| Repo | `cd-develop` | Images CI builds | +|------|--------------|------------------| +| `hiveops-incident` | **builds + DEPLOYS to bcos.dev** | backend + frontend | +| `hiveops-devices` | **builds + DEPLOYS to bcos.dev** | backend + frontend | +| `hiveops-recon` | build + push only | backend + frontend | +| `hiveops-reports` | build + push only | backend + frontend | +| `hiveops-journal` | build + push only | backend | +| `hiveops-auth` | build + push only | backend | +| `hiveops-dashboard` | build + push only | frontend | +| `hiveops-fleet` | build + push only | **frontend ONLY** | -The runner VM has **no internet access**. Nothing is pulled from the public net at job time: +For every repo except incident and devices, a green run means **the image exists in the registry — +nothing has been deployed.** Someone still has to pull and restart on `.251`. -- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7, - **Trivy** (DB baked in), Docker, openssh-client all pre-installed. -- **OWASP data** at `/opt/owasp-data` (mounted read-only into jobs). The `cd-develop` OWASP step - **skips with a warning** if that dir is absent — it never hard-fails on it. NVD refresh: - `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`). -- **Maven cache** at `/opt/maven-cache`, mounted into jobs. -- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` live under the Gitea - `actions` org; `DEFAULT_ACTIONS_URL = self` so the runner never reaches out to github.com. -- **Local registry** `192.168.200.227:5000`. +### 🕳️ Known gap — some backends are never built by CI -To update the job image: build on `dlx-proxmox-03` (has internet) → push to `227:5000`. +- **`hiveops-fleet`: the backend (`hiveiq-fleet`) is built by neither `cd-develop` nor `cd-main`.** + Only the frontend is. Any fleet backend image therefore *has* to be hand-built, which means it + never passes Trivy. +- **`hiveops-devices`: `cd-main` builds only the frontend** — the production devices backend image + is not produced by CI either. -## Access & diagnosis +Treat these as **defects, not as licence to hand-build**. Fix the workflow. + +## The runners + +Both are **Proxmox VMs on `dlx-proxmox-03`** (DLX network, no internet). Both are active — the +older "runner-02 is not the active worker" note is wrong: `hiveops-incident` built on **runner-02** +on 2026-07-14. + +| Runner | IP | Notes | +|--------|-----|-------| +| `dlx-hiveiq-runner-01` | `192.168.200.227` | 4 concurrent jobs. Also hosts the local registry on `:5000`. | +| `dlx-hiveiq-runner-02` | `192.168.200.228` | 4 concurrent jobs. | ```bash ssh -i ~/.ssh/id_ed25519_dlx -o IdentitiesOnly=yes dlxadmin@192.168.200.227 -sudo journalctl -u act_runner -n 40 --no-pager # runner activity / task fetch -sudo docker ps # running job containers (none = idle) -sudo systemctl restart act_runner # if it's wedged +sudo journalctl -u act_runner -n 40 --no-pager # runner activity / task fetch +sudo docker ps # job containers (none = idle) ``` -## Failure signatures (read this before blaming the build) +The Gitea API `online` field reads `false` even when a runner is active — a display quirk, ignore it. -- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev on 127.0.0.53:53: server misbehaving`** - → **the runner's DNS is down**, not a build break. The runner's only resolver is - `127.0.0.53` → AdGuard (`192.168.200.100`) with **no fallback**, so a DLX/AdGuard outage - (e.g. Proxmox maintenance) makes CI go red across *every* repo at once. This is a known - single-point-of-failure → **hiveiq-devops#11** and **platform.gotchas**. Confirm the build is - actually fine by running `mvn clean package` locally on the same commit. -- **Run shows "running" forever in Gitea but `docker ps` on the runner has no job container** - → an **orphaned/zombie run** — the runner lost its Gitea connection mid-job (DNS blip) and - couldn't report status. It's not really running; a fresh push on stable DNS supersedes it. -- **Sonar job red, build green** → the Sonar host (`192.168.200.230:9000`, DLX side) was - unreachable — again infra, not code. +### ⛔ The registry on runner-01 — never garbage-collect, never delete + +`192.168.200.227:5000` stores its data on a **bind mount at `/opt/docker-registry`** (~1.2G). It +holds `ubuntu-latest`, the job base image **every CI job depends on**. `docker volume prune` cannot +touch it (it is not a volume), but: + +- **Never** run `registry garbage-collect` against a live registry. On 2026-07-13 that reaped ~86GB + from the *dev* registry (88GB → 2.3GB); images not still in a local Docker cache were **lost + permanently**. +- **Never** `rm -rf /opt/docker-registry`. + +### No internet — everything is pre-baked or mirrored + +- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7, + **Trivy** (DB baked in), Docker, openssh-client. +- **OWASP data** `/opt/owasp-data` (ro mount). The step **skips with a warning** if absent. + Refresh: `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`). +- **Maven cache** `/opt/maven-cache`. +- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` under the `actions` org; + `DEFAULT_ACTIONS_URL = self`, so the runner never reaches github.com. + +## Failure signatures — read before blaming the build + +- **`network GITEA-ACTIONS-TASK-… not found`** / **`NotFound: lease does not exist`** mid-`docker + build` / `session healthcheck failed fatally` + → **the runner's Docker daemon is degraded, not your code.** Jobs share one + `/var/run/docker.sock`, so concurrent builds on a long-lived daemon race each other. + **Fix:** `sudo systemctl restart docker && sudo systemctl restart act_runner`, then reclaim the + stale state — orphaned `GITEA-ACTIONS-TASK-*` volumes, build cache, dangling images: + ```bash + sudo docker volume ls --format '{{.Name}}' | grep '^GITEA-ACTIONS-TASK-' | xargs -r sudo docker volume rm + sudo docker builder prune -af + sudo docker image prune -f # dangling ONLY — never -a (it evicts ubuntu-latest) + ``` + Keep the `act-toolcache` volume. The registry container is `restart=always` and comes back on its + own. (2026-07-14: runner-01 had 34 days of uptime, 497 orphaned volumes and 9.7GB of build cache; + four builds died on it.) + +- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev … server misbehaving`** + → **the runner's DNS is down**, not a build break. Its only resolver is AdGuard + (`192.168.200.100`) with **no fallback**, so DLX maintenance turns *every* repo red at once. + Tracked in **hiveiq-devops#11**. + +- **Run stuck "running" forever but `docker ps` shows no job container** → an orphaned run; the + runner lost Gitea mid-job and could not report status. A fresh push supersedes it. + +- **Sonar red, build green** → the Sonar host (`192.168.200.230:9000`) was unreachable. Infra, not code. + +## ⚠️ Do not merge a batch of PRs at once + +Each runner has **4 concurrent job slots against a single shared Docker daemon.** Merging six PRs +together on 2026-07-13 fired six workflows simultaneously and **broke runner-01's daemon** — four +builds failed with the lease/network errors above. **Stagger merges**, or accept that you will be +repairing the runner afterwards. ## Root fragility -The runner is a VM on the cluster it depends on, with a single non-HA resolver — so cluster -maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11** (secondary -resolver on the runner + AdGuard HA). Same DNS single-point-of-failure that hits the app boxes → -**platform.gotchas**. +The runners are VMs on the cluster they depend on, with a single non-HA resolver — so cluster +maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11**. Same DNS +single-point-of-failure that hits the app boxes → **platform.gotchas**. diff --git a/backend/src/main/resources/content/platform/deployment/overview.md b/backend/src/main/resources/content/platform/deployment/overview.md index 00ed927..0781f31 100644 --- a/backend/src/main/resources/content/platform/deployment/overview.md +++ b/backend/src/main/resources/content/platform/deployment/overview.md @@ -24,14 +24,30 @@ audience: dev - Live VM edits must be **committed back to `hiveiq-openmetal-dev`** or they drift. ## Deploy a service to bcos.dev + +> ⛔ **CI builds the image. You do not.** Merging to `develop` runs `cd-develop.yml` +> (npm audit → **Trivy** → build → push `:dev` **and `:`**). **Do NOT run `./build.sh`** — +> a hand-built image skips the security gate and races CI for the `:dev` tag. Full detail, the +> per-repo table of what CI actually builds, and the runner runbook are in **[[platform.ci-cd]]**. + +**1. Merge to `develop` and let CI build.** Confirm the image exists before deploying: + +```bash +curl -sk -u hiveiq: https://registry.bcos.dev/v2/hiveiq-/tags/list # expect your : +``` + +**2. Deploy — but check whether CI already did it.** `cd-develop` **only deploys for +`hiveops-incident` and `hiveops-devices`.** For every other repo CI stops at push, and you deploy: + ```bash -cd /source/hiveops-src/hiveops- -./build.sh --registry registry.bcos.dev --push # tag :dev ssh -i ~/.ssh/id_ed25519_hiveiq bcosadmin@173.231.195.251 cd ~/hiveiq/hiveops-openmetal/hiveops/instances/dev +# pin the immutable, scanned tag in .env — never deploy from :dev or :latest docker compose pull hiveiq- && docker compose up -d hiveiq- ``` +**Then smoke-test it** (see the bottom of this page). A green container is not a deploy. + ## Production blue/green cutover (services VM) Six prod services run blue/green (auth, incident, devices, fleet, transactions, agent-proxy). The routing mechanics — the `00-backends.conf` map, the targeted-flip gotcha, and how **internal diff --git a/backend/src/main/resources/content/processes/release/overview.md b/backend/src/main/resources/content/processes/release/overview.md index 27a03bb..caa800c 100644 --- a/backend/src/main/resources/content/processes/release/overview.md +++ b/backend/src/main/resources/content/processes/release/overview.md @@ -15,9 +15,16 @@ Milestones section in [[processes.gitea-issues]]. Roll up everything targeted at ## Microservice release 1. Merge to `main` via the [[processes.main]] gate. -2. Build + push: `./build.sh --registry registry.bcos.cloud --push` (image `hiveiq-`). -3. Deploy on the production services VM (`173.231.195.250`): `docker compose pull hiveiq-` - → `docker compose up -d hiveiq-` — blue/green rules apply (see [[processes.main]]). +2. **CI builds it — you do not.** `cd-main.yml` fires on the merge: npm audit → **Trivy** → + build → push `:latest` **and `:`** to `registry.bcos.cloud`. **Never run `./build.sh` + yourself** — a hand-built image skips the security gate. See **[[platform.ci-cd]]**, which also + lists the repos whose backend CI does *not* build (a defect to fix, not a licence to hand-build). +3. **Confirm the scanned image exists**, then deploy on the production services VM + (`173.231.195.250`) by **pinning `:`** — never `:latest`, which is mutable. + Blue/green rules apply: deploy the **STANDBY** colour, then flip the nginx map + (`scripts/bluegreen-deploy.sh`). **Never `docker compose up -d` the ACTIVE colour** — that + recreates the live container and is an outage, not a cutover (see [[processes.main]] and + [[platform.deployment]]). 4. Update **VERSIONS.md** and add a **What's New** entry (below). ## Agent release (`hiveops-agent`)