---
module: platform.ci-cd
title: CI/CD & Gitea Action Runners
tab: Overview
order: 35
audience: dev
---

> **CI IS ENABLED and runs on every push.** Any note claiming "CI is disabled, builds are manual"
> is **stale — do not believe it.** Verified against the workflow YAML and the runners on
> **2026-07-14**.

## Do not hand-build images

**Merge → let CI build → deploy the image CI produced.** Do **not** run `./build.sh` yourself.

A hand-built image **skips the security gate entirely** (npm audit + Trivy) and **races CI for the
mutable `:dev` tag**, so what ends up in the registry is nondeterministic. On 2026-07-13 a full
day's worth of images were hand-built on the false belief that CI was off; every one of them
reached bcos.dev **unscanned**.

The narrow exception — **CI is genuinely broken AND the user explicitly asks.** Confirm it is
broken with the actions API first, not from memory:
`GET /repos/hiveiq-src/<repo>/actions/tasks?limit=3`.

## What the workflows actually do

| Workflow | Trigger | Does |
|----------|---------|------|
| `cd-develop.yml` | push to `develop` | `npm audit --audit-level=critical` → **Trivy** (`--exit-code 1 --severity HIGH,CRITICAL`) → `docker build` → push `:dev` **and `:<sha8>`** |
| `cd-main.yml` | push to `main` | same, against the production registry |
| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube (`192.168.200.230:9000`) |
| `version-check.yml` | PR | guards the version bump |

**Immutable tags (added 2026-07-14):** every build now pushes `:<sha8>` alongside `:dev`, so a
deploy can pin a **scanned** artifact instead of chasing a mutable tag. Pin `:<sha8>` in the
instance `.env` — never deploy from `:dev` or `:latest`.

### ⚠️ `cd-develop` does NOT deploy for most repos

The old claim that "a green `cd-develop` run redeploys the service to bcos.dev" is **wrong for 6 of
8 repos.** Verified 2026-07-14:

| Repo | `cd-develop` | Images CI builds |
|------|--------------|------------------|
| `hiveops-incident` | **builds + DEPLOYS to bcos.dev** | backend + frontend |
| `hiveops-devices` | **builds + DEPLOYS to bcos.dev** | backend + frontend |
| `hiveops-recon` | build + push only | backend + frontend |
| `hiveops-reports` | build + push only | backend + frontend |
| `hiveops-journal` | build + push only | backend |
| `hiveops-auth` | build + push only | backend |
| `hiveops-dashboard` | build + push only | frontend |
| `hiveops-fleet` | build + push only | **frontend ONLY** |

For every repo except incident and devices, a green run means **the image exists in the registry —
nothing has been deployed.** Someone still has to pull and restart on `.251`.

### 🕳️ Known gap — some backends are never built by CI

- **`hiveops-fleet`: the backend (`hiveiq-fleet`) is built by neither `cd-develop` nor `cd-main`.**
  Only the frontend is. Any fleet backend image therefore *has* to be hand-built, which means it
  never passes Trivy.
- **`hiveops-devices`: `cd-main` builds only the frontend** — the production devices backend image
  is not produced by CI either.

Treat these as **defects, not as licence to hand-build**. Fix the workflow.

## The runners

Both are **Proxmox VMs on `dlx-proxmox-03`** (DLX network, no internet). Both are active — the
older "runner-02 is not the active worker" note is wrong: `hiveops-incident` built on **runner-02**
on 2026-07-14.

| Runner | IP | Notes |
|--------|-----|-------|
| `dlx-hiveiq-runner-01` | `192.168.200.227` | 4 concurrent jobs. Also hosts the local registry on `:5000`. |
| `dlx-hiveiq-runner-02` | `192.168.200.228` | 4 concurrent jobs. |

```bash
ssh -i ~/.ssh/id_ed25519_dlx -o IdentitiesOnly=yes dlxadmin@192.168.200.227
sudo journalctl -u act_runner -n 40 --no-pager   # runner activity / task fetch
sudo docker ps                                    # job containers (none = idle)
```

The Gitea API `online` field reads `false` even when a runner is active — a display quirk, ignore it.

### ⛔ The registry on runner-01 — never garbage-collect, never delete

`192.168.200.227:5000` stores its data on a **bind mount at `/opt/docker-registry`** (~1.2G). It
holds `ubuntu-latest`, the job base image **every CI job depends on**. `docker volume prune` cannot
touch it (it is not a volume), but:

- **Never** run `registry garbage-collect` against a live registry. On 2026-07-13 that reaped ~86GB
  from the *dev* registry (88GB → 2.3GB); images not still in a local Docker cache were **lost
  permanently**.
- **Never** `rm -rf /opt/docker-registry`.

### No internet — everything is pre-baked or mirrored

- **Job image** `192.168.200.227:5000/ubuntu-latest:latest` — Java 21 (Temurin), Maven 3.8.7,
  **Trivy** (DB baked in), Docker, openssh-client.
- **OWASP data** `/opt/owasp-data` (ro mount). The step **skips with a warning** if absent.
  Refresh: `sudo /opt/owasp-data/update.sh` (key in `/etc/owasp-nvd-api-key`).
- **Maven cache** `/opt/maven-cache`.
- **Actions mirrored on Gitea** — `actions/checkout`, `actions/cache` under the `actions` org;
  `DEFAULT_ACTIONS_URL = self`, so the runner never reaches github.com.

## Failure signatures — read before blaming the build

- **`network GITEA-ACTIONS-TASK-… not found`** / **`NotFound: lease does not exist`** mid-`docker
  build` / `session healthcheck failed fatally`
  → **the runner's Docker daemon is degraded, not your code.** Jobs share one
  `/var/run/docker.sock`, so concurrent builds on a long-lived daemon race each other.
  **Fix:** `sudo systemctl restart docker && sudo systemctl restart act_runner`, then reclaim the
  stale state — orphaned `GITEA-ACTIONS-TASK-*` volumes, build cache, dangling images:
  ```bash
  sudo docker volume ls --format '{{.Name}}' | grep '^GITEA-ACTIONS-TASK-' | xargs -r sudo docker volume rm
  sudo docker builder prune -af
  sudo docker image prune -f          # dangling ONLY — never -a (it evicts ubuntu-latest)
  ```
  Keep the `act-toolcache` volume. The registry container is `restart=always` and comes back on its
  own. (2026-07-14: runner-01 had 34 days of uptime, 497 orphaned volumes and 9.7GB of build cache;
  four builds died on it.)

- **`failed to fetch task … lookup hiveiq-gitea.directlx.dev … server misbehaving`**
  → **the runner's DNS is down**, not a build break. Its only resolver is AdGuard
  (`192.168.200.100`) with **no fallback**, so DLX maintenance turns *every* repo red at once.
  Tracked in **hiveiq-devops#11**.

- **Run stuck "running" forever but `docker ps` shows no job container** → an orphaned run; the
  runner lost Gitea mid-job and could not report status. A fresh push supersedes it.

- **Sonar red, build green** → the Sonar host (`192.168.200.230:9000`) was unreachable. Infra, not code.

## ⚠️ Do not merge a batch of PRs at once

Each runner has **4 concurrent job slots against a single shared Docker daemon.** Merging six PRs
together on 2026-07-13 fired six workflows simultaneously and **broke runner-01's daemon** — four
builds failed with the lease/network errors above. **Stagger merges**, or accept that you will be
repairing the runner afterwards.

## Root fragility

The runners are VMs on the cluster they depend on, with a single non-HA resolver — so cluster
maintenance removes the DNS the runner needs. Fix tracked in **hiveiq-devops#11**. Same DNS
single-point-of-failure that hits the app boxes → **platform.gotchas**.
