docs(guide): correct a false claim I just added — cd-main does NOT run, production is manual
CD - Develop (guide → bcos.dev) / build-and-deploy (push) Failing after 33m24s

The previous commit (8d7450d) said CI builds production images on merge to main. That is WRONG.

cd-main.yml exists in most repos but is dormant scaffolding: it targets runs-on: production, and
the only runner advertising that label (runner-02) is not an enabled service. ZERO cd-main runs
have ever executed in any repo. Merging to main builds nothing.

Only cd-develop runs. Production is built by hand, on purpose:
  develop -> bcos.dev  = CI builds + scans. Never hand-build.
  main    -> production = you build by hand. ./build.sh is CORRECT here.

Consequence now stated explicitly in both modules: the npm-audit + Trivy gate exists ONLY on
develop, so production images are unscanned by construction. That is why 'ship only what has been
verified on bcos.dev' is the entire security argument, not a nicety.

Also corrects the fleet gap to what it actually is — cd-develop never builds hiveiq-fleet — and
drops the devices cd-main 'gap', which was not a gap because that pipeline does not run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 10:22:45 -04:00
parent 8d7450d747
commit 74bcb97ecb
2 changed files with 59 additions and 37 deletions
@@ -10,31 +10,47 @@ audience: dev
> is **stale — do not believe it.** Verified against the workflow YAML and the runners on
> **2026-07-14**.
## Do not hand-build images
## ⚠️ ONLY `cd-develop` RUNS. Production is manual.
**Merge → let CI build → deploy the image CI produced.** Do **not** run `./build.sh` yourself.
This is the single most misunderstood fact about our CI. Get it right:
A hand-built image **skips the security gate entirely** (npm audit + Trivy) and **races CI for the
mutable `:dev` tag**, so what ends up in the registry is nondeterministic. On 2026-07-13 a full
day's worth of images were hand-built on the false belief that CI was off; every one of them
reached bcos.dev **unscanned**.
| Path | Who builds |
|------|-----------|
| **`develop` → bcos.dev** | **CI.** `cd-develop.yml` fires on every push. **Never hand-build.** |
| **`main` → production** | **You, by hand.** `./build.sh --registry registry.bcos.cloud --push` |
The narrow exception — **CI is genuinely broken AND the user explicitly asks.** Confirm it is
broken with the actions API first, not from memory:
`cd-main.yml` exists in most repos but is **dormant scaffolding — it has never run.** It targets
`runs-on: production`, and the only runner advertising that label (runner-02) is not an enabled
service. **Zero `cd-main` runs have ever executed in any repo.** Merging to `main` builds nothing.
**Consequence worth naming:** the npm-audit + Trivy gate exists **only on `develop`.** Production
images are hand-built and therefore unscanned by construction. That is why *"ship only what has been
verified on bcos.dev"* is not a nicety — it is the entire security argument.
## Do not hand-build for bcos.dev
**Merge to `develop` → let CI build → deploy the image CI produced.** Do **not** run `./build.sh`.
A hand-built dev image **skips the security gate** and **races CI for the mutable `:dev` tag**, so
what lands in the registry is nondeterministic. On 2026-07-13 a full day's images were hand-built on
the false belief that CI was off; every one reached bcos.dev **unscanned**.
The narrow exception — **CI is genuinely broken AND the user explicitly asks.** Confirm it is broken
with the actions API first, not from memory:
`GET /repos/hiveiq-src/<repo>/actions/tasks?limit=3`.
## What the workflows actually do
| Workflow | Trigger | Does |
|----------|---------|------|
| `cd-develop.yml` | push to `develop` | `npm audit --audit-level=critical` → **Trivy** (`--exit-code 1 --severity HIGH,CRITICAL`) → `docker build` → push `:dev` **and `:<sha8>`** |
| `cd-main.yml` | push to `main` | same, against the production registry |
| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | SonarQube (`192.168.200.230:9000`) |
| `version-check.yml` | PR | guards the version bump |
| Workflow | Trigger | Runs? | Does |
|----------|---------|-------|------|
| `cd-develop.yml` | push to `develop` | ✅ **yes** | `npm audit --audit-level=critical` → **Trivy** (`--exit-code 1 --severity HIGH,CRITICAL`) → `docker build` → push `:dev` **and `:<sha8>`** |
| `cd-main.yml` | push to `main` | ❌ **never has** | dormant — `runs-on: production`, no enabled runner |
| `sonar-pr.yml` / `sonar-nightly.yml` | PR / schedule | ✅ | SonarQube (`192.168.200.230:9000`) |
| `version-check.yml` | PR | ✅ | guards the version bump |
**Immutable tags (added 2026-07-14):** every build now pushes `:<sha8>` alongside `:dev`, so a
**Immutable tags (added 2026-07-14):** `cd-develop` now pushes `:<sha8>` alongside `:dev`, so a
deploy can pin a **scanned** artifact instead of chasing a mutable tag. Pin `:<sha8>` in the
instance `.env` — never deploy from `:dev` or `:latest`.
instance `.env` — never deploy from `:dev`.
### ⚠️ `cd-develop` does NOT deploy for most repos
@@ -44,26 +60,27 @@ The old claim that "a green `cd-develop` run redeploys the service to bcos.dev"
| Repo | `cd-develop` | Images CI builds |
|------|--------------|------------------|
| `hiveops-incident` | **builds + DEPLOYS to bcos.dev** | backend + frontend |
| `hiveops-devices` | **builds + DEPLOYS to bcos.dev** | backend + frontend |
| `hiveops-devices` | **builds + DEPLOYS the frontend** (backend image pushed only) | backend + frontend |
| `hiveops-recon` | build + push only | backend + frontend |
| `hiveops-reports` | build + push only | backend + frontend |
| `hiveops-journal` | build + push only | backend |
| `hiveops-auth` | build + push only | backend |
| `hiveops-dashboard` | build + push only | frontend |
| `hiveops-fleet` | build + push only | **frontend ONLY** |
| `hiveops-fleet` | build + push only | **frontend ONLY** — see below |
For every repo except incident and devices, a green run means **the image exists in the registry —
nothing has been deployed.** Someone still has to pull and restart on `.251`.
For every repo except incident (and the devices *frontend*), a green run means **the image exists in
the registry — nothing has been deployed.** Someone still has to pull and restart on `.251`.
### 🕳️ Known gap — some backends are never built by CI
Where a backend is blue/green, CI deliberately does **not** deploy it: CI cannot know which colour
is live, so the cutover stays a human's job.
- **`hiveops-fleet`: the backend (`hiveiq-fleet`) is built by neither `cd-develop` nor `cd-main`.**
Only the frontend is. Any fleet backend image therefore *has* to be hand-built, which means it
never passes Trivy.
- **`hiveops-devices`: `cd-main` builds only the frontend** — the production devices backend image
is not produced by CI either.
### 🕳️ Known gap — `hiveops-fleet`'s backend is not built by CI
Treat these as **defects, not as licence to hand-build**. Fix the workflow.
**`cd-develop` builds `hiveiq-fleet-frontend` only.** The backend image (`hiveiq-fleet`) is built by
no pipeline at all, so it can only be hand-built — meaning it never passes the gate. Tracked in
**hiveiq-ops/hiveiq-devops#46**, fixed by **hiveops-fleet#96**.
Treat this as a **defect, not a licence to hand-build.** Fix the workflow.
## The runners
@@ -14,17 +14,22 @@ Milestones section in [[processes.gitea-issues]]. Roll up everything targeted at
`GET /repos/issues/search?milestones=2026-07&owner=hiveiq-src`.
## Microservice release
> **Production is built BY HAND, on purpose.** `cd-main.yml` exists in most repos but **does not
> run** — it targets `runs-on: production`, and no enabled runner serves that label. **Zero
> `cd-main` runs have ever executed.** Do not assume merging to `main` builds anything.
>
> This is the one place `./build.sh` is correct. It is **not** correct for bcos.dev, where
> `cd-develop` builds and scans on every push — see [[platform.ci-cd]].
1. Merge to `main` via the [[processes.main]] gate.
2. **CI builds it — you do not.** `cd-main.yml` fires on the merge: npm audit → **Trivy** →
build → push `:latest` **and `:<sha8>`** to `registry.bcos.cloud`. **Never run `./build.sh`
yourself** — a hand-built image skips the security gate. See **[[platform.ci-cd]]**, which also
lists the repos whose backend CI does *not* build (a defect to fix, not a licence to hand-build).
3. **Confirm the scanned image exists**, then deploy on the production services VM
(`173.231.195.250`) by **pinning `:<sha8>`** — never `:latest`, which is mutable.
Blue/green rules apply: deploy the **STANDBY** colour, then flip the nginx map
(`scripts/bluegreen-deploy.sh`). **Never `docker compose up -d` the ACTIVE colour** — that
recreates the live container and is an outage, not a cutover (see [[processes.main]] and
[[platform.deployment]]).
2. **Build + push by hand:** `./build.sh --registry registry.bcos.cloud --push` (image `hiveiq-<svc>`).
Because CI is not in this path, **nothing has run npm audit or Trivy against this image** — the
security gate only exists on `develop`. Ship from a commit already verified on bcos.dev.
3. Deploy on the production services VM (`173.231.195.250`). Blue/green rules apply: deploy the
**STANDBY** colour, then flip the nginx map (`scripts/bluegreen-deploy.sh`). **Never
`docker compose up -d` the ACTIVE colour** — that recreates the live container and is an outage,
not a cutover (see [[processes.main]] and [[platform.deployment]]).
4. Update **VERSIONS.md** and add a **What's New** entry (below).
## Agent release (`hiveops-agent`)