From 1cd8f185bc288a97d1d1479536e18dae3465e4f6 Mon Sep 17 00:00:00 2001 From: Johannes Date: Sun, 5 Jul 2026 07:18:45 -0400 Subject: [PATCH] docs(guide): internal service-to-service calls route through nginx blue/green map MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit platform.nginx-routing: replace the stale 'flip map + internal callers in sync' guidance with the new pattern — internal callers point at hiveiq-nginx: and follow the 00-backends.conf map (10-internal-backends.conf). Document the targeted-flip gotcha (global sed corrupts the derived active_color table; bluegreen-deploy.sh has this bug) and 'never recreate the active color'. platform.deployment: add a prod blue/green cutover section + the 2026-07-03 incident lesson. Refs hiveiq-ops/hiveiq-devops#27, hiveiq-ops/hiveiq-openmetal-prod#20. Co-Authored-By: Claude Opus 4.8 --- .../content/platform/deployment/overview.md | 14 ++++++ .../platform/nginx-routing/overview.md | 48 +++++++++++++++++-- 2 files changed, 58 insertions(+), 4 deletions(-) diff --git a/backend/src/main/resources/content/platform/deployment/overview.md b/backend/src/main/resources/content/platform/deployment/overview.md index 4c5fbce..00ed927 100644 --- a/backend/src/main/resources/content/platform/deployment/overview.md +++ b/backend/src/main/resources/content/platform/deployment/overview.md @@ -32,6 +32,20 @@ cd ~/hiveiq/hiveops-openmetal/hiveops/instances/dev docker compose pull hiveiq- && docker compose up -d hiveiq- ``` +## Production blue/green cutover (services VM) +Six prod services run blue/green (auth, incident, devices, fleet, transactions, agent-proxy). The +routing mechanics — the `00-backends.conf` map, the targeted-flip gotcha, and how **internal +service-to-service calls now route through `hiveiq-nginx`** — live in **platform.nginx-routing**. + +- **Deploy to the STANDBY color only, then flip the map** — use `scripts/bluegreen-deploy.sh + `. **Never `docker compose up -d` the ACTIVE color** — that recreates the live container = + a full outage, not a cutover. +- **2026-07-03 incident:** a hand-deploy recreated the active `incident` container → ~35-min + fleet-wide agent-ingestion outage (11.7k dropped journal-event POSTs), because internal callers + hardwired the color and bypassed nginx. Fixed by routing internal calls through nginx + (`10-internal-backends.conf`, PR hiveiq-ops/hiveiq-openmetal-prod#20). Now one map flip + reload + cuts over both external and internal traffic. See hiveiq-ops/hiveiq-devops#27. + ## nginx routing gotchas - API routes live in `.../instances/dev/nginx/conf.d/api.conf` (bind-mounted into `hiveiq-nginx`). - Backends are set per-location: `set $backend "hiveiq-:"` then `proxy_pass`. **Include diff --git a/backend/src/main/resources/content/platform/nginx-routing/overview.md b/backend/src/main/resources/content/platform/nginx-routing/overview.md index 4cd7144..e188c27 100644 --- a/backend/src/main/resources/content/platform/nginx-routing/overview.md +++ b/backend/src/main/resources/content/platform/nginx-routing/overview.md @@ -38,16 +38,51 @@ when proxying to the container. ## Blue/Green backends -Seven+ services run blue/green. The active upstream is selected by a **`map` variable** in -`00-backends.conf` (e.g. `set $backend hiveiq--blue:PORT`), not a hardcoded hostname. +Six services run blue/green: **auth, incident, devices, fleet, transactions, agent-proxy**. The +active upstream is selected by a **`map` variable** in `00-backends.conf` +(e.g. `$incident_backend → "hiveiq-incident-blue:8081"`), not a hardcoded hostname. Every other +service is single-instance. -- To switch slots: deploy to the **standby** color, verify, then flip the map var **and any - internal callers** in sync — keep the old slot until verified. +- **To cut over:** deploy to the **standby** color, health-gate it, then flip **only the + `map $host $_backend { default ... }` source line** and `nginx -s reload`. Keep the old + slot warm until verified — rollback is flip-back + reload (seconds). +- **Never recreate the ACTIVE color to deploy** — that is a full outage for the service, not a + cutover. `scripts/bluegreen-deploy.sh ` does the standby-only deploy + flip correctly; + use it. (A hand-deploy that recreated the active `incident` container caused a ~35-min fleet + outage on 2026-07-03 — see platform.deployment.) +- **Flip the SOURCE map line only — never a global search/replace of the color.** Each + `$_backend` has a derived `$_active_color` lookup map just below it; a global + `blue→green` sed corrupts that table (two identical keys). Target the line: + `sed -i '/\$_backend .*default/ s/-blue/-green/' 00-backends.conf`. + ⚠️ `bluegreen-deploy.sh` currently uses a global sed and has this bug — flip by hand or fix the script. - **Stale hardcoded hostnames = 502 on relative `/api/`.** Per-domain confs (devices/incident/dashboard/aria) once pinned pre-blue/green hostnames and 502'd; resolved 2026-06-30 by moving them to the `map` vars. **If a domain 502s on `/api/` again, recheck this first.** +## Internal service-to-service calls route through the map too (since 2026-07-05) + +Internal calls between services (agent-proxy → incident, analytics → devices, etc.) travel over the +docker network, **not** through the external `api.conf`. They used to hardwire a color in compose +env (`INCIDENT_SERVICE_URL=http://hiveiq-incident-blue:8081`), so a cutover moved only external +traffic — and recreating the active color dropped every internal caller (the 2026-07-03 outage; +blue/green gave the agent path zero protection). + +Now internal callers point at **`http://hiveiq-nginx:`** and nginx forwards to the +active color via the same `00-backends.conf` map. **One map flip + `nginx -s reload` governs +internal AND external traffic; internal callers are never recreated on cutover.** + +- Internal listeners live in **`nginx/conf.d/10-internal-backends.conf`** — docker-network-only + `server` blocks on `8081` (→ `$incident_backend`), `8096` (→ `$devices_backend`), `8097` + (→ `$fleet_backend`). Not published to the host; `resolver 127.0.0.11` is required for the + variable `proxy_pass`. +- Ports mirror each service's real port, so migrating a caller is a **hostname swap only** + (`hiveiq-incident-blue` → `hiveiq-nginx`, port unchanged). +- Callers on this pattern today: **agent-proxy, analytics, transactions, ai, reports**. When adding + a new internal caller of a blue/green service, point it at `hiveiq-nginx:` — never a + `-blue`/`-green` hostname. +- Deployed by hiveiq-openmetal-prod PR #20 (issue hiveiq-ops/hiveiq-devops#27). + ## Health-check noise Several backends returned 401/403 to Uptime Kuma because the NGINX catch-all prefix injected a @@ -59,4 +94,9 @@ not "down." - Do not add CORS handling in a backend — NGINX owns it. - Do not add an `add_header` in a location without re-adding CORS. - Do not hardcode a blue/green hostname in a per-domain conf — use the `00-backends.conf` map var. +- Do not hardcode a blue/green hostname in an **internal** caller's compose env — use + `http://hiveiq-nginx:` so it follows the map. +- Do not global-search-replace a color in `00-backends.conf` — flip the source map line only + (a global sed corrupts the derived `$_active_color` table). +- Do not recreate the **active** color to deploy — deploy to the standby via `bluegreen-deploy.sh`. - Do not place an API `location` after `location /`.