docs(guide): internal service-to-service calls route through nginx blue/green map

platform.nginx-routing: replace the stale 'flip map + internal callers in sync' guidance with the
new pattern — internal callers point at hiveiq-nginx:<port> and follow the 00-backends.conf map
(10-internal-backends.conf). Document the targeted-flip gotcha (global sed corrupts the derived
active_color table; bluegreen-deploy.sh has this bug) and 'never recreate the active color'.
platform.deployment: add a prod blue/green cutover section + the 2026-07-03 incident lesson.

Refs hiveiq-ops/hiveiq-devops#27, hiveiq-ops/hiveiq-openmetal-prod#20.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

backend/src/main/resources/content/platform/deployment/overview.md

@@ -32,6 +32,20 @@ cd ~/hiveiq/hiveops-openmetal/hiveops/instances/dev
 docker compose pull hiveiq-<svc> && docker compose up -d hiveiq-<svc>
 ```
 
+## Production blue/green cutover (services VM)
+Six prod services run blue/green (auth, incident, devices, fleet, transactions, agent-proxy). The
+routing mechanics — the `00-backends.conf` map, the targeted-flip gotcha, and how **internal
+service-to-service calls now route through `hiveiq-nginx`** — live in **platform.nginx-routing**.
+
+- **Deploy to the STANDBY color only, then flip the map** — use `scripts/bluegreen-deploy.sh <svc>
+  <ver>`. **Never `docker compose up -d` the ACTIVE color** — that recreates the live container =
+  a full outage, not a cutover.
+- **2026-07-03 incident:** a hand-deploy recreated the active `incident` container → ~35-min
+  fleet-wide agent-ingestion outage (11.7k dropped journal-event POSTs), because internal callers
+  hardwired the color and bypassed nginx. Fixed by routing internal calls through nginx
+  (`10-internal-backends.conf`, PR hiveiq-ops/hiveiq-openmetal-prod#20). Now one map flip + reload
+  cuts over both external and internal traffic. See hiveiq-ops/hiveiq-devops#27.
+
 ## nginx routing gotchas
 - API routes live in `.../instances/dev/nginx/conf.d/api.conf` (bind-mounted into `hiveiq-nginx`).
 - Backends are set per-location: `set $backend "hiveiq-<svc>:<port>"` then `proxy_pass`. **Include

backend/src/main/resources/content/platform/nginx-routing/overview.md

@@ -38,16 +38,51 @@ when proxying to the container.
 
 ## Blue/Green backends
 
-Seven+ services run blue/green. The active upstream is selected by a **`map` variable** in
-`00-backends.conf` (e.g. `set $backend hiveiq-<svc>-blue:PORT`), not a hardcoded hostname.
+Six services run blue/green: **auth, incident, devices, fleet, transactions, agent-proxy**. The
+active upstream is selected by a **`map` variable** in `00-backends.conf`
+(e.g. `$incident_backend → "hiveiq-incident-blue:8081"`), not a hardcoded hostname. Every other
+service is single-instance.
 
-- To switch slots: deploy to the **standby** color, verify, then flip the map var **and any
-  internal callers** in sync — keep the old slot until verified.
+- **To cut over:** deploy to the **standby** color, health-gate it, then flip **only the
+  `map $host $<svc>_backend { default ... }` source line** and `nginx -s reload`. Keep the old
+  slot warm until verified — rollback is flip-back + reload (seconds).
+- **Never recreate the ACTIVE color to deploy** — that is a full outage for the service, not a
+  cutover. `scripts/bluegreen-deploy.sh <svc> <ver>` does the standby-only deploy + flip correctly;
+  use it. (A hand-deploy that recreated the active `incident` container caused a ~35-min fleet
+  outage on 2026-07-03 — see platform.deployment.)
+- **Flip the SOURCE map line only — never a global search/replace of the color.** Each
+  `$<svc>_backend` has a derived `$<svc>_active_color` lookup map just below it; a global
+  `blue→green` sed corrupts that table (two identical keys). Target the line:
+  `sed -i '/\$<svc>_backend .*default/ s/-blue/-green/' 00-backends.conf`.
+  ⚠️ `bluegreen-deploy.sh` currently uses a global sed and has this bug — flip by hand or fix the script.
 - **Stale hardcoded hostnames = 502 on relative `/api/`.** Per-domain confs
   (devices/incident/dashboard/aria) once pinned pre-blue/green hostnames and 502'd; resolved
   2026-06-30 by moving them to the `map` vars. **If a domain 502s on `/api/` again, recheck this
   first.**
 
+## Internal service-to-service calls route through the map too (since 2026-07-05)
+
+Internal calls between services (agent-proxy → incident, analytics → devices, etc.) travel over the
+docker network, **not** through the external `api.conf`. They used to hardwire a color in compose
+env (`INCIDENT_SERVICE_URL=http://hiveiq-incident-blue:8081`), so a cutover moved only external
+traffic — and recreating the active color dropped every internal caller (the 2026-07-03 outage;
+blue/green gave the agent path zero protection).
+
+Now internal callers point at **`http://hiveiq-nginx:<service-port>`** and nginx forwards to the
+active color via the same `00-backends.conf` map. **One map flip + `nginx -s reload` governs
+internal AND external traffic; internal callers are never recreated on cutover.**
+
+- Internal listeners live in **`nginx/conf.d/10-internal-backends.conf`** — docker-network-only
+  `server` blocks on `8081` (→ `$incident_backend`), `8096` (→ `$devices_backend`), `8097`
+  (→ `$fleet_backend`). Not published to the host; `resolver 127.0.0.11` is required for the
+  variable `proxy_pass`.
+- Ports mirror each service's real port, so migrating a caller is a **hostname swap only**
+  (`hiveiq-incident-blue` → `hiveiq-nginx`, port unchanged).
+- Callers on this pattern today: **agent-proxy, analytics, transactions, ai, reports**. When adding
+  a new internal caller of a blue/green service, point it at `hiveiq-nginx:<port>` — never a
+  `-blue`/`-green` hostname.
+- Deployed by hiveiq-openmetal-prod PR #20 (issue hiveiq-ops/hiveiq-devops#27).
+
 ## Health-check noise
 
 Several backends returned 401/403 to Uptime Kuma because the NGINX catch-all prefix injected a
@@ -59,4 +94,9 @@ not "down."
 - Do not add CORS handling in a backend — NGINX owns it.
 - Do not add an `add_header` in a location without re-adding CORS.
 - Do not hardcode a blue/green hostname in a per-domain conf — use the `00-backends.conf` map var.
+- Do not hardcode a blue/green hostname in an **internal** caller's compose env — use
+  `http://hiveiq-nginx:<port>` so it follows the map.
+- Do not global-search-replace a color in `00-backends.conf` — flip the source map line only
+  (a global sed corrupts the derived `$<svc>_active_color` table).
+- Do not recreate the **active** color to deploy — deploy to the standby via `bluegreen-deploy.sh`.
 - Do not place an API `location` after `location /`.