---
module: platform.nginx-routing
title: NGINX Routing, CORS & Blue/Green
tab: Overview
order: 10
audience: dev
---

> **DRAFT · internal.** Written 2026-07-01. NGINX on the services VM (`173.231.195.250`) is the
> single entry point for every `*.bcos.cloud` API. Most "CORS error" / "502 on /api" reports are
> NGINX config, not the backend. Verify against the live `api.conf` before asserting a fix.

## NGINX is the CORS authority — not the services

CORS headers are set **once, in NGINX**, via the `cors_map.conf` + `cors_include.conf` pattern.
The allowed origin is chosen by a `map` on the request `Origin`.

- **Every new API `location` block must `include` the CORS snippet** — a location without it
  emits no CORS headers and the browser call fails.
- **A per-location `add_header` RESETS all inherited headers.** If you add any `add_header` inside
  a location, you must re-add the CORS headers there too, or CORS silently breaks for that path
  only. This is the #1 cause of "works on endpoint A, CORS-fails on endpoint B, same domain."
- Backends should **not** also emit `Access-Control-*` — double headers = browser rejects.

## Path prefixing — the double-prefix trap

External API paths are namespaced by service: `https://api.bcos.cloud/<service>/api/...`
(e.g. `/incident/api/...`, `/journal/api/...`). NGINX strips/rewrites the `<service>` segment
when proxying to the container.

- Some per-domain confs **prepend** a service prefix. If the frontend also sends the prefix you
  get a **double prefix** (`/devices/api/devices/api/...`) → 404. Location blocks like
  `/devices/api/`, `/fleet/api/`, `/transactions/api/` exist in `api.conf` to normalise this.
- **Journal agent block MUST come before the generic UI block.** Generic path-prepend rules
  otherwise double-prefix agent paths. Admin journal API external paths **drop** the
  `/api/journal/` segment — do not add it back.
- Order matters: **specific API `location`s first, SPA catch-all `location /` LAST.**

## Blue/Green backends

Six services run blue/green: **auth, incident, devices, fleet, transactions, agent-proxy**. The
active upstream is selected by a **`map` variable** in `00-backends.conf`
(e.g. `$incident_backend → "hiveiq-incident-blue:8081"`), not a hardcoded hostname. Every other
service is single-instance.

- **To cut over:** deploy to the **standby** color, health-gate it, then flip **only the
  `map $host $<svc>_backend { default ... }` source line** and `nginx -s reload`. Keep the old
  slot warm until verified — rollback is flip-back + reload (seconds).
- **Never recreate the ACTIVE color to deploy** — that is a full outage for the service, not a
  cutover. `scripts/bluegreen-deploy.sh <svc> <ver>` does the standby-only deploy + flip correctly;
  use it. (A hand-deploy that recreated the active `incident` container caused a ~35-min fleet
  outage on 2026-07-03 — see platform.deployment.)
- **Flip the SOURCE map line only — never a global search/replace of the color.** Each
  `$<svc>_backend` has a derived `$<svc>_active_color` lookup map just below it; a global
  `blue→green` sed corrupts that table (two identical keys). Target the line:
  `sed -i '/\$<svc>_backend .*default/ s/-blue/-green/' 00-backends.conf`.
  ⚠️ `bluegreen-deploy.sh` currently uses a global sed and has this bug — flip by hand or fix the script.
- **Stale hardcoded hostnames = 502 on relative `/api/`.** Per-domain confs
  (devices/incident/dashboard/aria) once pinned pre-blue/green hostnames and 502'd; resolved
  2026-06-30 by moving them to the `map` vars. **If a domain 502s on `/api/` again, recheck this
  first.**

## Internal service-to-service calls route through the map too (since 2026-07-05)

Internal calls between services (agent-proxy → incident, analytics → devices, etc.) travel over the
docker network, **not** through the external `api.conf`. They used to hardwire a color in compose
env (`INCIDENT_SERVICE_URL=http://hiveiq-incident-blue:8081`), so a cutover moved only external
traffic — and recreating the active color dropped every internal caller (the 2026-07-03 outage;
blue/green gave the agent path zero protection).

Now internal callers point at **`http://hiveiq-nginx:<service-port>`** and nginx forwards to the
active color via the same `00-backends.conf` map. **One map flip + `nginx -s reload` governs
internal AND external traffic; internal callers are never recreated on cutover.**

- Internal listeners live in **`nginx/conf.d/10-internal-backends.conf`** — docker-network-only
  `server` blocks on `8081` (→ `$incident_backend`), `8096` (→ `$devices_backend`), `8097`
  (→ `$fleet_backend`). Not published to the host; `resolver 127.0.0.11` is required for the
  variable `proxy_pass`.
- Ports mirror each service's real port, so migrating a caller is a **hostname swap only**
  (`hiveiq-incident-blue` → `hiveiq-nginx`, port unchanged).
- Callers on this pattern today: **agent-proxy, analytics, transactions, ai, reports**. When adding
  a new internal caller of a blue/green service, point it at `hiveiq-nginx:<port>` — never a
  `-blue`/`-green` hostname.
- Deployed by hiveiq-openmetal-prod PR #20 (issue hiveiq-ops/hiveiq-devops#27).

## Health-check noise

Several backends returned 401/403 to Uptime Kuma because the NGINX catch-all prefix injected a
path they auth-gate. Fix = an **exact-match `location =` passthrough** in `api.conf` for the
health path. A 401 on a health probe often means "NGINX reached the backend and it challenged,"
not "down."

## Do NOT
- Do not add CORS handling in a backend — NGINX owns it.
- Do not add an `add_header` in a location without re-adding CORS.
- Do not hardcode a blue/green hostname in a per-domain conf — use the `00-backends.conf` map var.
- Do not hardcode a blue/green hostname in an **internal** caller's compose env — use
  `http://hiveiq-nginx:<port>` so it follows the map.
- Do not global-search-replace a color in `00-backends.conf` — flip the source map line only
  (a global sed corrupts the derived `$<svc>_active_color` table).
- Do not recreate the **active** color to deploy — deploy to the standby via `bluegreen-deploy.sh`.
- Do not place an API `location` after `location /`.
