bluegreen-deploy.sh cannot deploy devices, and its cutover sed corrupts the colour maps #5

New Issue

2026-07-09T17:51:05-04:00

hiveiq commented

2026-07-09 17:51:05 -04:00

Found while deploying the devices#121 hotfix (2026-07-09). Two independent bugs in instances/services/scripts/bluegreen-deploy.sh.

1. Devices is undeployable by the script.
It derives the container as hiveiq-${SERVICE}-${COLOR}, but the real containers are hiveiq-devices-backend-blue/green. With SERVICE=devices, grep -o "hiveiq-devices-[a-z]*" matches hiveiq-devices-backend, so ACTIVE_COLOR resolves to the literal string backend and STANDBY_CONTAINER becomes hiveiq-devices-blue — which does not exist. With SERVICE=devices-backend the map key becomes devices_backend_backend and the lookup fails.

2. The step-5 cutover sed is a global replace.

sed -i "s|hiveiq-${SERVICE}-${ACTIVE_COLOR}|hiveiq-${SERVICE}-${STANDBY_COLOR}|g" "$BACKENDS_CONF"

00-backends.conf also contains the derived colour maps, which list both colours as literal keys:

map $incident_backend $incident_active_color {
    "hiveiq-incident-blue:8081"   "blue";
    "hiveiq-incident-green:8081"  "green";
}

A blue→green cutover rewrites the blue key too, producing two identical "hiveiq-incident-green:8081" keys. nginx rejects duplicate map keys, so nginx -t fails — after the file has already been modified, and with set -e the script exits leaving the config broken.

Worked around by hand-editing only the map $host $X_backend default line, with a backup and nginx -t before reload.

Also, separately: compose/recon.yml sets SERVICES_INCIDENT_URL=http://hiveiq-incident-blue:8081, pinning the running hiveiq-recon to one colour instead of nginx's internal listener http://hiveiq-nginx:8081. It survives cutover only because the old slot stays warm. Same pattern as the stale-backend issue fixed on 2026-06-30. hiveiq-agent-proxy-blue and hiveiq-transactions-blue carry the same hardcoding but are currently standby.

Found while deploying the devices#121 hotfix (2026-07-09). Two independent bugs in `instances/services/scripts/bluegreen-deploy.sh`. **1. Devices is undeployable by the script.** It derives the container as `hiveiq-${SERVICE}-${COLOR}`, but the real containers are `hiveiq-devices-backend-blue/green`. With `SERVICE=devices`, `grep -o "hiveiq-devices-[a-z]*"` matches `hiveiq-devices-backend`, so `ACTIVE_COLOR` resolves to the literal string `backend` and `STANDBY_CONTAINER` becomes `hiveiq-devices-blue` — which does not exist. With `SERVICE=devices-backend` the map key becomes `devices_backend_backend` and the lookup fails. **2. The step-5 cutover `sed` is a global replace.** ```bash sed -i "s|hiveiq-${SERVICE}-${ACTIVE_COLOR}|hiveiq-${SERVICE}-${STANDBY_COLOR}|g" "$BACKENDS_CONF" ``` `00-backends.conf` also contains the *derived* colour maps, which list both colours as literal keys: ```nginx map $incident_backend $incident_active_color { "hiveiq-incident-blue:8081" "blue"; "hiveiq-incident-green:8081" "green"; } ``` A blue→green cutover rewrites the `blue` key too, producing two identical `"hiveiq-incident-green:8081"` keys. nginx rejects duplicate map keys, so `nginx -t` fails — after the file has already been modified, and with `set -e` the script exits leaving the config broken. Worked around by hand-editing only the `map $host $X_backend` default line, with a backup and `nginx -t` before reload. **Also, separately:** `compose/recon.yml` sets `SERVICES_INCIDENT_URL=http://hiveiq-incident-blue:8081`, pinning the running `hiveiq-recon` to one colour instead of nginx's internal listener `http://hiveiq-nginx:8081`. It survives cutover only because the old slot stays warm. Same pattern as the stale-backend issue fixed on 2026-06-30. `hiveiq-agent-proxy-blue` and `hiveiq-transactions-blue` carry the same hardcoding but are currently standby.

hiveiq added the Bug label 2026-07-09 17:51:05 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hiveiq-src/hiveiq-ops#5