hiveiq-src/hiveiq-ops
4
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

bluegreen-deploy.sh does not update env vars in dependent services on slot flip #3

New Issue
Open
opened 2026-06-29 07:23:08 -04:00 by hiveiq · 0 comments
hiveiq commented 2026-06-29 07:23:08 -04:00
Owner
Copy Link
Copy Source

Bug

When bluegreen-deploy.sh flips the active slot (blue → green or vice versa), it updates nginx 00-backends.conf but does not update the hardcoded backend URLs in dependent services that call the blue/green services directly (bypassing nginx).

Affected compose files

  • compose/agent-proxy.yml — INCIDENT_SERVICE_URL, SERVICES_DEVICES_URL, SERVICES_FLEET_URL
  • compose/analytics.yml — INCIDENT_SERVICE_URL, SERVICES_DEVICES_URL, SERVICES_FLEET_URL
  • compose/recon.yml — SERVICES_INCIDENT_URL
  • compose/reports.yml — INCIDENT_SERVICE_URL

Impact

Discovered 2026-06-29 after stopping the green standby instances post-maintenance-window. Agent-proxy was forwarding all agent heartbeats and journal events to the stopped hiveiq-incident-green container, silently failing (agent got 200, internal call dropped). Result: 572 AGENT_OFFLINE incidents stuck open, plus dead internal calls in analytics, recon, and reports.

Fix required

bluegreen-deploy.sh must sed the active color in all dependent compose files in addition to updating 00-backends.conf, then run docker compose up -d --no-deps on the affected containers so they pick up the corrected env vars on each slot flip.

Workaround applied 2026-06-29

Manually sed all 4 compose files (green → blue) and restarted agent-proxy, analytics, recon, reports.

## Bug When `bluegreen-deploy.sh` flips the active slot (blue → green or vice versa), it updates nginx `00-backends.conf` but does **not** update the hardcoded backend URLs in dependent services that call the blue/green services directly (bypassing nginx). ## Affected compose files - `compose/agent-proxy.yml` — `INCIDENT_SERVICE_URL`, `SERVICES_DEVICES_URL`, `SERVICES_FLEET_URL` - `compose/analytics.yml` — `INCIDENT_SERVICE_URL`, `SERVICES_DEVICES_URL`, `SERVICES_FLEET_URL` - `compose/recon.yml` — `SERVICES_INCIDENT_URL` - `compose/reports.yml` — `INCIDENT_SERVICE_URL` ## Impact Discovered 2026-06-29 after stopping the green standby instances post-maintenance-window. Agent-proxy was forwarding all agent heartbeats and journal events to the stopped `hiveiq-incident-green` container, silently failing (agent got 200, internal call dropped). Result: 572 AGENT_OFFLINE incidents stuck open, plus dead internal calls in analytics, recon, and reports. ## Fix required `bluegreen-deploy.sh` must sed the active color in all dependent compose files in addition to updating `00-backends.conf`, then run `docker compose up -d --no-deps` on the affected containers so they pick up the corrected env vars on each slot flip. ## Workaround applied 2026-06-29 Manually sed all 4 compose files (green → blue) and restarted agent-proxy, analytics, recon, reports.
hiveiq added the BugPriority/High labels 2026-06-29 07:23:08 -04:00
hiveiq self-assigned this 2026-06-29 07:23:08 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Abandoned
Somebody has started to work on this but abandoned work
 Blocked
Something is blocking this issue or pull request
 Breaking
Breaking change that won't be backward compatible
 Bug
Something is not working
 Confirmed
Issue has been confirmed
 Documentation
Documentation changes
 Duplicate
This issue or pull request already exists
 Enhancement
Improve existing functionality
 Feature
New functionality
 Invalid
Invalid issue
 Need More Info
Feedback is required to reproduce issue or to continue work
 Priority/Critical
The priority is critical
 Priority/High
The priority is high
 Priority/Low
The priority is low
 Priority/Medium
The priority is medium
 Security
This is security issue
 Testing
Issue or pull request related to testing
 Won't Fix
This issue won't be fixed
Backend
Service, API, or data-layer work
 Breaking
Breaking change that won't be backward compatible
 Bug
Something is not working
 Dev Verified
Fix verified on bcos.dev by QA — cleared for production release
 Documentation
Documentation changes
 Enhancement
Improve existing functionality
 Feature
New functionality
 Frontend
UI / Svelte SPA work
 Pattern
For changes to our agent Pattern for detecting hardware faults
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reopen
Fix was verified but the issue persists or regressed. Reopen and reassign for investigation.
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Security
This is security issue
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
TEST
Tested
Not Fixed
Testing
Issue or pull request related to testing
 Verify Fix
Fix has been deployed to production. Verify the issue is resolved before closing.
 Verify/Production
Fix deployed to production — verify before closing
No Label Bug Priority/High
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
hiveiq hiveiq-browser johannes tristan (Tristan Smith)
No Assignees hiveiq
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: hiveiq-src/hiveiq-ops#3
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?