bcos.dev: no container memory limits — every JVM sizes its heap to 3.9GB, causing host-wide OOM cascades #6

New Issue

2026-07-18T15:23:43-04:00

hiveiq commented

2026-07-18 15:23:43 -04:00

Summary

All 56 containers on bcos.dev (.251) run with MemLimit=0. Because the JVM's UseContainerSupport has no limit to detect, it falls back to host RAM and sizes MaxHeapSize to 25% of 15.6GB = 3.9GB per service. With 26 JVM-class containers, that is ~100GB of theoretical heap entitlement on a 15.6GB box.

They coexist only because they don't all claim it simultaneously. When one service takes sustained load it grows unbounded and drags the host into global_oom, killing bystanders by OOM score.

Impact (observed 2026-07-18)

guide.bcos.dev returned 502 for a period this morning. Root cause was a global OOM burst, not a Guide fault. Kernel log shows repeated kills roughly every 5 minutes:

oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task=java
Out of memory: Killed process (java) anon-rss:1625000kB
Out of memory: Killed process (ollama) anon-rss:1619104kB

constraint=CONSTRAINT_NONE + global_oom confirms host-wide exhaustion, not a per-cgroup limit.

Cumulative restart counts show this is chronic, not a one-off:

Container	RestartCount
hiveiq-incident-blue	344
hiveiq-incident-green	182 (currently exited)
hiveiq-kafka	127

Evidence

Inside hiveiq-incident-blue (Temurin 21.0.11):

MaxHeapSize         = 4192206848   (3.9 GB)  {ergonomic}
MaxRAMPercentage    = 25.0                   {default}
UseContainerSupport = true                   {default}
JAVA_OPTS           = (unset)

Config tree hiveiq-openmetal-dev/ contains zero occurrences of mem_limit, resources:, MaxRAMPercentage, or Xmx.

Current footprint

14.32 GiB used across 56 containers on a 15.61 GiB box, with zero swap. Largest consumers:

Container	MB
hiveiq-kafka	1215
hiveiq-incident-blue	1163
hiveiq-ollama	672
hiveiq-recon	596
hiveiq-auth-green	563
hiveiq-auth-blue	506

Proposed fix

Memory limits do not create capacity — they contain blast radius, converting a host-wide outage into a single-container restart, and they make each JVM size its heap correctly. Limits are ceilings, not reservations, so the sum may exceed physical RAM.

Add mem_limit to the x-*-base anchors in hiveiq-openmetal-dev/hiveops/instances/dev/compose/*.yml.
Set JAVA_TOOL_OPTIONS=-XX:MaxRAMPercentage=50 so heap is bounded by the container limit.
Stop the idle standby slots hiveiq-auth-green + hiveiq-devices-backend-green (~985MB reclaimed; blue/green does not need both hot on dev).
Add swap to give the kernel headroom instead of jumping straight to the OOM killer.
Evaluate whether hiveiq-ollama (672MB) is still in use on this box.

Out of scope / follow-ups

hiveiq-incident-blue emits Spring Security DEBUG for every heartbeat and journal-event POST despite SPRING_PROFILES_ACTIVE=prod. Not set in compose, so it originates from Spring Cloud Config or the image default. Worth a separate ticket — it is significant allocation churn on a box with no headroom.
Disk is healthy (root 38%, /data/hiveiq 35%); ~31GB of dangling images is reclaimable but unrelated to this incident.

Filed by session 1.

## Summary All 56 containers on bcos.dev (`.251`) run with `MemLimit=0`. Because the JVM's `UseContainerSupport` has no limit to detect, it falls back to **host** RAM and sizes `MaxHeapSize` to 25% of 15.6GB = **3.9GB per service**. With 26 JVM-class containers, that is ~100GB of theoretical heap entitlement on a 15.6GB box. They coexist only because they don't all claim it simultaneously. When one service takes sustained load it grows unbounded and drags the **host** into `global_oom`, killing bystanders by OOM score. ## Impact (observed 2026-07-18) `guide.bcos.dev` returned **502** for a period this morning. Root cause was a global OOM burst, not a Guide fault. Kernel log shows repeated kills roughly every 5 minutes: ``` oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task=java Out of memory: Killed process (java) anon-rss:1625000kB Out of memory: Killed process (ollama) anon-rss:1619104kB ``` `constraint=CONSTRAINT_NONE` + `global_oom` confirms host-wide exhaustion, not a per-cgroup limit. Cumulative restart counts show this is chronic, not a one-off: | Container | RestartCount | |---|---| | hiveiq-incident-blue | 344 | | hiveiq-incident-green | 182 (currently exited) | | hiveiq-kafka | 127 | ## Evidence Inside `hiveiq-incident-blue` (Temurin 21.0.11): ``` MaxHeapSize = 4192206848 (3.9 GB) {ergonomic} MaxRAMPercentage = 25.0 {default} UseContainerSupport = true {default} JAVA_OPTS = (unset) ``` Config tree `hiveiq-openmetal-dev/` contains **zero** occurrences of `mem_limit`, `resources:`, `MaxRAMPercentage`, or `Xmx`. ## Current footprint **14.32 GiB used across 56 containers on a 15.61 GiB box, with zero swap.** Largest consumers: | Container | MB | |---|---| | hiveiq-kafka | 1215 | | hiveiq-incident-blue | 1163 | | hiveiq-ollama | 672 | | hiveiq-recon | 596 | | hiveiq-auth-green | 563 | | hiveiq-auth-blue | 506 | ## Proposed fix Memory limits do not create capacity — they **contain blast radius**, converting a host-wide outage into a single-container restart, and they make each JVM size its heap correctly. Limits are ceilings, not reservations, so the sum may exceed physical RAM. 1. Add `mem_limit` to the `x-*-base` anchors in `hiveiq-openmetal-dev/hiveops/instances/dev/compose/*.yml`. 2. Set `JAVA_TOOL_OPTIONS=-XX:MaxRAMPercentage=50` so heap is bounded by the container limit. 3. Stop the idle standby slots `hiveiq-auth-green` + `hiveiq-devices-backend-green` (~985MB reclaimed; blue/green does not need both hot on dev). 4. Add swap to give the kernel headroom instead of jumping straight to the OOM killer. 5. Evaluate whether `hiveiq-ollama` (672MB) is still in use on this box. ## Out of scope / follow-ups - `hiveiq-incident-blue` emits Spring Security `DEBUG` for every heartbeat and journal-event POST despite `SPRING_PROFILES_ACTIVE=prod`. Not set in compose, so it originates from Spring Cloud Config or the image default. Worth a separate ticket — it is significant allocation churn on a box with no headroom. - Disk is healthy (root 38%, `/data/hiveiq` 35%); ~31GB of dangling images is reclaimable but unrelated to this incident. Filed by session 1.

hiveiq added the Bug Priority/High labels 2026-07-18 15:23:43 -04:00

hiveiq commented

2026-07-18 18:16:32 -04:00

Phase 1 done + PR open

Reclaimed ~474Mi on .251: stopped hiveiq-devices-backend-green (available 408Mi -> 882Mi). Verified by behaviour, not status code — blue processed 320 heartbeat lines in the following 60s, devices API and SPA unaffected.

PR: hiveiq-ops/hiveiq-openmetal-dev#10 — mem_limit + MaxRAMPercentage=50 on kafka, incident b/g, recon, auth b/g. Not yet merged, and merging does not deploy (the .251 tree is a hand-managed copy, not a git checkout).

Correction to this issue's framing

The original text called the green slots "idle standby". That is wrong, and it matters:

hiveiq-auth-green is the active auth backend per the live nginx map, not blue. Stopping it would have taken down auth for all of bcos.dev.
Blue/green on dev is not active/passive for internal callers. Both colours share a network alias (hiveiq-auth, hiveiq-devices-backend, hiveiq-incident), and Docker DNS round-robins across it. Measured: hiveiq-auth resolved to blue 3/10 and green 7/10. The nginx $*_backend map only governs traffic that passes through nginx; anything resolving the alias hits both colours regardless of which is "active".

So devices-backend-green was serving roughly half the alias-addressed internal traffic — including agent heartbeats — right up to being stopped. Blue absorbed it cleanly, but it was not a cold spare.

Consequence for anyone acting on this issue: do not assume a colour is idle. Check the nginx map AND the shared alias before stopping anything.

Not done

auth-blue not stopped (~506MB). It serves live internal traffic via the alias. Stopping it shifts all internal auth to green one day after cutover and removes the rollback slot.
Remaining ~50 containers deferred until the four in PR #10 prove stable.
Swap — still zero on .251. Separate change.
24GB resize considered and declined for now. Worth noting it would raise the per-JVM default heap from 3.9GB to 6GB, so it does not fix the mechanism on its own.

Separate bug found

devices-blue logs java.net.UnknownHostException: kafka — consumers point at kafka:29092, container is hiveiq-kafka. Devices' Kafka consumers have never connected. Plausibly related to kafka's 127 restarts. Needs its own ticket.

Not a factor

Disk. Root 38%, /data/hiveiq 35%. ~31GB of dangling images is reclaimable but irrelevant to this.

## Phase 1 done + PR open **Reclaimed ~474Mi** on `.251`: stopped `hiveiq-devices-backend-green` (available 408Mi -> 882Mi). Verified by behaviour, not status code — blue processed **320 heartbeat lines in the following 60s**, devices API and SPA unaffected. **PR:** hiveiq-ops/hiveiq-openmetal-dev#10 — `mem_limit` + `MaxRAMPercentage=50` on kafka, incident b/g, recon, auth b/g. Not yet merged, and **merging does not deploy** (the `.251` tree is a hand-managed copy, not a git checkout). ## Correction to this issue's framing The original text called the green slots "idle standby". **That is wrong**, and it matters: - `hiveiq-auth-green` is the **active** auth backend per the live nginx map, not blue. Stopping it would have taken down auth for all of bcos.dev. - Blue/green on dev is **not active/passive for internal callers**. Both colours share a network alias (`hiveiq-auth`, `hiveiq-devices-backend`, `hiveiq-incident`), and Docker DNS round-robins across it. Measured: `hiveiq-auth` resolved to blue 3/10 and green 7/10. The nginx `$*_backend` map only governs traffic that passes through nginx; anything resolving the alias hits **both** colours regardless of which is "active". So `devices-backend-green` was serving roughly half the alias-addressed internal traffic — including agent heartbeats — right up to being stopped. Blue absorbed it cleanly, but it was not a cold spare. **Consequence for anyone acting on this issue: do not assume a colour is idle. Check the nginx map AND the shared alias before stopping anything.** ## Not done - **`auth-blue` not stopped** (~506MB). It serves live internal traffic via the alias. Stopping it shifts all internal auth to green one day after cutover and removes the rollback slot. - **Remaining ~50 containers** deferred until the four in PR #10 prove stable. - **Swap** — still zero on `.251`. Separate change. - **24GB resize** considered and declined for now. Worth noting it would *raise* the per-JVM default heap from 3.9GB to 6GB, so it does not fix the mechanism on its own. ## Separate bug found `devices-blue` logs `java.net.UnknownHostException: kafka` — consumers point at `kafka:29092`, container is `hiveiq-kafka`. Devices' Kafka consumers have never connected. Plausibly related to kafka's 127 restarts. Needs its own ticket. ## Not a factor Disk. Root 38%, `/data/hiveiq` 35%. ~31GB of dangling images is reclaimable but irrelevant to this.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hiveiq-src/hiveiq-ops#6