guide: document supervisor/service-state mirror (devices#121)
CD - Develop (guide → bcos.dev) / build-and-deploy (push) Failing after 36m22s

incident.journal-events (Claude tab): the state flip is a dual-write — incident
writes its own atm_properties, then mirrors to devices, and it is devices the UI
reads. Adds the best-effort/afterCommit semantics and the nginx-listener URL rule.

technical.devices: document POST /api/internal/atms/{agentAtmId}/service-state,
that devices does not ingest journal events, the now()-stamped supervisor_entered_at,
and that the status badge is derived client-side (agent-offline masks IN_SUPERVISOR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 17:54:39 -04:00
parent 7429f1413b
commit ffc06e2ef5
2 changed files with 28 additions and 0 deletions
@@ -51,6 +51,16 @@ Act-without-guessing reference. Everything below is confirmed in code.
- `createEvent` is `@Retryable` on `PessimisticLockingFailureException` (incident#212 deadlock guard).
- `eventTime` parsed as `LocalDateTime` (no timezone); unparseable/blank → `now()`.
- Auto-registers ATM on `agentAtmId` via `findOrCreateAtm`; also updates last heartbeat + may flip service state (`ATM_IN_SERVICE`/`ATM_OUT_OF_SERVICE`/`SUPERVISOR_ENTER`/`SUPERVISOR_EXIT`) + trigger auto-incident.
- **The state flip is DUAL-WRITE (devices#121).** Incident writes its own `atm_properties`, then mirrors to
**hiveops-devices** via `POST /api/internal/atms/{agentAtmId}/service-state` (`DevicesInternalService`).
Devices is what every status badge reads — nginx rewrites `/incident/api/atms*` to the devices backend —
so **incident's own copy is NOT what the UI shows.** Before 2026-07-09 the mirror did not exist and no
device ever displayed IN SUPERVISOR. Incident keeps its local write only because
`resolveOpenSupervisorTimeout` depends on it.
- The mirror is **best-effort** (a devices outage logs a warning, never fails ingest) and fires on
`afterCommit` — `createEvent` is `@Retryable`, so a rolled-back attempt must not mirror.
- `devices.service.url` defaults to `http://hiveiq-nginx:8096` (nginx internal listener), **never** a
container name — devices runs blue/green and the active colour flips on cutover.
- Category filter in UI is an allowlist of exact type strings (8 groups); types outside the groups are hidden when any category is ticked.
- Per-ATM read endpoints filter by `atmId` only — they do NOT apply `InstitutionContext` scope (only `/activity` and `/fleet/cassette-inventory` do).
@@ -61,3 +71,5 @@ Act-without-guessing reference. Everything below is confirmed in code.
- Do NOT expect a Kafka topic for journal ingest — there is none.
- Do NOT assume this view is admin-gated — any authenticated user role can read it.
- Do NOT treat a 202 on ingest as "stored" — it means the type was unknown.
- Do NOT read `hiveiq_incident.atm_properties` to answer "is this ATM in supervisor?" — the UI reads
`hiveiq_devices.atm_properties`. The two can diverge; devices is what the customer sees.
@@ -81,6 +81,7 @@ Base paths: `/api/atms` + `/api/devices`, `/api/atm-groups`, `/api/atm-makes`, `
| GET/POST/PUT/DELETE `/api/agent-profiles` · GET `/api/agent-profiles/modules-manifest` | Agent profiles + CDN modules manifest proxy |
| POST `/api/internal/atms/register` | Agent auto-registration (INTERNAL) |
| POST `/api/internal/atms/agent/{country}/{atm}/heartbeat` | Agent heartbeat (INTERNAL) |
| POST `/api/internal/atms/{agentAtmId}/service-state` | Service-state / supervisor mirror from incident (INTERNAL). Body `{"eventType":"SUPERVISOR_ENTER"}`. 404 if the ATM does not exist — never auto-creates |
| POST `/api/internal/agent-tokens/validate` | Token validation (INTERNAL) |
## Gotchas
@@ -91,4 +92,19 @@ Base paths: `/api/atms` + `/api/devices`, `/api/atm-groups`, `/api/atm-makes`, `
- **Phase-2 path aliases.** `/api/atms`, `/api/atm-groups`, `/api/atm-makes`, `/api/atm-models`, `/api/atm-field-definitions` each also respond on a `/api/device*` alias for the ongoing ATM→device rename; both must stay routable in nginx.
- **CDN token.** `agent.cdn.token` (env `CDN_TOKEN`) has a hardcoded fallback in `application.properties` used by the `/api/agent-profiles/modules-manifest` proxy.
- **Publish-on-every-change.** Device mirrors in incident/fleet/reports/recon depend on `hiveops.device.events`; any write path that skips `DeviceEventPublisher` leaves those mirrors stale (no reconciler).
- **Supervisor / service state is written from OUTSIDE this service.** Journal events land in
`hiveops-incident`, which applies the transition to its own tables and then mirrors it here via
`POST /api/internal/atms/{agentAtmId}/service-state` (`ATM_IN_SERVICE` / `ATM_OUT_OF_SERVICE` /
`SUPERVISOR_ENTER` / `SUPERVISOR_EXIT` → `AtmService.setAtmInService` / `setAtmOutOfService` /
`setSupervisorEnter` / `setSupervisorExit`). Devices does **not** ingest journal events itself.
Added 2026-07-09 (devices#121): those four methods had existed with **zero callers** since the
extraction, so `in_supervisor` was never set and no device displayed IN SUPERVISOR for ~a month,
while `hiveiq_incident.atm_properties` held the correct value that nothing read.
- **`supervisor_entered_at` is stamped `now()`** by the mirror endpoint — it takes no timestamp
argument. A backfill therefore records the backfill time, not the true entry time; correct it
directly if the "in supervisor since" value matters. Exposed on `/api/atms/{id}/properties`
(not on the list DTO).
- **Status badge is derived client-side**, not stored: `AGENT_OFFLINE` if the agent is disconnected/never-connected,
else `IN_SUPERVISOR` if `inSupervisor`, else `serviceState || status`. An offline agent therefore **masks**
supervisor mode in the UI.
- **Error message hiding.** `server.error.include-message=never` and `management.endpoint.health.show-details=never` — expect terse errors in prod.