docs(guide): institution write paths + the institution_locked operator lock (devices#170)
CD - Develop (guide → bcos.dev) / build-and-deploy (push) Has started running

Records what actually governs atms.institution, after feedback #792 (NC000937
kept reverting from Peoples Bank to Five Star).

- Three writers, one lock: updateLastHeartbeat() (guarded, never overwrites),
  syncAtmConfig() (agent-reported key; now skips a locked device and WARNs),
  moveInstitution() (operator intent — sets atms.institution_locked).
- The fingerprint for debugging "it changed back": `Assigned institution ...` is
  syncAtmConfig(); `Auto-assigned institution ...` is the guarded heartbeat path
  and is not the culprit. devices#93 blamed the wrong one for weeks.
- The key a device reports is NOT its authenticated identity: the agent reads it
  from `incident.institution.key` in its own hiveops.properties, and agent-proxy's
  putIfAbsent keeps it over the token-derived key. Fixing agent.auth.token alone
  does not stop a mis-imaged box re-asserting the wrong institution.
- Pre-V19 rows ship unlocked; re-run the move to lock one.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-17 10:26:37 -04:00
parent ab717062d6
commit 6c728bd96e
2 changed files with 30 additions and 4 deletions
@@ -21,7 +21,7 @@ Act-without-guessing reference for `devices.device-detail` (`AtmDetail.svelte`).
| `GET /api/atms/{id}/properties` | AtmProperties | authed |
| `PUT /api/atms/{id}` | Update / rename | `MSP_ADMIN`\|`BCOS_ADMIN` |
| `PUT /api/atms/{id}/properties` | Update properties | admin |
| `PATCH /api/atms/{id}/institution` | Move institution (upper-cases key) | admin |
| `PATCH /api/atms/{id}/institution` | Move institution (upper-cases key); **sets `institution_locked`** | admin |
| `PATCH /api/atms/{id}/status` | Set status | admin |
| `POST /api/atms/{id}/reset-state` | → IN_SERVICE, clear supervisor | admin |
| `PUT /api/atms/{id}/hardware-profile` | Set profile (does NOT queue task) | admin |
@@ -61,7 +61,13 @@ Act-without-guessing reference for `devices.device-detail` (`AtmDetail.svelte`).
- Hardware profile = **two calls**: `PUT .../hardware-profile` THEN `POST /fleet/tasks` (`configPatch:{'atm.hardware.profile':<key>}`). Backend alone does not queue the task.
- `DELETE` on an active device is a **soft delete** (status→INACTIVE) yet still emits `DEVICE_DELETED`; call again to hard-delete.
- `moveInstitution` upper-cases + trims the key; blank is rejected.
- `moveInstitution` upper-cases + trims the key; blank is rejected. It also **sets `atms.institution_locked = true`** — an explicit move is operator intent, and the lock is what makes it stick (#170).
- **Institution has TWO auto-write paths, and a lock that beats both** (#170):
- `updateLastHeartbeat()` — assigns only when institution is null/blank. Never overwrites. Logs `Auto-assigned institution ...`.
- `syncAtmConfig()` — assigns from the agent-reported key, but **skips a locked device** and logs a WARN (`reports institution key X but is locked to Y — ignoring`). Logs `Assigned institution ...` on the normal path.
- Before #170 `syncAtmConfig()` had no guard, so a `PATCH .../institution` reverted within the hour on any **online** box and only appeared to hold while the device was offline. If you are debugging "the institution changed back", `Assigned institution ...` in the devices log is your fingerprint.
- **The key a device reports is NOT its authenticated identity.** The agent sources it from `incident.institution.key` in its own `hiveops.properties` (agent `AtmConfigSyncModule`), and agent-proxy's `putIfAbsent` (`AtmSyncController`) keeps the agent-supplied value over the token-derived one. So fixing a mis-imaged box's `agent.auth.token` alone does **not** stop it re-asserting the wrong institution — push `incident.institution.key` too (CONFIG_UPDATE fleet task).
- Rows predating the V19 migration are **unlocked** (we cannot tell retroactively which were operator-set). Re-run the move to acquire the lock.
- `latest-log*` are `FileSystemResource` reads → error/404 if no file uploaded yet (not a bug).
- Auto-refresh every **60s**, paused while `editing`.
- `server.error.include-message=never` → terse 5xx; get real errors from Loki (`localhost:3100`, services VM).
@@ -21,7 +21,7 @@ audience: dev
| Table | Entity | Purpose |
|-------|--------|---------|
| `atms` | `Atm` | Core device inventory (source of truth) |
| `atms` | `Atm` | Core device inventory (source of truth). `institution_key` = the multi-tenant key; `institution_locked` pins an operator's move against config sync (#170) |
| `atm_properties` | `AtmProperties` | Per-device properties |
| `atm_groups` | `AtmGroup` | Device groupings |
| `atm_make` / `atm_model` | `AtmMake` / `AtmModel` | Make/model catalog |
@@ -71,7 +71,7 @@ Base paths: `/api/atms` + `/api/devices`, `/api/atm-groups`, `/api/atm-makes`, `
| GET `/api/atms/dashboard/stats` | Fleet dashboard stats |
| GET `/api/atms/summary` | Device summary |
| POST `/api/atms` | Create device (admin) |
| PATCH `/api/atms/{id}/institution` | Reassign institution (admin) |
| PATCH `/api/atms/{id}/institution` | Reassign institution (admin); sets `institution_locked` |
| POST `/api/atms/config/sync` · `/api/atms/hardware/sync` | Config/hardware sync ingest |
| GET `/api/atm-groups` · POST `/api/atm-groups/{id}/atms/{atmId}` | Groups + membership |
| GET/POST `/api/cassette-configurations` | Cassette templates |
@@ -93,6 +93,26 @@ Base paths: `/api/atms` + `/api/devices`, `/api/atm-groups`, `/api/atm-makes`, `
- **Phase-2 path aliases.** `/api/atms`, `/api/atm-groups`, `/api/atm-makes`, `/api/atm-models`, `/api/atm-field-definitions` each also respond on a `/api/device*` alias for the ongoing ATM→device rename; both must stay routable in nginx.
- **CDN token.** `agent.cdn.token` (env `CDN_TOKEN`) has a hardcoded fallback in `application.properties` used by the `/api/agent-profiles/modules-manifest` proxy.
- **Publish-on-every-change.** Device mirrors in incident/fleet/reports/recon depend on `hiveops.device.events`; any write path that skips `DeviceEventPublisher` leaves those mirrors stale (no reconciler).
- **Institution: three writers, one lock** (#170, V19). `atms.institution` / `institution_key` can be
written by (1) `updateLastHeartbeat()` — only when null/blank, never overwrites; (2) `syncAtmConfig()`
— from the key the agent reports; (3) `moveInstitution()` — the operator's explicit move, which sets
**`atms.institution_locked = true`**. A locked device is skipped by (2), which logs a WARN
(`reports institution key X but is locked to Y — ignoring`) instead of applying it.
- **Before #170, (2) had no guard at all**, so an operator's `PATCH .../institution` reverted on the
next config sync and only appeared to hold while the box was **offline**. That is the NC000937 /
feedback#792 case: a Peoples Bank ATM carrying the Five Star token re-stamped itself Five Star 7×
in one day. If a device "changes institution back", grep the devices log for
`Assigned institution ...` — that line is `syncAtmConfig()`; `Auto-assigned institution ...` is the
guarded heartbeat path and is **not** the culprit.
- **The reported key is not the authenticated identity.** The agent reads it from
`incident.institution.key` in its own `hiveops.properties` (`AtmConfigSyncModule`), and
agent-proxy's `putIfAbsent` (`AtmSyncController`) keeps the agent-supplied value over the
token-derived one. Correcting a mis-imaged box's `agent.auth.token` alone is therefore **not
enough** — push `incident.institution.key` in the same CONFIG_UPDATE, or it keeps asserting the
wrong institution.
- Rows created before V19 are **unlocked** (there is no way to tell retroactively which were
operator-set, and locking all of them would freeze fleet auto-assign). Re-run the move to lock one.
- Auto-assign for never-classified devices is unchanged: unlocked devices still follow their config.
- **Supervisor / service state is written from OUTSIDE this service.** Journal events land in
`hiveops-incident`, which applies the transition to its own tables and then mirrors it here via
`POST /api/internal/atms/{agentAtmId}/service-state` (`ATM_IN_SERVICE` / `ATM_OUT_OF_SERVICE` /