docs(guide): incident.processing-rules — institution match is KEY to KEY

The module described pre-#290 behaviour: rule key resolved to a display NAME,
compared to the ATM's institution name. The code has not worked that way since
2d2dee0 — it compares rule.institution to atm.institutionKey directly,
case-insensitively, and never resolves a name. institutionKeyService is still
injected but the matcher does not call it.

This was not harmless. Investigating incident#241 (a PBNC EMV rule a customer
reported twice), the stale gotcha led to the conclusion that the rule was
silently falling through to global and that #232 was the root cause. Both wrong
— the rule fires correctly in prod (Loki: SUPPRESS 'EMV Fallback Suppression
for Peoples Bank ONLY' on NC000936, institution=PBNC). That wrong conclusion was
posted to a customer-facing ticket and had to be retracted. The Guide exists to
prevent exactly that.

- claude.md   — gotcha rewritten to key-to-key; adds a do-not-re-diagnose note
- internal.md — failure-mode row + Matching gotcha section rewritten; keeps the
                silent-fallback-to-global warning, which is still real when a
                key genuinely mismatches, and marks the name-based version as
                historical (#232/#153, fixed 2d2dee0, live 2026-07-16)
- architect.md — evaluation order step 3 corrected

claude.md's "rule institution is stored as the KEY, not a display name" was
already correct and is unchanged.

Refs #29
This commit is contained in:
2026-07-17 08:20:56 -04:00
parent acab7befa4
commit e52635191c
3 changed files with 9 additions and 4 deletions
@@ -49,7 +49,7 @@ device event ──► hiveops.device.events ──► DeviceEventConsumer
Rule selection precedence inside `findApplicableRule`:
1. Filter enabled rules for the event type (`findEnabledByEventType`).
2. Drop rules whose `text_pattern` (if set) is not `contains`-ed by the lowercased event details.
3. Prefer **institution-scoped** rules (rule institution key → institution name, compared to ATM institution) over **Global** (`institution` null/blank).
3. Prefer **institution-scoped** rules (rule's `institution` key compared case-insensitively to the ATM's `institution_key` — key to key, no name resolution) over **Global** (`institution` null/blank).
4. Within a scope, **text-pattern-specific** rules win over catch-all (`Comparator` on pattern presence).
5. No match → built-in default incident behaviour.
@@ -71,7 +71,8 @@ Adoons — `@RequestMapping("/api/adoons")` + `/api/adoons/rule-suggestions`:
## Gotchas
- Opening-rule match precedence: institution-scoped > global; within scope, text-pattern-specific > catch-all.
- `text_pattern` match = case-insensitive `contains` on event details; blank/null = matches any.
- Institution match resolves rule's stored `institution` as a **KEY** → institution NAME, then compares to ATM's institution name; unresolved key falls back to raw value. Key-vs-name mismatch = rule silently falls through to global.
- Institution match is **KEY to KEY**: `rule.institution` (a tenant key, e.g. `PBNC`) is compared case-insensitively to `atm.institutionKey`. No name resolution — `institutionKeyService` is injected but NOT used by the matcher. A rule whose key doesn't match the ATM's key **silently falls through to global** (no error, no log of the miss).
- ⚠️ Pre-#290 this resolved the key to a display NAME and compared it to `atms.institution` — NULL for migrated devices, so NO scoped rule could fire. That was the #232/#153 defect, fixed in `2d2dee0` and live in prod since 2026-07-16. **Do not re-diagnose a non-firing rule as this bug** — verify with the `Rule eval:` log line first, which prints the institution actually matched.
- Recovery/close path applies built-in `RECOVERY_MAP` BEFORE configurable closing rules.
- Unrecognised event types are recorded to `event_exceptions` by `JournalEventService`, not incidents.
- `event_type` strings are SCREAMING_SNAKE_CASE; exact match required.
@@ -59,12 +59,16 @@ Rule suggestion enqueue/poll (`/api/adoons/rule-suggestions`) is **not** admin-g
|---------|-------------|-----|
| Rule mutation returns 403 | Caller is not MSP_ADMIN/BCOS_ADMIN | Use an admin JWT |
| Rule saved but never fires | Disabled, wrong event type, or over-specific text pattern | Re-check enabled flag, exact event type, and pattern |
| Institution rule ignored | ATM institution name doesn't equal the rule's institution *name* (rule stores institution **key**, resolved to name at match time — see gotcha) | Verify the institution key resolves to the ATM's institution name |
| Institution rule ignored | The rule's institution **key** doesn't equal the ATM's `institution_key` (matched key-to-key, case-insensitive) — the rule then silently falls through to Global | Compare the rule's `institution` value against the ATM's `institution_key`; check the `Rule eval:` log line for which institution actually matched |
| Unknown event floods Exceptions | Agent emitting an event type with no `event_type` lookup / no rule | Create a rule or add the event type; then Dismiss the exception |
| Suggestion never returns | hiveops-ai down or Kafka topic not consumed | Check hiveops-ai health + `adoons.rule-suggestion.ready` |
## Matching gotcha (institution scope)
In `findApplicableRule`, a rule's stored `institution` value is treated as an **institution key**, resolved via `institutionKeyService.findByKey(...).getInstitutionName()` and compared to the ATM's `institution` (a **name**). If the key doesn't resolve, it falls back to comparing the raw stored value. A mismatch between what's stored (key vs name) and the ATM's institution name will cause an institution-scoped rule to silently fall through to Global. Verify with the `Rule eval:` log line, which prints the ATM institution it tried to match.
In `findApplicableRule` (and identically in `processConfigurableClosingRules`), a rule's stored `institution` value is an **institution key** and is compared **directly, case-insensitively, to the ATM's `institution_key`**. There is no name resolution — `institutionKeyService` is injected on the class but the matcher does not call it.
If the rule's key doesn't equal the ATM's key, the scoped rule **silently falls through to Global** — no error and no log line for the miss. That fallback is by design (institution-scoped wins over global; if nothing scoped matches, global applies), but it means a typo'd key looks identical to "no scoped rule exists". Verify with the `Rule eval:` log line, which prints the institution that actually matched (`institution=PBNC`, or `institution=global`).
> **⚠️ Historical — do not re-diagnose this.** Before #290, the matcher resolved the rule's key to a display *name* and compared it to `atms.institution`, which the device-identity migration left NULL for migrated devices — so **no** institution-scoped rule could ever fire. That was #232/#153, fixed in `2d2dee0` and live in production since 2026-07-16 (verified: PBNC's EMV suppression rule firing on NC000936). This stale description previously caused a wrong root-cause call on a customer ticket (#241) — if a scoped rule isn't firing today, check the key values and the `Rule eval:` line, not this.
Related: [technical.incident] · [platform.service-ownership]