docs(guide): add IAT_SERVER_RECOVERED / IAT_CORE_RECOVERED to activeteller module

The activeteller agent module now emits two connectivity-recovery events
(hiveops-agent#68) that drive incident auto-close. Adds them to the event lists
(internal + claude), bumps the count 9→11, and corrects the SymXchange behaviour
in architect.md: the connection-error streak resets (and IAT_CORE_RECOVERED fires)
on the next message header, not on a fault line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

backend/src/main/resources/content/agent/activeteller/architect.md

@@ -70,7 +70,7 @@ Line format: `yyyy-MM-dd HH:mm:ss <message>`. Maintains single-customer session
 - A payload containing `"TellerSessionRequestId"` → session assigned (extracts `"TellerName"`; dedups repeated echoes of the same payload); clears `requestAt`, resets video flags.
 - `OnAdd for RemoteControlSession` (while a session is active) → `videoUp = true`.
 - `Ending teller session` / `OnDelete for TellerSessionRequest` → session end; if video never came up, emit `IAT_VIDEO_SESSION_FAIL`.
-- REST outcome `(https?://…) server response (\w+)` → success statuses `{OK, Created, Accepted, NoContent, NotModified, ResetContent}` reset the failure streak; anything else emits `IAT_SERVER_ERROR` and, at `failureThreshold` consecutive failures, `IAT_SERVER_UNREACHABLE` (once, until a success resets it).
+- REST outcome `(https?://…) server response (\w+)` → success statuses `{OK, Created, Accepted, NoContent, NotModified, ResetContent}` reset the failure streak (and, if `IAT_SERVER_UNREACHABLE` had been reported, emit `IAT_SERVER_RECOVERED`); anything else emits `IAT_SERVER_ERROR` and, at `failureThreshold` consecutive failures, `IAT_SERVER_UNREACHABLE` (once, until a success resets it).
 - A payload containing `"ModeType"` with `"ModeName":"(\w+)"` → `IAT_MODE_CHANGE` (suppressed on first observation and on unchanged mode).
 - **`checkTimers()`** runs after each batch, comparing `requestAt`/`sessionAt` against **`logClock`** (not wall-clock): overdue request → `IAT_TELLER_UNANSWERED`; session with no video past `videoTimeout` → `IAT_VIDEO_SESSION_FAIL`. Driving timers off the log clock means replaying a backlog can't raise spurious timeouts.
 
@@ -80,7 +80,8 @@ The SymXchange log alternates header lines (`[yyyy-MM-dd HH:mm:ss-nnn][OpName]`)
 
 - A header sets `currentTs`/`currentOp`, resets `faultReportedForBlock`, bumps `messageCount`.
 - A `CONN_ERROR`-matching line (unable to connect / no connection / refused / timed out / DNS) increments a streak; at `failureThreshold` consecutive → `IAT_CORE_UNREACHABLE` (once until reset).
-- A `FAULT`-matching line (`faultstring`, `<faultcode`, `:Fault>`, `<Fault>`, `<ErrorMessage>`, non-zero `<ErrorCode>`) → `IAT_CORE_TXN_FAIL` (at most once per header block); a fault/response also resets the connection-error streak. The fault/conn regexes are conservative heuristics noted for tuning against a real failure sample.
+- Each SymXchange message **header** clears the connection-error streak — any response (incl. a fault) proves the core is reachable — and, if `IAT_CORE_UNREACHABLE` had been reported, emits `IAT_CORE_RECOVERED`.
+- A `FAULT`-matching line (`faultstring`, `<faultcode`, `:Fault>`, `<Fault>`, `<ErrorMessage>`, non-zero `<ErrorCode>`) → `IAT_CORE_TXN_FAIL` (at most once per header block). The fault/conn regexes are conservative heuristics noted for tuning against a real failure sample.
 
 ### `CheckProcessingMonitor` — pipe-delimited records
 

backend/src/main/resources/content/agent/activeteller/claude.md

@@ -39,9 +39,11 @@ Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix = `
 | `IAT_VIDEO_SESSION_FAIL` | `IAT_ACTIVETELLER` | session assigned but no video/remote-control within `video.timeout.sec`, or session ended with no video |
 | `IAT_SERVER_ERROR` | `IAT_ACTIVETELLER` | Active Teller REST call returned non-2xx |
 | `IAT_SERVER_UNREACHABLE` | `IAT_ACTIVETELLER` | `server.failure.threshold` consecutive REST failures |
+| `IAT_SERVER_RECOVERED` | `IAT_ACTIVETELLER` | first REST success after an `IAT_SERVER_UNREACHABLE` (→ incident auto-close) |
 | `IAT_MODE_CHANGE` | `IAT_ACTIVETELLER` | `"ModeType"` payload, `ModeName` changed (not first observation) |
 | `IAT_CORE_TXN_FAIL` | `IAT_SYMXCHANGE` | SOAP fault / error line in SymXchange log (once per header block) |
 | `IAT_CORE_UNREACHABLE` | `IAT_SYMXCHANGE` | `core.failure.threshold` consecutive connection errors |
+| `IAT_CORE_RECOVERED` | `IAT_SYMXCHANGE` | next SymXchange message after an `IAT_CORE_UNREACHABLE` (→ incident auto-close) |
 | `IAT_CHECK_IMAGE_QUALITY` | `IAT_CHECK21` | Check21 status≠Processed, invalidity>threshold, or CAR/LAR gap≥threshold |
 | `IAT_HEARTBEAT` | `IAT_ACTIVETELLER` | every `heartbeat.interval.sec`, only if a monitor saw a line today |
 
@@ -57,14 +59,14 @@ Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix = `
 
 ## Behavior facts
 - Answer/video timers use the **log clock** (timestamp of last processed line), not wall-clock — backlog replay won't raise false timeouts.
-- Dedup guards: `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_UNREACHABLE`, `IAT_CORE_UNREACHABLE`, and per-block `IAT_CORE_TXN_FAIL` each fire once until state resets.
+- Dedup guards: `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_UNREACHABLE`, `IAT_CORE_UNREACHABLE`, and per-block `IAT_CORE_TXN_FAIL` each fire once until state resets. When an `*_UNREACHABLE` state resets (first REST success / next SymXchange message), the matching `IAT_SERVER_RECOVERED` / `IAT_CORE_RECOVERED` is emitted.
 - REST "success" whitelist: `OK, Created, Accepted, NoContent, NotModified, ResetContent`.
 - SymXchange fault/conn regexes are conservative heuristics — may need tuning against a real failure sample.
 
 ## Deployment
 - Delivered fleet-wide via **`INSTALL_MODULE`** fleet task (drops JAR into agent `modules/` dir, restarts agent — `ProcessInstallModule` in `hiveops-file-collection`).
 - Disable without uninstalling: add `activeteller` to `modules.disabled`, push `CONFIG_UPDATE`.
-- Register the nine `IAT_*` event types in hiveops-incident → Settings → Event Types (DB `event_type` table) for correct labeling.
+- Register the eleven `IAT_*` event types in hiveops-incident → Settings → Event Types (DB `event_type` table) for correct labeling.
 
 ## Do NOT
 - Do NOT expect the module to run on any `device.type` other than `NH_IAT` — inert by design elsewhere, not a bug.

backend/src/main/resources/content/agent/activeteller/internal.md

@@ -23,10 +23,12 @@ All three logs live under the same base folder configured by `activeteller.log.d
    - **`IAT_VIDEO_SESSION_FAIL`** — a teller session started but the video/remote-control session never came up (within `activeteller.video.timeout.sec`, or the session ended without video).
    - **`IAT_SERVER_ERROR`** — an Active Teller REST call returned a non-2xx status.
    - **`IAT_SERVER_UNREACHABLE`** — `activeteller.server.failure.threshold` consecutive REST failures.
+   - **`IAT_SERVER_RECOVERED`** — first successful Active Teller REST call after an `IAT_SERVER_UNREACHABLE`; lets the incident side auto-close the connectivity ticket.
    - **`IAT_MODE_CHANGE`** — scheduled operating-mode transition (e.g. Standard ↔ SelfService).
 2. **`SymXchange{date}.log`** (fixed daily name) — the Jack Henry **Symitar (SymXchange)** core-banking conversation. Reports:
    - **`IAT_CORE_TXN_FAIL`** — a SOAP fault / error response from the core (at most once per header block).
    - **`IAT_CORE_UNREACHABLE`** — `activeteller.core.failure.threshold` consecutive connection-level errors (unable to connect / timed out / refused / DNS).
+   - **`IAT_CORE_RECOVERED`** — next SymXchange message after an `IAT_CORE_UNREACHABLE` (a response, incl. a SOAP fault, proves the core is reachable); lets the incident side auto-close the core-banking ticket.
 3. **`CheckProcessingResults{date}.log`** (fixed daily name) — Check21 image capture. Emits informational **`IAT_CHECK_IMAGE_QUALITY`** when a check deposit's invalidity score is high, the CAR/LAR (courtesy/legal amount) disagree, or the status isn't "Processed" — cases a back-office reviewer would otherwise have to catch by hand.
 
 A rollup **`IAT_HEARTBEAT`** is emitted every `activeteller.heartbeat.interval.sec` (default 900s / 15 min), but **only if at least one log produced a line today** — it will not send an empty heartbeat on an idle/offline machine.
@@ -53,7 +55,7 @@ It also reuses the shared agent HTTP settings — `agent.endpoint` (preferred) o
 - Activates only on ATMs configured with `device.type=NH_IAT`. On any other device type it logs `activeteller: device.type='<X>', not NH_IAT — module disabled` and does nothing — expected, not a bug.
 - Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which drops the module JAR into the agent's `modules/` directory and restarts the agent service (see `ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order: lab ATM → pilot device → fleet.
 - Can be disabled fleet-wide without removing the JAR by adding `activeteller` to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE` (the id is **`activeteller`**, the module's `getName()`).
-- The nine event types it emits (`IAT_HEARTBEAT`, `IAT_TELLER_UNANSWERED`, `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_ERROR`, `IAT_SERVER_UNREACHABLE`, `IAT_MODE_CHANGE`, `IAT_CORE_TXN_FAIL`, `IAT_CORE_UNREACHABLE`, `IAT_CHECK_IMAGE_QUALITY`) are `IAT_*` custom types — they must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) to be recognized/labeled correctly downstream. They are not standard shared types.
+- The eleven event types it emits (`IAT_HEARTBEAT`, `IAT_TELLER_UNANSWERED`, `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_ERROR`, `IAT_SERVER_UNREACHABLE`, `IAT_SERVER_RECOVERED`, `IAT_MODE_CHANGE`, `IAT_CORE_TXN_FAIL`, `IAT_CORE_UNREACHABLE`, `IAT_CORE_RECOVERED`, `IAT_CHECK_IMAGE_QUALITY`) are `IAT_*` custom types — they must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) to be recognized/labeled correctly downstream. They are not standard shared types.
 
 ## Common failure modes / what to check