diff --git a/backend/src/main/resources/content/agent/activeteller/architect.md b/backend/src/main/resources/content/agent/activeteller/architect.md index 4c13cae..16571ab 100644 --- a/backend/src/main/resources/content/agent/activeteller/architect.md +++ b/backend/src/main/resources/content/agent/activeteller/architect.md @@ -70,7 +70,7 @@ Line format: `yyyy-MM-dd HH:mm:ss `. Maintains single-customer session - A payload containing `"TellerSessionRequestId"` → session assigned (extracts `"TellerName"`; dedups repeated echoes of the same payload); clears `requestAt`, resets video flags. - `OnAdd for RemoteControlSession` (while a session is active) → `videoUp = true`. - `Ending teller session` / `OnDelete for TellerSessionRequest` → session end; if video never came up, emit `IAT_VIDEO_SESSION_FAIL`. -- REST outcome `(https?://…) server response (\w+)` → success statuses `{OK, Created, Accepted, NoContent, NotModified, ResetContent}` reset the failure streak; anything else emits `IAT_SERVER_ERROR` and, at `failureThreshold` consecutive failures, `IAT_SERVER_UNREACHABLE` (once, until a success resets it). +- REST outcome `(https?://…) server response (\w+)` → success statuses `{OK, Created, Accepted, NoContent, NotModified, ResetContent}` reset the failure streak (and, if `IAT_SERVER_UNREACHABLE` had been reported, emit `IAT_SERVER_RECOVERED`); anything else emits `IAT_SERVER_ERROR` and, at `failureThreshold` consecutive failures, `IAT_SERVER_UNREACHABLE` (once, until a success resets it). - A payload containing `"ModeType"` with `"ModeName":"(\w+)"` → `IAT_MODE_CHANGE` (suppressed on first observation and on unchanged mode). - **`checkTimers()`** runs after each batch, comparing `requestAt`/`sessionAt` against **`logClock`** (not wall-clock): overdue request → `IAT_TELLER_UNANSWERED`; session with no video past `videoTimeout` → `IAT_VIDEO_SESSION_FAIL`. Driving timers off the log clock means replaying a backlog can't raise spurious timeouts. @@ -80,7 +80,8 @@ The SymXchange log alternates header lines (`[yyyy-MM-dd HH:mm:ss-nnn][OpName]`) - A header sets `currentTs`/`currentOp`, resets `faultReportedForBlock`, bumps `messageCount`. - A `CONN_ERROR`-matching line (unable to connect / no connection / refused / timed out / DNS) increments a streak; at `failureThreshold` consecutive → `IAT_CORE_UNREACHABLE` (once until reset). -- A `FAULT`-matching line (`faultstring`, ``, ``, ``, non-zero ``) → `IAT_CORE_TXN_FAIL` (at most once per header block); a fault/response also resets the connection-error streak. The fault/conn regexes are conservative heuristics noted for tuning against a real failure sample. +- Each SymXchange message **header** clears the connection-error streak — any response (incl. a fault) proves the core is reachable — and, if `IAT_CORE_UNREACHABLE` had been reported, emits `IAT_CORE_RECOVERED`. +- A `FAULT`-matching line (`faultstring`, ``, ``, ``, non-zero ``) → `IAT_CORE_TXN_FAIL` (at most once per header block). The fault/conn regexes are conservative heuristics noted for tuning against a real failure sample. ### `CheckProcessingMonitor` — pipe-delimited records diff --git a/backend/src/main/resources/content/agent/activeteller/claude.md b/backend/src/main/resources/content/agent/activeteller/claude.md index d7e026c..84b65ec 100644 --- a/backend/src/main/resources/content/agent/activeteller/claude.md +++ b/backend/src/main/resources/content/agent/activeteller/claude.md @@ -39,9 +39,11 @@ Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix = ` | `IAT_VIDEO_SESSION_FAIL` | `IAT_ACTIVETELLER` | session assigned but no video/remote-control within `video.timeout.sec`, or session ended with no video | | `IAT_SERVER_ERROR` | `IAT_ACTIVETELLER` | Active Teller REST call returned non-2xx | | `IAT_SERVER_UNREACHABLE` | `IAT_ACTIVETELLER` | `server.failure.threshold` consecutive REST failures | +| `IAT_SERVER_RECOVERED` | `IAT_ACTIVETELLER` | first REST success after an `IAT_SERVER_UNREACHABLE` (→ incident auto-close) | | `IAT_MODE_CHANGE` | `IAT_ACTIVETELLER` | `"ModeType"` payload, `ModeName` changed (not first observation) | | `IAT_CORE_TXN_FAIL` | `IAT_SYMXCHANGE` | SOAP fault / error line in SymXchange log (once per header block) | | `IAT_CORE_UNREACHABLE` | `IAT_SYMXCHANGE` | `core.failure.threshold` consecutive connection errors | +| `IAT_CORE_RECOVERED` | `IAT_SYMXCHANGE` | next SymXchange message after an `IAT_CORE_UNREACHABLE` (→ incident auto-close) | | `IAT_CHECK_IMAGE_QUALITY` | `IAT_CHECK21` | Check21 status≠Processed, invalidity>threshold, or CAR/LAR gap≥threshold | | `IAT_HEARTBEAT` | `IAT_ACTIVETELLER` | every `heartbeat.interval.sec`, only if a monitor saw a line today | @@ -57,14 +59,14 @@ Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix = ` ## Behavior facts - Answer/video timers use the **log clock** (timestamp of last processed line), not wall-clock — backlog replay won't raise false timeouts. -- Dedup guards: `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_UNREACHABLE`, `IAT_CORE_UNREACHABLE`, and per-block `IAT_CORE_TXN_FAIL` each fire once until state resets. +- Dedup guards: `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_UNREACHABLE`, `IAT_CORE_UNREACHABLE`, and per-block `IAT_CORE_TXN_FAIL` each fire once until state resets. When an `*_UNREACHABLE` state resets (first REST success / next SymXchange message), the matching `IAT_SERVER_RECOVERED` / `IAT_CORE_RECOVERED` is emitted. - REST "success" whitelist: `OK, Created, Accepted, NoContent, NotModified, ResetContent`. - SymXchange fault/conn regexes are conservative heuristics — may need tuning against a real failure sample. ## Deployment - Delivered fleet-wide via **`INSTALL_MODULE`** fleet task (drops JAR into agent `modules/` dir, restarts agent — `ProcessInstallModule` in `hiveops-file-collection`). - Disable without uninstalling: add `activeteller` to `modules.disabled`, push `CONFIG_UPDATE`. -- Register the nine `IAT_*` event types in hiveops-incident → Settings → Event Types (DB `event_type` table) for correct labeling. +- Register the eleven `IAT_*` event types in hiveops-incident → Settings → Event Types (DB `event_type` table) for correct labeling. ## Do NOT - Do NOT expect the module to run on any `device.type` other than `NH_IAT` — inert by design elsewhere, not a bug. diff --git a/backend/src/main/resources/content/agent/activeteller/internal.md b/backend/src/main/resources/content/agent/activeteller/internal.md index e2a4326..db619be 100644 --- a/backend/src/main/resources/content/agent/activeteller/internal.md +++ b/backend/src/main/resources/content/agent/activeteller/internal.md @@ -23,10 +23,12 @@ All three logs live under the same base folder configured by `activeteller.log.d - **`IAT_VIDEO_SESSION_FAIL`** — a teller session started but the video/remote-control session never came up (within `activeteller.video.timeout.sec`, or the session ended without video). - **`IAT_SERVER_ERROR`** — an Active Teller REST call returned a non-2xx status. - **`IAT_SERVER_UNREACHABLE`** — `activeteller.server.failure.threshold` consecutive REST failures. + - **`IAT_SERVER_RECOVERED`** — first successful Active Teller REST call after an `IAT_SERVER_UNREACHABLE`; lets the incident side auto-close the connectivity ticket. - **`IAT_MODE_CHANGE`** — scheduled operating-mode transition (e.g. Standard ↔ SelfService). 2. **`SymXchange{date}.log`** (fixed daily name) — the Jack Henry **Symitar (SymXchange)** core-banking conversation. Reports: - **`IAT_CORE_TXN_FAIL`** — a SOAP fault / error response from the core (at most once per header block). - **`IAT_CORE_UNREACHABLE`** — `activeteller.core.failure.threshold` consecutive connection-level errors (unable to connect / timed out / refused / DNS). + - **`IAT_CORE_RECOVERED`** — next SymXchange message after an `IAT_CORE_UNREACHABLE` (a response, incl. a SOAP fault, proves the core is reachable); lets the incident side auto-close the core-banking ticket. 3. **`CheckProcessingResults{date}.log`** (fixed daily name) — Check21 image capture. Emits informational **`IAT_CHECK_IMAGE_QUALITY`** when a check deposit's invalidity score is high, the CAR/LAR (courtesy/legal amount) disagree, or the status isn't "Processed" — cases a back-office reviewer would otherwise have to catch by hand. A rollup **`IAT_HEARTBEAT`** is emitted every `activeteller.heartbeat.interval.sec` (default 900s / 15 min), but **only if at least one log produced a line today** — it will not send an empty heartbeat on an idle/offline machine. @@ -53,7 +55,7 @@ It also reuses the shared agent HTTP settings — `agent.endpoint` (preferred) o - Activates only on ATMs configured with `device.type=NH_IAT`. On any other device type it logs `activeteller: device.type='', not NH_IAT — module disabled` and does nothing — expected, not a bug. - Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which drops the module JAR into the agent's `modules/` directory and restarts the agent service (see `ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order: lab ATM → pilot device → fleet. - Can be disabled fleet-wide without removing the JAR by adding `activeteller` to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE` (the id is **`activeteller`**, the module's `getName()`). -- The nine event types it emits (`IAT_HEARTBEAT`, `IAT_TELLER_UNANSWERED`, `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_ERROR`, `IAT_SERVER_UNREACHABLE`, `IAT_MODE_CHANGE`, `IAT_CORE_TXN_FAIL`, `IAT_CORE_UNREACHABLE`, `IAT_CHECK_IMAGE_QUALITY`) are `IAT_*` custom types — they must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) to be recognized/labeled correctly downstream. They are not standard shared types. +- The eleven event types it emits (`IAT_HEARTBEAT`, `IAT_TELLER_UNANSWERED`, `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_ERROR`, `IAT_SERVER_UNREACHABLE`, `IAT_SERVER_RECOVERED`, `IAT_MODE_CHANGE`, `IAT_CORE_TXN_FAIL`, `IAT_CORE_UNREACHABLE`, `IAT_CORE_RECOVERED`, `IAT_CHECK_IMAGE_QUALITY`) are `IAT_*` custom types — they must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) to be recognized/labeled correctly downstream. They are not standard shared types. ## Common failure modes / what to check