content(guide): agent.* module guides — 15 modules (Internal/Architect/Claude, audience dev)
Closes the agent-module coverage gap the reconciler flagged (activeteller + 14). Grounded per module in hiveops-agent/hiveops-module-*; audience dev (dev-only). Reconciler now green. Grounding surfaced agent-side bugs (log-error route mismatch, atec poll-interval hardcoded) — to verify + file next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
---
|
||||
module: agent.activeteller
|
||||
title: Active Teller (IAT) — teller-session, core-banking & Check21 monitoring for Nautilus Hyosung ITMs
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · architecture.** Written 2026-07-01 from `hiveops-module-activeteller` source. Verify against code before relying on specifics.
|
||||
|
||||
## Module shape
|
||||
|
||||
Package `com.hiveops.activeteller`, six classes, no controllers/DB/Kafka (this is an agent module, not a microservice — see [technical.agent]):
|
||||
|
||||
- **`ActiveTellerModule`** — the `AgentModule` SPI entry point (registered via `META-INF/services/com.hiveops.core.module.AgentModule`). Owns lifecycle (`initialize()` → `isEnabled()` → `start()` → `stop()`), the shared single-thread `ScheduledExecutorService` (daemon thread named `activeteller`), and constructs the three monitors + the emitter.
|
||||
- **`TellerSessionMonitor`** (package-private) — parses `ActiveTellerAgent_*.log`.
|
||||
- **`SymXchangeMonitor`** (package-private) — parses `SymXchange{date}.log`.
|
||||
- **`CheckProcessingMonitor`** (package-private) — parses `CheckProcessingResults{date}.log`.
|
||||
- **`RollingLogTailer`** (package-private) — shared tail-by-offset reader; each monitor owns one instance.
|
||||
- **`IatEventEmitter`** (package-private) — thin wrapper over `IncidentEventClient` that builds the `CreateJournalEventRequest` and centralizes send-error handling.
|
||||
|
||||
The three monitors are plain constructed objects, not their own `AgentModule`s — `ActiveTellerModule` drives all three from a single poll loop.
|
||||
|
||||
## Lifecycle & config gating
|
||||
|
||||
`initialize(ModuleContext)` is defensive and never throws for a mis-provisioned box — it just leaves `ready=false`, which `isEnabled()` returns. It bails (module stays disabled) when any of these fail, each with a distinct log line:
|
||||
|
||||
1. `device.type != NH_IAT`
|
||||
2. `activeteller.log.dir` unset
|
||||
3. `activeteller.log.dir` is not an existing directory
|
||||
4. neither `agent.endpoint` nor `incident.endpoint` is configured
|
||||
|
||||
Endpoint prefix is chosen the standard way: `prefix = agentEndpoint != null ? "agent" : "incident"`, then `HttpClientSettings settings = HttpClientHelper.loadSettingsFromProperties(props, prefix)`. The `IncidentEventClient` is built once with `(settings, atmName, country)` from `ModuleContext`. `isHealthy()` = scheduler alive and not terminated.
|
||||
|
||||
## Flow: poll → tail → detect → emit
|
||||
|
||||
```
|
||||
ActiveTellerModule.start()
|
||||
→ ScheduledExecutorService (daemon "activeteller")
|
||||
├─ scheduleAtFixedRate(poll, 0, pollSec, SECONDS) // default 30s
|
||||
└─ scheduleAtFixedRate(heartbeat, heartbeatSec, heartbeatSec, SECONDS) // default 900s
|
||||
poll():
|
||||
tellerMonitor.poll(); symXchangeMonitor.poll(); checkMonitor.poll();
|
||||
heartbeat():
|
||||
if any monitor isAlive() → emit IAT_HEARTBEAT with concatenated fragments
|
||||
```
|
||||
|
||||
`poll()` wraps all three in a try/catch so one monitor's parse error can't kill the scheduler. `heartbeat()` skips entirely if no monitor has seen a line today (`isAlive()` = `tailer.currentFile() != null`), so idle machines stay quiet.
|
||||
|
||||
### `RollingLogTailer` — the shared tail engine
|
||||
|
||||
Every monitor reads through one `RollingLogTailer` constructed with `(baseLogDir, tag, FileResolver)`. On each `poll(Consumer<String>)`:
|
||||
|
||||
- Computes `dailyFolder = baseLogDir.resolve(yyyyMMdd)`; if it isn't a directory, no-ops.
|
||||
- Calls the `FileResolver` to pick the concrete file inside that folder:
|
||||
- `fixedName(prefix, suffix)` → `{folder}/{prefix}{yyyyMMdd}{suffix}` (SymXchange, CheckProcessingResults).
|
||||
- `newestMatching(prefix, suffix)` → the most-recently-modified file matching prefix+suffix (session-stamped `ActiveTellerAgent_*.log`).
|
||||
- Tail-by-offset: opens with `RandomAccessFile`, `seek(fileOffset)`, reads new lines to EOF, advances `fileOffset = raf.getFilePointer()`.
|
||||
- **Rotation/new file:** when the resolved path changes (new day or new session file), logs `watching <path>`, sets `currentFile` and resets `fileOffset = 0` (reads the new file from the top).
|
||||
- **Truncation guard:** if `raf.length() < fileOffset`, resets offset to 0.
|
||||
- **Encoding fix:** `RandomAccessFile.readLine()` decodes as ISO-8859-1; each line is re-decoded to UTF-8 (`new String(raw.getBytes(ISO_8859_1), UTF_8)`) before the consumer sees it.
|
||||
|
||||
`RollingLogTailer` holds **no** detection logic — the monitors own all parsing.
|
||||
|
||||
### `TellerSessionMonitor` — session state machine
|
||||
|
||||
Line format: `yyyy-MM-dd HH:mm:ss <message>`. Maintains single-customer session state (an IAT serves one customer at a time): `requestAt`, `sessionAt`, `videoUp`, `currentTeller`, plus a `logClock` (timestamp of the last processed line).
|
||||
|
||||
- `POST TellerSessionRequest message received` → `requestAt = ts` (idle `DELETE` heartbeats are ignored).
|
||||
- A payload containing `"TellerSessionRequestId"` → session assigned (extracts `"TellerName"`; dedups repeated echoes of the same payload); clears `requestAt`, resets video flags.
|
||||
- `OnAdd for RemoteControlSession` (while a session is active) → `videoUp = true`.
|
||||
- `Ending teller session` / `OnDelete for TellerSessionRequest` → session end; if video never came up, emit `IAT_VIDEO_SESSION_FAIL`.
|
||||
- REST outcome `(https?://…) server response (\w+)` → success statuses `{OK, Created, Accepted, NoContent, NotModified, ResetContent}` reset the failure streak; anything else emits `IAT_SERVER_ERROR` and, at `failureThreshold` consecutive failures, `IAT_SERVER_UNREACHABLE` (once, until a success resets it).
|
||||
- A payload containing `"ModeType"` with `"ModeName":"(\w+)"` → `IAT_MODE_CHANGE` (suppressed on first observation and on unchanged mode).
|
||||
- **`checkTimers()`** runs after each batch, comparing `requestAt`/`sessionAt` against **`logClock`** (not wall-clock): overdue request → `IAT_TELLER_UNANSWERED`; session with no video past `videoTimeout` → `IAT_VIDEO_SESSION_FAIL`. Driving timers off the log clock means replaying a backlog can't raise spurious timeouts.
|
||||
|
||||
### `SymXchangeMonitor` — header-block parsing
|
||||
|
||||
The SymXchange log alternates header lines (`[yyyy-MM-dd HH:mm:ss-nnn][OpName]`) and XML payloads (PANs/SSNs masked to `****` at source). Per line:
|
||||
|
||||
- A header sets `currentTs`/`currentOp`, resets `faultReportedForBlock`, bumps `messageCount`.
|
||||
- A `CONN_ERROR`-matching line (unable to connect / no connection / refused / timed out / DNS) increments a streak; at `failureThreshold` consecutive → `IAT_CORE_UNREACHABLE` (once until reset).
|
||||
- A `FAULT`-matching line (`faultstring`, `<faultcode`, `:Fault>`, `<Fault>`, `<ErrorMessage>`, non-zero `<ErrorCode>`) → `IAT_CORE_TXN_FAIL` (at most once per header block); a fault/response also resets the connection-error streak. The fault/conn regexes are conservative heuristics noted for tuning against a real failure sample.
|
||||
|
||||
### `CheckProcessingMonitor` — pipe-delimited records
|
||||
|
||||
Each `Process Date:`-prefixed line is split on `|` into `Key: Value` fields. A check is flagged (→ `IAT_CHECK_IMAGE_QUALITY`, informational) when any of: `Status` doesn't start with `Processed`; `Invalidity Score` > `invalidityThreshold`; or `|Amount CAR − Amount LAR| >= amountMismatchCents`. Amounts are in cents and rendered as dollars in the detail text. Clean checks emit nothing.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
All three monitors emit through the single shared `IatEventEmitter`, which wraps the same `IncidentEventClient` class used by `journal-events` and other event-reporting modules — see [technical.agent] for the full outbound API surface.
|
||||
|
||||
```
|
||||
monitor → emitter.emit(eventType, details, eventSource, eventTime)
|
||||
→ CreateJournalEventRequest.builder()
|
||||
.agentAtmId(atmId) // ATM name string, resolved to numeric atmId server-side
|
||||
.eventType(eventType) // free-form String, e.g. "IAT_CORE_UNREACHABLE"
|
||||
.eventDetails(details)
|
||||
.eventSource(eventSource) // IAT_ACTIVETELLER | IAT_SYMXCHANGE | IAT_CHECK21
|
||||
.eventTime(ts.format(ISO_LOCAL_DATE_TIME) | null)
|
||||
.build()
|
||||
→ IncidentEventClient.sendEvent(req)
|
||||
→ HTTP POST {agent.endpoint or incident.endpoint}/api/journal-events
|
||||
```
|
||||
|
||||
`eventType` is a plain `String` on `CreateJournalEventRequest` (the `hiveops-agent-journal` DTO) — not validated client-side. Nothing stops an unregistered `IAT_*` type from being sent; recognition/labeling happens server-side against hiveops-incident's DB-backed `event_type` table. On `IOException`, `emit()` logs a warning and drops the event — no retry, no queue.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
None used directly. The module reads plain files via `java.nio.file` / `RandomAccessFile` — no `PlatformService`/`PathResolver` dependency. The Hyosung/Windows constraint is enforced by deployment convention (installed only on `device.type=NH_IAT` machines) and the configured `activeteller.log.dir`, not by a runtime OS check in this module's code.
|
||||
|
||||
## Cross-links
|
||||
- [technical.agent] — module system, lifecycle, shared HTTP client settings, full outbound API surface
|
||||
- [platform.module-map]
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
module: agent.activeteller
|
||||
title: Active Teller (IAT) — teller-session, core-banking & Check21 monitoring for Nautilus Hyosung ITMs
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-activeteller` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Maven artifact: **`hiveops-module-activeteller`** (`com.hiveops`, version `1.0.0`, packaged as a standalone `jar`, parent `hiveops-agent-parent` `4.4.3`).
|
||||
- Agent module id (`getName()`, and the string to use in **`modules.disabled`**): **`activeteller`**. Module `getVersion()` = `1.0.0`.
|
||||
- Entry class: `com.hiveops.activeteller.ActiveTellerModule`, registered via `META-INF/services/com.hiveops.core.module.AgentModule`.
|
||||
- Log package (`getLogPackage()`): `com.hiveops.activeteller`.
|
||||
- Activation gate: `device.type` must equal `NH_IAT`; any other value → module inert, `isEnabled()` returns `false`.
|
||||
- Classes: `ActiveTellerModule`, `TellerSessionMonitor`, `SymXchangeMonitor`, `CheckProcessingMonitor`, `RollingLogTailer`, `IatEventEmitter`.
|
||||
|
||||
## Config properties (exact keys)
|
||||
| Key | Default | Required | Notes |
|
||||
|---|---|---|---|
|
||||
| `device.type` | *(none)* | **yes** | must be `NH_IAT` |
|
||||
| `activeteller.log.dir` | *(none)* | **yes** | base folder holding `yyyyMMdd/` daily subfolders; must exist & be a dir or module disables (warn, no crash) |
|
||||
| `activeteller.poll.interval.sec` | `30` | no | **applied** in `start()` |
|
||||
| `activeteller.heartbeat.interval.sec` | `900` | no | rollup heartbeat cadence |
|
||||
| `activeteller.answer.timeout.sec` | `180` | no | request→assignment deadline |
|
||||
| `activeteller.video.timeout.sec` | `60` | no | session→video deadline |
|
||||
| `activeteller.server.failure.threshold` | `3` | no | consecutive REST fails → `IAT_SERVER_UNREACHABLE` |
|
||||
| `activeteller.core.failure.threshold` | `3` | no | consecutive SymXchange conn errors → `IAT_CORE_UNREACHABLE` |
|
||||
| `activeteller.check.invalidity.threshold` | `50` | no | Check21 invalidity flag threshold |
|
||||
| `activeteller.check.amount.mismatch.cents` | `500` | no | CAR/LAR gap (cents) that flags a check |
|
||||
|
||||
Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix = `agent` if `agent.endpoint` set, else `incident`), resolved via `HttpClientHelper.loadSettingsFromProperties(props, prefix)` — same bearer-token / institution-key chain as every other event module. No IAT-specific auth keys.
|
||||
|
||||
## Event types emitted (exact strings)
|
||||
| Event type | `eventSource` | Trigger |
|
||||
|---|---|---|
|
||||
| `IAT_TELLER_UNANSWERED` | `IAT_ACTIVETELLER` | teller requested, no session assigned within `answer.timeout.sec` |
|
||||
| `IAT_VIDEO_SESSION_FAIL` | `IAT_ACTIVETELLER` | session assigned but no video/remote-control within `video.timeout.sec`, or session ended with no video |
|
||||
| `IAT_SERVER_ERROR` | `IAT_ACTIVETELLER` | Active Teller REST call returned non-2xx |
|
||||
| `IAT_SERVER_UNREACHABLE` | `IAT_ACTIVETELLER` | `server.failure.threshold` consecutive REST failures |
|
||||
| `IAT_MODE_CHANGE` | `IAT_ACTIVETELLER` | `"ModeType"` payload, `ModeName` changed (not first observation) |
|
||||
| `IAT_CORE_TXN_FAIL` | `IAT_SYMXCHANGE` | SOAP fault / error line in SymXchange log (once per header block) |
|
||||
| `IAT_CORE_UNREACHABLE` | `IAT_SYMXCHANGE` | `core.failure.threshold` consecutive connection errors |
|
||||
| `IAT_CHECK_IMAGE_QUALITY` | `IAT_CHECK21` | Check21 status≠Processed, invalidity>threshold, or CAR/LAR gap≥threshold |
|
||||
| `IAT_HEARTBEAT` | `IAT_ACTIVETELLER` | every `heartbeat.interval.sec`, only if a monitor saw a line today |
|
||||
|
||||
- `eventType` is a free-form `String` on `CreateJournalEventRequest` (`hiveops-agent-journal` DTO) — **not validated client-side**. These `IAT_*` strings are custom (not in any agent enum) — recognition is hiveops-incident's DB-backed `event_type` table.
|
||||
- `CreateJournalEventRequest` fields set: `agentAtmId`, `eventType`, `eventDetails`, `eventSource`, `eventTime` (ISO_LOCAL_DATE_TIME, or null). No cassette/card fields set.
|
||||
- Transport: `POST {endpoint}/api/journal-events` via `IncidentEventClient.sendEvent()` — same client/endpoint as `journal-events`.
|
||||
|
||||
## Files monitored (under `{activeteller.log.dir}/{yyyyMMdd}/`)
|
||||
- `ActiveTellerAgent_*.log` — newest-matching (session-stamped); `TellerSessionMonitor`, tag `agent`.
|
||||
- `SymXchange{yyyyMMdd}.log` — fixed daily name; `SymXchangeMonitor`, tag `symx`.
|
||||
- `CheckProcessingResults{yyyyMMdd}.log` — fixed daily name; `CheckProcessingMonitor`, tag `check`.
|
||||
- All via `RollingLogTailer`: tail-by-offset (`RandomAccessFile` + `fileOffset`), reset on new-day/new-file or truncation; lines re-decoded ISO-8859-1 → UTF-8.
|
||||
|
||||
## Behavior facts
|
||||
- Answer/video timers use the **log clock** (timestamp of last processed line), not wall-clock — backlog replay won't raise false timeouts.
|
||||
- Dedup guards: `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_UNREACHABLE`, `IAT_CORE_UNREACHABLE`, and per-block `IAT_CORE_TXN_FAIL` each fire once until state resets.
|
||||
- REST "success" whitelist: `OK, Created, Accepted, NoContent, NotModified, ResetContent`.
|
||||
- SymXchange fault/conn regexes are conservative heuristics — may need tuning against a real failure sample.
|
||||
|
||||
## Deployment
|
||||
- Delivered fleet-wide via **`INSTALL_MODULE`** fleet task (drops JAR into agent `modules/` dir, restarts agent — `ProcessInstallModule` in `hiveops-file-collection`).
|
||||
- Disable without uninstalling: add `activeteller` to `modules.disabled`, push `CONFIG_UPDATE`.
|
||||
- Register the nine `IAT_*` event types in hiveops-incident → Settings → Event Types (DB `event_type` table) for correct labeling.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT expect the module to run on any `device.type` other than `NH_IAT` — inert by design elsewhere, not a bug.
|
||||
- Do NOT expect retry/queueing on send failure — a failed `sendEvent()` is logged (`activeteller: failed to send ...`) and dropped.
|
||||
- Do NOT assume the `IAT_*` strings exist in an agent `EventType` enum — they don't; the incident DB `event_type` table is what gates recognition.
|
||||
- Do NOT point `activeteller.log.dir` at the daily `yyyyMMdd` subfolder — it must be the **base** folder; the tailer appends `{yyyyMMdd}/` itself.
|
||||
- Do NOT assume `IAT_HEARTBEAT` fires on an idle/offline machine — it's suppressed unless a monitor saw a line today.
|
||||
- Do NOT treat the SymXchange fault detection as authoritative — the markers are heuristics tuned without a real fault sample.
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
module: agent.activeteller
|
||||
title: Active Teller (IAT) — teller-session, core-banking & Check21 monitoring for Nautilus Hyosung ITMs
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support.** Written 2026-07-01 from `hiveops-module-activeteller` source. Verify against code before relying on specifics.
|
||||
|
||||
## What this module is
|
||||
|
||||
`hiveops-module-activeteller` (Maven artifact `hiveops-module-activeteller`, agent module id **`activeteller`**) is an **agent drop-in module**, not a backend service. It runs inside the HiveIQ Agent process on **Nautilus Hyosung Active Teller Interactive/Video Teller Machines (IAT/ITM) only** — it self-disables unless the ATM's `device.type` is `NH_IAT`.
|
||||
|
||||
It tails three log files that the Hyosung MoniPlus2s / Active Teller software already writes to disk and turns specific lines into HiveIQ journal events (`IAT_*`) sent to the incident pipeline via agent-proxy. The module does **not** talk to the ITM hardware, the teller/video server, or the core banking host directly — it is purely a read-only log observer.
|
||||
|
||||
## What it monitors and why
|
||||
|
||||
All three logs live under the same base folder configured by `activeteller.log.dir`, inside a per-day `yyyyMMdd` subfolder (the standard MoniPlus2s convention). One scheduler tick (default 30s) polls all three:
|
||||
|
||||
1. **`ActiveTellerAgent_{date}_{time}.log`** (session-stamped; newest match wins) — the Active Teller client log. Reconstructs the interactive-teller session flow and reports:
|
||||
- **`IAT_TELLER_UNANSWERED`** — a customer requested a teller but no teller session was assigned within `activeteller.answer.timeout.sec`.
|
||||
- **`IAT_VIDEO_SESSION_FAIL`** — a teller session started but the video/remote-control session never came up (within `activeteller.video.timeout.sec`, or the session ended without video).
|
||||
- **`IAT_SERVER_ERROR`** — an Active Teller REST call returned a non-2xx status.
|
||||
- **`IAT_SERVER_UNREACHABLE`** — `activeteller.server.failure.threshold` consecutive REST failures.
|
||||
- **`IAT_MODE_CHANGE`** — scheduled operating-mode transition (e.g. Standard ↔ SelfService).
|
||||
2. **`SymXchange{date}.log`** (fixed daily name) — the Jack Henry **Symitar (SymXchange)** core-banking conversation. Reports:
|
||||
- **`IAT_CORE_TXN_FAIL`** — a SOAP fault / error response from the core (at most once per header block).
|
||||
- **`IAT_CORE_UNREACHABLE`** — `activeteller.core.failure.threshold` consecutive connection-level errors (unable to connect / timed out / refused / DNS).
|
||||
3. **`CheckProcessingResults{date}.log`** (fixed daily name) — Check21 image capture. Emits informational **`IAT_CHECK_IMAGE_QUALITY`** when a check deposit's invalidity score is high, the CAR/LAR (courtesy/legal amount) disagree, or the status isn't "Processed" — cases a back-office reviewer would otherwise have to catch by hand.
|
||||
|
||||
A rollup **`IAT_HEARTBEAT`** is emitted every `activeteller.heartbeat.interval.sec` (default 900s / 15 min), but **only if at least one log produced a line today** — it will not send an empty heartbeat on an idle/offline machine.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
| Property | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `device.type` | *(none)* | Activation gate — must equal `NH_IAT` or the module is inert. |
|
||||
| `activeteller.log.dir` | *(none — required)* | Base folder that contains the `yyyyMMdd` daily subfolders (e.g. `C:/Hyosung/MoniPlus2s/MoniPlus2Slog`). If unset or not a real directory, the module logs a warning and disables itself — it does **not** crash agent startup. |
|
||||
| `activeteller.poll.interval.sec` | `30` | Log poll interval. **Applied** (unlike some sibling modules). |
|
||||
| `activeteller.heartbeat.interval.sec` | `900` | Rollup-heartbeat interval. |
|
||||
| `activeteller.answer.timeout.sec` | `180` | Teller-request → assignment deadline before `IAT_TELLER_UNANSWERED`. |
|
||||
| `activeteller.video.timeout.sec` | `60` | Session-start → video/remote-control-up deadline before `IAT_VIDEO_SESSION_FAIL`. |
|
||||
| `activeteller.server.failure.threshold` | `3` | Consecutive REST failures → `IAT_SERVER_UNREACHABLE`. |
|
||||
| `activeteller.core.failure.threshold` | `3` | Consecutive SymXchange connection errors → `IAT_CORE_UNREACHABLE`. |
|
||||
| `activeteller.check.invalidity.threshold` | `50` | Check21 invalidity score above which a check is flagged. |
|
||||
| `activeteller.check.amount.mismatch.cents` | `500` | CAR/LAR gap (in cents) that counts as a bad read. |
|
||||
|
||||
It also reuses the shared agent HTTP settings — `agent.endpoint` (preferred) or `incident.endpoint` (legacy fallback), resolved through `HttpClientHelper.loadSettingsFromProperties(props, prefix)` (the same bearer-token / institution-key chain every event-reporting module uses). There are no IAT-specific auth properties.
|
||||
|
||||
## How it's enabled / deployed
|
||||
|
||||
- Activates only on ATMs configured with `device.type=NH_IAT`. On any other device type it logs `activeteller: device.type='<X>', not NH_IAT — module disabled` and does nothing — expected, not a bug.
|
||||
- Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which drops the module JAR into the agent's `modules/` directory and restarts the agent service (see `ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order: lab ATM → pilot device → fleet.
|
||||
- Can be disabled fleet-wide without removing the JAR by adding `activeteller` to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE` (the id is **`activeteller`**, the module's `getName()`).
|
||||
- The nine event types it emits (`IAT_HEARTBEAT`, `IAT_TELLER_UNANSWERED`, `IAT_VIDEO_SESSION_FAIL`, `IAT_SERVER_ERROR`, `IAT_SERVER_UNREACHABLE`, `IAT_MODE_CHANGE`, `IAT_CORE_TXN_FAIL`, `IAT_CORE_UNREACHABLE`, `IAT_CHECK_IMAGE_QUALITY`) are `IAT_*` custom types — they must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) to be recognized/labeled correctly downstream. They are not standard shared types.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **No IAT events ever appear for a customer ITM** — first check `device.type` on that ATM's config; if it isn't `NH_IAT` the module is intentionally inert. Not a bug.
|
||||
- **Module loaded but silent** — check `activeteller.log.dir` is set and points at a real, existing base directory on that machine. A missing/wrong path disables the module at startup with a log warning (no crash). Also confirm the daily `yyyyMMdd` subfolder actually exists — the tailer resolves `{log.dir}/{yyyyMMdd}/` each poll and silently no-ops if that folder isn't there.
|
||||
- **Events stopped after midnight / around log rotation** — all three monitors roll to a new `yyyyMMdd` folder (and SymXchange/Check21 to a new dated filename) at local midnight; the ActiveTeller log is session-stamped, so the newest matching file wins. If the Hyosung file-naming convention ever changes, the module watches a file that never appears and goes silent. Grep agent logs for `activeteller[...]: watching <path>` to confirm which file each monitor is tailing.
|
||||
- **Spurious `IAT_TELLER_UNANSWERED` / `IAT_VIDEO_SESSION_FAIL`** — the answer/video timers are driven off the **log clock** (timestamp of the last processed line), not wall-clock, so replaying a backlog cannot raise false timeouts. If real timeouts fire wrongly, re-check `activeteller.answer.timeout.sec` / `activeteller.video.timeout.sec` against the site's actual teller-answer SLA.
|
||||
- **SymXchange faults missed or over-reported** — the fault / connection-error regexes are conservative heuristics (no real SymXchange failure was in the reference capture). If a customer's real fault has a different shape, the markers may need tuning; check `IAT_CORE_TXN_FAIL` details against the raw log.
|
||||
- **Events sent but not visible/categorized in Incident** — confirm the `IAT_*` types are registered in hiveops-incident's Event Types settings; these are DB-driven, not hardcoded.
|
||||
- **HTTP send failures** — `IatEventEmitter` catches and logs send failures (`activeteller: failed to send ...`) but does **not** retry or queue — a transient outage silently drops that single event rather than backing it up.
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
module: agent.aria-signals
|
||||
title: ARIA Security Signals — ATM threat-signal collector
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · how it's built.** Confirmed against `hiveops-module-aria-signals` source (`com.hiveops.aria`) 2026-07-01. See [technical.agent] for agent-wide module-system context.
|
||||
|
||||
## Maven module
|
||||
|
||||
`hiveops-module-aria-signals` (artifact `hiveops-module-aria-signals`, own version `1.0.0`, parented by `hiveops-agent-parent` — currently `4.4.3`). Packaged as a standalone jar (`<finalName>${project.artifactId}</finalName>` → `hiveops-module-aria-signals.jar`). Dependencies are all **`provided`** scope (`hiveops-core`, `jackson-databind`, `log4j-api`, `commons-lang3`) — nothing is shaded into the module JAR; it relies on the agent runtime classpath. No dependency on `hiveops-journal`, `hiveops-images`, or `hiveops-file-collection`.
|
||||
|
||||
Registered via ServiceLoader SPI: `src/main/resources/META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.aria.AriaSignalModule`. This is how `ModuleLoader` discovers it from the `modules/` directory at startup.
|
||||
|
||||
## Classes and flow
|
||||
|
||||
Four runtime classes plus three DTOs:
|
||||
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `AriaSignalModule` | `AgentModule` implementation — reads config, wires collaborators, owns the scheduler and lifecycle |
|
||||
| `AriaSignalCollector` | `Runnable` — polls the Windows Security Event Log + scans staging dirs, batches, POSTs events |
|
||||
| `AriaPostureReporter` | `Runnable` — builds and POSTs the device posture snapshot |
|
||||
| `AriaHttpClient` | Thin `HttpURLConnection` JSON POST client to the ARIA base URL |
|
||||
| `dto.AriaIngestRequest` | Envelope: `deviceAgentId`, `deviceId`, `institutionKey`, `List<AriaSignalItem> events` |
|
||||
| `dto.AriaSignalItem` | One signal: `signalKey`, `eventType`, `Map<String,Object> rawPayload`, `detectedAt` (ISO-8601 string) |
|
||||
| `dto.AriaPostureReportDto` | Posture snapshot (see fields below) |
|
||||
|
||||
### Lifecycle (`AriaSignalModule`)
|
||||
|
||||
`getName()` = `"aria-signals"`; `getLogPackage()` = `"com.hiveops.aria"` → the framework routes its output to `logs/modules/aria-signals.log`.
|
||||
|
||||
- `initialize(ModuleContext)` — reads `aria.signals.enabled`; if false, returns (leaves `ready=false`). Reads `aria.endpoint`; if blank, logs a warning and returns. Otherwise builds one `AriaHttpClient(endpoint, serviceSecret, skipSsl)`, resolves `atmName` (`context.getAtmName()`), `institutionKey` (`agent.institution.key` → `incident.institution.key`), `deviceId` (`aria.device.id`, parsed to `Long` or null), intervals and batch size, then constructs the collector and reporter and sets `ready=true`.
|
||||
- `isEnabled(context)` returns the `ready` flag (so a mis-configured module self-disables rather than erroring).
|
||||
- `start()` — creates a **2-thread** daemon `ScheduledExecutorService` and schedules both `Runnable`s with `scheduleAtFixedRate`: the collector at `[0, signalIntervalSec]` and the reporter at `[0, postureIntervalSec]`. So the posture report fires **immediately**; the collector's first run just **anchors** the log (see below).
|
||||
- `stop()` — `scheduler.shutdownNow()`.
|
||||
- `isHealthy()` — `scheduler == null || !scheduler.isTerminated()`; consulted by the framework `ModuleWatchdog`.
|
||||
|
||||
### Signal collection cycle (`AriaSignalCollector`, poll-based)
|
||||
|
||||
The trigger model is **poll**, not log-tail or hook. Each `run()`:
|
||||
|
||||
1. **First tick anchors, doesn't collect.** If not yet `anchored`, `anchorSecurityLog()` runs `wevtutil qe Security /rd:true /f:XML /c:1 /q:<query>` (newest first, query with no `EventRecordID` bound), parses the max `EventRecordID`, stores it in `lastSecurityRecordId`, sets `anchored=true`, and returns. This prevents replaying historical events. On non-Windows it just sets `anchored=true` and returns.
|
||||
2. **Subsequent ticks:** `collectSecurityEvents()` builds a query `*[System[(EventID=1102 or … ) and EventRecordID > <lastSecurityRecordId>]]`, runs `wevtutil qe Security /rd:false /f:XML /c:<batchSize> /q:<query>`, and parses the returned XML with **hand-rolled string scanning** (`parseEvents`/`parseBlock`/`extractTag`/`extractAttr` — no XML parser dependency). Each event becomes an `AriaSignalItem` with `signalKey = <eventId>`, `eventType = "OS_EVENT"`, and `rawPayload = {eventId, recordId, provider?}`. The bookmark `lastSecurityRecordId` advances to the max record seen.
|
||||
3. **Staging-dir scan** (`scanStagingDirs()`, runs on all OSes but paths are Windows `C:/...`): for each existing directory, any regular file whose name matches `SUSPICIOUS_FILENAMES` (case-insensitive) emits `AriaSignalItem(signalKey="IOC_FILENAME_MATCH", eventType="FILE_SYSTEM", payload={path, filename})`. Separately, if the directory contains **any** file at all, it emits `AriaSignalItem(signalKey="SUSPICIOUS_DIRECTORY", eventType="FILE_SYSTEM", payload={directory})`.
|
||||
4. If the combined list is non-empty, wrap in `AriaIngestRequest(deviceAgentId, deviceId, institutionKey, events)` and `client.sendEvents(...)`.
|
||||
|
||||
Note the bookmark advances **before** the POST, so a failed send does not re-queue those Security events on the next tick.
|
||||
|
||||
### Posture cycle (`AriaPostureReporter`)
|
||||
|
||||
`buildReport()` populates `AriaPostureReportDto`: `deviceAgentId`/`deviceId`/`institutionKey`; `osVersion` (`os.name os.version`) and `osEolStatus` (`classifyOsEol` → `EOL` for Win 7/XP/Vista/8, else `SUPPORTED`/`UNKNOWN`). On Windows it adds `diskEncryptionEnabled` (BitLocker `manage-bde -status C:` → "protection on"), `auditPolicyCompliant` (`auditpol` → not "no auditing"), `softwareWhitelistEnabled` (`sadmin status` → "solid", fallback AppLocker registry `EnforcementMode 0x1`). If `aria.gold.image.hash` is set, it SHA-256-hashes `aria.image.path` (`sha256:<hex>`) into `currentImageHash` for drift comparison. Each check returns `null` on failure, and `@JsonInclude(NON_NULL)` drops null fields from the JSON.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
`AriaHttpClient` posts JSON via `HttpURLConnection` (connect 10s / read 15s) with `Content-Type: application/json; charset=UTF-8` and header `X-Service-Secret: <aria.service.secret>`:
|
||||
|
||||
| Method | Path (relative to `aria.endpoint`) | Body |
|
||||
|---|---|---|
|
||||
| `sendEvents` | `POST /api/aria/ingest/events` | `AriaIngestRequest` |
|
||||
| `sendPostureReport` | `POST /api/aria/posture/report` | `AriaPostureReportDto` |
|
||||
|
||||
This is a **direct** agent → ARIA backend path. It does **not** traverse agent-proxy/incident and does **not** use `agent.auth.token`, `FileUploadClient`, or the fleet-task pipeline that other modules share. Non-2xx responses are logged as warnings; there is no retry/backoff.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
This module deliberately **does not** use the agent's `PlatformService`/`SystemCommands` abstraction. OS branching is done inline with `System.getProperty("os.name")` and raw `ProcessBuilder` calls to `wevtutil`, `manage-bde`, `auditpol`, `sadmin`, `reg`. Command output is read with the JVM `file.encoding` charset. Consequence: all monitoring except the OS/EOL and image-hash posture fields is effectively Windows-only.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
module: agent.aria-signals
|
||||
title: ARIA Security Signals — ATM threat-signal collector
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-aria-signals` source (`com.hiveops.aria`) 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Module name (`AgentModule.getName()`): **`aria-signals`**
|
||||
- Maven artifact: **`hiveops-module-aria-signals`** (own version `1.0.0`; parent `hiveops-agent-parent` `4.4.3`)
|
||||
- Entry class: `com.hiveops.aria.AriaSignalModule` (SPI file: `META-INF/services/com.hiveops.core.module.AgentModule`)
|
||||
- Other classes: `AriaSignalCollector`, `AriaPostureReporter`, `AriaHttpClient`; DTOs `AriaIngestRequest`, `AriaSignalItem`, `AriaPostureReportDto`
|
||||
- Log package (`getLogPackage()`): `com.hiveops.aria` → `logs/modules/aria-signals.log`
|
||||
- Drop-in JAR: `modules/hiveops-module-aria-signals.jar` (all deps `provided`; not in fat JAR)
|
||||
- **Disabled by default.** No fleet-task kind; passive scheduled collector.
|
||||
|
||||
## Config property keys (all in `hiveops.properties`)
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `aria.signals.enabled` | `false` | master switch |
|
||||
| `aria.endpoint` | *(none, required)* | ARIA base URL; blank → module inactive |
|
||||
| `aria.service.secret` | `""` | sent as `X-Service-Secret` header |
|
||||
| `aria.signals.poll.interval.sec` | `60` | event/staging poll |
|
||||
| `aria.posture.report.interval.sec` | `86400` | posture report (24h) |
|
||||
| `aria.signals.batch.size` | `50` | `wevtutil /c:` cap per poll |
|
||||
| `aria.gold.image.hash` | *(none)* | enables image-drift check when set |
|
||||
| `aria.image.path` | *(none)* | file hashed (SHA-256) vs gold hash |
|
||||
| `aria.ssl.skip.verify` | `false` | installs **JVM-wide** trust-all SSL |
|
||||
| `aria.device.id` | *(none)* | numeric; else omitted from reports |
|
||||
| `agent.institution.key` → `incident.institution.key` | *(none)* | institution key |
|
||||
|
||||
- `isEnabled()` == true only when `aria.signals.enabled=true` AND `aria.endpoint` non-blank.
|
||||
- `deviceAgentId` = `ModuleContext.getAtmName()`. No dedicated `aria.*` interval for staging scan (shares `aria.signals.poll.interval.sec`).
|
||||
|
||||
## Signal strings emitted (`AriaSignalItem.signalKey` / `.eventType`)
|
||||
| signalKey | eventType | Source |
|
||||
|---|---|---|
|
||||
| `"1102"`,`"4719"`,`"6416"`,`"2003"`,`"4663"`,`"4688"`,`"4697"` | `OS_EVENT` | Windows Security Event Log (`wevtutil`) |
|
||||
| `IOC_FILENAME_MATCH` | `FILE_SYSTEM` | suspicious filename in a staging dir |
|
||||
| `SUSPICIOUS_DIRECTORY` | `FILE_SYSTEM` | staging dir contains ANY file |
|
||||
|
||||
- Monitored Security Event IDs: `1102, 4719, 6416, 2003, 4663, 4688, 4697` (`SECURITY_EVENT_IDS`).
|
||||
- Staging dirs (`STAGING_DIRS`): `C:/Temp/bin`, `C:/Windows/Temp/s`, `C:/ProgramData/s`, `C:/ProgramData/Microsoft/Wms`, `C:/Windows/system32/bin`.
|
||||
- Suspicious filenames (case-insensitive): `Newage.exe`, `Color.exe`, `sdelete.exe`, `atmapp.exe`, `mainclientd.exe`, `ATMContent.exe`.
|
||||
- Posture DTO fields: `diskEncryptionEnabled`, `auditPolicyCompliant`, `softwareWhitelistEnabled`, `osVersion`, `osEolStatus` (`SUPPORTED|EOL|UNKNOWN`), `goldImageHash`, `currentImageHash` (`sha256:<hex>`).
|
||||
|
||||
## HTTP endpoints (agent → ARIA backend, DIRECT)
|
||||
| Call | Method + path (relative to `aria.endpoint`) |
|
||||
|---|---|
|
||||
| Events | `POST /api/aria/ingest/events` (`AriaIngestRequest`) |
|
||||
| Posture | `POST /api/aria/posture/report` (`AriaPostureReportDto`) |
|
||||
- Header: `X-Service-Secret: <aria.service.secret>`. Timeouts: connect 10s / read 15s. Non-2xx logged as warning, **no retry**.
|
||||
- Does **NOT** go through agent-proxy/incident; does **NOT** use `agent.auth.token` or `FleetTask`/`FileUploadClient`.
|
||||
|
||||
## Posture check commands (Windows, raw `ProcessBuilder`)
|
||||
- BitLocker: `manage-bde -status C:` → true if output contains "protection on"
|
||||
- Audit policy: `auditpol /get /subcategory:"Audit Policy Change"` → true unless "no auditing"
|
||||
- Whitelisting: `sadmin status` → "solid"; fallback `reg query HKLM\...\SrpV2\Exe /v EnforcementMode` → contains `0x1`
|
||||
- Each returns `null` on failure; `@JsonInclude(NON_NULL)` drops nulls.
|
||||
|
||||
## Deployment
|
||||
- Enable: `CONFIG_UPDATE` fleet task setting `aria.signals.enabled=true` + `aria.endpoint` (+ secret), then agent restart.
|
||||
- Retrofit missing JAR: `INSTALL_MODULE` fleet task (drops JAR into `modules/`, restarts agent) or full agent update.
|
||||
- Disable: `aria.signals.enabled=false` OR add `aria-signals` to `modules.disabled`.
|
||||
|
||||
## Gotchas
|
||||
- **First collector tick only anchors the log** (records max `EventRecordID`, sends nothing). First real events arrive on the 2nd tick (after `aria.signals.poll.interval.sec`). Posture report fires immediately on start.
|
||||
- **Windows-only in practice:** event-log poll + staging scan produce nothing on Linux. Posture report runs anywhere but only OS/EOL + image-hash fields populate off-Windows (BitLocker/audit/whitelist are Windows-guarded).
|
||||
- `SUSPICIOUS_DIRECTORY` fires if the staging dir holds **any** file, not just a suspicious one → noisy.
|
||||
- Record bookmark (`lastSecurityRecordId`) advances **before** the POST; a failed send loses those events (no re-queue).
|
||||
- `aria.ssl.skip.verify=true` installs a trust-all `SSLContext` via `HttpsURLConnection.setDefault*` — affects the **entire JVM**, not just ARIA calls.
|
||||
- XML is parsed by hand-rolled string scanning (`extractTag`/`extractAttr`), not a real XML parser.
|
||||
- No `PlatformService`/`SystemCommands` usage — inline `os.name` + `ProcessBuilder`.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT expect `EventType` journal/incident events (`CARD_READER_FAIL`, `IAT_*`, etc.) — this module emits ARIA `signalKey`/`eventType` (`OS_EVENT`/`FILE_SYSTEM`) only, direct to ARIA.
|
||||
- Do NOT look for these calls in agent-proxy/incident or fleet-task logs — they bypass that pipeline entirely.
|
||||
- Do NOT assume `agent.auth.token` authenticates ARIA calls — it's `X-Service-Secret` only.
|
||||
- Do NOT expect data on the first poll or on Linux — anchor-first + Windows-centric.
|
||||
- Do NOT enable `aria.ssl.skip.verify` casually — it disables TLS verification JVM-wide.
|
||||
- Do NOT assume delivery is guaranteed — best-effort, no retry, bookmark advances past failed sends.
|
||||
- Do NOT invent extra `aria.*` keys — only the eleven above exist in code.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
module: agent.aria-signals
|
||||
title: ARIA Security Signals — ATM threat-signal collector
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-aria-signals` source (`com.hiveops.aria`) 2026-07-01.
|
||||
|
||||
## What it does (and doesn't)
|
||||
|
||||
`hiveops-module-aria-signals` (module name `aria-signals`, class `AriaSignalModule`) is a **passive security-signal collector** that feeds the HiveIQ **ARIA** threat-intelligence backend. On each ATM it runs two scheduled jobs:
|
||||
|
||||
1. **Signal collector** (`AriaSignalCollector`, default every **60s**) — polls the Windows **Security Event Log** for a fixed set of threat-relevant Event IDs, and scans a fixed list of known malware **staging directories** for suspicious filenames. Batches whatever it finds and POSTs it to ARIA.
|
||||
2. **Posture reporter** (`AriaPostureReporter`, default every **24h**) — collects a device **security posture** snapshot (BitLocker disk encryption, audit policy, software whitelisting, OS/EOL status, gold-image hash drift) and POSTs it to ARIA.
|
||||
|
||||
It is **disabled by default** and is **Windows-centric**: the event-log poll and staging-dir scan only produce data on Windows; on Linux the posture reporter still runs but only fills in OS/EOL and image-hash fields.
|
||||
|
||||
Important boundary: unlike the journal/incident modules, this module does **not** emit `EventType` journal events to hiveops-incident and does **not** go through agent-proxy. It POSTs **directly** to the ARIA base URL (`aria.endpoint`, e.g. `https://api.bcos.cloud`) authenticated with an `X-Service-Secret` header. See [architect.md] for the flow.
|
||||
|
||||
### What it monitors
|
||||
|
||||
**Security Event Log IDs** (`AriaSignalCollector.SECURITY_EVENT_IDS`):
|
||||
|
||||
| Event ID | Meaning |
|
||||
|---|---|
|
||||
| 1102 | Audit log cleared |
|
||||
| 4719 | Audit policy changed |
|
||||
| 6416 | New removable (USB) device recognised |
|
||||
| 2003 | Network setting changed |
|
||||
| 4663 | File/object access attempt |
|
||||
| 4688 | New process created |
|
||||
| 4697 | Service installed on system |
|
||||
|
||||
**Staging directories scanned** (jackpotting / Ploutus-family paths — `STAGING_DIRS`): `C:/Temp/bin`, `C:/Windows/Temp/s`, `C:/ProgramData/s`, `C:/ProgramData/Microsoft/Wms`, `C:/Windows/system32/bin`. Suspicious filenames flagged (`SUSPICIOUS_FILENAMES`, case-insensitive): `Newage.exe`, `Color.exe`, `sdelete.exe`, `atmapp.exe`, `mainclientd.exe`, `ATMContent.exe`.
|
||||
|
||||
**Posture checks** (Windows, via `ProcessBuilder`): BitLocker (`manage-bde -status C:`), audit policy (`auditpol /get /subcategory:"Audit Policy Change"`), software whitelisting (McAfee Solidcore/Trellix `sadmin status`, falling back to Windows AppLocker registry `SrpV2\Exe EnforcementMode`). OS/EOL classification and optional gold-image SHA-256 drift run on all platforms.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
All keys live in `hiveops.properties` and are read in `AriaSignalModule.initialize()`:
|
||||
|
||||
| Key | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `aria.signals.enabled` | `false` | Master switch — module stays inactive unless `true` |
|
||||
| `aria.endpoint` | *(none — required)* | ARIA base URL, e.g. `https://api.bcos.cloud`. If unset, module logs a warning and stays inactive |
|
||||
| `aria.service.secret` | `""` | Value sent in the `X-Service-Secret` header on every ARIA POST |
|
||||
| `aria.signals.poll.interval.sec` | `60` | Event-log / staging-dir poll interval |
|
||||
| `aria.posture.report.interval.sec` | `86400` | Posture report interval (default 24h) |
|
||||
| `aria.signals.batch.size` | `50` | Max Security events pulled per poll (`wevtutil /c:`) |
|
||||
| `aria.gold.image.hash` | *(none)* | Expected SHA-256 for image-drift check; only checked if set |
|
||||
| `aria.image.path` | *(none)* | File hashed and compared against `aria.gold.image.hash` |
|
||||
| `aria.ssl.skip.verify` | `false` | Skip TLS cert verification (installs a JVM-wide trust-all — see gotchas) |
|
||||
| `aria.device.id` | *(none)* | Optional numeric HiveIQ device id included in reports |
|
||||
| `agent.institution.key` (falls back to `incident.institution.key`) | *(none)* | Institution key stamped on every report |
|
||||
|
||||
Device identity on the wire: `deviceAgentId` = the agent's ATM name (`ModuleContext.getAtmName()`); `deviceId` = `aria.device.id` if numeric, else omitted.
|
||||
|
||||
## Enabled / deployed to ATMs
|
||||
|
||||
- **Enable:** push `aria.signals.enabled=true` **and** `aria.endpoint=...` (plus `aria.service.secret`) via a `CONFIG_UPDATE` fleet task, then restart the agent so the module re-initializes. `isEnabled()` returns true only when both `aria.signals.enabled=true` and `aria.endpoint` is non-blank.
|
||||
- **Shipped how:** it is a **drop-in JAR** — `modules/hiveops-module-aria-signals.jar` (own artifact version `1.0.0`, parented by `hiveops-agent-parent`). It is **not** baked into the fat JAR. Discovered at startup via ServiceLoader from the `modules/` directory.
|
||||
- **Retrofit to an ATM that doesn't have the JAR yet:** deliver via a fleet **`INSTALL_MODULE`** task (drops the JAR into `modules/` and restarts the agent), or via a full agent update that includes it in the module set.
|
||||
- **Disable again:** either set `aria.signals.enabled=false`, or add `aria-signals` to `modules.disabled` (comma-separated) via `CONFIG_UPDATE`.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---|---|---|
|
||||
| Module logs "disabled via aria.signals.enabled" and does nothing | `aria.signals.enabled` not `true` | Confirm the property on that ATM after the `CONFIG_UPDATE` landed and the agent restarted |
|
||||
| Module logs "aria.endpoint not configured, module inactive" | `aria.endpoint` blank even though enabled | Set `aria.endpoint` and restart |
|
||||
| No events ever arrive at ARIA, no errors | ATM is Linux, or genuinely no matching events; also the **first poll only anchors** the log (no data until the 2nd tick) | Confirm OS is Windows; wait past one `aria.signals.poll.interval.sec`; check `logs/modules/aria-signals.log` for "anchored Security log at RecordID" |
|
||||
| Posture report fields all null on Windows | `manage-bde` / `auditpol` / `sadmin` not on PATH or agent lacks privileges | Each check returns null on failure — run the commands manually as the agent service account |
|
||||
| `ARIA POST ... returned HTTP 4xx/5xx` in log | Wrong `aria.endpoint`, bad/missing `aria.service.secret`, or ARIA backend down | Non-2xx is logged as a warning and **not retried**; verify endpoint + secret |
|
||||
| TLS errors on the POST | ARIA cert not trusted | Fix the trust store; only use `aria.ssl.skip.verify=true` as a last resort (it disables verification JVM-wide) |
|
||||
| Flood of `SUSPICIOUS_DIRECTORY` signals | A monitored staging dir contains *any* file — the directory is flagged even when the file itself isn't on the suspicious list | Inspect the actual `directory` path; this is expected behaviour, not a bug |
|
||||
| Events dropped after a failed send | The record bookmark advances before the POST; if the POST fails those events aren't re-sent | Check network to `aria.endpoint`; treat ARIA as best-effort telemetry, not guaranteed delivery |
|
||||
|
||||
## Not applicable here
|
||||
|
||||
No journal/incident (`EventType`) integration, no fleet-task execution. Its only output is HTTP POSTs to ARIA (`/api/aria/ingest/events`, `/api/aria/posture/report`).
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
module: agent.atec-tcr-events
|
||||
title: ATEC TCR Events — cassette + process monitoring for LG Teller Cash Recyclers
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · architecture.** Written 2026-07-01 from `hiveops-module-atec-tcr-events` source. Verify against code before relying on specifics.
|
||||
|
||||
## Module shape
|
||||
|
||||
Package `com.hiveops.tcrevents`, three classes, no controllers/DB/Kafka (this is an agent module, not a microservice — see [technical.agent]):
|
||||
|
||||
- **`TcrEventsModule`** — the `AgentModule` SPI entry point (registered via `META-INF/services/com.hiveops.core.module.AgentModule`). Owns lifecycle (`initialize()` → `isEnabled()` → `start()` → `stop()`) and the shared `ScheduledExecutorService`.
|
||||
- **`CstSnsLogMonitor`** (package-private) — parses `CST_SNR_MMDD.log`.
|
||||
- **`TcrAccessLogMonitor`** (package-private) — parses `ACESSLog<MMDD>.txt` (code: `logDir.resolve("ACESSLog" + mmdd + ".txt")` — **no underscore** before the date; the class javadoc's `ACESSLog_MMDD.txt` is stale).
|
||||
|
||||
Both monitors are plain constructed objects (`new CstSnsLogMonitor(logDir, atmName, client)`), not their own `AgentModule`s — `TcrEventsModule` drives both from a single poll loop.
|
||||
|
||||
## Flow: poll → parse → detect → emit
|
||||
|
||||
```
|
||||
TcrEventsModule.start()
|
||||
→ ScheduledExecutorService (daemon thread "tcr-events", fixed-rate, every 30s — see internal.md re: config bug)
|
||||
→ poll()
|
||||
→ cstMonitor.poll() // CstSnsLogMonitor
|
||||
→ accessMonitor.poll() // TcrAccessLogMonitor
|
||||
```
|
||||
|
||||
Each monitor uses **tail-by-offset** log reading, the same pattern as the agent's journal readers:
|
||||
- Tracks `currentFile` + `fileOffset` (byte position) per monitor.
|
||||
- On each `poll()`, opens the file with `RandomAccessFile`, seeks to `fileOffset`, reads new lines to EOF, advances `fileOffset` to `raf.getFilePointer()`.
|
||||
- Daily rotation: `rotateDailyFileIfNeeded()` compares `LocalDate.now()` to the last-seen date; on a new day it recomputes the filename (`CST_SNR_<MMDD>.log` / `ACESSLog<MMDD>.txt` — the date is `MMdd` via `DateTimeFormatter.ofPattern("MMdd")`) and resets `fileOffset` to 0.
|
||||
- Truncation guard: if `raf.length() < fileOffset` (file was truncated/replaced mid-session), `CstSnsLogMonitor` resets offset + in-progress parse state; `TcrAccessLogMonitor` resets offset only (no block state to clear).
|
||||
|
||||
### `CstSnsLogMonitor` — block-based parsing
|
||||
|
||||
The CST_SNR log groups related lines into operation blocks:
|
||||
```
|
||||
>>>>> HH:MM:SS.mmm - <OP> START (OP = RP/Replenishment, TD/Teller Deposit, BC/Bill Count, ...)
|
||||
...
|
||||
[N_E] AH: NORM AL: EMPT BH: NORM ...
|
||||
...
|
||||
>>>>> HH:MM:SS.mmm - <OP> END
|
||||
```
|
||||
- `BLOCK_START` / `BLOCK_END` regexes toggle an `inBlock` flag and accumulate `blockLines`.
|
||||
- On `BLOCK_END`, `parseBlock()` finds the single `[N_E]` line, tokenizes `SLOT: STATUS` pairs (`SLOT_TOKEN` regex), and diffs each slot against an in-memory `Map<String,String> slotStatus` — **only emits when status actually changes** (no dedup on the server side, so this in-agent memory is the only guard against event storms).
|
||||
- Status → event mapping: `EMPT`→`CASSETTE_EMPTY`, `SNRF`→`TCR_SENSOR_FAULT`, `TMAN`→`TCR_MANUAL_REQUIRED`, `NORM`→ log-only (no event) recovery notice.
|
||||
- A separate `MISMATCH` regex is checked on **every line independent of block state** (not gated by `inBlock`) — catches `** CST <slot> - SNR deposit count & 80 data deposit count mismatch! ... - <dataCount> - <snsCount>` and emits `TCR_SENSOR_MISMATCH` immediately.
|
||||
- `slotStatus` map is per-`CstSnsLogMonitor` instance (i.e. per-agent-process lifetime), cleared on file truncation or day rollover — so a slot that was `EMPT` yesterday will re-emit `CASSETTE_EMPTY` today if it's still empty at the first block of the new day (this is intentional: state doesn't carry across the daily file rotation).
|
||||
|
||||
### `TcrAccessLogMonitor` — line-based parsing
|
||||
|
||||
Simpler: no blocks, just substring matches per line:
|
||||
- `"Process restarted from abnormal termination"` → `TCR_PROCESS_RESTART`
|
||||
- `"## Shutdown ##"` → `TCR_SHUTDOWN` (message: "TCR operator shutdown")
|
||||
- `"## Shutdown for reboot ##"` → `TCR_SHUTDOWN` (message: "TCR shutdown for reboot" — same event type, different `eventDetails` text based on whether the line contains `"reboot"`)
|
||||
|
||||
No de-dup logic here — every matching line emits, every poll cycle it appears in the unread tail.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
Both monitors emit through the same shared `IncidentEventClient` instance (constructed once in `TcrEventsModule.initialize()`), which is the same client class used by `journal-events` and other event-reporting modules — see [technical.agent] for the full outbound API surface.
|
||||
|
||||
```
|
||||
CstSnsLogMonitor/TcrAccessLogMonitor.emit(eventType, details, ...)
|
||||
→ CreateJournalEventRequest.builder()
|
||||
.agentAtmId(atmId) // ATM name string, not incident's numeric atmId — resolved server-side
|
||||
.eventType(eventType) // free-form String, e.g. "TCR_SENSOR_FAULT" — not validated client-side
|
||||
.eventDetails(details)
|
||||
.cassettePosition(slot) // CstSnsLogMonitor only; null for TcrAccessLogMonitor events
|
||||
.eventSource("TCR_CST_SNR" | "TCR_ACESS_LOG")
|
||||
.build()
|
||||
→ IncidentEventClient.sendEvent(req)
|
||||
→ HTTP POST {agent.endpoint or incident.endpoint}/api/journal-events
|
||||
```
|
||||
|
||||
`eventType` on `CreateJournalEventRequest` is a plain `String` field (`hiveops-agent-journal`'s DTO) — the agent's own `EventType` enum (`com.hiveops.events.EventType`) does **not** include the five `TCR_*` values this module emits (only `CASSETTE_EMPTY` overlaps with the enum). Nothing client-side stops an unregistered type from being sent; recognition/labeling happens server-side against hiveops-incident's DB-backed `event_type` table.
|
||||
|
||||
## Auth / endpoint selection
|
||||
|
||||
`initialize()` picks the property prefix based on whether `agent.endpoint` is set:
|
||||
```java
|
||||
String prefix = agentEndpoint != null ? "agent" : "incident";
|
||||
HttpClientSettings incidentSettings = HttpClientHelper.loadSettingsFromProperties(props, prefix);
|
||||
```
|
||||
`HttpClientHelper.loadSettingsFromProperties` resolves institution key with fallback order `{prefix}.institution.key` → `agent.institution.key` → `incident.institution.key`, and always reads the bearer token from `agent.auth.token` regardless of prefix. This module has no bespoke auth handling — it's 100% the shared agent HTTP settings pattern.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
None used directly. The module reads plain files via `java.nio.file` / `RandomAccessFile` — no `PlatformService`/`PathResolver` dependency, even though it's Windows-only in practice (the LG TCR software and its log format only exist on ATEC Windows kiosks). The Windows-only constraint is enforced by convention/deployment (only installed on `device.type=ATEC_TCR` machines) rather than by a runtime OS check in this module's code.
|
||||
|
||||
## Cross-links
|
||||
- [technical.agent] — module system, lifecycle, shared HTTP client settings, full outbound API surface
|
||||
- [platform.module-map]
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
module: agent.atec-tcr-events
|
||||
title: ATEC TCR Events — cassette + process monitoring for LG Teller Cash Recyclers
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-atec-tcr-events` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Maven artifact: **`hiveops-module-atec-tcr-events`** (`com.hiveops`, version `1.0.0`, packaged as a standalone `jar`, parent `hiveops-agent-parent`).
|
||||
- Agent module id (`getName()`, and the string to use in **`modules.disabled`**): **`tcr-events`** — NOT `atec-tcr-events`.
|
||||
- Entry class: `com.hiveops.tcrevents.TcrEventsModule`, registered via `META-INF/services/com.hiveops.core.module.AgentModule`.
|
||||
- Log package: `com.hiveops.tcrevents`.
|
||||
- Activation gate: `device.type` must equal `ATEC_TCR` (or legacy `TCR`); any other value → module inert, `isEnabled()` returns `false`.
|
||||
- Platform: Windows-only in practice (no runtime OS check in code — enforced by deployment convention).
|
||||
|
||||
## Config properties (exact keys)
|
||||
| Key | Default | Required | Notes |
|
||||
|---|---|---|---|
|
||||
| `tcr.events.enabled` | `true` | no | `false` (case-insensitive) disables the module |
|
||||
| `tcr.log.dir` | *(none)* | **yes** | must exist and be a directory or module disables itself with a warning log, no crash |
|
||||
| `tcr.events.poll.interval.sec` | `30` | no | **read but NOT applied** — `start()` hardcodes `DEFAULT_POLL_SEC=30`; changing this property does nothing today |
|
||||
|
||||
Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix selection: `agent` if `agent.endpoint` is set, else `incident`), `agent.auth.token`, `agent.institution.key` / `incident.institution.key`.
|
||||
|
||||
## Event types emitted (exact strings)
|
||||
| Event type | Source file | Trigger |
|
||||
|---|---|---|
|
||||
| `CASSETTE_EMPTY` | `CST_SNR_MMDD.log` | slot status transitions to `EMPT` |
|
||||
| `TCR_SENSOR_FAULT` | `CST_SNR_MMDD.log` | slot status transitions to `SNRF` |
|
||||
| `TCR_MANUAL_REQUIRED` | `CST_SNR_MMDD.log` | slot status transitions to `TMAN` |
|
||||
| `TCR_SENSOR_MISMATCH` | `CST_SNR_MMDD.log` | line matches `** CST <slot> - SNR deposit count ... mismatch ...` |
|
||||
| `TCR_PROCESS_RESTART` | `ACESSLog<MMDD>.txt` | line contains `Process restarted from abnormal termination` |
|
||||
| `TCR_SHUTDOWN` | `ACESSLog<MMDD>.txt` | line contains `## Shutdown ##` or `## Shutdown for reboot ##` |
|
||||
|
||||
- `eventType` is a free-form `String` on `CreateJournalEventRequest` (`hiveops-agent-journal` DTO) — **not validated client-side**.
|
||||
- The agent's own `com.hiveops.events.EventType` enum only contains `CASSETTE_EMPTY` from this list — the five `TCR_*` strings are **not** in that enum. Recognition happens in hiveops-incident's DB-backed `event_type` table, not agent code.
|
||||
- `eventSource` sent: `"TCR_CST_SNR"` (cassette log events) or `"TCR_ACESS_LOG"` (access log events).
|
||||
- `cassettePosition` (slot, e.g. `AH`, `AL`) is only populated for `CstSnsLogMonitor` events; null for access-log events.
|
||||
- Transport: `POST {endpoint}/api/journal-events` via `IncidentEventClient.sendEvent()` — same client/endpoint as `journal-events` module.
|
||||
- Recovery to `NORM` status emits **no event** — agent-log-only (`slot {} recovered`).
|
||||
|
||||
## Files monitored
|
||||
- `{tcr.log.dir}/CST_SNR_<MMDD>.log` — cassette sensor/status log (daily rotation; date = `MMdd`, e.g. `CST_SNR_0701.log`).
|
||||
- `{tcr.log.dir}/ACESSLog<MMDD>.txt` — TCR process/access log (daily rotation; **no underscore** in code: `ACESSLog0701.txt`).
|
||||
- Both use tail-by-offset (`RandomAccessFile` + stored `fileOffset`), reset on daily rollover or file truncation.
|
||||
|
||||
## Deployment
|
||||
- Delivered fleet-wide via **`INSTALL_MODULE`** fleet task (drops JAR into agent's `modules/` dir, restarts agent — `ProcessInstallModule` in `hiveops-file-collection`).
|
||||
- Disable without uninstalling: add `tcr-events` to `modules.disabled`, push `CONFIG_UPDATE`.
|
||||
- Event types must be registered in hiveops-incident → Settings → Event Types (DB `event_type` table) to be recognized/labeled correctly.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT use `atec-tcr-events` as the `modules.disabled` value — the actual module id is `tcr-events`.
|
||||
- Do NOT expect changing `tcr.events.poll.interval.sec` to change the poll cadence — it's a known no-op bug; cadence is always 30s.
|
||||
- Do NOT assume the five `TCR_*` event type strings exist in the agent's `EventType` enum — they don't; only `CASSETTE_EMPTY` does. Don't "fix" this by adding them there — the enum isn't what gates recognition (the DB `event_type` table in hiveops-incident is).
|
||||
- Do NOT expect this module to run on non-`ATEC_TCR` device types — it's a no-op by design elsewhere, not a bug to chase.
|
||||
- Do NOT expect retry/queueing on send failure — a failed `sendEvent()` call is logged and dropped, not persisted or retried on the next poll.
|
||||
- Do NOT look for an underscore in the access-log filename — the code is `ACESSLog<MMDD>.txt` (e.g. `ACESSLog0701.txt`), not `ACESSLog_0701.txt`; the class javadoc's `ACESSLog_MMDD.txt` is stale.
|
||||
- Do NOT confuse this with `hiveops-module-nh-tcr-events` (`NhTcrEventsModule`) or `hiveops-module-nh-tcr-journal` (`NhTcrJournalModule`) — separate modules, separate vendor (NH TCR), different log formats and event pipelines.
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
module: agent.atec-tcr-events
|
||||
title: ATEC TCR Events — cassette + process monitoring for LG Teller Cash Recyclers
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support.** Written 2026-07-01 from `hiveops-module-atec-tcr-events` source. Verify against code before relying on specifics.
|
||||
|
||||
## What this module is
|
||||
|
||||
`hiveops-module-atec-tcr-events` (Maven artifact `hiveops-module-atec-tcr-events`, agent module id **`tcr-events`**) is an **agent drop-in module**, not a backend service. It runs inside the HiveIQ Agent process on **ATEC/LG TCR (Teller Cash Recycler) devices only** — Windows-only, and it self-disables unless the ATM's `device.type` is `ATEC_TCR` (legacy alias `TCR`).
|
||||
|
||||
It tails two log files the LG TCR software already writes on disk and turns specific lines into HiveIQ journal events sent to the incident pipeline — the module does not talk to the recycler hardware directly, and it does not touch cassette counts/cash totals (that's reconciliation's job, not this module's).
|
||||
|
||||
## What it monitors
|
||||
|
||||
Two independent pollers, both driven by a single scheduler tick (default every 30s):
|
||||
|
||||
1. **`CST_SNR_MMDD.log`** (daily-rotated) — cassette sensor/status log. The module watches for:
|
||||
- Per-slot status transitions inside `[N_E]` lines (e.g. `AH: NORM AL: EMPT`) — only emits when a slot's status *changes*, so a slot stuck in `EMPT` doesn't re-fire every poll.
|
||||
- A `SNR deposit count ... mismatch` warning line (deposit-count sensor vs. transaction-data mismatch).
|
||||
2. **`ACESSLog<MMDD>.txt`** (daily-rotated — note the firmware spelling `ACESS` and, in the code, **no underscore** before the date: e.g. `ACESSLog0701.txt`) — TCR process/access log. The module watches for:
|
||||
- `Process restarted from abnormal termination`
|
||||
- `## Shutdown ##` / `## Shutdown for reboot ##`
|
||||
|
||||
Both files are expected in the **same directory**, configured once via `tcr.log.dir`.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
| Property | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `tcr.events.enabled` | `true` | Master on/off switch for this module (checked in addition to the `device.type` gate). |
|
||||
| `tcr.log.dir` | *(none — required)* | Directory containing `CST_SNR_<MMDD>.log` and `ACESSLog<MMDD>.txt`. If unset or not a real directory, the module logs a warning and disables itself — it does **not** throw/crash agent startup. |
|
||||
| `tcr.events.poll.interval.sec` | `30` | Poll interval in seconds. **Known bug:** `initialize()` reads this value only to log it — `start()` schedules the poller with the hardcoded `DEFAULT_POLL_SEC` (30s) regardless of what's configured. Changing this property currently has no effect. |
|
||||
|
||||
It also reuses the shared agent HTTP settings — `agent.endpoint` (preferred) or `incident.endpoint` (legacy fallback), `agent.auth.token`, and the institution-key chain (`agent.institution.key` / `incident.institution.key`) — the same properties every other event-reporting module uses. There are no TCR-specific auth properties.
|
||||
|
||||
## How it's enabled / deployed
|
||||
|
||||
- It only activates on ATMs configured with `device.type=ATEC_TCR` (or the legacy `TCR` value). On any other device type it logs `device.type is '<X>', not ATEC_TCR — module disabled` and does nothing — this is expected, not a bug, on non-TCR machines.
|
||||
- Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which drops the module JAR into the agent's `modules/` directory and restarts the agent service (see `ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order applies: lab ATM → pilot device → fleet.
|
||||
- Can be disabled fleet-wide without removing the JAR by adding `tcr-events` to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE` (module id for `modules.disabled` is **`tcr-events`**, not `atec-tcr-events` — see Claude doc for the exact string).
|
||||
- Event types it emits (`CASSETTE_EMPTY`, `TCR_SENSOR_FAULT`, `TCR_MANUAL_REQUIRED`, `TCR_SENSOR_MISMATCH`, `TCR_PROCESS_RESTART`, `TCR_SHUTDOWN`) must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) for them to be recognized/labeled properly downstream. `CASSETTE_EMPTY` is a standard type shared with other modules; the five `TCR_*` types are specific to this module and may need to be added manually on a new environment.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **No TCR events ever appear for a customer ATM** — first check `device.type` on that ATM's config; if it isn't `ATEC_TCR`/`TCR`, the module is intentionally inert. Not a bug.
|
||||
- **Module loaded but silent** — check `tcr.log.dir` is set and points at a real, existing directory on that machine; a missing/wrong path disables the module at startup with a log warning, no crash.
|
||||
- **Events stopped after midnight / around log rotation** — both monitors roll to a new `MMDD`-named file at local midnight; if the LG TCR software's file-naming convention ever changes (e.g. non-standard rotation, DST edge case), the module will watch a file that never appears and go silent. Check agent logs for `tcr-events: watching <path>` to confirm which file it's tailing.
|
||||
- **Events sent but not visible/categorized in Incident** — check that `CASSETTE_EMPTY`, `TCR_SENSOR_FAULT`, `TCR_MANUAL_REQUIRED`, `TCR_SENSOR_MISMATCH`, `TCR_PROCESS_RESTART`, `TCR_SHUTDOWN` are registered in hiveops-incident's Event Types settings — these are DB-driven, not hardcoded in incident's code.
|
||||
- **HTTP send failures** — `IncidentEventClient.sendEvent()` failures are caught and logged (`tcr-events: failed to send ...`) but not retried or queued; a transient outage silently drops that single event rather than backing it up for a later poll.
|
||||
- **Slot recovered (NORM) events don't create tickets** — by design; recovery only writes an agent-side log line (`slot {} recovered`), no event is emitted for a return to `NORM`.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
module: agent.atm-config-sync
|
||||
title: ATM Config Sync — pushes ATM hardware/software/cassette config to the platform
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · architecture.** Written 2026-07-01 from `hiveops-module-atm-config-sync` source. Verify against code before relying on specifics.
|
||||
|
||||
## Module shape
|
||||
|
||||
Package `com.hiveops.configsync`, a **single class**, no controllers/DB/Kafka (this is an agent module, not a microservice — see [technical.agent]):
|
||||
|
||||
- **`AtmConfigSyncModule`** — the `AgentModule` SPI entry point (registered via `META-INF/services/com.hiveops.core.module.AgentModule`). Owns the full lifecycle (`initialize()` → `isEnabled()` → `start()` → `stop()`), the `IncidentEventClient`, and a single-thread `ScheduledExecutorService`.
|
||||
|
||||
No separate monitors or parsers of its own — it **orchestrates existing agent parsers/readers** (from `hiveops-version`, `hiveops-agent-journal`, and the platform layer) and assembles their output into one DTO. All heavy lifting (applog parsing, MoniPlus2S parsing, cassette/version reads) is delegated to shared, `provided`-scope components supplied by the fat JAR at runtime.
|
||||
|
||||
## Lifecycle & flow
|
||||
|
||||
Unlike the log-tailing modules, this is a **poll-and-assemble-then-POST** module, not a log tail:
|
||||
|
||||
```
|
||||
initialize(ctx)
|
||||
→ resolve endpoint: agent.endpoint (preferred) else incident.endpoint; if neither → return (ready=false)
|
||||
→ prefix = agent.endpoint != null ? "agent" : "incident"
|
||||
→ HttpClientSettings via HttpClientHelper.loadSettingsFromProperties(props, prefix)
|
||||
→ new IncidentEventClient(settings, atmName, country)
|
||||
→ ready = true
|
||||
→ syncAtmConfig() // one immediate sync at startup
|
||||
|
||||
isEnabled(ctx) → ready // true only if an endpoint was resolved
|
||||
|
||||
start()
|
||||
→ ConfigResyncRegistry.register(this::syncAtmConfig) // on-demand RESYNC_CONFIG hook
|
||||
→ ScheduledExecutorService (daemon thread "config-resync")
|
||||
.scheduleAtFixedRate(syncAtmConfig, initialDelay=0, period=6h)
|
||||
|
||||
stop() → scheduler.shutdownNow()
|
||||
```
|
||||
|
||||
### `syncAtmConfig()` — the one method that does everything
|
||||
|
||||
```
|
||||
awaitReconnectJitter() // ConnectionStateTracker — post-reconnect anti-stampede
|
||||
AtmConfigSyncRequest req = new(...)
|
||||
req.atmName / country / agentVersion // from ModuleContext + agentContext.getVersion()
|
||||
req.startupDetectionMethod // startup-info.properties → startup.detection.method (if file exists)
|
||||
req.timezone = TimeZone.getDefault().getID()
|
||||
req.institutionKey // incident.institution.key (only)
|
||||
// platform ConfigurationReader (Optional-returning reads, each set only if present):
|
||||
req.systemName / customizingVersion / hotfixVersion / countrySpecificVersion
|
||||
req.cassettes = readCassetteDenominations() → List<CassetteConfig>(position, currency, value)
|
||||
// applog file (only if applog.events.dir set):
|
||||
req.hyosungVersion = HyosungVersionParser.parseFromFile(applogFile) // app/SDK/GSDK/Nextware
|
||||
req.systemInfo = AppLogHeaderParser.parseFromFile(applogFile) // OS + installed programs/packages
|
||||
// MoniPlus2S dir (moniplus2s.dir, or C:\Hyosung\MoniPlus2S on Windows):
|
||||
req.moniplus2sConfig = MoniPlus2SConfigParser.parse()
|
||||
req.cassettes (fallback) = parseCassettesFromHostNoteTypes(...) // only if cassettes still empty
|
||||
req.moniplus2sLayeredConfig = MoniPlus2SConfigParser.parseAllLayers()
|
||||
client.syncConfig(req) // POST — see below
|
||||
```
|
||||
|
||||
Every step is best-effort: a null/absent source is skipped, and the whole body is wrapped in try/catch that logs `ATM config sync failed: <msg>` (no rethrow, no retry queue). The scheduler's next 6h tick is the retry.
|
||||
|
||||
### Applog file resolution
|
||||
`applog.events.dir` + `applog.events.filename.format` (default `APLog{YYYY}{MM}{DD}.log`) feed an `AppLogSource(dir, format, atmName)`, whose `getCurrentLogFile()` returns the file for today. That single file is handed to both `HyosungVersionParser` and `AppLogHeaderParser`.
|
||||
|
||||
### MoniPlus2S path handling
|
||||
Java `.properties` parsing strips single backslashes, so `moniplus2s.dir=C:\Hyosung\MoniPlus2S` arrives as `C:HyosungMoniPlus2S`. The module checks `new File(dir).isDirectory()`; if it fails it logs a warning, nulls the value, and (on Windows) falls back to the hardcoded `C:\Hyosung\MoniPlus2S`.
|
||||
|
||||
## How the payload reaches the platform
|
||||
|
||||
Transport is the shared `IncidentEventClient` (from `hiveops-agent-journal`) — the same client used by the event modules, but a **different method and endpoint**:
|
||||
|
||||
```
|
||||
IncidentEventClient.syncConfig(AtmConfigSyncRequest req)
|
||||
→ POST {agent.endpoint | incident.endpoint}/api/atms/config/sync
|
||||
Content-Type: application/json (Jackson-serialized AtmConfigSyncRequest)
|
||||
→ accepts HTTP 200 / 201 / 202; anything else → logErrorResponse + throw IOException
|
||||
```
|
||||
|
||||
Note this is **not** `/api/journal-events` — config sync has its own `/api/atms/config/sync` endpoint. The backend keys off `atmName`/`country` in the body to find/update the device record. In production the base URL points at **agent-proxy** (`agent.endpoint`), which forwards to the device/incident backends; the legacy `incident.endpoint` path targets incident directly.
|
||||
|
||||
## On-demand resync wiring
|
||||
|
||||
`ConfigResyncRegistry` (in `hiveops-core`) is a static single-slot registry decoupling the `RESYNC_CONFIG` command handler from the resync implementation:
|
||||
|
||||
```
|
||||
AtmConfigSyncModule.start() → ConfigResyncRegistry.register(this::syncAtmConfig) // sets one volatile callback
|
||||
CommandCenterModule (RESYNC_CONFIG task) → ConfigResyncRegistry.trigger() → callback.run()
|
||||
```
|
||||
|
||||
It holds exactly **one** `volatile Runnable` — last registrant wins. The registry's javadoc references `JournalEventModule` as the historical registrant, but in the current tree **`AtmConfigSyncModule` is the only caller of `register()`**, so it owns the `RESYNC_CONFIG` trigger. If another module ever also registers, whichever `start()`s last silently overwrites the other. If nothing registered (module disabled), `trigger()` logs a warning and does nothing.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
Uses the platform layer directly (unlike the file-tailing modules):
|
||||
|
||||
- `context.getPlatformService().getConfigurationReader()` → `readSystemName / readCustomizingVersion / readHotfixVersion / readCountrySpecificVersion` (all `Optional`) and `readCassetteDenominations()`.
|
||||
- `context.getPlatformService().isWindows()` → gates the `C:\Hyosung\MoniPlus2S` fallback.
|
||||
|
||||
The `ConfigurationReader` abstracts Windows (registry) vs Linux (file/env) config sources, so the same module works cross-platform without OS branching in this class (aside from the Windows-only MoniPlus2S default).
|
||||
|
||||
## Cross-links
|
||||
- [technical.agent] — module system, lifecycle, `ModuleContext`, shared `IncidentEventClient` / HTTP settings, platform abstraction
|
||||
- [platform.module-map]
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
module: agent.atm-config-sync
|
||||
title: ATM Config Sync — pushes ATM hardware/software/cassette config to the platform
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-atm-config-sync` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Maven artifact: **`hiveops-module-atm-config-sync`** (`com.hiveops`, version `1.0.0`, packaged as a standalone `jar`, parent `hiveops-agent-parent` `4.4.3`).
|
||||
- Agent module id (`getName()`, and the string for **`modules.disabled`**): **`device-config-sync`** — NOT `atm-config-sync`.
|
||||
- Entry class: `com.hiveops.configsync.AtmConfigSyncModule`, registered via `META-INF/services/com.hiveops.core.module.AgentModule`.
|
||||
- Log package (`getLogPackage()`): `com.hiveops.configsync`.
|
||||
- Dependencies (`getDependencies()`): none (empty list).
|
||||
- Single class — no separate monitor/parser classes of its own.
|
||||
|
||||
## Behavior (not an event/log-tail module)
|
||||
- Emits **NO** journal/incident events. Sends one consolidated config payload.
|
||||
- `isEnabled()` = `ready` = true only if `agent.endpoint` OR `incident.endpoint` resolved at `initialize()`. No `device.type` gate, no dedicated enable flag.
|
||||
- Sync triggers: (1) once at `initialize()`; (2) every **6h** fixed-rate (`RESYNC_INTERVAL_HOURS=6`, initialDelay 0, daemon thread `config-resync`); (3) on-demand via **`RESYNC_CONFIG`** fleet command → `ConfigResyncRegistry.trigger()`.
|
||||
- Each sync calls `ConnectionStateTracker.awaitReconnectJitter()` first (post-reconnect anti-stampede).
|
||||
- Failure = logged (`ATM config sync failed: ...`), NOT retried/queued; next 6h tick or `RESYNC_CONFIG` recovers.
|
||||
|
||||
## Config properties (exact keys)
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `agent.endpoint` | *(none)* | preferred base URL; if set → HTTP prefix `agent` |
|
||||
| `incident.endpoint` | *(none)* | legacy fallback; used only if `agent.endpoint` unset → prefix `incident`. Neither set → module disabled |
|
||||
| `incident.institution.key` | *(none)* | stamped on payload. Reads ONLY this key (not `agent.institution.key`) |
|
||||
| `applog.events.dir` | *(none)* | dir for applog file; unset → Hyosung version + system info skipped |
|
||||
| `applog.events.filename.format` | `APLog{YYYY}{MM}{DD}.log` | applog filename pattern |
|
||||
| `moniplus2s.dir` | Win fallback `C:\Hyosung\MoniPlus2S` | if value isn't a real dir (backslash stripping) → warn + Windows default |
|
||||
|
||||
Bearer token / HTTP params: shared `HttpClientHelper.loadSettingsFromProperties(props, prefix)` (`agent.auth.token` chain). No module-specific auth.
|
||||
|
||||
## Transport (exact)
|
||||
- Method: `IncidentEventClient.syncConfig(AtmConfigSyncRequest)`.
|
||||
- Endpoint: **`POST {endpoint}/api/atms/config/sync`** (NOT `/api/journal-events`).
|
||||
- Body: Jackson-serialized `AtmConfigSyncRequest` (`com.hiveops.events.dto`).
|
||||
- Success: HTTP 200/201/202; else throws `IOException`.
|
||||
- Backend resolves device by `atmName` + `country` in body.
|
||||
|
||||
## Payload fields (`AtmConfigSyncRequest`)
|
||||
- Identity: `atmName`, `country`, `agentVersion`, `timezone` (`TimeZone.getDefault().getID()`), `institutionKey`, `startupDetectionMethod` (from `startup-info.properties` → `startup.detection.method`).
|
||||
- Versions (platform `ConfigurationReader`): `systemName`, `customizingVersion`, `hotfixVersion`, `countrySpecificVersion`.
|
||||
- `cassettes`: `List<CassetteConfig>{position, currency, denomination}` from `readCassetteDenominations()`; fallback = MoniPlus2S `HostNoteTypes` parse only if still empty.
|
||||
- `hyosungVersion` (`HyosungVersionParser`): app/SDK/GSDK/Nextware/componentVersions.
|
||||
- `systemInfo` (`AppLogHeaderParser`): OS + installed programs/packages.
|
||||
- `moniplus2sConfig` + `moniplus2sLayeredConfig` (`MoniPlus2SConfigParser.parse()` / `.parseAllLayers()`).
|
||||
|
||||
## Deployment
|
||||
- Delivered via **`INSTALL_MODULE`** fleet task (drops JAR into agent `modules/` dir — `ProcessInstallModule` in `hiveops-file-collection`).
|
||||
- Disable without uninstall: add `device-config-sync` to `modules.disabled`, push `CONFIG_UPDATE`.
|
||||
- Force immediate resync: `RESYNC_CONFIG` fleet command (no restart).
|
||||
|
||||
## Do NOT
|
||||
- Do NOT use `atm-config-sync` as the `modules.disabled` value — the real id is **`device-config-sync`**.
|
||||
- Do NOT look for emitted events — this module sends config, not `*_EVENT`s; there are no event-type strings.
|
||||
- Do NOT expect it to POST to `/api/journal-events` — it uses `/api/atms/config/sync`.
|
||||
- Do NOT expect `agent.institution.key` to be picked up for the payload — only `incident.institution.key` is read here.
|
||||
- Do NOT put a single-backslash Windows path in `moniplus2s.dir` — properties strip it; double the backslashes or rely on the `C:\Hyosung\MoniPlus2S` default.
|
||||
- Do NOT expect send failures to retry — they're logged and dropped; recovery is the next 6h tick / `RESYNC_CONFIG`.
|
||||
- Do NOT assume `RESYNC_CONFIG` works if this module is disabled — it's the sole `ConfigResyncRegistry` registrant; disabled → `trigger()` no-ops with a warning.
|
||||
- Do NOT expect a `device.type`/enable gate — the only gate is "an endpoint is configured."
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
module: agent.atm-config-sync
|
||||
title: ATM Config Sync — pushes ATM hardware/software/cassette config to the platform
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support.** Written 2026-07-01 from `hiveops-module-atm-config-sync` source. Verify against code before relying on specifics.
|
||||
|
||||
## What this module is
|
||||
|
||||
`hiveops-module-atm-config-sync` (Maven artifact `hiveops-module-atm-config-sync`, agent module id **`device-config-sync`**) is an **agent drop-in module**, not a backend service. It runs inside the HiveIQ Agent process and periodically **snapshots the ATM's configuration and inventory** and POSTs it to the HiveIQ backend so the device record (Devices / ATM detail) stays current: software versions, timezone, cassette denominations, and Hyosung MoniPlus2S settings.
|
||||
|
||||
Unlike the event-reporting modules (journal-events, tcr-events), it emits **no journal/incident events**. It sends one consolidated **config-sync payload** (`AtmConfigSyncRequest`) to a dedicated endpoint. Think of it as "device metadata sync," not "alerting."
|
||||
|
||||
## What it collects and syncs
|
||||
|
||||
Each sync builds one `AtmConfigSyncRequest` from whatever sources resolve on that machine (all optional — missing sources are simply omitted):
|
||||
|
||||
- **Identity / basics** — `atmName`, `country`, `agentVersion` (from the agent context), `timezone` (`TimeZone.getDefault().getID()`), and `institutionKey` (only from `incident.institution.key`).
|
||||
- **Startup detection method** — read from `startup-info.properties` (`startup.detection.method`) in the agent's start dir, if present.
|
||||
- **Software versions** — via the platform `ConfigurationReader`: `systemName`, `customizingVersion`, `hotfixVersion`, `countrySpecificVersion`.
|
||||
- **Cassette denominations** — from `ConfigurationReader.readCassetteDenominations()`; if that yields nothing, it falls back to parsing cassettes out of the MoniPlus2S `HostNoteTypes`.
|
||||
- **Hyosung version info** — parsed from the current applog file (`HyosungVersionParser`): application / SDK / GSDK / Nextware / component versions.
|
||||
- **System info** — parsed from the applog header (`AppLogHeaderParser`): OS version, installed programs and packages.
|
||||
- **MoniPlus2S config** — parsed from the MoniPlus2S directory (`MoniPlus2SConfigParser`): serial, model, host IP/port, plus a full **layered config** view (all config folders / XML files).
|
||||
|
||||
Payload is POSTed to **`{endpoint}/api/atms/config/sync`** via `IncidentEventClient.syncConfig(...)`. The backend resolves the device by `atmName`/`country`.
|
||||
|
||||
## When it syncs
|
||||
|
||||
Three triggers, all running the same `syncAtmConfig()` method:
|
||||
|
||||
1. **On agent startup** — `initialize()` runs one sync immediately (after the endpoint is resolved).
|
||||
2. **On a fixed schedule** — a daemon scheduler (`config-resync` thread) re-syncs **every 6 hours** (`RESYNC_INTERVAL_HOURS = 6`), starting immediately at `start()`.
|
||||
3. **On demand** — the module registers a callback with `ConfigResyncRegistry`, so a **`RESYNC_CONFIG`** fleet command (handled by the command processor) triggers an immediate re-sync without restarting the agent.
|
||||
|
||||
Every sync first calls `ConnectionStateTracker.awaitReconnectJitter()` — after a reconnect it waits a randomized jitter so a fleet coming back online doesn't stampede the backend all at once.
|
||||
|
||||
## Config properties it reads
|
||||
|
||||
This module has **no dedicated `enabled` flag and no `device.type` gate** — it activates whenever an endpoint is configured. Keys read from `hiveops.properties`:
|
||||
|
||||
| Property | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `agent.endpoint` | *(none)* | Preferred backend base URL (agent-proxy). If set, HTTP settings use the `agent` prefix. |
|
||||
| `incident.endpoint` | *(none)* | Legacy fallback base URL, used only if `agent.endpoint` is unset (`incident` prefix). If **both** are unset, the module returns early at init and stays disabled (`isEnabled()` → false). |
|
||||
| `incident.institution.key` | *(none)* | Institution key stamped onto the sync payload. Note: this method reads **only** `incident.institution.key`, not `agent.institution.key`. |
|
||||
| `applog.events.dir` | *(none)* | Directory holding the applog file used for Hyosung version + system info. If unset, version/system info are skipped. |
|
||||
| `applog.events.filename.format` | `APLog{YYYY}{MM}{DD}.log` | Applog filename pattern used to resolve the current file. |
|
||||
| `moniplus2s.dir` | Windows fallback `C:\Hyosung\MoniPlus2S` | MoniPlus2S config directory. If the property value doesn't resolve to a real directory (Java properties strip single backslashes, e.g. `C:\Hyosung` → `C:Hyosung`), it logs a warning and falls back to the hardcoded Windows path. |
|
||||
|
||||
Bearer token and other HTTP params come from the shared agent HTTP settings (`HttpClientHelper.loadSettingsFromProperties(props, prefix)`) — the same `agent.auth.token` chain every module uses. No config-sync-specific auth.
|
||||
|
||||
## How it's enabled / deployed
|
||||
|
||||
- **No per-device gate** — as long as `agent.endpoint` (or legacy `incident.endpoint`) is set, the module runs on that ATM. It is safe on any device type; it only reports what it can read.
|
||||
- Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which drops the module JAR into the agent's `modules/` directory (handled by `ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order applies: lab ATM → pilot device → fleet.
|
||||
- Can be disabled fleet-wide without removing the JAR by adding **`device-config-sync`** to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE`. (Disable string is `device-config-sync` — the `getName()` value — **not** `atm-config-sync`.)
|
||||
- Force a fresh push at any time with a **`RESYNC_CONFIG`** fleet command — no restart needed.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **Device metadata never populates for an ATM** — confirm `agent.endpoint`/`incident.endpoint` is set; with neither, the module is inert at startup (`isEnabled()` false). Also check the agent log for `ATM config sync complete` vs `ATM config sync failed: ...`.
|
||||
- **Sync runs but versions/system info are blank** — usually `applog.events.dir` is unset or points at a directory with no matching `APLog…` file; the Hyosung/system-info blocks are simply skipped. Cassette/MoniPlus2S data can still be present.
|
||||
- **MoniPlus2S config missing on a Hyosung box** — check the `moniplus2s.dir` warning in the log (`… not found (backslash escaping issue?)`). A single-backslash path in `hiveops.properties` gets mangled; either double the backslashes or rely on the `C:\Hyosung\MoniPlus2S` default.
|
||||
- **Cassettes wrong/empty** — the platform `ConfigurationReader` is tried first; the MoniPlus2S `HostNoteTypes` fallback only fires when the reader returns none. If both are empty, no cassettes are sent.
|
||||
- **Whole sync fails** — `syncAtmConfig()` wraps everything in a try/catch and logs `ATM config sync failed: <message>`; a failed HTTP call (`syncConfig` throws on non-2xx) is logged but **not** queued/retried — the next scheduled 6h tick (or a `RESYNC_CONFIG`) is the recovery path.
|
||||
- **On-demand resync does nothing** — a `RESYNC_CONFIG` only fires if this module registered its callback (i.e. it's enabled and started). If it's in `modules.disabled`, the command center logs "no callback registered."
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
module: agent.ejournal-db-monitor
|
||||
title: EJournal DB Monitor — how it's built
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
# EJournal DB Monitor (Architecture)
|
||||
|
||||
A single-class agent module, `com.hiveops.ejournal.EJournalDbMonitorModule`,
|
||||
registered as an SPI service via
|
||||
`META-INF/services/com.hiveops.core.module.AgentModule`. It plugs into the standard
|
||||
agent module lifecycle (`initialize` → `isEnabled` → `start` → `stop`) — see
|
||||
[technical.agent] for the module system as a whole.
|
||||
|
||||
## Module shape
|
||||
|
||||
- **Artifact:** `hiveops-module-ejournal-db-monitor.jar` (Maven `finalName`), module
|
||||
version `1.0.0`, parent `hiveops-agent-parent` `4.4.3`.
|
||||
- **Dependencies** (all `provided` — supplied by the fat JAR at runtime, not shaded
|
||||
in): `hiveops-core` (module API), `hiveops-agent-journal` (`EventType`,
|
||||
`IncidentEventClient`, `CreateJournalEventRequest`, `HttpClientHelper`),
|
||||
`log4j-api`, `commons-lang3`.
|
||||
- `getName()` → `ejournal-db-monitor`; `getDependencies()` → empty (no ordering
|
||||
constraints); `getLogPackage()` → `com.hiveops.ejournal`.
|
||||
|
||||
## Flow: poll → detect → emit
|
||||
|
||||
The module is a **poller**, not a log-tail or file-watch. There is no hook into the
|
||||
ATM app; it just stats a file on a schedule.
|
||||
|
||||
1. **`initialize(ModuleContext)`**
|
||||
- Reads `ejournal.db.monitor.enabled`; returns early if disabled.
|
||||
- Resolves the event endpoint: `agent.endpoint` preferred, else
|
||||
`incident.endpoint`. If neither is set, returns early (module stays not-ready →
|
||||
no-op). HTTP config loaded via `HttpClientHelper.loadSettingsFromProperties`
|
||||
with prefix `agent` or `incident` accordingly.
|
||||
- Builds an `IncidentEventClient` from those settings plus
|
||||
`context.getAtmName()` and `context.getCountry()`.
|
||||
- Parses thresholds/interval/cooldown into bytes/minutes/hours and sets
|
||||
`ready = true`.
|
||||
2. **`isEnabled(ModuleContext)`** → returns the `ready` flag set in `initialize`.
|
||||
3. **`start()`** — creates a single-thread `ScheduledExecutorService` (daemon thread
|
||||
named `ejournal-db-monitor`) and schedules `checkDatabaseSize()` at fixed rate,
|
||||
starting immediately (`initialDelay = 0`), every `intervalMinutes`.
|
||||
4. **`checkDatabaseSize()`** (each tick):
|
||||
- `resolveDbFile()` — if `ejournal.db.monitor.filename` set, use it; otherwise
|
||||
list the directory filtered to `.mdb/.sdf/.db/.sqlite` and pick the **largest**.
|
||||
- `classify(sizeBytes)` → `NONE` / `WARNING` / `CRITICAL` against the byte
|
||||
thresholds.
|
||||
- State reset: if now `NONE` but previously alerted, log and clear
|
||||
`lastAlertLevel`/`lastAlertTime` (DB was cleared).
|
||||
- Suppression: send only if the level **escalated** (`ordinal()` increased) OR the
|
||||
cooldown (`cooldownHours`) has elapsed since `lastAlertTime`.
|
||||
- On send, record `lastAlertLevel` + `lastAlertTime`.
|
||||
- The whole body is wrapped in try/catch so a bad tick logs and the schedule
|
||||
continues.
|
||||
5. **`sendAlert(level, file, sizeBytes)`** — maps level to `EventType`
|
||||
(`EJOURNAL_DB_CRITICAL` or `EJOURNAL_DB_WARNING`), formats a human `eventDetails`
|
||||
string (name, MB, % of 2 GB), and builds a `CreateJournalEventRequest`.
|
||||
6. **`stop()`** — `scheduler.shutdownNow()`.
|
||||
|
||||
Alert state (`lastAlertLevel`, `lastAlertTime`, the `AlertLevel` enum
|
||||
`NONE/WARNING/CRITICAL`) is **in-memory only** — it resets on agent restart, so the
|
||||
first post-restart check that is over-threshold will alert again.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
The module reuses the shared journal-events pipeline rather than any bespoke
|
||||
transport:
|
||||
|
||||
```
|
||||
EJournalDbMonitorModule.sendAlert()
|
||||
→ CreateJournalEventRequest.builder()
|
||||
.agentAtmId(atmName)
|
||||
.eventType(EJOURNAL_DB_WARNING | EJOURNAL_DB_CRITICAL)
|
||||
.eventDetails(...)
|
||||
.eventSource("HIVEOPS_AGENT_EJDB")
|
||||
.build()
|
||||
→ IncidentEventClient.sendEvent(event)
|
||||
→ POST {endpoint}/api/journal-events (JSON)
|
||||
```
|
||||
|
||||
`endpoint` is `agent.endpoint` (agent-proxy) when configured, else
|
||||
`incident.endpoint` (direct to hiveops-incident). The ATM is resolved server-side by
|
||||
`agentAtmId`; the module does not resolve a numeric `atmId` itself. From there the
|
||||
event follows the normal HiveIQ incident path (auto-register + incident creation).
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
The module itself uses **no** `PlatformService` — it only does `java.io.File`
|
||||
size/listing, and the default directory is Windows-shaped (`D:\EJOURNAL`), matching
|
||||
its Hyosung target. Platform-specific pieces (delivery to `modules/`, agent restart)
|
||||
live in the `INSTALL_MODULE` handler (`ProcessInstallModule`), which does use
|
||||
`PlatformService.getSystemCommands()` / `getPathResolver()`. For non-Windows or
|
||||
non-standard installs, point `ejournal.db.monitor.dir` at the real path.
|
||||
|
||||
Cross-reference: [technical.agent] for the ServiceLoader module registry, lifecycle,
|
||||
and the `IncidentEventClient` / `/api/journal-events` contract shared with the
|
||||
`journal-events` module.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
module: agent.ejournal-db-monitor
|
||||
title: EJournal DB Monitor — act-without-guessing reference
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
# EJournal DB Monitor (Claude quick-reference)
|
||||
|
||||
## Identity
|
||||
|
||||
- Module name (`getName()`): `ejournal-db-monitor`
|
||||
- Entry class: `com.hiveops.ejournal.EJournalDbMonitorModule`
|
||||
- Maven artifact / JAR: `hiveops-module-ejournal-db-monitor.jar` (module version `1.0.0`)
|
||||
- Log package / thread name: `com.hiveops.ejournal` / `ejournal-db-monitor`
|
||||
- Dependencies: none (`getDependencies()` empty)
|
||||
- Purpose: poll Hyosung EJournal DB file size, alert before the **2 GB** write limit
|
||||
|
||||
## Config property keys (exact) — defaults from code
|
||||
|
||||
| Key | Default |
|
||||
|-----|---------|
|
||||
| `ejournal.db.monitor.enabled` | `true` |
|
||||
| `ejournal.db.monitor.dir` | `D:\EJOURNAL` |
|
||||
| `ejournal.db.monitor.filename` | *(blank → auto-pick largest DB file)* |
|
||||
| `ejournal.db.monitor.warn.mb` | `1536` |
|
||||
| `ejournal.db.monitor.critical.mb` | `2048` |
|
||||
| `ejournal.db.monitor.interval.min` | `15` |
|
||||
| `ejournal.db.monitor.cooldown.hours` | `24` |
|
||||
|
||||
Endpoint resolution (not module-specific keys, but required): uses `agent.endpoint`
|
||||
if set, else `incident.endpoint`. Neither set → module is a silent no-op.
|
||||
|
||||
## Event types emitted (exact strings)
|
||||
|
||||
- `EJOURNAL_DB_WARNING` — size ≥ `warn.mb` (default 1.5 GB)
|
||||
- `EJOURNAL_DB_CRITICAL` — size ≥ `critical.mb` (default 2.0 GB)
|
||||
|
||||
Both defined in `com.hiveops.events.EventType`. Sent via
|
||||
`IncidentEventClient.sendEvent()` → `POST {endpoint}/api/journal-events`.
|
||||
|
||||
## Event payload fields
|
||||
|
||||
`CreateJournalEventRequest` with: `agentAtmId` = ATM name, `eventType` = one of the
|
||||
two above (`.name()`), `eventDetails` = formatted string (filename, MB, % of 2 GB),
|
||||
`eventSource` = `HIVEOPS_AGENT_EJDB` (constant — use to filter these events).
|
||||
|
||||
## Auto-detect file rule
|
||||
|
||||
If `ejournal.db.monitor.filename` is blank, picks the **largest** file in
|
||||
`ejournal.db.monitor.dir` matching extensions: `.mdb`, `.sdf`, `.db`, `.sqlite`.
|
||||
|
||||
## Alert de-dup logic
|
||||
|
||||
- Fire only if level escalated (Warning→Critical) OR cooldown elapsed.
|
||||
- Size dropping below warning threshold resets state (treated as DB cleared).
|
||||
- State is in-memory only → resets on agent restart.
|
||||
|
||||
## Deploy / disable
|
||||
|
||||
- Deploy: fleet `INSTALL_MODULE` task (artifact `task.paths` or legacy
|
||||
`configPatch.url`+`filename`, must end `.jar`) → lands in agent `modules/` →
|
||||
agent restarts. Handler: `ProcessInstallModule`.
|
||||
- Disable: `ejournal.db.monitor.enabled=false` OR add `ejournal-db-monitor` to
|
||||
`modules.disabled` (push via `CONFIG_UPDATE`).
|
||||
|
||||
## Do NOT
|
||||
|
||||
- Do NOT assume events go to a dedicated endpoint — they use the shared
|
||||
`/api/journal-events`, same as the `journal-events` module.
|
||||
- Do NOT expect a numeric `atmId` from the module — it sends `agentAtmId` (name);
|
||||
the server resolves the ATM.
|
||||
- Do NOT expect persisted alert state across restarts — it is in-memory only.
|
||||
- Do NOT expect Linux path defaults — default dir is `D:\EJOURNAL` (Hyosung/Windows);
|
||||
set `ejournal.db.monitor.dir` for other layouts.
|
||||
- Do NOT expect alert spam — one alert per level per cooldown (default 24 h) unless
|
||||
escalating.
|
||||
- Do NOT rely on it detecting DBs with extensions outside
|
||||
`.mdb/.sdf/.db/.sqlite` unless you pin `ejournal.db.monitor.filename`.
|
||||
- Do NOT count on any `PlatformService` use inside the module — it uses plain
|
||||
`java.io.File` only.
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
module: agent.ejournal-db-monitor
|
||||
title: EJournal DB Monitor — guards the Hyosung 2 GB journal write limit
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
# EJournal DB Monitor (Internal / Ops view)
|
||||
|
||||
## What it monitors and why
|
||||
|
||||
Hyosung MoniPlus2S ATMs store their electronic journal in a local database file
|
||||
(Access `.mdb`, SQL Compact `.sdf`, or a plain `.db`/`.sqlite`). That engine has a
|
||||
hard **2 GB** file-size limit — once the file reaches it, the ATM application
|
||||
**stops writing journal data** and journal capture silently fails. There is no
|
||||
customer-visible alarm on the machine itself.
|
||||
|
||||
The `ejournal-db-monitor` agent module watches that database file size on a timer
|
||||
and raises HiveIQ incident events **before** the ATM hits the wall, so helpdesk can
|
||||
schedule the journal-clear/maintenance while the ATM is still capturing.
|
||||
|
||||
Two thresholds (both grounded in `EJournalDbMonitorModule`):
|
||||
|
||||
| Level | Default trigger | Event emitted | Meaning |
|
||||
|----------|-----------------|-------------------------|---------|
|
||||
| Warning | 1.5 GB (1536 MB)| `EJOURNAL_DB_WARNING` | Early notice — plan maintenance |
|
||||
| Critical | 2.0 GB (2048 MB)| `EJOURNAL_DB_CRITICAL` | At/near the limit — journal writing may stop, act immediately |
|
||||
|
||||
The event `eventDetails` string reports the file name, current size in MB, and the
|
||||
percentage of the 2 GB limit consumed. Events are tagged with `eventSource =
|
||||
HIVEOPS_AGENT_EJDB` so they can be told apart from normal journal-parsed events.
|
||||
|
||||
### Alert de-duplication
|
||||
|
||||
Once an alert level fires it is **not** repeated until one of:
|
||||
|
||||
- The level **escalates** (Warning → Critical), or
|
||||
- The **cooldown** window elapses (default 24 h), or
|
||||
- The file drops back **below** the warning threshold (i.e. the DB was cleared) —
|
||||
this resets the alert state so the next growth cycle alerts fresh.
|
||||
|
||||
So a stuck-full ATM produces at most one Critical event per cooldown window, not a
|
||||
flood.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
All read from the agent's main properties (`hiveops.properties`). Keys and defaults
|
||||
come straight from `EJournalDbMonitorModule.initialize()`:
|
||||
|
||||
| Property | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `ejournal.db.monitor.enabled` | `true` | Master on/off. If `false`, module no-ops (never becomes ready). |
|
||||
| `ejournal.db.monitor.dir` | `D:\EJOURNAL` | Directory to scan. |
|
||||
| `ejournal.db.monitor.filename` | *(blank)* | Specific file to watch. If blank, auto-detects the **largest** file matching `.mdb` / `.sdf` / `.db` / `.sqlite`. |
|
||||
| `ejournal.db.monitor.warn.mb` | `1536` | Warning threshold (MB). |
|
||||
| `ejournal.db.monitor.critical.mb` | `2048` | Critical threshold (MB). |
|
||||
| `ejournal.db.monitor.interval.min` | `15` | Check interval (minutes). First check runs immediately at start. |
|
||||
| `ejournal.db.monitor.cooldown.hours` | `24` | Hours before re-alerting at the same level. |
|
||||
|
||||
The module also needs an event endpoint — it uses `agent.endpoint` if present
|
||||
(preferred, agent-proxy), else falls back to `incident.endpoint`. HTTP settings are
|
||||
loaded with the matching prefix (`agent.*` or `incident.*`). If **neither** endpoint
|
||||
is set the module initializes to a no-op and never sends events.
|
||||
|
||||
## How it's enabled / deployed to ATMs
|
||||
|
||||
This is a **standalone drop-in module** (`pom.xml` `finalName =
|
||||
hiveops-module-ejournal-db-monitor`, artifact `hiveops-module-ejournal-db-monitor.jar`,
|
||||
module version `1.0.0`). It is not bundled in the fat JAR by default; it is discovered
|
||||
at runtime via `ServiceLoader` from the agent's `modules/` directory.
|
||||
|
||||
Two ways to get it onto an ATM:
|
||||
|
||||
1. **Fleet `INSTALL_MODULE` task** (preferred) — delivers the module JAR to the
|
||||
ATM's `modules/` directory, then the agent restarts to pick it up. Supports both
|
||||
artifact-store delivery (chunked download, `task.paths`) and legacy URL delivery
|
||||
(`configPatch.url` + `filename`, must end in `.jar`). See
|
||||
`ProcessInstallModule` in `hiveops-file-collection`.
|
||||
2. **Manual drop-in** — copy `hiveops-module-ejournal-db-monitor.jar` into the
|
||||
agent install's `modules/` folder and restart the agent.
|
||||
|
||||
To turn it off without removing the JAR: either set
|
||||
`ejournal.db.monitor.enabled=false`, or add `ejournal-db-monitor` to the
|
||||
`modules.disabled` property in `hiveops.properties`. Both are pushable via a
|
||||
`CONFIG_UPDATE` fleet task.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **No events ever arrive** — check that `agent.endpoint` (or `incident.endpoint`)
|
||||
is set; with neither, the module silently becomes a no-op. Also confirm the module
|
||||
is present in `modules/` and not listed in `modules.disabled`.
|
||||
- **"No EJournal database file found"** (debug log) — `ejournal.db.monitor.dir`
|
||||
wrong for this box, or the DB uses an extension outside `.mdb/.sdf/.db/.sqlite`.
|
||||
Set `ejournal.db.monitor.filename` explicitly.
|
||||
- **Wrong file watched** — with no `filename` set, the module picks the *largest*
|
||||
matching file in the directory. If backups or unrelated DBs share the dir, pin the
|
||||
exact filename.
|
||||
- **Expected an alert but none came** — the level may be inside its cooldown (24 h
|
||||
default) and not yet escalated. Escalation Warning→Critical always fires
|
||||
immediately regardless of cooldown.
|
||||
- **Alerts stopped after a clear** — expected: dropping below the warning threshold
|
||||
resets alert state; growth back up re-alerts.
|
||||
- **Send failures** — `Failed to send EJournal DB ... event` in the log means the
|
||||
size check ran but the HTTP POST to `/api/journal-events` failed; check agent-proxy
|
||||
reachability. The check will retry on the next interval.
|
||||
|
||||
Log package for this module is `com.hiveops.ejournal` (thread name
|
||||
`ejournal-db-monitor`).
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
module: agent.hw-inventory
|
||||
title: Hardware Inventory — ATM hardware/OS inventory sync + CDN bandwidth probe
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · architecture view.** Confirmed against `hiveops-module-hw-inventory` source 2026-07-01. See [technical.agent] for the module system as a whole.
|
||||
|
||||
## Shape
|
||||
|
||||
A tiny 4-type module: one `AgentModule` entry point, a collector SPI interface, and two platform collectors.
|
||||
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `HardwareInventoryModule` | `AgentModule` entry point — lifecycle, scheduling, CDN probe, POST orchestration |
|
||||
| `HardwareCollector` (interface) | `collect(atmName, country, institutionKey) → HardwareInventoryRequest` |
|
||||
| `LinuxHardwareCollector` | Reads `/proc`, `/sys`, `/etc/os-release`, `lsblk`, Java `File.listRoots()` |
|
||||
| `WindowsHardwareCollector` | Shells out to **PowerShell** (`Get-CimInstance`, `Get-HotFix`, registry reads) |
|
||||
|
||||
The transport (`IncidentEventClient.syncHardware`) and the payload DTO (`HardwareInventoryRequest`) live in the shared `hiveops-agent-journal` module (`com.hiveops.events`), reused here as a `provided` dependency.
|
||||
|
||||
## Lifecycle & flow
|
||||
|
||||
Standard SPI lifecycle (`initialize → isEnabled → start → stop`), driven by `ModuleRegistry`:
|
||||
|
||||
1. **`initialize(ModuleContext)`**
|
||||
- Reads `hw.inventory.enabled`; if `false`, returns early (module never activates).
|
||||
- Resolves the base URL: prefers `agent.endpoint`, falls back to `incident.endpoint`. If neither is set → inactive.
|
||||
- Loads HTTP settings via `HttpClientHelper.loadSettingsFromProperties(props, prefix)` where `prefix` = `"agent"` or `"incident"` to match the chosen endpoint.
|
||||
- Pulls identity from context: `getAtmName()`, `getCountry()`, and `incident.institution.key` from props.
|
||||
- Picks the collector from `context.getPlatformService().isWindows()` → `WindowsHardwareCollector` or `LinuxHardwareCollector`.
|
||||
- Sets `ready = true` and **calls `syncInventory()` once immediately** (a first sync happens during init, before `start()`).
|
||||
2. **`isEnabled(context)`** → returns the `ready` flag.
|
||||
3. **`start()`** — creates a single daemon `ScheduledExecutorService` (thread name `hw-inventory`) and `scheduleAtFixedRate(this::syncInventory, 0, intervalHours, HOURS)`. Because initial delay is `0`, this fires an **additional** sync right at startup (so a fresh agent syncs twice close together — once from `initialize`, once from the delay-0 schedule).
|
||||
4. **`stop()`** — `scheduler.shutdownNow()`.
|
||||
|
||||
Each `syncInventory()`:
|
||||
|
||||
```
|
||||
collector.collect(atmName, country, institutionKey) // build HardwareInventoryRequest
|
||||
→ req.setCdnDownloadSpeedBps(probeCdnDownload()) // measure CDN bandwidth
|
||||
→ client.syncHardware(req) // POST /api/atms/hardware/sync
|
||||
```
|
||||
|
||||
The whole method is wrapped in try/catch; a failure logs `sync failed` and is swallowed so the schedule keeps running. Inside each collector, every sub-step (CPU, system, disks, drives, NICs, BIOS/boot, patches, …) is independently try/caught — a partial snapshot is still sent.
|
||||
|
||||
## How data reaches the platform
|
||||
|
||||
`IncidentEventClient.syncHardware(req)` does a plain `HttpURLConnection` **POST** of the JSON `HardwareInventoryRequest` to:
|
||||
|
||||
```
|
||||
{endpoint}/api/atms/hardware/sync
|
||||
```
|
||||
|
||||
with `Content-Type: application/json` and the shared HTTP settings (auth header, host header) applied by `HttpClientHelper`. `{endpoint}` is the agent-proxy base (`agent.endpoint`) on current ATMs, so the request lands on **agent-proxy** and is forwarded to the incident/devices hardware-sync handler — the same agent→agent-proxy path every other agent call uses. Accepts `200/201/202`; anything else throws `IOException`.
|
||||
|
||||
The ATM is identified by `atmName` + `country` in the payload (plus `institutionKey`); there is no client-side `atmId` resolution here — the platform maps it server-side.
|
||||
|
||||
## CDN bandwidth probe (`probeCdnDownload()`)
|
||||
|
||||
Runs inside the same sync, purely to populate `cdnDownloadSpeedBps`:
|
||||
|
||||
1. GET the manifest at `agent.update.check.url` (default `https://cdn.bcos.cloud/downloads/agent/latest.json`), sending `X-Institution-Key` if an institution key is set. Parsed leniently (`FAIL_ON_UNKNOWN_PROPERTIES=false`).
|
||||
2. Resolve `patches.windows` or `patches.linux` from the manifest (chosen by `isWindows`); relative filenames are resolved against the manifest base URL.
|
||||
3. HTTP **Range** request `bytes=0-524287` (first 512 KB) of that patch file; accepts `206 Partial` or `200`.
|
||||
4. Read the bytes, time the transfer, compute `rawBps = bytes*1000/elapsedMs`, then `speedBps = rawBps * task.hotfix.network.cap` (cap clamped `0.01`–`1.0`, default `0.9`).
|
||||
5. Any failure (bad HTTP, no patch URL, exception) returns `0`.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
The module uses `PlatformService` **only** for OS detection (`isWindows()`) to select the collector. The collectors themselves do **not** go through the platform abstraction — they read OS sources directly:
|
||||
|
||||
- **Linux:** `/proc/cpuinfo`, `/sys/devices/.../cpufreq`, `/sys/class/dmi/id/*`, `lsblk -b -o … -n -d`, `/sys/block/*/queue/rotational` (SSD/HDD), `/sys/class/net/*` (physical-NIC filter via a `device` symlink), `/proc/net/dev` (throughput), `/proc/stat` `btime` (boot time), `/etc/os-release`, `File.listRoots()` (drives).
|
||||
- **Windows:** PowerShell (`-NoProfile -NonInteractive -ExecutionPolicy Bypass`) with `Get-CimInstance` (`Win32_Processor`, `Win32_ComputerSystem`, `Win32_DiskDrive`, `Win32_LogicalDisk`, `Win32_NetworkAdapter`, `Win32_BIOS`, `Win32_OperatingSystem`), `Get-NetAdapterStatistics` (throughput), `Get-HotFix` (patches), registry `Uninstall` keys (programs), and `USBSTOR`/`Control Panel\Desktop` registry reads (security posture). stderr is drained on a background thread so it can't block stdout.
|
||||
|
||||
> Note: the pom `<description>` says Windows uses `wmic` — that's stale; the code uses PowerShell/CIM.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
module: agent.hw-inventory
|
||||
title: Hardware Inventory — ATM hardware/OS inventory sync + CDN bandwidth probe
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-hw-inventory` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Module name (`AgentModule.getName()`): **`hw-inventory`**
|
||||
- Maven artifact: **`hiveops-module-hw-inventory`** (own version `1.0.0`; parent `hiveops-agent-parent` currently `4.4.3`)
|
||||
- Entry class: `com.hiveops.hwinventory.HardwareInventoryModule` (SPI file: `META-INF/services/com.hiveops.core.module.AgentModule`)
|
||||
- Other classes: `HardwareCollector` (interface), `LinuxHardwareCollector`, `WindowsHardwareCollector`
|
||||
- Log package (`getLogPackage()`): `com.hiveops.hwinventory` (→ per-module log, convention `logs/modules/hw-inventory.log` — path unverified)
|
||||
- Dependencies (`getDependencies()`): **none** (empty list)
|
||||
- Drop-in JAR: `modules/hiveops-module-hw-inventory.jar`; in **both** build-dist module sets (`atm` default **and** `srv`)
|
||||
|
||||
## Config properties
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `hw.inventory.enabled` | `true` | `false` → module inactive |
|
||||
| `hw.inventory.sync.interval.hours` | `24` | fixed-rate resync period (hours) |
|
||||
|
||||
Reused (not owned):
|
||||
- `agent.endpoint` (preferred) **or** `incident.endpoint` (fallback) — HTTP base URL; **neither set → inactive**
|
||||
- `incident.institution.key` — institution key; sent in payload + as `X-Institution-Key` on the CDN probe
|
||||
- `agent.update.check.url` — CDN manifest for probe; default `https://cdn.bcos.cloud/downloads/agent/latest.json`
|
||||
- `task.hotfix.network.cap` — probe speed multiplier; default `0.9`, clamped `0.01`–`1.0`
|
||||
- `modules.disabled` — comma list; include `hw-inventory` to disable
|
||||
- auth token / host / timeouts via `HttpClientHelper.loadSettingsFromProperties(props, "agent"|"incident")`
|
||||
|
||||
## Events / status strings emitted
|
||||
- **None.** No `EventType`, no `IAT_*`, no journal/incident events, no fleet task status. This module does **not** call the journal-event path.
|
||||
- Sole output = one JSON POST of `HardwareInventoryRequest`.
|
||||
|
||||
## HTTP endpoint used (agent → agent-proxy → incident)
|
||||
| Call | Method + Path |
|
||||
|---|---|
|
||||
| Inventory sync | `POST {endpoint}/api/atms/hardware/sync` (`IncidentEventClient.syncHardware`) |
|
||||
| CDN probe (manifest) | `GET {agent.update.check.url}` |
|
||||
| CDN probe (speed test) | `GET <patch url>` with header `Range: bytes=0-524287` |
|
||||
|
||||
Accepts `200/201/202` on sync; else throws `IOException`.
|
||||
|
||||
## Payload fields (`HardwareInventoryRequest`)
|
||||
- Identity: `atmName`, `country`, `institutionKey`
|
||||
- Both OS: `cpu{name,cores,maxClockMhz}`, `system{manufacturer,model,type}`, `disks[]{model,serialNumber,sizeBytes,mediaType}`, `drives[]{letter,fileSystem,totalBytes,freeBytes}`, `nics[]{name,macAddress,speedBps,rxBytesPerSec,txBytesPerSec}`, `biosSerial`, `lastBootTime`, `osVersion`, `cdnDownloadSpeedBps`
|
||||
- Windows-only: `osInstallDate`, `installedPatches[]`, `installedPatchDates{}`, `installedPrograms[]{name,version,installedDate}`, `usbLockStatus`, `windowsLockStatus`
|
||||
|
||||
## Gotchas
|
||||
- **Syncs twice at startup:** once in `initialize()`, then again from `start()`'s `scheduleAtFixedRate(..., 0, ...)` (initial delay 0).
|
||||
- **~5s added per sync:** NIC RX/TX throughput is measured by sampling twice, 5s apart (`Thread.sleep(5000)`).
|
||||
- **Partial payloads are normal:** every collector step is independently try/caught; failures are logged and the rest is still sent.
|
||||
- **CDN probe never blocks the sync:** on any failure it returns `0` for `cdnDownloadSpeedBps`; inventory still POSTs.
|
||||
- **Windows path = PowerShell/CIM**, not `wmic` (pom `<description>` is stale on this).
|
||||
- **Server (`srv`) devices also run this module** — it's not ATM-only.
|
||||
- Linux physical-NIC filter: skips `lo`, virtual ifaces (no `/sys/class/net/<n>/device` symlink), and MACs starting `00:00:00`.
|
||||
- No client-side `atmId` resolution — platform maps `atmName`+`country` server-side.
|
||||
|
||||
## Do NOT
|
||||
- Do **not** expect `IAT_*`/journal events from this module — it emits none.
|
||||
- Do **not** look for a poll loop / fleet-task handling — it doesn't poll; it's push-only on a timer.
|
||||
- Do **not** add a `hw-inventory.*`-namespaced key expecting it to work; only `hw.inventory.enabled` and `hw.inventory.sync.interval.hours` exist.
|
||||
- Do **not** assume the collectors go through `PlatformService` — only `isWindows()` does; collection reads OS sources directly.
|
||||
- Do **not** point this at incident directly on modern ATMs — route via `agent.endpoint` (agent-proxy).
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
module: agent.hw-inventory
|
||||
title: Hardware Inventory — ATM hardware/OS inventory sync + CDN bandwidth probe
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-hw-inventory` source 2026-07-01.
|
||||
|
||||
## What it does (and doesn't)
|
||||
|
||||
`hiveops-module-hw-inventory` (module name `hw-inventory`, entry class `HardwareInventoryModule`) is a **periodic push module**, not an event/incident monitor. Once per run it takes a full snapshot of the ATM's hardware and OS and **POSTs it to the platform** at `/api/atms/hardware/sync` (via `IncidentEventClient.syncHardware`). It does **not** tail journals, does **not** emit `IAT_*`/journal-incident events, and does **not** poll for fleet tasks. It is fire-and-forget inventory reporting.
|
||||
|
||||
It runs the collection **once on startup** and then **re-syncs on a fixed interval** (default every 24h).
|
||||
|
||||
What each sync collects:
|
||||
|
||||
| Area | Both platforms | Windows-only extras |
|
||||
|---|---|---|
|
||||
| CPU | model name, cores, max clock (MHz) | — |
|
||||
| System board | manufacturer, model, type | — |
|
||||
| Physical disks | model, serial, size, media type (SSD/HDD) | — |
|
||||
| Logical drives | mount/letter, filesystem, total/free bytes | — |
|
||||
| NICs | name, MAC, link speed, **measured RX/TX throughput** (sampled over 5s) | — |
|
||||
| BIOS / boot | BIOS/product serial, last boot time | OS install date |
|
||||
| OS | OS version string | installed patches (KB IDs + dates), installed programs, USB-lock status, Windows screen-lock status |
|
||||
| CDN | effective CDN download speed (bytes/s) — see below | — |
|
||||
|
||||
**CDN bandwidth probe.** On every sync the module also measures how fast this ATM can pull from the CDN: it fetches the agent manifest (`latest.json`), resolves the platform patch file URL, issues a **512 KB HTTP Range request**, times it, multiplies the raw rate by a configurable cap, and ships the result as `cdnDownloadSpeedBps` on the same payload. The platform uses this so hotfix/software delivery can be paced per-device to the ATM's real link speed. A failed probe just sends `0` — the inventory sync still succeeds.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `hw.inventory.enabled` | `true` | Master on/off. `false` → module logs "disabled" and stays inactive. |
|
||||
| `hw.inventory.sync.interval.hours` | `24` | Hours between re-syncs (also the fixed-rate schedule period). |
|
||||
|
||||
Properties it **reuses** (does not own):
|
||||
|
||||
| Key | Used for |
|
||||
|---|---|
|
||||
| `agent.endpoint` | Preferred HTTP base URL (agent-proxy). If set, module uses the `agent` client prefix. |
|
||||
| `incident.endpoint` | Fallback base URL if `agent.endpoint` is absent (older ATMs). If neither is set, the module goes inactive. |
|
||||
| `incident.institution.key` | Institution key; sent as `X-Institution-Key` header on the CDN probe and included in the inventory payload. |
|
||||
| `agent.update.check.url` | CDN manifest URL for the probe. Default `https://cdn.bcos.cloud/downloads/agent/latest.json`. |
|
||||
| `task.hotfix.network.cap` | Multiplier applied to the measured raw CDN speed (default `0.9`, clamped to `0.01`–`1.0`). Same knob the hotfix delivery path uses. |
|
||||
| `agent.*` / `incident.*` HTTP settings | Auth token, host header, timeouts — loaded via `HttpClientHelper.loadSettingsFromProperties(props, prefix)` under whichever prefix (`agent` or `incident`) is active. |
|
||||
|
||||
## Enabled / deployed to ATMs
|
||||
|
||||
- **Enable condition:** on by default. It self-activates whenever `hw.inventory.enabled` is not `false` **and** an `agent.endpoint` or `incident.endpoint` is configured. To disable on a specific ATM/institution, push `hw.inventory.enabled=false` (or add `hw-inventory` to `modules.disabled`) via a `CONFIG_UPDATE` fleet task.
|
||||
- **Shipped how:** it is a drop-in JAR (`modules/hiveops-module-hw-inventory.jar`), discovered via ServiceLoader — **not** baked into the fat JAR. `deployment/build-dist.sh` includes it in **both** module sets: the default `atm` set and the `srv` (server-monitoring) set. So every standard install/patch ZIP carries it.
|
||||
- **Retrofitting an ATM that doesn't have it yet:** deliver via a fleet `INSTALL_MODULE` task (drops the JAR into `modules/` and restarts the agent), or via a full agent update that includes the current module set.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---|---|---|
|
||||
| No hardware ever appears for an ATM | Module inactive — no endpoint configured, or `hw.inventory.enabled=false`, or JAR missing | Confirm `agent.endpoint`/`incident.endpoint` is set; grep the module log for `hw-inventory initialized`; confirm `modules/hiveops-module-hw-inventory.jar` is present |
|
||||
| Sync logs `Failed to sync hardware inventory: HTTP <code>` | agent-proxy/incident rejected the POST (auth, routing, or `/api/atms/hardware/sync` not reachable) | Verify the endpoint resolves through agent-proxy and the agent's auth token/institution key are valid |
|
||||
| Windows fields (patches, programs, lock status) are empty but Linux-style fields present | Wrong collector, or PowerShell failing | These come from PowerShell (`Get-CimInstance`, `Get-HotFix`, registry reads); check the module log for `PowerShell exited <n>` warnings |
|
||||
| CPU/disk/NIC fields blank | A single collector step failed but sync still ran | Each collector step is independently try/caught and logs `failed to collect <x>`; a partial payload is still sent |
|
||||
| `cdnDownloadSpeedBps` is 0 | CDN probe failed or manifest had no patch URL | Non-fatal; check log for `CDN probe returned HTTP <n>` / `CDN probe failed`; verify the ATM can reach `cdn.bcos.cloud` and the manifest has `patches.windows`/`patches.linux` |
|
||||
| Inventory looks ~24h stale | That's the default resync cadence | Lower `hw.inventory.sync.interval.hours` via `CONFIG_UPDATE`, or restart the agent to force an immediate sync |
|
||||
| Each NIC sync adds ~5s of latency | Expected — throughput is measured by sampling `/proc/net/dev` (Linux) / `Get-NetAdapterStatistics` (Windows) twice, 5s apart | Not a bug; it's the RX/TX measurement window |
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
module: agent.log-collect
|
||||
title: Log Collect Path — on-demand file/log pull from an arbitrary ATM path
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · how it's built.** Confirmed against `hiveops-module-log-collect` source (`com.hiveops.logcollect`) 2026-07-01. See [technical.agent] for agent-wide context (module system, HTTP layer, platform abstraction).
|
||||
|
||||
## Maven module
|
||||
|
||||
`hiveops-module-log-collect` (artifact `hiveops-module-log-collect`, own version `1.0.0`, parented by `hiveops-agent-parent` — currently `4.4.3`). Packaged as a standalone jar (`<finalName>${project.artifactId}</finalName>` → `hiveops-module-log-collect.jar`).
|
||||
|
||||
Dependencies (all `provided` scope — supplied by the agent runtime, not shaded in):
|
||||
|
||||
- `hiveops-core` — module SPI + HTTP layer (`FileUploadClient`, `HttpClientHelper`, `HttpClientSettings`, `CurrentAtmPendingTask`).
|
||||
- `hiveops-agent-journal` (`1.0.0`) — for `com.hiveops.logs.AgentLogUploader`, which does the actual ZIP-and-upload.
|
||||
- `log4j-api`, `commons-lang3`.
|
||||
|
||||
Registered via ServiceLoader SPI: `src/main/resources/META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.logcollect.LogCollectPathModule`.
|
||||
|
||||
## The one class
|
||||
|
||||
There is a single class, `LogCollectPathModule`, implementing `AgentModule`. No sub-packages, no monitors, no emitters — the heavy lifting lives in `AgentLogUploader` (from `hiveops-agent-journal`).
|
||||
|
||||
### Lifecycle (`AgentModule` contract)
|
||||
|
||||
- `getName()` → **`log-collect-path`**
|
||||
- `getVersion()` → `1.0.0`
|
||||
- `getLogPackage()` → `com.hiveops.logcollect`
|
||||
- `getDependencies()` → empty (no ordering constraints)
|
||||
- `isEnabled(ctx)` → true when `agent.endpoint` **or** `incident.endpoint` is non-blank in the main properties.
|
||||
- `initialize(ctx)` → picks the settings prefix (`agent` if `agent.endpoint` is set, else `incident`), builds an `HttpClientSettings` via `HttpClientHelper.loadSettingsFromProperties(props, prefix)`, and caches `country` + `atmName` from the `ModuleContext`.
|
||||
- `start()` → creates a single-thread daemon `ScheduledExecutorService` (thread name `log-collect-path`) and schedules `poll()` at a fixed `POLL_INTERVAL_SECONDS = 30` rate.
|
||||
- `stop()` → `scheduler.shutdownNow()`.
|
||||
|
||||
## Poll → detect → collect → emit flow
|
||||
|
||||
```
|
||||
every 30s poll()
|
||||
└─ FileUploadClient.queryCurrentAtmPendingTask(country, atmName, "LOG_COLLECT_PATH")
|
||||
GET {endpoint}/api/agent/{country}/{atm}/task?kind=LOG_COLLECT_PATH
|
||||
└─ null / blank taskKind → return (nothing pending; HTTP 204 → null)
|
||||
└─ task present → handle(client, task)
|
||||
├─ targetPath = task.configPatch.get("path")
|
||||
│ └─ blank → statusReport(FAILED, "No path specified…")
|
||||
├─ statusReport(tid, aid, "RUNNING", "Collecting logs from: <path>")
|
||||
├─ AgentLogUploader.uploadPath(targetPath, settings, country, atmName, tid)
|
||||
│ POST {endpoint}/api/agent/{country}/{atm}/log/upload?taskId={tid}
|
||||
└─ statusReport(tid, aid, "COMPLETED", "…uploaded successfully")
|
||||
└─ on exception → statusReport(FAILED, e.getMessage())
|
||||
```
|
||||
|
||||
### `AgentLogUploader.uploadPath(...)` (in `hiveops-agent-journal`)
|
||||
|
||||
1. Normalize `\`→`/` and resolve the path. Missing path → `IOException("Path does not exist…")`.
|
||||
2. If it's a regular file, collect just that file; otherwise `Files.walkFileTree` collecting **every** file. `visitFileFailed` logs and skips unreadable files (`CONTINUE`).
|
||||
3. Empty result → `IOException("No files found at path…")`.
|
||||
4. Build a temp ZIP (`hiveops-path-logs-*.zip`) with entries relative to the target root (single-file case roots at the parent).
|
||||
5. Multipart-POST the ZIP to `…/log/upload?taskId={tid}` — boundary `----HiveOpsLogUploadBoundary`, form part name `file`, filename `hiveops-logs.zip`, `Content-Type: application/zip`. Auth headers/host applied via `HttpClientHelper.applyParameters` / `applyParameterHost`.
|
||||
6. Non-2xx logs the error body; the temp ZIP is always deleted in `finally`.
|
||||
|
||||
The `taskId` query param is what lets the backend attach the upload to a trackable `AtmLogCollect` record for that task.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
This module reaches the platform over the **agent HTTP task API**, not the journal-event pipeline. Three calls, all against `settings.getEndpoint()` (the `agent.endpoint`/`incident.endpoint` base — served by **agent-proxy** in production, which proxies to incident):
|
||||
|
||||
| Purpose | Method + path |
|
||||
|---------|---------------|
|
||||
| Poll for work | `GET /api/agent/{country}/{atm}/task?kind=LOG_COLLECT_PATH` |
|
||||
| Status updates | `POST /api/agent/tasks/{tid}/status` body `{ status, message }` |
|
||||
| Deliver the bundle | `POST /api/agent/{country}/{atm}/log/upload?taskId={tid}` (multipart ZIP) |
|
||||
|
||||
Errors in `poll()` are classified: `IOException`/`SocketException` → warn ("transient network error"); `com.hiveops.http.Utils.MyException` (backend error) → warn ("server error"); anything else → error with stack trace. The poll loop survives all of them and retries on the next tick.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
The module itself uses **no** `PlatformService` — it operates on whatever path string the task supplies and relies on `AgentLogUploader`'s `\`→`/` normalization to accept Windows paths. (By contrast the `INSTALL_MODULE` handler that *delivers* this jar does use `PlatformService` for `restartService` / scripts path.)
|
||||
|
||||
## `CurrentAtmPendingTask` fields used
|
||||
|
||||
`tid`, `aid`, `taskKind`, and `configPatch` (a `Map<String,String>`; only key `path` is read). Other fields on the DTO (`kb`, `paths`, `artifactCdnUrl`, `cdnToken`) are ignored by this module.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
module: agent.log-collect
|
||||
title: Log Collect Path — on-demand file/log pull from an arbitrary ATM path
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-log-collect` source 2026-07-01. Every key/string below is from code; nothing invented.
|
||||
|
||||
## Identity
|
||||
- Module name (`AgentModule.getName()`): **`log-collect-path`**
|
||||
- Maven artifact / jar: **`hiveops-module-log-collect`** → `hiveops-module-log-collect.jar` (own version `1.0.0`; parent `hiveops-agent-parent` `4.4.3`)
|
||||
- Entry class: `com.hiveops.logcollect.LogCollectPathModule` (SPI: `META-INF/services/com.hiveops.core.module.AgentModule`)
|
||||
- Uploader: `com.hiveops.logs.AgentLogUploader.uploadPath(...)` (from `hiveops-agent-journal`)
|
||||
- Log package (`getLogPackage()`): `com.hiveops.logcollect`
|
||||
- Fleet task kind it implements: **`LOG_COLLECT_PATH`**
|
||||
- Dependencies: none (`getDependencies()` empty)
|
||||
|
||||
## Enable / disable
|
||||
- Enabled when **`agent.endpoint`** OR **`incident.endpoint`** is non-blank. Both blank → `isEnabled` false, module never starts.
|
||||
- Prefix selection: `agent.endpoint` set → prefix `agent`; else prefix `incident`.
|
||||
- Disable explicitly: add `log-collect-path` to `modules.disabled` in `hiveops.properties`.
|
||||
|
||||
## Config property keys (read, not registered)
|
||||
No `log-collect.*` keys exist. It reuses HTTP settings via `HttpClientHelper.loadSettingsFromProperties(props, prefix)`:
|
||||
|
||||
| Key | Notes |
|
||||
|-----|-------|
|
||||
| `agent.endpoint` / `incident.endpoint` | base URL; also the enable switch |
|
||||
| `{prefix}.host` | host header |
|
||||
| `{prefix}.sim` | routing hint |
|
||||
| `{prefix}.to.connect` | default `10000` (ms) |
|
||||
| `{prefix}.to.read` | default `10000` (ms) |
|
||||
| `{prefix}.institution.key` | fallback → `agent.institution.key` → `incident.institution.key` |
|
||||
| `agent.auth.token` | bearer, always this key regardless of prefix |
|
||||
|
||||
- Poll interval = **30s**, hardcoded constant `POLL_INTERVAL_SECONDS`. NOT configurable.
|
||||
|
||||
## Task-status strings it emits (`POST /api/agent/tasks/{tid}/status`)
|
||||
- `RUNNING` — "Collecting logs from: <path>"
|
||||
- `COMPLETED` — "Log collection uploaded successfully"
|
||||
- `FAILED` — "No path specified in task configPatch" | `<exception message>`
|
||||
|
||||
It emits **no** journal/incident events, no `IAT_*` events. Status reports only.
|
||||
|
||||
## HTTP endpoints (base = `settings.getEndpoint()`)
|
||||
| Call | Verb + path |
|
||||
|------|-------------|
|
||||
| Poll | `GET /api/agent/{country}/{atm}/task?kind=LOG_COLLECT_PATH` |
|
||||
| Status | `POST /api/agent/tasks/{tid}/status` → `{status,message}` |
|
||||
| Upload | `POST /api/agent/{country}/{atm}/log/upload?taskId={tid}` (multipart) |
|
||||
|
||||
- Multipart: boundary `----HiveOpsLogUploadBoundary`, part name `file`, filename `hiveops-logs.zip`, `Content-Type: application/zip`.
|
||||
|
||||
## Task input contract
|
||||
- Target read from `task.configPatch.get("path")` only. Blank → `FAILED`.
|
||||
- Accepts a directory (recursive, **all** files, any extension) or a single regular file.
|
||||
- Windows paths OK: `\` is normalized to `/`.
|
||||
- Unreadable files skipped (logged), rest still uploaded. Empty/missing path → `IOException` → `FAILED`.
|
||||
|
||||
## Fleet delivery (`INSTALL_MODULE`)
|
||||
- Push jar `hiveops-module-log-collect.jar` via artifact (`task.paths[0]`) or url (`configPatch.url` + `filename`).
|
||||
- Filename MUST end `.jar`. Agent writes to `modules/`, then **restarts the agent service** to load it (`ProcessInstallModule`).
|
||||
|
||||
## Gotchas
|
||||
- Silent-ish poll failures: network/backend errors are `warn`-logged and retried next tick, not surfaced as task FAILED.
|
||||
- Whole target tree is zipped to a temp file first — large dirs = large temp file + memory/disk pressure. Temp ZIP deleted in `finally`.
|
||||
- `queryCurrentAtmPendingTask` returns `null` on HTTP 204 (nothing pending) — normal, not an error.
|
||||
- `taskId` query param is required for the backend to create the trackable `AtmLogCollect` record; don't strip it.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT expect `log-collect.*` config keys — none exist; don't add one for poll interval without code.
|
||||
- Do NOT assume it tails/watches logs continuously — it is poll-then-pull, task-driven only.
|
||||
- Do NOT confuse with the built-in `LOG_COLLECT` command (`AgentLogUploader.uploadDirectory`, agent `logs/` only, `.log`/`.log.gz`). This module is `LOG_COLLECT_PATH` = arbitrary path, all files.
|
||||
- Do NOT look for journal/incident events from this module — there are none.
|
||||
- Do NOT rely on `configPatch` keys other than `path` (e.g. `url`/`filename` belong to `INSTALL_MODULE`, not this task).
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
module: agent.log-collect
|
||||
title: Log Collect Path — on-demand file/log pull from an arbitrary ATM path
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-log-collect` source (`com.hiveops.logcollect.LogCollectPathModule`) 2026-07-01. Dev-facing; not customer material.
|
||||
|
||||
## What it does
|
||||
|
||||
The Log Collect Path module lets a fleet operator pull **any file or directory** off an ATM on demand — not just the agent's own logs. It implements the `LOG_COLLECT_PATH` fleet task kind:
|
||||
|
||||
1. The module polls the agent endpoint every **30 seconds** for a pending `LOG_COLLECT_PATH` task for this ATM.
|
||||
2. When one arrives, it reads the target path from the task's `configPatch.path`.
|
||||
3. It walks that path recursively, ZIPs every readable file, and uploads the bundle to the backend tagged with the task id so operators can download it (e.g. the **Other Logs** tab in device detail).
|
||||
4. It reports task status back (`RUNNING` → `COMPLETED`, or `FAILED`).
|
||||
|
||||
It does **not** monitor anything continuously and it emits **no journal/incident events** — it is a task-driven, request/response file collector. There is no schedule, watch, or tail; the only trigger is an operator-created fleet task.
|
||||
|
||||
## What it monitors / why
|
||||
|
||||
Nothing passively. The "monitor" is a 30-second poll for pending tasks. Use cases:
|
||||
|
||||
- Grab a vendor log directory (dispenser, EPP, XFS logs) that the standard `LOG_COLLECT` command doesn't bundle.
|
||||
- Pull a specific config/trace file from an arbitrary path during an investigation.
|
||||
- Retrieve a single file (the module handles a regular-file target as well as a directory).
|
||||
|
||||
Contrast with the sibling built-in `LOG_COLLECT` command (in `AgentLogUploader.uploadDirectory`), which only bundles the agent's own `logs/` tree (`.log` / `.log.gz`). `LOG_COLLECT_PATH` collects **all files** under the requested path regardless of extension.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
The module registers **no dedicated `log-collect.*` properties**. It reuses the agent's existing HTTP client settings. It is **enabled** whenever either endpoint property is set:
|
||||
|
||||
| Property | Purpose | Default |
|
||||
|----------|---------|---------|
|
||||
| `agent.endpoint` | Agent-proxy base URL (preferred). If set, the module uses the `agent.*` settings prefix. | — |
|
||||
| `incident.endpoint` | Legacy incident base URL. Used only if `agent.endpoint` is blank. | — |
|
||||
|
||||
Derived HTTP settings (loaded via the chosen `{prefix}` = `agent` or `incident`):
|
||||
|
||||
| Property | Purpose | Default |
|
||||
|----------|---------|---------|
|
||||
| `{prefix}.host` | Host header override | — |
|
||||
| `{prefix}.sim` | Sim/routing hint | — |
|
||||
| `{prefix}.to.connect` | Connect timeout (ms) | `10000` |
|
||||
| `{prefix}.to.read` | Read timeout (ms) | `10000` |
|
||||
| `{prefix}.institution.key` | Institution key; falls back to `agent.institution.key` then `incident.institution.key` | — |
|
||||
| `agent.auth.token` | Bearer token — always read as `agent.auth.token` regardless of prefix | — |
|
||||
|
||||
The **30-second poll interval is a hardcoded constant** (`POLL_INTERVAL_SECONDS`), not a property — it cannot be tuned via config.
|
||||
|
||||
## How it's enabled / deployed to ATMs
|
||||
|
||||
The module ships as its own jar, `hiveops-module-log-collect.jar`, discovered at agent startup via the ServiceLoader SPI (`META-INF/services/com.hiveops.core.module.AgentModule`).
|
||||
|
||||
- **Standard build:** it's a `<module>` in the agent parent build; if the jar is on the classpath / in the agent `modules/` directory, it auto-registers.
|
||||
- **Fleet delivery (`INSTALL_MODULE`):** push an `INSTALL_MODULE` task carrying `hiveops-module-log-collect.jar` (artifact or `configPatch.url`+`filename`). The agent drops it into `modules/` and **restarts the agent service** to load it (see `ProcessInstallModule`). Filename must end in `.jar`.
|
||||
- **Disable:** add `log-collect-path` to `modules.disabled` in `hiveops.properties`, or unset both `agent.endpoint` and `incident.endpoint` (then `isEnabled` returns false).
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause / check |
|
||||
|---------|----------------------|
|
||||
| Task stuck `FAILED` immediately, message "No path specified in task configPatch" | The task was created without `configPatch.path`. Recreate with a path. |
|
||||
| `FAILED` "Path does not exist" / "No files found at path" | The path is wrong for that OS, or empty. Windows paths are accepted with `\` (the module normalizes `\`→`/`). |
|
||||
| Task never picked up | Module disabled, agent offline, or neither `agent.endpoint`/`incident.endpoint` set → module didn't start. Check `logs/modules/…` for "LogCollectPathModule started". |
|
||||
| Poll warns "transient network error" / "server error" | Endpoint unreachable or backend 4xx/5xx; poll retries on the next 30s tick — not fatal. |
|
||||
| Upload returns non-2xx | Check the backend `/api/agent/{country}/{atm}/log/upload` route (served by agent-proxy) and the bearer token / institution key. |
|
||||
| Huge/locked files | Unreadable files are skipped with a warning (`visitFileFailed`); the ZIP still uploads with the rest. Be mindful of very large target dirs — the whole tree is zipped to a temp file first. |
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
module: agent.log
|
||||
title: Agent Log — self-log upload & error-to-incident monitoring
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · how it's built.** Confirmed against `hiveops-module-log` source (`com.hiveops.logs`) 2026-07-01. See [technical.agent] for agent-wide module/lifecycle context.
|
||||
|
||||
## Maven module & packaging
|
||||
|
||||
`hiveops-module-log` (artifact `hiveops-module-log`, own version `1.0.0`, parented by `hiveops-agent-parent` `4.4.3`). Built as a standalone jar with `<finalName>${project.artifactId}</finalName>` → `hiveops-module-log.jar`, a drop-in for the agent's `modules/` directory (not merged into the fat JAR).
|
||||
|
||||
Dependencies are all **`provided`** scope — supplied by the running fat JAR, not bundled:
|
||||
- `hiveops-core` — the `AgentModule` SPI + `HttpClientHelper`/`HttpClientSettings`.
|
||||
- `hiveops-agent-journal` — provides **`AgentLogUploader`** (the multipart/zip upload helper `LogUploadModule` delegates to). *Note: `hiveops-agent-journal` is in the `atm` build set but not the `srv` set, while `hiveops-module-log` is in both; whether the srv fat JAR still carries `AgentLogUploader` on the classpath is **unverified** — flag if a srv device throws `NoClassDefFoundError` on log upload.*
|
||||
- `log4j-api`, `commons-lang3`, `commons-io`.
|
||||
|
||||
Two implementations are registered via the ServiceLoader SPI file `META-INF/services/com.hiveops.core.module.AgentModule`:
|
||||
```
|
||||
com.hiveops.logs.LogUploadModule
|
||||
com.hiveops.logs.LogMonitorModule
|
||||
```
|
||||
|
||||
## Classes and flow
|
||||
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `LogUploadModule` | `AgentModule` `log-upload` — scheduler that ships the whole log file on an interval. |
|
||||
| `LogMonitorModule` | `AgentModule` `log-monitor` (depends on `log-upload`) — scheduler that tails the log for errors and reports summaries. |
|
||||
| `AgentLogUploader` | *(in `hiveops-agent-journal`)* static helper doing the actual multipart POST / ZIP bundling. Shared with the fleet log-collect commands. |
|
||||
|
||||
Neither module overrides `getLogPackage()`, so their own log output goes to the **root** logger (`hiveops-agent.log`) rather than a dedicated `logs/modules/*.log` — which is deliberate here, since that root file is exactly what they monitor and upload. Neither overrides `isHealthy()` either, so the `ModuleWatchdog` treats them as healthy by default even if their scheduler thread dies *(potential blind spot — unverified as an operational issue)*.
|
||||
|
||||
### log-upload cycle (poll/push, interval-driven)
|
||||
|
||||
1. `initialize()`: resolve endpoint (`agent.endpoint` else `incident.endpoint`; disable if neither), build `HttpClientSettings` via `HttpClientHelper.loadSettingsFromProperties(props, prefix)` where `prefix` is `"agent"` or `"incident"`, read `country`/`atmName` from `ModuleContext`, read `log.upload.interval.minutes` (default 15), resolve `logFilePath`.
|
||||
2. `start()`: single daemon thread `log-upload`, `scheduleAtFixedRate(doUpload, 0, interval, MINUTES)` — **fires immediately** then every interval.
|
||||
3. `doUpload()` → `AgentLogUploader.upload(logFilePath, settings, country, atmName)`:
|
||||
- Skips silently if the file is absent.
|
||||
- Multipart POST to `{endpoint}/api/agent/{country}/{atm}/log/upload` (boundary `----HiveOpsLogUploadBoundary`, part `file`, filename `hiveops-agent.log`, `Content-Type: text/plain`); logs HTTP status/elapsed/bytes.
|
||||
4. `stop()`: `scheduler.shutdownNow()`.
|
||||
|
||||
### log-monitor cycle (tail → detect → report)
|
||||
|
||||
1. `initialize()`: same endpoint/settings resolution; read the four `log.monitor.*` tunables; resolve `logFilePath`.
|
||||
2. `start()`: daemon thread `log-monitor`, `scheduleAtFixedRate(doMonitor, interval, interval, MINUTES)` — **first run delayed one interval** so the log settles.
|
||||
3. `doMonitor()`:
|
||||
- Track `lastCheckedPosition` (byte offset). If file shrank (`position > size`) → treat as rotation, reset to 0. If `position == size` → nothing new, return.
|
||||
- `scanNewLines(fileSize)`: `RandomAccessFile.seek(lastCheckedPosition)`, read line-by-line, re-decode each line from ISO-8859-1 bytes → UTF-8 (works around `RandomAccessFile.readLine()`'s charset), match against `LOG_LEVEL_PATTERN` = `^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[.,]\d+\s+(ERROR|FATAL|WARN)\s+(.+)$`. `WARN` → `warnCount`; `ERROR`/`FATAL` → `errorCount` (+ up to `maxSamples` sample lines, each truncated to 200 chars).
|
||||
- Advance `lastCheckedPosition = fileSize`.
|
||||
- If `errorCount < errorThreshold` → return. If `lastReportedAt > 0` and within `cooldownMinutes` → return.
|
||||
- Else `reportToBackend()` and set `lastReportedAt = now`.
|
||||
4. `reportToBackend()`: hand-builds JSON `{"errorCount":N,"warnCount":N,"sampleErrors":[...]}` (no Jackson — its own `jsonEscape`), `POST` with `Content-Type: application/json` to `String.format("%s/atm/log-error?country=%s&name=%s", endpoint, country, name)`. Accepts HTTP 200/201; logs (truncated) error body otherwise. Never throws out of the scheduler.
|
||||
|
||||
All state (`lastCheckedPosition`, `lastReportedAt`) is **in-memory only** — an agent restart resets the tail offset to 0, so the first post-restart scan re-reads the whole current file.
|
||||
|
||||
## How events/results reach the platform
|
||||
|
||||
This module has **no** journal/incident `EventType` (`IAT_*`, `CARD_READER_*`, …) integration and does **not** use `IncidentEventClient` — see [technical.agent] "API surface" for that separate path. It also does not report fleet task status. It has exactly two HTTP touchpoints, both terminating at **agent-proxy** (which owns agent log endpoints natively and forwards to incident's internal API):
|
||||
|
||||
```
|
||||
LogUploadModule ──► AgentLogUploader.upload()
|
||||
POST {endpoint}/api/agent/{country}/{atm}/log/upload (multipart)
|
||||
│
|
||||
▼ agent-proxy AgentLogController.uploadLog() (50 MB cap → 413)
|
||||
→ IncidentInternalService.uploadLog() → incident internal store
|
||||
|
||||
LogMonitorModule ──► reportToBackend()
|
||||
POST {endpoint}/atm/log-error?country=&name= (JSON summary)
|
||||
│ ⚠ mismatch: agent-proxy's error endpoint is
|
||||
│ POST /api/agent/{country}/{atm}/log/errors
|
||||
▼ agent-proxy AgentLogController.logErrors()
|
||||
→ IncidentInternalService.logErrors()
|
||||
→ incident InternalAgentController POST /api/internal/agent/{country}/{atm}/log/errors
|
||||
→ LogErrorReportRequest → IncidentAutoCreationService → SOFTWARE_ERROR incident
|
||||
```
|
||||
|
||||
**Path discrepancy to be aware of:** the *upload* URL matches agent-proxy's route exactly, but the *monitor's* report URL (`/atm/log-error?country=&name=`) is a legacy shape that does **not** match agent-proxy's `/api/agent/{country}/{atm}/log/errors`. Whether an nginx/proxy rewrite bridges the two, or whether error reporting silently 404s on the current fleet, is **unverified** — verify against the live agent-proxy/nginx config before relying on error-to-incident from this module.
|
||||
|
||||
## Related helper capabilities (not driven by this module)
|
||||
|
||||
`AgentLogUploader` also exposes `uploadDirectory()` (bundles every `*.log`/`*.log.gz` under `logs/` incl. `logs/modules/` into `hiveops-logs.zip`) and `uploadPath(..., taskId)` (zips an arbitrary path, appends `?taskId=`). These back the **`LOG_COLLECT` / `LOG_COLLECT_PATH` fleet commands** and are invoked from the fleet command handlers, **not** from `log-upload`/`log-monitor`. The scheduled upload only ever ships the single current `hiveops-agent.log`.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
The module does **not** use `PlatformServiceFactory` / `PlatformService` at all — it needs no registry reads, path resolution, or shell-outs. Log path resolution is a plain JVM-system-property (`hiveops-agent.log`) lookup with a `ModuleContext.getStartedFromDir()` fallback, and all I/O is JDK (`RandomAccessFile`, `HttpURLConnection`, `Files`).
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
module: agent.log
|
||||
title: Agent Log — self-log upload & error-to-incident monitoring
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-log` source (`com.hiveops.logs`) 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Maven artifact: **`hiveops-module-log`** (own version `1.0.0`; parent `hiveops-agent-parent` `4.4.3`)
|
||||
- Drop-in JAR: **`modules/hiveops-module-log.jar`** (finalName = artifactId; NOT in fat JAR)
|
||||
- SPI file `META-INF/services/com.hiveops.core.module.AgentModule` registers **two** modules:
|
||||
- `com.hiveops.logs.LogUploadModule` → `AgentModule.getName()` = **`log-upload`** (no deps)
|
||||
- `com.hiveops.logs.LogMonitorModule` → `AgentModule.getName()` = **`log-monitor`** (depends on `log-upload`)
|
||||
- Upload helper: `com.hiveops.logs.AgentLogUploader` — lives in **`hiveops-agent-journal`**, not this jar (provided scope).
|
||||
- `getLogPackage()` NOT overridden → own logging goes to root **`logs/hiveops-agent.log`** (no `logs/modules/log*.log`).
|
||||
- `isHealthy()` NOT overridden → always `true` (watchdog won't catch a dead scheduler).
|
||||
- Monitored/uploaded file: `logs/hiveops-agent.log` under sysprop `hiveops-agent.log` base, else `{startedFromDir}/logs/hiveops-agent.log`.
|
||||
|
||||
## Config property keys (exact)
|
||||
| Key | Default | Module |
|
||||
|---|---|---|
|
||||
| `incident.log.upload.enabled` | `true` | log-upload master switch |
|
||||
| `log.upload.interval.minutes` | `15` | log-upload cadence (also fires once on start) |
|
||||
| `incident.log.monitor.enabled` | `true` | log-monitor master switch |
|
||||
| `log.monitor.interval.minutes` | `5` | log-monitor scan cadence (first scan delayed 1 interval) |
|
||||
| `log.monitor.cooldown.minutes` | `120` | min gap between error reports |
|
||||
| `log.monitor.error.threshold` | `1` | min errors in a scan to report |
|
||||
| `log.monitor.max.samples` | `5` | max sample error lines sent (each ≤200 chars) |
|
||||
| `agent.endpoint` (pref) / `incident.endpoint` (fallback) | — | HTTP base URL; if both blank → both modules self-disable |
|
||||
| `agent.*` / `incident.*` client settings | — | auth token / host header via `HttpClientHelper.loadSettingsFromProperties(props, "agent"\|"incident")` |
|
||||
| `modules.disabled` | — | comma list; include `log-upload` and/or `log-monitor` to disable |
|
||||
|
||||
- `isEnabled()` = endpoint set **AND** the module's own `enabled` flag true **AND** settings built.
|
||||
- No `getDefaultConfiguration()` override — defaults are the inline `getProperty(key, default)` values above only.
|
||||
|
||||
## Events / reports emitted
|
||||
- **NO journal `EventType` / `IAT_*` events.** Does not call `IncidentEventClient`. Does NOT report fleet task status.
|
||||
- **log-upload:** multipart `POST {endpoint}/api/agent/{country}/{atm}/log/upload` — boundary `----HiveOpsLogUploadBoundary`, part `file`, filename `hiveops-agent.log`, `text/plain`. (matches agent-proxy `AgentLogController`.)
|
||||
- **log-monitor:** JSON `POST {endpoint}/atm/log-error?country={c}&name={n}` — body `{"errorCount":N,"warnCount":N,"sampleErrors":[...]}`. Backend → **`SOFTWARE_ERROR`** incident (dedup + parked/OOS suppression in `IncidentAutoCreationService`).
|
||||
|
||||
## Error-line detection
|
||||
- Regex: `^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[.,]\d+\s+(ERROR|FATAL|WARN)\s+(.+)$`
|
||||
- `ERROR`+`FATAL` → `errorCount`; `WARN` → `warnCount` (counted, not reported as errors).
|
||||
- Tails via `RandomAccessFile.seek(lastCheckedPosition)`; re-decodes each line ISO-8859-1 bytes → UTF-8.
|
||||
- Rotation handling: if file size < last offset → reset offset to 0. Offsets are **in-memory only** (restart re-reads whole current file).
|
||||
|
||||
## Deployment
|
||||
- In **BOTH** `atm` and `srv` module sets of `deployment/build-dist.sh` → ships in every install/patch ZIP.
|
||||
- On by default on any device with an endpoint set.
|
||||
- Disable: `modules.disabled=log-upload,log-monitor` via `CONFIG_UPDATE` fleet task.
|
||||
- Retrofit: `INSTALL_MODULE` fleet task (drops jar, restarts agent) or full agent update.
|
||||
- Upload cap: agent-proxy rejects >**50 MB** with `413`.
|
||||
|
||||
## Gotchas
|
||||
- **Monitor endpoint mismatch:** module POSTs `/atm/log-error?country=&name=`, but agent-proxy's error route is `/api/agent/{country}/{atm}/log/errors`. Whether a rewrite bridges them / whether error-to-incident currently works is **unverified** — check live agent-proxy + nginx before trusting it. (Upload path is fine — it matches.)
|
||||
- Two module names, one jar — `modules.disabled` must name `log-upload` and/or `log-monitor`, NOT `hiveops-module-log` / `log`.
|
||||
- Only the **single current** `hiveops-agent.log` is uploaded on schedule — rotated `.gz` files are NOT (those come via the separate `LOG_COLLECT`/`LOG_COLLECT_PATH` fleet commands through the same `AgentLogUploader`).
|
||||
- First error scan is delayed one full `log.monitor.interval.minutes` after start; a startup error burst won't report until then.
|
||||
- Sample lines truncated to 200 chars; only up to `max.samples` errors sent — the count is full, the samples are not.
|
||||
- Cooldown + backend dedup mean a chronically-failing agent yields **one** `SOFTWARE_ERROR` incident, not a stream.
|
||||
- `AgentLogUploader` is in `hiveops-agent-journal` (atm set), not this jar — srv-set classpath carrying it is **unverified**.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT expect journal/hardware events (`IAT_*`, `CARD_READER_*`, `CASSETTE_*`, any `EventType`) from this module — self-log only.
|
||||
- Do NOT expect fleet task status reports — this module reports none.
|
||||
- Do NOT invent extra property namespaces — only the `incident.log.*` / `log.upload.*` / `log.monitor.*` keys above exist.
|
||||
- Do NOT assume error-to-incident works without verifying the `/atm/log-error` vs `/api/agent/.../log/errors` routing on the live proxy.
|
||||
- Do NOT disable via `modules.disabled=log` — use the exact names `log-upload` / `log-monitor`.
|
||||
- Do NOT look for a `logs/modules/log*.log` — this module logs to the root `hiveops-agent.log`.
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
module: agent.log
|
||||
title: Agent Log — self-log upload & error-to-incident monitoring
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-log` source (`com.hiveops.logs`) 2026-07-01.
|
||||
|
||||
## What it does (and doesn't)
|
||||
|
||||
`hiveops-module-log` is the agent's **self-observability** module. It watches and ships the agent's **own** operational log file (`logs/hiveops-agent.log`) — it does **not** parse ATM journals, count cassettes, or emit hardware/`IAT_*` events. It exists so HiveIQ can see when the agent itself is unhealthy without SSHing to the ATM.
|
||||
|
||||
It packs **two independent `AgentModule` implementations** into one drop-in JAR:
|
||||
|
||||
| Module name | Class | Job |
|
||||
|---|---|---|
|
||||
| `log-upload` | `LogUploadModule` | Periodically POSTs the whole `hiveops-agent.log` to the backend so support can read an ATM's agent log from the platform. |
|
||||
| `log-monitor` | `LogMonitorModule` | Tails the same log for `ERROR`/`FATAL` lines and, when enough pile up, reports a summary that the backend turns into a **`SOFTWARE_ERROR` incident**. |
|
||||
|
||||
`log-monitor` declares a dependency on `log-upload` (so the shared log-path resolution runs first), but the two run on separate schedulers and can be enabled/disabled independently.
|
||||
|
||||
Flow at a glance:
|
||||
- **log-upload:** on start, and then every `log.upload.interval.minutes` (default 15), multipart-POST `logs/hiveops-agent.log` → `{endpoint}/api/agent/{country}/{atm}/log/upload`.
|
||||
- **log-monitor:** every `log.monitor.interval.minutes` (default 5), read only the bytes appended since last check, count `ERROR`+`FATAL` (and `WARN` separately). If errors ≥ `log.monitor.error.threshold` **and** the cooldown has elapsed, POST a JSON summary `{errorCount, warnCount, sampleErrors[]}` → the backend creates/dedups a `SOFTWARE_ERROR` incident for that ATM.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
All read from `hiveops.properties` (main properties). There is no `getDefaultConfiguration()` override — defaults below are the inline `getProperty(key, default)` fallbacks in code.
|
||||
|
||||
**log-upload:**
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `incident.log.upload.enabled` | `true` | Master switch for the upload module. |
|
||||
| `log.upload.interval.minutes` | `15` | Upload cadence (also fires once immediately on start). |
|
||||
|
||||
**log-monitor:**
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `incident.log.monitor.enabled` | `true` | Master switch for the monitor module. |
|
||||
| `log.monitor.interval.minutes` | `5` | How often the log is scanned for new error lines. |
|
||||
| `log.monitor.cooldown.minutes` | `120` | Minimum gap between two error reports (anti-spam). |
|
||||
| `log.monitor.error.threshold` | `1` | Minimum error count in a scan before a report is sent. |
|
||||
| `log.monitor.max.samples` | `5` | Max error lines included as `sampleErrors` (each truncated to 200 chars). |
|
||||
|
||||
**Shared (not owned by this module):** both modules pick the HTTP base URL from `agent.endpoint` (preferred) or `incident.endpoint` (fallback), and build their HTTP client via the matching `agent.*` / `incident.*` settings (auth token, host header, etc.). If neither endpoint is set, both modules self-disable.
|
||||
|
||||
The monitored/uploaded file path is `logs/hiveops-agent.log` under the base given by the JVM system property `hiveops-agent.log` (set by `AgentApplication`), falling back to `{startedFromDir}/logs/hiveops-agent.log`.
|
||||
|
||||
## Enabled / deployed to ATMs
|
||||
|
||||
- **Enable condition:** each sub-module self-enables when `agent.endpoint` **or** `incident.endpoint` is set **and** its own `incident.log.upload.enabled` / `incident.log.monitor.enabled` flag is `true` (both default true). So on any normally-configured device both are on by default.
|
||||
- **Ships everywhere:** `hiveops-module-log` is in **both** the `atm` and `srv` module sets of `deployment/build-dist.sh`, so it's in every standard install/patch ZIP as the drop-in `modules/hiveops-module-log.jar` (not baked into the fat JAR).
|
||||
- **Disable per ATM/institution:** push `modules.disabled=log-upload,log-monitor` (comma list; include either or both names) via a `CONFIG_UPDATE` fleet task. Turning off just the noise: set `incident.log.monitor.enabled=false` but leave upload on, or vice-versa.
|
||||
- **Retrofit an ATM without it:** deliver `modules/hiveops-module-log.jar` via a fleet `INSTALL_MODULE` task (drops the JAR and restarts the agent), or via a full agent update.
|
||||
|
||||
## What the backend does with it
|
||||
|
||||
- **Log upload** → agent-proxy `AgentLogController` `POST /api/agent/{country}/{atm}/log/upload` → forwarded to incident's internal API. Agent-proxy rejects uploads over **50 MB** with `413 Payload Too Large`.
|
||||
- **Error report** → the backend maps the report to a **`SOFTWARE_ERROR`** incident via `IncidentAutoCreationService`. That path **suppresses** creation if the ATM is parked or `OUT_OF_SERVICE`, and **dedups** against an already-active `SOFTWARE_ERROR` incident for the same ATM (so a flapping agent gives one incident, not hundreds).
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---|---|---|
|
||||
| No agent log ever appears in the platform for an ATM | Module disabled, or no `agent.endpoint`/`incident.endpoint` on that ATM | Check `modules.disabled`; confirm an endpoint is set; look for "Log upload module initialized" in the ATM's `hiveops-agent.log`. |
|
||||
| Log upload rejected | File exceeds 50 MB | Agent-proxy returns 413; check log-rotation on the ATM — the module uploads the single current `hiveops-agent.log`, not rotated `.gz` files. |
|
||||
| Expected a `SOFTWARE_ERROR` incident but none created | ATM parked/`OUT_OF_SERVICE`, or an active `SOFTWARE_ERROR` incident already open (dedup), or still inside the 120-min cooldown | Check ATM status and existing open incidents; check `log.monitor.cooldown.minutes`. |
|
||||
| Error incidents never fire even though the agent logs errors | Errors don't match the expected Log4j2 line shape (`yyyy-MM-dd HH:mm:ss.SSS LEVEL ...`), so the scanner doesn't count them | Confirm the ATM's `log4j2.xml` pattern starts with timestamp + level; non-standard patterns are silently skipped. |
|
||||
| Flood of errors right after startup produced nothing | First monitor scan is delayed one full interval (default 5 min) after start | Expected; give it an interval. |
|
||||
| Report seems to go nowhere / backend 404 | **Path mismatch (see gotcha):** the monitor POSTs to `{endpoint}/atm/log-error?country=&name=`, but agent-proxy exposes the error endpoint at `/api/agent/{country}/{atm}/log/errors` | If error-to-incident stops working after an endpoint/proxy change, verify how `/atm/log-error` is routed for that fleet — see the Claude tab "Do NOT / gotchas". *(routing of the legacy path is unverified.)* |
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
module: agent.nh-tcr-events
|
||||
title: NH TCR Events — cassette + firmware monitoring for Nautilus Hyosung / Glory BRM30 recyclers
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · architecture.** Written 2026-07-01 from `hiveops-module-nh-tcr-events` source. Verify against code before relying on specifics.
|
||||
|
||||
## Module shape
|
||||
|
||||
Package `com.hiveops.nhtcr`, three classes, no controllers/DB/Kafka (this is an agent module, not a microservice — see [technical.agent]):
|
||||
|
||||
- **`NhTcrEventsModule`** — the `AgentModule` SPI entry point (registered via `META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.nhtcr.NhTcrEventsModule`). `getName()` = `nh-tcr-events`, `getVersion()` = `1.0.0`, `getLogPackage()` = `com.hiveops.nhtcr`. Owns lifecycle (`initialize()` → `isEnabled()` → `start()` → `stop()`) and the single shared `ScheduledExecutorService`.
|
||||
- **`ApLogMonitor`** (package-private) — parses the MoniPlus2S APLog and syncs firmware inventory.
|
||||
- **`FwEventLogMonitor`** (package-private) — parses the Glory BRM30 FWLogSave circular buffer.
|
||||
|
||||
Both monitors are plain constructed objects (`new ApLogMonitor(dir, atmName, client)`), not their own `AgentModule`s — `NhTcrEventsModule` drives both from one poll loop. Each monitor is created only if its directory property is set **and** resolves to an existing directory; `ready = apLogMonitor != null || fwEventLogMonitor != null` gates `isEnabled()`.
|
||||
|
||||
## Lifecycle & gating (`initialize()`)
|
||||
|
||||
1. Reads `device.type`; if not `NH_TCR` (uppercased/trimmed) → logs and returns (module stays disabled, `ready=false`).
|
||||
2. Reads `nh.tcr.ap.log.dir` and `nh.tcr.fw.log.dir`; if **both** null → disabled.
|
||||
3. Resolves the endpoint: `agent.endpoint` if present, else `incident.endpoint`; if neither → disabled. `prefix` is `"agent"` or `"incident"` accordingly.
|
||||
4. Builds one `IncidentEventClient` from `HttpClientHelper.loadSettingsFromProperties(props, prefix)` + `context.getAtmName()` + `context.getCountry()`, shared by both monitors.
|
||||
5. Constructs each monitor only if its dir exists (else a per-monitor warning; the other monitor can still run).
|
||||
6. Reads `nh.tcr.events.poll.interval.sec` (default `DEFAULT_POLL_SEC=30`) into `pollSec`.
|
||||
|
||||
`start()` spins one daemon thread `"nh-tcr-events"` and `scheduleAtFixedRate(this::poll, 0, pollSec, SECONDS)` — **`pollSec` is honored here** (contrast the `atec-tcr-events` sibling, whose poll interval is a no-op bug). `poll()` calls `apLogMonitor.poll()` then `fwEventLogMonitor.poll()`, wrapping both in a try/catch so one monitor's exception can't kill the scheduler.
|
||||
|
||||
## Flow: poll → parse → detect → emit
|
||||
|
||||
```
|
||||
NhTcrEventsModule.start()
|
||||
→ ScheduledExecutorService (daemon "nh-tcr-events", fixed-rate, every pollSec sec)
|
||||
→ poll()
|
||||
→ apLogMonitor.poll() // cassette status + firmware inventory
|
||||
→ fwEventLogMonitor.poll() // EP restarts
|
||||
```
|
||||
|
||||
### `ApLogMonitor` — date-subfolder tail + header scrape
|
||||
|
||||
- **File selection:** `rotateDailyFileIfNeeded()` compares `LocalDate.now()` to `currentDate`; on a new day it recomputes `currentFile = baseLogDir/{YYYYMMDD}/APLog{YYYYMMDD}.log` (`DateTimeFormatter.ofPattern("yyyyMMdd")`), resets `fileOffset=0`, `headerParsed=false`, and clears the `slotStatus` map.
|
||||
- **Header sync (once per daily file):** on first poll of a file, `parseAndSyncHeader()` opens a **separate** `RandomAccessFile` from offset 0 (so it doesn't disturb `fileOffset`), reads the block between the two `={20,}` delimiter lines, and extracts versions. Two line shapes: `- <label> = SP[..], EP[..]` (→ `key.sp` / `key.ep`) and simple `- <label> = <value>`. `labelToKey()` strips a trailing ` Version`, lowercases, dots spaces, and skips `Machine Number` / `Model` / `IP Address` / `Port Number` / `Time Zone`. If any versions found → `HardwareInventoryRequest{atmName, peripheralVersions}` → `client.syncHardware(req)`. `headerParsed=true` is set even on partial failure so it won't loop.
|
||||
- **Tail-by-offset:** opens `currentFile` with `RandomAccessFile`, guards truncation (`raf.length() < fileOffset` → reset offset + clear `slotStatus`), seeks `fileOffset`, reads new lines to EOF, advances `fileOffset = raf.getFilePointer()`.
|
||||
- **Per-line detection (`processLine`):** `STACK_BLOCK` regex captures `denom | count | status | type | slot` from `** IStackedBillsBlock = $|...`. Diffs `status` against `slotStatus.put(slot, status)` — **no emit if unchanged** (this in-agent map is the only storm guard; there's no server-side dedup). On a real change:
|
||||
- Always emit **`CASSETTE_INVENTORY`** first (keeps the cassette-details tab current on any transition, including recoveries).
|
||||
- Then by status: `LOW`→`CASSETTE_LOW`, `EMPTY`→`CASSETTE_EMPTY`, `MISSING`→`TCR_CASSETTE_MISSING`, `MANIP`→`TCR_CASSETTE_MANIP`, `OK`→log-only recovery notice (no event).
|
||||
- `slotStatus` is per-instance/per-agent-process lifetime, cleared on truncation or day rollover — so a slot still `EMPTY` at the first block of a new day re-emits (state doesn't carry across daily rotation).
|
||||
|
||||
> Note: `NhTcrEventsModule`'s class-level Javadoc lists a stale subset (`CASSETTE_LOW`, `CASSETTE_EMPTY`, `TCR_CASSETTE_MISSING`, `TCR_EP_RESTART`) — the authoritative emit set is what `ApLogMonitor.processLine` / `emit` and `FwEventLogMonitor` actually send.
|
||||
|
||||
### `FwEventLogMonitor` — circular-buffer tail
|
||||
|
||||
- 10 fixed files `00FWEvent00.log`…`00FWEvent09.log` (`FILE_PREFIX="00FWEvent"`), one `long[] offsets` per index.
|
||||
- **First poll:** seeks every existing file to its current size (`offsets[i]=Files.size`) and marks `seenOnInit`, so only entries written **after** agent start are processed; then returns.
|
||||
- **Subsequent polls:** for each index, `scanFile()` opens the file, resets offset to 0 if `fileLen < offsets[index]` (file recycled), seeks and reads new lines. `processLine` matches `LogManager_StartUp\(\)` → emit **`TCR_EP_RESTART`** (with `FWEventNN` label in details). No de-dup beyond offset tracking.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
Both monitors emit through the same shared `IncidentEventClient` (constructed once in `initialize()`), the same client class used by `journal-events` and other event-reporting modules — see [technical.agent] for the full outbound surface.
|
||||
|
||||
```
|
||||
ApLogMonitor/FwEventLogMonitor.emit(...)
|
||||
→ CreateJournalEventRequest.builder()
|
||||
.agentAtmId(atmId) // ATM name string; resolved to incident's numeric atmId server-side
|
||||
.eventType(eventType) // free-form String — not validated client-side
|
||||
.eventDetails(details)
|
||||
// cassette fields (ApLogMonitor only):
|
||||
.cassettePosition(slot) // e.g. CST_A / CST_C
|
||||
.cassetteDenomination(int) .cassetteBillCount(int)
|
||||
.cassetteType(type) // RECYCLING / REJECT / REPCONTAINER
|
||||
.cassetteCurrency("USD") // hardcoded
|
||||
.eventSource("NH_TCR_APLOG" | "NH_TCR_FWEVENT")
|
||||
.build()
|
||||
→ IncidentEventClient.sendEvent(req) → POST {endpoint}/api/journal-events
|
||||
```
|
||||
|
||||
Firmware inventory takes a different path in the same client: `client.syncHardware(req)` → `POST {endpoint}/api/atms/hardware/sync`.
|
||||
|
||||
`eventType` is a plain `String` on `CreateJournalEventRequest` (the `hiveops-agent-journal` DTO). Nothing client-side blocks an unregistered type; recognition/labeling happens server-side against hiveops-incident's DB-backed `event_type` table. Send/sync failures throw `IOException`, which `emit()` / `parseAndSyncHeader()` catch and log — no retry or queueing.
|
||||
|
||||
## Auth / endpoint selection
|
||||
|
||||
`HttpClientHelper.loadSettingsFromProperties(props, prefix)` resolves institution key with fallback `{prefix}.institution.key` → `agent.institution.key` → `incident.institution.key`, and reads the bearer token from `agent.auth.token`. No bespoke auth in this module — 100% the shared agent HTTP settings pattern.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
None used directly. The module reads files via `java.nio.file` / `RandomAccessFile` — no `PlatformService` / `PathResolver` dependency, even though NH TCR software is Windows-only in practice. The Windows-only constraint is enforced by deployment convention (`device.type=NH_TCR`), not a runtime OS check in this module's code.
|
||||
|
||||
## Cross-links
|
||||
- [technical.agent] — module system, lifecycle, shared HTTP client settings, full outbound API surface
|
||||
- [platform.module-map]
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
module: agent.nh-tcr-events
|
||||
title: NH TCR Events — cassette + firmware monitoring for Nautilus Hyosung / Glory BRM30 recyclers
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-nh-tcr-events` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Maven artifact: **`hiveops-module-nh-tcr-events`** (`com.hiveops`, version `1.0.0`, packaged as a standalone `jar`, parent `hiveops-agent-parent` `4.4.3`).
|
||||
- Agent module id (`getName()`, and the string for **`modules.disabled`**): **`nh-tcr-events`**.
|
||||
- `getVersion()` = `1.0.0`; `getLogPackage()` = `com.hiveops.nhtcr`.
|
||||
- Entry class: `com.hiveops.nhtcr.NhTcrEventsModule`, registered via `META-INF/services/com.hiveops.core.module.AgentModule`.
|
||||
- Vendor target: Nautilus Hyosung TCR + Glory BRM30 external processor (EP).
|
||||
- Activation gate: `device.type` must equal **`NH_TCR`** exactly (uppercased/trimmed). No legacy alias. Any other value → module inert, `isEnabled()` = `false`.
|
||||
- Platform: Windows-only in practice (no runtime OS check in code — enforced by deployment convention).
|
||||
|
||||
## Config properties (exact keys)
|
||||
| Key | Default | Required | Notes |
|
||||
|---|---|---|---|
|
||||
| `device.type` | *(none)* | **yes** | must be `NH_TCR` or module disables |
|
||||
| `nh.tcr.ap.log.dir` | *(none)* | one of ap/fw | MoniPlus2S APLog root; date subfolders under it. Not-a-dir → APLog monitor off (warn) |
|
||||
| `nh.tcr.fw.log.dir` | *(none)* | one of ap/fw | Glory FWLogSave dir. Not-a-dir → FW monitor off (warn) |
|
||||
| `nh.tcr.events.poll.interval.sec` | `30` | no | **applied** here (unlike sibling `atec-tcr-events`, where it's a no-op) |
|
||||
|
||||
- If **both** `nh.tcr.ap.log.dir` and `nh.tcr.fw.log.dir` are unset → module disabled. Neither dir existing → module disabled.
|
||||
- Shared (not module-specific): `agent.endpoint` / `incident.endpoint` (prefix = `agent` if `agent.endpoint` set, else `incident`; if neither → disabled), `agent.auth.token`, `agent.institution.key` / `incident.institution.key`.
|
||||
|
||||
## Event types emitted (exact strings)
|
||||
| Event type | Source | Trigger |
|
||||
|---|---|---|
|
||||
| `CASSETTE_INVENTORY` | APLog | **any** slot status change (emitted first, always) |
|
||||
| `CASSETTE_LOW` | APLog | slot status → `LOW` |
|
||||
| `CASSETTE_EMPTY` | APLog | slot status → `EMPTY` |
|
||||
| `TCR_CASSETTE_MISSING` | APLog | slot status → `MISSING` |
|
||||
| `TCR_CASSETTE_MANIP` | APLog | slot status → `MANIP` (suspect bills) |
|
||||
| `TCR_EP_RESTART` | FWEvent | line matches `LogManager_StartUp()` |
|
||||
|
||||
- Status `OK` (recovery) → **no event**, agent-log-only (`slot {} recovered`).
|
||||
- `eventType` is a free-form `String` on `CreateJournalEventRequest` (`hiveops-agent-journal` DTO) — **not validated client-side**. Recognition/labeling is server-side via hiveops-incident's DB `event_type` table.
|
||||
- `eventSource` sent: **`NH_TCR_APLOG`** (cassette) or **`NH_TCR_FWEVENT`** (EP restart).
|
||||
- Cassette events also set `cassettePosition` (slot, e.g. `CST_C`), `cassetteDenomination` (int), `cassetteBillCount` (int), `cassetteType` (`RECYCLING`/`REJECT`/`REPCONTAINER`), `cassetteCurrency="USD"` (hardcoded). FW events set none of these.
|
||||
- Transport: `IncidentEventClient.sendEvent()` → `POST {endpoint}/api/journal-events`.
|
||||
- Separate path — firmware inventory: `client.syncHardware()` → `POST {endpoint}/api/atms/hardware/sync`, fired **once per daily APLog file** from the header block (`peripheralVersions` map).
|
||||
|
||||
## Files monitored
|
||||
- APLog: `{nh.tcr.ap.log.dir}/{YYYYMMDD}/APLog{YYYYMMDD}.log` — **date subfolder**, daily rotation. Tail-by-offset; offset/`slotStatus` reset on rollover or truncation.
|
||||
- FWEvent: `{nh.tcr.fw.log.dir}/00FWEvent00.log` … `00FWEvent09.log` — 10-file **circular buffer**. Per-file offset; first poll seeks all to EOF (only NEW entries emit).
|
||||
|
||||
## Deployment
|
||||
- Delivered fleet-wide via **`INSTALL_MODULE`** fleet task (drops JAR into agent's `modules/` dir, restarts agent — `ProcessInstallModule` in `hiveops-file-collection`).
|
||||
- Disable without uninstalling: add `nh-tcr-events` to `modules.disabled`, push `CONFIG_UPDATE`.
|
||||
- Emitted event types must be registered in hiveops-incident → Settings → Event Types (DB `event_type` table) to be recognized/labeled.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT run this on non-`NH_TCR` device types — it's a no-op by design elsewhere, not a bug to chase.
|
||||
- Do NOT use anything but **`nh-tcr-events`** as the `modules.disabled` / module id string.
|
||||
- Do NOT expect `TCR_EP_RESTART` for restarts that happened **before** the agent came up — the FW monitor seeks to EOF on first poll; only post-start `LogManager_StartUp()` lines emit.
|
||||
- Do NOT assume firmware versions refresh continuously — the header sync runs **once per daily APLog file** and sets `headerParsed=true` even on partial failure (no same-day retry).
|
||||
- Do NOT expect retry/queueing on send/sync failure — a failed `sendEvent()` / `syncHardware()` is logged and dropped.
|
||||
- Do NOT assume `cassetteCurrency` reflects the real currency — it's hardcoded `"USD"`.
|
||||
- Do NOT trust `NhTcrEventsModule`'s class Javadoc emit list — it's stale (omits `CASSETTE_INVENTORY` and `TCR_CASSETTE_MANIP`). Use the table above / the monitor code.
|
||||
- Do NOT confuse with `hiveops-module-atec-tcr-events` (`tcr-events`, LG/ATEC vendor) or `hiveops-module-nh-tcr-journal` — separate modules, different log formats and event pipelines.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
module: agent.nh-tcr-events
|
||||
title: NH TCR Events — cassette + firmware monitoring for Nautilus Hyosung / Glory BRM30 recyclers
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support.** Written 2026-07-01 from `hiveops-module-nh-tcr-events` source (agent parent `4.4.3`). Verify against code before relying on specifics.
|
||||
|
||||
## What this module is
|
||||
|
||||
`hiveops-module-nh-tcr-events` (Maven artifact `hiveops-module-nh-tcr-events`, agent module id **`nh-tcr-events`**) is an **agent drop-in module**, not a backend service. It runs inside the HiveIQ Agent process on **Nautilus Hyosung TCR (Teller Cash Recycler) devices** that use the **Glory BRM30** external processor (EP). It self-disables unless the ATM's `device.type` is exactly `NH_TCR` (no legacy alias).
|
||||
|
||||
It tails log files that the Hyosung MoniPlus2S and Glory firmware already write to disk, turns specific lines into HiveIQ journal events, and additionally scrapes the daily APLog header for component firmware versions and pushes those to the incident hardware inventory. It does not talk to the recycler hardware directly, and it does not own cash totals / cassette reconciliation (that's reconciliation's job).
|
||||
|
||||
## What it monitors
|
||||
|
||||
Two independent monitors, both driven by one scheduler tick:
|
||||
|
||||
1. **MoniPlus2S APLog** (`ApLogMonitor`) — dir `nh.tcr.ap.log.dir`.
|
||||
- File pattern: `{nh.tcr.ap.log.dir}\{YYYYMMDD}\APLog{YYYYMMDD}.log` (each day gets its own **date subfolder**, e.g. `C:\Hyosung\MoniPlus2S\MoniPlus2Slog\20230330\APLog20230330.log`).
|
||||
- Parses `** IStackedBillsBlock = $|{denom}|{count}|{status}|{type}|{slot}|...` lines for per-slot cassette **level and status** (`OK` / `LOW` / `EMPTY` / `MISSING` / `MANIP`). Emits only when a slot's status **changes** (in-memory per-slot last-status guard), so a slot stuck `EMPTY` doesn't re-fire every poll.
|
||||
- On the first read of each daily file it parses the **header block** (bounded by `====` delimiter lines) for component firmware versions (SIU, BRM, SDK, Application, Nextware, etc.) and syncs them to the incident backend hardware inventory.
|
||||
2. **Glory BRM30 FWLogSave** (`FwEventLogMonitor`) — dir `nh.tcr.fw.log.dir`.
|
||||
- Files: `00FWEvent00.log` … `00FWEvent09.log`, a **10-file circular buffer** in that directory.
|
||||
- Watches for `LogManager_StartUp()` lines, which indicate the Glory EP firmware process (re)started.
|
||||
|
||||
The two directories are configured separately and may point anywhere; either one being missing just disables that one monitor.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
| Property | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `device.type` | *(none)* | Activation gate. Must equal `NH_TCR` (uppercased/trimmed). Any other value → module logs `device.type='<X>', not NH_TCR — module disabled` and does nothing. |
|
||||
| `nh.tcr.ap.log.dir` | *(none)* | Directory root for the MoniPlus2S APLog (date subfolders live under it). If set but not an existing directory → `APLog monitoring disabled` warning, module keeps running for FW events. |
|
||||
| `nh.tcr.fw.log.dir` | *(none)* | Directory for the Glory `00FWEventNN.log` files. If set but not an existing directory → `FW event monitoring disabled` warning. |
|
||||
| `nh.tcr.events.poll.interval.sec` | `30` | Poll interval in seconds. **This module actually applies the configured value** (`start()` schedules at `pollSec`) — unlike the sibling `atec-tcr-events` module, where the same-named property is a no-op. |
|
||||
|
||||
If **both** `nh.tcr.ap.log.dir` and `nh.tcr.fw.log.dir` are unset, the module disables itself (`neither nh.tcr.ap.log.dir nor nh.tcr.fw.log.dir configured — module disabled`) — it never throws or crashes agent startup.
|
||||
|
||||
It reuses the shared agent HTTP settings — `agent.endpoint` (preferred) or `incident.endpoint` (legacy fallback), `agent.auth.token`, and the institution-key chain (`agent.institution.key` / `incident.institution.key`) — the same properties every other event-reporting module uses. There are no NH-TCR-specific auth properties. If neither endpoint is set the module disables itself.
|
||||
|
||||
## How it's enabled / deployed
|
||||
|
||||
- Activates only on ATMs configured with `device.type=NH_TCR`. On any other device type it is intentionally inert — this is expected, not a bug.
|
||||
- Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which drops the module JAR into the agent's `modules/` directory and restarts the agent (`ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order applies: lab ATM → pilot device → fleet.
|
||||
- Can be disabled fleet-wide **without** removing the JAR by adding `nh-tcr-events` to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE`. The module id for `modules.disabled` is the `getName()` value, **`nh-tcr-events`** (see the Claude doc for the exact string).
|
||||
- Event types it emits (`CASSETTE_INVENTORY`, `CASSETTE_LOW`, `CASSETTE_EMPTY`, `TCR_CASSETTE_MISSING`, `TCR_CASSETTE_MANIP`, `TCR_EP_RESTART`) must exist as rows in hiveops-incident's `event_type` table (Incident → Settings → Event Types) to be recognized/labeled downstream. `CASSETTE_LOW` / `CASSETTE_EMPTY` are standard shared types; the `TCR_*` / `CASSETTE_INVENTORY` types may need adding on a fresh environment.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **No NH TCR events ever appear for a customer ATM** — first check `device.type` on that ATM's config; if it isn't exactly `NH_TCR`, the module is intentionally inert. Not a bug.
|
||||
- **Module loaded but silent** — check that `nh.tcr.ap.log.dir` / `nh.tcr.fw.log.dir` are set and point at real existing directories on that machine; a missing/wrong path disables that monitor at startup with a log warning, no crash. Confirm via agent logs: `APLog watching <path>` and `FWEvent monitor initialized`.
|
||||
- **Events stopped after midnight** — the APLog monitor rolls to a new **date subfolder** at local midnight (`rotateDailyFileIfNeeded()`). If the Hyosung software's folder/file-naming convention ever changes (non-standard rotation, DST edge case), the module will watch a path that never appears and go silent.
|
||||
- **No `TCR_EP_RESTART` for restarts that happened before the agent came up** — by design: on its first poll the FW monitor **seeks each `00FWEventNN.log` to its end**, so only `LogManager_StartUp()` lines written *after* agent start are emitted. Pre-existing startup entries are ignored.
|
||||
- **Events sent but not visible/categorized in Incident** — confirm the emitted event types are registered in hiveops-incident's Event Types settings; these are DB-driven, not hardcoded in incident's code.
|
||||
- **Firmware versions not updating on the device detail** — version sync only fires on the **first read of each daily APLog file** (header parse). It runs once per day per file; a partial/failed header parse still marks `headerParsed=true` so it won't retry until the next day's file. Check for `synced N peripheral version(s) from APLog header` or `failed to sync peripheral versions` in agent logs.
|
||||
- **HTTP send failures** — `IncidentEventClient.sendEvent()` / `syncHardware()` failures are caught and logged (`failed to send ...` / `failed to sync ...`) but **not retried or queued**; a transient outage silently drops that single event/sync rather than backing it up.
|
||||
- **Slot recovered (`OK`) events don't create tickets** — by design; a return to `OK` only writes an agent-side log line (`slot {} recovered`), no event is emitted.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
module: agent.nh-tcr-journal
|
||||
title: NH TCR Journal — transaction + fault events from Nautilus Hyosung Teller Cash Recyclers
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · architecture.** Written 2026-07-01 from `hiveops-module-nh-tcr-journal` source. Verify against code before relying on specifics.
|
||||
|
||||
## Module shape
|
||||
|
||||
Package `com.hiveops.nhtcr`, two classes, no controllers/DB/Kafka (this is an agent module, not a microservice — see [technical.agent]):
|
||||
|
||||
- **`NhTcrJournalModule`** — the `AgentModule` SPI entry point (registered via `META-INF/services/com.hiveops.core.module.AgentModule`). Owns lifecycle (`initialize()` → `isEnabled()` → `start()` → `stop()`), the shared `ScheduledExecutorService`, config resolution, and the `IncidentEventClient` construction.
|
||||
- **`TcrJournalMonitor`** — the actual ejournal follower + parser. A plain constructed object (`new TcrJournalMonitor(journalSource, atmName, client)`), not its own `AgentModule`.
|
||||
|
||||
Module identity from `NhTcrJournalModule`: `getName()` = `nh-tcr-journal`, `getVersion()` = `1.0.0`, `getLogPackage()` = `com.hiveops.nhtcr`.
|
||||
|
||||
## Activation gating (in `initialize()`)
|
||||
|
||||
Three gates, all fail-soft (log + return, `ready` stays `false`, `isEnabled()` returns `false` — no exception thrown to agent startup):
|
||||
|
||||
1. `device.type` (uppercased/trimmed) must equal `NH_TCR`.
|
||||
2. A `JournalSource` must be resolvable from the ext properties — `findJournalSource()` scans `context.getExtensionProperties()` for the first entry with `journal.dir` set and `type` in `{NH_TCR, MAINJOURNAL}`, then builds `new JournalSource(journalDir, type, fileFormat, filenameFormat, atmName)` (`file.format` defaults to `UTF8`).
|
||||
3. An endpoint must exist: `agent.endpoint` if set (prefix `agent`), else `incident.endpoint` (prefix `incident`).
|
||||
|
||||
On success it reads `nh.tcr.journal.poll.interval.sec` (default `60`), constructs the monitor, and schedules `poll()` on a single daemon thread named `nh-tcr-journal` via `scheduleAtFixedRate(this::poll, 0, pollSec, SECONDS)`. Note: unlike the sibling ATEC module, the poll interval property here **is** applied.
|
||||
|
||||
## Flow: tail → parse → detect → emit
|
||||
|
||||
```
|
||||
NhTcrJournalModule (scheduler, every nh.tcr.journal.poll.interval.sec)
|
||||
→ poll() → TcrJournalMonitor.poll()
|
||||
→ journalSource.getCurrentJournalFile() // resolves today's ej_..._{YYYY}{MM}{DD} file
|
||||
→ RandomAccessFile tail from stored byte offset (UTF-16 LE, 2 MB/poll cap)
|
||||
→ split into complete lines → processLine() per line
|
||||
```
|
||||
|
||||
### Tail-by-offset, UTF-16 LE
|
||||
|
||||
`TcrJournalMonitor` uses byte-offset tailing, but with a TCR-specific twist — the file is decoded as **UTF-16 LE**:
|
||||
- Tracks `currentFile` + `filePosition` (raw byte offset).
|
||||
- Daily rollover: when `getCurrentJournalFile()` returns a different file, it resets `filePosition=0`, clears block state, logs `switched to ...`.
|
||||
- Each poll seeks to `filePosition`, reads up to 2 MB, trims to an even byte count (UTF-16 needs pairs), decodes via `StandardCharsets.UTF_16LE.newDecoder()`, and splits on `\r?\n`.
|
||||
- Only **complete** lines are processed; a trailing partial line is left unread. It advances `filePosition` by `charsProcessed * 2` (2 bytes/char) so the next poll resumes exactly where parsing stopped — the partial line is re-read once it completes.
|
||||
- No explicit truncation guard: if `fileLength <= filePosition` the poll simply returns (no work), and a same-name rewrite would only be reprocessed if the file object changes at rollover.
|
||||
|
||||
### Line parsing (`processLine`)
|
||||
|
||||
`JOURNAL_LINE` regex: `^?\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}-\d+\]\[(\d+)\]\[TCR\](\w*)(.*)` — captures `level`, `eventName`, `params`. (The optional `?` absorbs a leading BOM.)
|
||||
|
||||
- A `level == 5` line with an **empty** event name opens a TRANSACTION RECORD block (`inTransactionBlock = true`); subsequent non-matching lines are buffered.
|
||||
- Any other timestamp line first flushes an open block, then routes the event name through `handleEvent()`.
|
||||
- Non-matching lines while a block is open are appended to the buffer; a blank line inside a block is buffered too.
|
||||
|
||||
### Detect → event mapping
|
||||
|
||||
`handleEvent()` (single-line device events):
|
||||
|
||||
| Journal `EventName` | Condition | Emitted event type |
|
||||
|---|---|---|
|
||||
| `CashDispenserStatusChanged` | params == `NODISPENSE` | `DISPENSER_JAM` |
|
||||
| `BNAItemsRefused` | always | `DISPENSER_JAM` |
|
||||
| `CashUnitThresholdChanged` | always | `CASSETTE_LOW` |
|
||||
| `AcceptorStatusChanged` | params != `OK` | `SELF_TEST_FAILED` |
|
||||
|
||||
`flushTransactionBlock()` (multi-line): `extract()` pulls `KEY : value` fields (`TRANSACTION NO.`, `TRANSACTION TYPE`, `TRANSACTION RESULT`, `TRANSACTION AMOUNT`, `MACHINE BALANCE`, `TELLER ID`, `BRANCH NUMBER`) and `extractBracketed()` pulls `Error Code [..]`. If both transaction number and type are missing the block is dropped. Otherwise it builds a `TCR Transaction #.. | Type: .. | Result: ..` summary string and emits it as a single **`OPERATOR_ACTION`** event (error code appended only if non-zero).
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
Every event goes through the shared `IncidentEventClient` (constructed once in `initialize()`), the same client class used by `journal-events` and the other event-reporting modules — see [technical.agent] for the full outbound API surface.
|
||||
|
||||
```
|
||||
TcrJournalMonitor.sendEvent(eventType, details)
|
||||
→ CreateJournalEventRequest.builder()
|
||||
.agentAtmId(atmId) // ATM name string, resolved to incident's numeric atmId server-side
|
||||
.eventType(eventType) // free-form String, not validated client-side
|
||||
.eventDetails(details)
|
||||
.eventSource("NH_TCR_JOURNAL")
|
||||
.build()
|
||||
→ IncidentEventClient.sendEvent(req)
|
||||
→ HTTP POST {agent.endpoint or incident.endpoint}/api/journal-events
|
||||
```
|
||||
|
||||
`eventType` is a plain `String` on `CreateJournalEventRequest` (`hiveops-agent-journal` DTO). The four types this module emits (`OPERATOR_ACTION`, `DISPENSER_JAM`, `CASSETTE_LOW`, `SELF_TEST_FAILED`) are all standard HiveIQ event types recognized server-side; there is no TCR-specific event string here. Send failures (`IOException`) are caught and logged — **no retry or queueing**.
|
||||
|
||||
## Auth / endpoint selection
|
||||
|
||||
`initialize()` picks the property prefix from whether `agent.endpoint` is set (`agent`) or not (`incident`), then `HttpClientHelper.loadSettingsFromProperties(props, prefix)` resolves timeouts, institution key, and the bearer token (`agent.auth.token`). No bespoke auth in this module — it is 100% the shared agent HTTP settings pattern.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
None used directly. The follower reads files via `RandomAccessFile` / `java.nio` and resolves the daily filename through `JournalSource` — no `PlatformService`/`PathResolver` dependency. In practice NH TCR machines are Windows, but the constraint is enforced by deployment convention (`device.type=NH_TCR` gating) rather than a runtime OS check.
|
||||
|
||||
## Cross-links
|
||||
- [technical.agent] — module system, lifecycle, shared HTTP client settings, full outbound API surface
|
||||
- [platform.module-map]
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
module: agent.nh-tcr-journal
|
||||
title: NH TCR Journal — transaction + fault events from Nautilus Hyosung Teller Cash Recyclers
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-nh-tcr-journal` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Maven artifact: **`hiveops-module-nh-tcr-journal`** (`com.hiveops`, version `1.0.0`, standalone `jar`, parent `hiveops-agent-parent` 4.4.3).
|
||||
- Agent module id (`getName()`, and the string for **`modules.disabled`**): **`nh-tcr-journal`**.
|
||||
- Entry class: `com.hiveops.nhtcr.NhTcrJournalModule`, registered via `META-INF/services/com.hiveops.core.module.AgentModule`.
|
||||
- Follower/parser class: `com.hiveops.nhtcr.TcrJournalMonitor`.
|
||||
- Log package (`getLogPackage()`): `com.hiveops.nhtcr`.
|
||||
- Activation gate: `device.type` must equal `NH_TCR` **and** an ext properties entry with `type=NH_TCR` (or legacy `MAINJOURNAL`) + `journal.dir` must exist, **and** `agent.endpoint` or `incident.endpoint` must be set. Any gate failing → module inert, `isEnabled()` returns `false` (no crash).
|
||||
|
||||
## Config properties (exact keys)
|
||||
| Key | File | Default | Required | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `device.type` | `hiveops.properties` | *(none)* | **yes** | must be `NH_TCR` (uppercased/trimmed) |
|
||||
| `nh.tcr.journal.poll.interval.sec` | `hiveops.properties` | `60` | no | poll cadence; **is** applied (`scheduleAtFixedRate`) |
|
||||
| `type` | ext (`ej.properties`) | *(none)* | **yes** | `NH_TCR` preferred, `MAINJOURNAL` legacy fallback |
|
||||
| `journal.dir` | ext | *(none)* | **yes** | TCR ejournal directory |
|
||||
| `filename.format` | ext | *(none)* | recommended | daily template, e.g. `ej_TCR1_{YYYY}{MM}{DD}.txt` |
|
||||
| `file.format` | ext | `UTF8` | no | `JournalSource` descriptor; follower still decodes bytes as UTF-16 LE |
|
||||
|
||||
Shared (not module-specific): `agent.endpoint` (prefix `agent`) else `incident.endpoint` (prefix `incident`); `agent.auth.token`; institution-key chain — all via `HttpClientHelper.loadSettingsFromProperties(props, prefix)`.
|
||||
|
||||
## Event types emitted (exact strings)
|
||||
| Event type | Trigger | Source |
|
||||
|---|---|---|
|
||||
| `DISPENSER_JAM` | `CashDispenserStatusChanged` with params `NODISPENSE` | single-line device event |
|
||||
| `DISPENSER_JAM` | `BNAItemsRefused` (always) | single-line device event |
|
||||
| `CASSETTE_LOW` | `CashUnitThresholdChanged` (always) | single-line device event |
|
||||
| `SELF_TEST_FAILED` | `AcceptorStatusChanged` with params != `OK` | single-line device event |
|
||||
| `OPERATOR_ACTION` | completed TRANSACTION RECORD block (`level 5`, empty name) | multi-line block |
|
||||
|
||||
- All four are standard HiveIQ event types (no TCR-specific event strings). `eventType` is a free-form `String` on `CreateJournalEventRequest` (`hiveops-agent-journal` DTO), not validated client-side.
|
||||
- `eventSource` sent for all events: **`"NH_TCR_JOURNAL"`**.
|
||||
- `OPERATOR_ACTION` `eventDetails` = `TCR Transaction #<no> | Type: .. | Result: .. | Amount: .. | Balance: .. | Teller: .. | Branch: ..` (`| Error: <code>` appended only if error code present and != `0`).
|
||||
- Transport: `POST {endpoint}/api/journal-events` via `IncidentEventClient.sendEvent()` — same client/endpoint as `journal-events`.
|
||||
|
||||
## Journal format facts
|
||||
- File: UTF-16 LE text, daily-rotated, resolved via `JournalSource.getCurrentJournalFile()` (filename fixed per model; ATM identity comes from the upload folder, not the filename).
|
||||
- Line regex: `[YYYY-MM-DD HH:MM:SS-nnn][<level>][TCR]<EventName><params>` (optional leading BOM tolerated).
|
||||
- TRANSACTION RECORD block opens on a `level 5` line with empty event name; buffered until the next timestamp line, then flushed. Block dropped if both `TRANSACTION NO.` and `TRANSACTION TYPE` are missing.
|
||||
- Tail-by-byte-offset (`filePosition`), 2 MB read cap per poll, only complete lines processed; offset advanced by `chars * 2`.
|
||||
|
||||
## Deployment
|
||||
- Delivered fleet-wide via **`INSTALL_MODULE`** fleet task (downloads JAR into agent `modules/` dir, restarts agent — `ProcessInstallModule` in `hiveops-file-collection`).
|
||||
- Disable without uninstalling: add `nh-tcr-journal` to `modules.disabled`, push `CONFIG_UPDATE`.
|
||||
- Rollout order: lab ATM → pilot → fleet.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT expect events on non-`NH_TCR` devices — inert by design, not a bug to chase.
|
||||
- Do NOT assume plain UTF-8/ASCII parsing — the follower decodes **UTF-16 LE**; a differently-encoded ejournal parses to nothing silently.
|
||||
- Do NOT expect retry/queueing on send failure — a failed `sendEvent()` is logged and dropped, not persisted or retried next poll.
|
||||
- Do NOT expect in-agent de-dup for single-line events (`DISPENSER_JAM` / `CASSETTE_LOW` / `SELF_TEST_FAILED`) — every matching line emits; only the transaction-block buffer holds state.
|
||||
- Do NOT rely on `MAINJOURNAL` for new installs — use `type=NH_TCR` so `journal-upload` tags uploads for the TCR parser; `MAINJOURNAL` is a compatibility fallback only.
|
||||
- Do NOT confuse this with `hiveops-module-nh-tcr-events` (`NhTcrEventsModule`) or `hiveops-module-atec-tcr-events` (`tcr-events`) — separate modules, separate formats/pipelines.
|
||||
- Do NOT assume the poll-interval property is a no-op — here it **is** applied (unlike the ATEC sibling).
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
module: agent.nh-tcr-journal
|
||||
title: NH TCR Journal — transaction + fault events from Nautilus Hyosung Teller Cash Recyclers
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support.** Written 2026-07-01 from `hiveops-module-nh-tcr-journal` source. Verify against code before relying on specifics.
|
||||
|
||||
## What this module is
|
||||
|
||||
`hiveops-module-nh-tcr-journal` (Maven artifact `hiveops-module-nh-tcr-journal`, agent module id **`nh-tcr-journal`**) is an **agent drop-in module**, not a backend service. It runs inside the HiveIQ Agent process on **Nautilus Hyosung TCR (Teller Cash Recycler) devices only**, and it self-disables unless the ATM's `device.type` is `NH_TCR`.
|
||||
|
||||
It tails the NH TCR ejournal file the recycler software already writes on disk (UTF-16 LE text), turns each teller transaction record and specific device-fault lines into HiveIQ journal events, and sends them to the incident pipeline via the shared agent HTTP client. It does not talk to the recycler hardware directly and it does not reconcile cash totals — it is a read-only journal follower.
|
||||
|
||||
> Not to be confused with `hiveops-module-nh-tcr-events` (a separate NH TCR module) or `hiveops-module-atec-tcr-events` (ATEC/LG TCRs). Different modules, different log formats.
|
||||
|
||||
## What it monitors
|
||||
|
||||
A single poll loop (default every 60s) tails one daily-rotated ejournal file resolved through the agent's standard `JournalSource` (directory + `filename.format`, e.g. `ej_TCR1_{YYYY}{MM}{DD}.txt`). The filename is fixed per machine model — ATM identity comes from the device-specific upload folder, not the filename.
|
||||
|
||||
The follower recognizes two kinds of content:
|
||||
|
||||
1. **Single-line device events** — lines shaped `[timestamp][level][TCR]EventName,params`. It maps four of these to HiveIQ events:
|
||||
- `CashDispenserStatusChanged` with params `NODISPENSE` → dispenser fault.
|
||||
- `BNAItemsRefused` → dispenser fault (possible note jam / validation failure).
|
||||
- `CashUnitThresholdChanged` → cash-unit low.
|
||||
- `AcceptorStatusChanged` with any non-`OK` status → acceptor self-test failure.
|
||||
- All other event names are ignored.
|
||||
2. **Multi-line TRANSACTION RECORD blocks** — a `level 5` line with no event name opens a block; the module buffers following lines until the next timestamp line, then extracts transaction number, type, result, amount, machine balance, teller ID, branch number and error code into a one-line summary and emits it as a teller-action event.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
| Property | Where | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `nh.tcr.journal.poll.interval.sec` | `hiveops.properties` | `60` | Poll interval in seconds for the ejournal tail loop. |
|
||||
| `device.type` | `hiveops.properties` | *(none — required)* | Must equal `NH_TCR` for the module to activate; any other value → module inert. |
|
||||
|
||||
The journal location and filename come from an **ext properties file** (the standard `ext/ej.properties`), not from `hiveops.properties`:
|
||||
|
||||
| ext key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `type` | *(required)* | Must be `NH_TCR` (preferred) or `MAINJOURNAL` (legacy fallback). |
|
||||
| `journal.dir` | *(required)* | Directory containing the TCR ejournal files. |
|
||||
| `filename.format` | *(none)* | Daily filename template, e.g. `ej_TCR1_{YYYY}{MM}{DD}.txt`. |
|
||||
| `file.format` | `UTF8` | Journal file encoding descriptor read by `JournalSource` (the follower itself decodes the raw bytes as UTF-16 LE). |
|
||||
|
||||
It also reuses the shared agent HTTP settings — `agent.endpoint` (preferred) or `incident.endpoint` (legacy fallback), `agent.auth.token`, and the institution-key chain — the same properties every other event-reporting module uses. There are no TCR-specific auth properties.
|
||||
|
||||
> Prefer `type=NH_TCR` over `MAINJOURNAL` on new installs: the `journal-upload` module uses the same `type` to tag uploads so the server routes them to the TCR parser. `MAINJOURNAL` only exists so pre-existing ATM configs keep working.
|
||||
|
||||
## How it's enabled / deployed
|
||||
|
||||
- It only activates on ATMs configured with `device.type=NH_TCR` **and** a matching ext properties entry (`type=NH_TCR`/`MAINJOURNAL` with `journal.dir` set). Missing either → the module logs a warning and disables itself; it does **not** crash agent startup.
|
||||
- Delivered like any other agent module: uploaded to the fleet artifact store and pushed via an **`INSTALL_MODULE`** fleet task, which downloads the module JAR into the agent's `modules/` directory and restarts the agent (see `ProcessInstallModule` in `hiveops-file-collection`). Standard rollout order applies: lab ATM → pilot device → fleet.
|
||||
- Can be disabled fleet-wide without removing the JAR by adding `nh-tcr-journal` to the comma-separated `modules.disabled` property and pushing a `CONFIG_UPDATE`.
|
||||
- Event types it emits (`OPERATOR_ACTION`, `DISPENSER_JAM`, `CASSETTE_LOW`, `SELF_TEST_FAILED`) are all standard HiveIQ event types shared with other modules and normally already registered in hiveops-incident's Event Types settings.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **No TCR events ever appear for a customer ATM** — first check `device.type` on that ATM's config; if it isn't `NH_TCR`, the module is intentionally inert. Not a bug.
|
||||
- **Module loaded but silent** — confirm the ext properties file exists with `type=NH_TCR` (or `MAINJOURNAL`) and a valid `journal.dir`; a missing/wrong ext entry disables the module at startup with a log warning, no crash. Look for `nh-tcr-journal: monitoring <dir>` in agent logs to confirm what it is tailing.
|
||||
- **Events stopped after midnight / around rollover** — the follower resets its read position on daily filename rollover. If the recycler's filename convention differs from the configured `filename.format`, it will look for a file that never appears and go silent. Check `nh-tcr-journal: switched to <file>` in agent logs.
|
||||
- **Garbled details / no events despite an active journal** — this follower decodes the file as **UTF-16 LE**. If a given customer machine writes the ejournal in a different encoding, line parsing will fail silently. Confirm the raw file encoding before assuming a bug.
|
||||
- **HTTP send failures** — `IncidentEventClient.sendEvent()` failures are caught and logged (`nh-tcr-journal: failed to send ...`) but **not** retried or queued; a transient outage drops that single event rather than backing it up for a later poll.
|
||||
- **A slot/dispenser fault fires once but not repeatedly** — single-line events are emitted every time the matching line appears in the unread tail; there is no in-agent de-dup for these (unlike the block state). High-frequency `CashUnitThresholdChanged` lines can therefore produce repeated `CASSETTE_LOW` events.
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
module: agent.packet-capture
|
||||
title: Packet Capture — on-demand network capture for ATM diagnostics
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · how it's built.** Confirmed against `hiveops-module-packet-capture` source (`com.hiveops.packetcapture`) 2026-07-01. See [technical.agent] for agent-wide context.
|
||||
|
||||
## Maven module
|
||||
|
||||
`hiveops-module-packet-capture` (artifact `hiveops-module-packet-capture`, own version `1.0.0`, parented by `hiveops-agent-parent`). Packaged as a standalone jar (`<finalName>${project.artifactId}</finalName>` → `hiveops-module-packet-capture.jar`), depending only on `hiveops-core` (provided scope) — no dependency on `hiveops-journal`, `hiveops-images`, or `hiveops-file-collection`. It's listed in the parent `pom.xml` `<modules>` alongside the other module-set jars (`hiveops-module-log`, `hiveops-module-hw-inventory`, etc.).
|
||||
|
||||
Registered via ServiceLoader SPI: `src/main/resources/META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.packetcapture.PacketCaptureModule`.
|
||||
|
||||
## Classes and flow
|
||||
|
||||
Three classes, no sub-packages:
|
||||
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `PacketCaptureModule` | `AgentModule` implementation — lifecycle, scheduler, poll loop |
|
||||
| `PacketCaptureHandler` | Executes one `PACKET_CAPTURE` task: parses `configPatch`, runs the tool, uploads the result, reports status |
|
||||
| `PacketCaptureTool` | Platform-specific process runner: picks and drives `tcpdump` (Linux) or `tshark`/`pktmon`/`netsh trace` (Windows, in that preference order) |
|
||||
|
||||
**Poll → detect → capture → upload** cycle (poll-based, not log-tail or hook-based):
|
||||
|
||||
1. `PacketCaptureModule.start()` schedules `poll()` on a single-thread daemon executor every **30s** (`POLL_INTERVAL_SECONDS`).
|
||||
2. `poll()` calls `FileUploadClient.queryCurrentAtmPendingTask(country, atmName, "PACKET_CAPTURE")` — a **kind-filtered** poll (`GET /api/agent/{country}/{atm}/task?kind=PACKET_CAPTURE`), so this module only ever sees tasks of its own kind; it does not compete with `CommandProcessor`'s generic poll for other task kinds.
|
||||
3. If a task is returned, it builds a `PacketCaptureHandler` and calls `execute(task)`. A static `AtomicBoolean captureInProgress` shared across polls ensures only one capture runs at a time per ATM — a second concurrent task is failed immediately with "Another capture is already in progress on this ATM".
|
||||
4. `PacketCaptureHandler.doExecute()`:
|
||||
- Reads `duration` (default 60, clamped to `MAX_DURATION=300`), `interface` (default `"auto"`), `filter` (default empty) from `task.configPatch`.
|
||||
- Probes which capture tool/format will be used (`PacketCaptureTool.detectFormat()`) **before** creating the output file, so the extension (`.pcap` vs `.etl`) matches the actual tool up front.
|
||||
- Reports `RUNNING` via `statusReport`, then runs `PacketCaptureTool.capture()` synchronously, with a progress callback firing roughly every 10s that updates the task's status message with remaining seconds.
|
||||
- On success, uploads the file in `100_000`-byte chunks (`uploadFile()` / `FileUploadClient.fleetManagementUpload`), tracking a server-confirmed offset and retrying a stalled chunk up to 5 times before failing.
|
||||
- Always deletes the local capture file in a `finally` block, whether the task succeeded or failed.
|
||||
5. `PacketCaptureTool.capture()` branches on OS (`System.getProperty("os.name")`):
|
||||
- **Linux:** requires `tcpdump` (`which tcpdump`); throws if missing. Runs `tcpdump -U -q [-i <iface>] -w <file> [filter tokens]` as a foreground process for the requested duration, then force-kills it.
|
||||
- **Windows:** tries `tshark` first (`-a duration:<n> -w <file> [-i <iface>] [-f <filter>]`, produces `.pcap`); falls back to `pktmon start --capture --file-name <file>` / `stop` (produces `.etl`, ignores the BPF filter — captures everything); falls back further to `netsh trace start capture=yes traceFile=<file> maxSize=50` / `stop` (also `.etl`, also ignores filter, and `stop` can take up to 90s to flush).
|
||||
- `abort()` (called from `PacketCaptureModule.stop()` on agent shutdown) force-kills the foreground process and, on Windows, best-effort stops `pktmon`/`netsh trace` too.
|
||||
|
||||
## How events/results reach the platform
|
||||
|
||||
This module does **not** integrate with the journal/incident event pipeline (`IncidentEventClient`, `EventType`) at all — see [technical.agent] "API surface" for that separate path. Its only platform integration is the **fleet task protocol**, shared with every other fleet-task-driven module (`command-processor`, log-collect, etc.). The agent-side calls (verified in `FileUploadClient`, `hiveops-core`) are:
|
||||
|
||||
```
|
||||
PacketCaptureModule / PacketCaptureHandler
|
||||
│ (FileUploadClient, HttpClientSettings built from agent.endpoint/incident.endpoint)
|
||||
▼
|
||||
{endpoint} GET /api/agent/{country}/{atm}/task?kind=PACKET_CAPTURE (poll)
|
||||
POST /api/agent/tasks/{tid}/status (RUNNING/COMPLETED/FAILED + msg)
|
||||
POST /api/agent/tasks/{tid}/upload?o=<base36offset>&f=<enc> (chunked file upload)
|
||||
│ (endpoint = agent.endpoint → agent-proxy in current topology)
|
||||
▼
|
||||
agent-proxy → incident's internal fleet API → fleet task store / Fleet UI
|
||||
(downstream controller name + internal path unverified, cross-repo)
|
||||
```
|
||||
|
||||
- `HttpClientSettings incidentSettings` is built once in `initialize()` via `HttpClientHelper.loadSettingsFromProperties(props, prefix)`, where `prefix` is `"agent"` if `agent.endpoint` is set, else `"incident"`. This is the same pattern used by other modules that need to talk to the fleet/incident base URL.
|
||||
- Endpoints hit (all via `FileUploadClient`, defined in `hiveops-core`):
|
||||
- `GET {endpoint}/api/agent/{country}/{atm}/task?kind=PACKET_CAPTURE` — poll
|
||||
- `POST {endpoint}/api/agent/tasks/{tid}/status` — RUNNING/COMPLETED/FAILED + message
|
||||
- `POST {endpoint}/api/agent/tasks/{tid}/upload?o=...&f=...` — chunked capture-file upload
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
The module does **not** use `com.hiveops.platform.PlatformServiceFactory` / `PlatformService` at all — it does its own OS detection (`PacketCaptureTool.isWindows()` checking `os.name`) and shells out directly via `ProcessBuilder`, rather than going through the shared `SystemCommands` abstraction other modules (e.g. `file-collection`'s reboot/patch handling) use. This is a deliberate/simple design: the module only ever needs to run one of four well-known external tools and doesn't need registry/config-file reads or path resolution.
|
||||
|
||||
## Coordination with the legacy command-processor
|
||||
|
||||
`CommandProcessor` (in `hiveops-file-collection`) also long-polls the server for pending tasks (unfiltered by kind). When its poll loop encounters a `PACKET_CAPTURE` task, it explicitly does **not** process it — it reports the task back to `PENDING` ("Released to packet-capture module") and sleeps, leaving it for this module's kind-filtered poll to pick up. This avoids two different modules racing to claim the same task.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
module: agent.packet-capture
|
||||
title: Packet Capture — on-demand network capture for ATM diagnostics
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-packet-capture` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Module name (`AgentModule.getName()`): **`packet-capture`**
|
||||
- Maven artifact: **`hiveops-module-packet-capture`** (own version `1.0.0`; parent `hiveops-agent-parent` currently `4.4.3`)
|
||||
- Entry class: `com.hiveops.packetcapture.PacketCaptureModule` (SPI file: `META-INF/services/com.hiveops.core.module.AgentModule`)
|
||||
- Other classes: `PacketCaptureHandler` (task execution), `PacketCaptureTool` (OS-specific capture runner)
|
||||
- Log package (`getLogPackage()`): `com.hiveops.packetcapture` → `logs/modules/packet-capture.log`
|
||||
- Fleet task kind it implements: **`PACKET_CAPTURE`** (exact string, passed to `queryCurrentAtmPendingTask(country, atm, "PACKET_CAPTURE")`; the matching `FleetTaskKind` enum lives in `hiveops-fleet` — *(unverified, cross-repo)*)
|
||||
|
||||
## Config properties
|
||||
- **No dedicated `packet-capture.*` keys exist in code.** Reuses whichever is set:
|
||||
- `agent.endpoint` (preferred) or `incident.endpoint` (fallback) — HTTP base URL
|
||||
- `agent.auth.token` / `agent.institution.key` — auth (shared across modules)
|
||||
- `modules.disabled` — comma list; include `packet-capture` to disable
|
||||
- `isEnabled()` = true whenever `agent.endpoint` OR `incident.endpoint` is non-blank (on by default).
|
||||
- Hardcoded (not configurable): `POLL_INTERVAL_SECONDS = 30`, `MAX_DURATION = 300` (seconds, hard cap), `SEGMENT_SIZE = 100_000` (upload chunk bytes).
|
||||
|
||||
## Fleet task `configPatch` keys (set by the operator in Fleet UI)
|
||||
| Key | Default if absent | Notes |
|
||||
|---|---|---|
|
||||
| `duration` | `60` | seconds; clamped **agent-side** to ≤300 (`Math.min(parsed, MAX_DURATION)`) |
|
||||
| `interface` | `"auto"` | blank or `"auto"` → OS default interface |
|
||||
| `filter` | `""` | BPF filter string, e.g. `tcp port 443`; **ignored** by `pktmon`/`netsh` Windows fallbacks |
|
||||
|
||||
## Event / status strings emitted
|
||||
- **No `EventType` / `IAT_*` / journal-incident events.** This module never calls `IncidentEventClient`.
|
||||
- Only emits **fleet task status** via `FileUploadClient.statusReport(tid, aid, status, message)`: `"RUNNING"`, `"COMPLETED"`, `"FAILED"` (plus free-text progress messages while running).
|
||||
|
||||
## HTTP endpoints used (agent → agent-proxy)
|
||||
| Call | Path |
|
||||
|---|---|
|
||||
| Poll (kind-filtered) | `GET {endpoint}/api/agent/{country}/{atm}/task?kind=PACKET_CAPTURE` |
|
||||
| Status report | `POST {endpoint}/api/agent/tasks/{tid}/status` |
|
||||
| Upload chunk | `POST {endpoint}/api/agent/tasks/{tid}/upload?o=<base36offset>&f=<filename>` |
|
||||
|
||||
Base URL = `agent.endpoint` (preferred) or `incident.endpoint` (fallback), from `HttpClientHelper.loadSettingsFromProperties(props, prefix)`. In current topology `agent.endpoint` points at `agent-proxy`, which forwards to incident's internal fleet API — *(agent-proxy controller/internal path unverified, cross-repo)*. Same transport as every other fleet-task module.
|
||||
|
||||
## Capture tool selection (in order tried)
|
||||
- **Linux:** `tcpdump` only — no fallback; module throws if `tcpdump` isn't installed. Output: `.pcap`.
|
||||
- **Windows:** `tshark` → `.pcap` (supports `-f <BPF filter>`) → else `pktmon` → `.etl` (no filter support) → else `netsh trace` → `.etl` (no filter support, `stop` can take up to 90s).
|
||||
- Output file extension is decided by `PacketCaptureTool.detectFormat()` **before** the real capture file is created, so it matches the tool that will actually run.
|
||||
|
||||
## Concurrency / lifecycle
|
||||
- `AtomicBoolean captureInProgress` (static in `PacketCaptureModule`, shared with `PacketCaptureHandler`) allows only **one capture per ATM at a time**. A second concurrent `PACKET_CAPTURE` task fails immediately: "Another capture is already in progress on this ATM".
|
||||
- Capture file is **always deleted** locally after upload (success or failure) — never left on ATM disk.
|
||||
- `stop()` (agent shutdown) calls `PacketCaptureHandler.abort()` → `PacketCaptureTool.abort()`, force-killing the running process (and best-effort `pktmon stop`/`netsh trace stop` on Windows).
|
||||
|
||||
## Deployment
|
||||
- Ships in the **`atm`** module set of `deployment/build-dist.sh` (default), NOT the `srv` (server-monitor) set.
|
||||
- Delivered as drop-in JAR `modules/hiveops-module-packet-capture.jar` — not bundled in the fat JAR.
|
||||
- Retrofit to an ATM missing it: `INSTALL_MODULE` fleet task, or a full standard-set agent update.
|
||||
|
||||
## Gotchas
|
||||
- **Legacy `command-processor` explicitly ignores `PACKET_CAPTURE`:** when its own unfiltered poll sees a `PACKET_CAPTURE` task, it reports it back to `PENDING` ("Released to packet-capture module") and sleeps 35s — it never executes the capture itself. If tasks never leave PENDING, check that the `packet-capture` module (not just `command-processor`) is actually running, not that command-processor is "stuck."
|
||||
- Requesting `duration` > 300 does **not** error — it silently clamps to 300 with no warning back to Fleet.
|
||||
- Windows without `tshark` installed silently degrades to `.etl` capture with **no BPF filter applied** — a requested `filter` is quietly ignored, not rejected.
|
||||
- No `PlatformService`/`SystemCommands` usage here — OS branching is done locally via `os.name` + raw `ProcessBuilder`, unlike most other modules.
|
||||
- This module has zero config-property surface of its own; don't go looking for `packet-capture.*` in `hiveops.properties` — there isn't any.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT treat this as a continuous/passive network monitor — it only runs when a `PACKET_CAPTURE` fleet task is queued.
|
||||
- Do NOT expect journal/incident events (`IAT_*` or any `EventType`) from this module — it reports fleet task status only.
|
||||
- Do NOT assume a BPF `filter` works on Windows unless `tshark` is confirmed installed — `pktmon`/`netsh` ignore it silently.
|
||||
- Do NOT invent a `packet-capture.*` properties namespace — none exists; config comes from `agent.endpoint`/`incident.endpoint`/`agent.auth.token`/`modules.disabled` only.
|
||||
- Do NOT expect the captured file to persist on the ATM after task completion — it's deleted immediately after upload, success or failure.
|
||||
- Do NOT add this module to the `srv` (server-monitor) build set without checking with the team — it's ATM-set only today.
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
module: agent.packet-capture
|
||||
title: Packet Capture — on-demand network capture for ATM diagnostics
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-packet-capture` source 2026-07-01.
|
||||
|
||||
## What it does (and doesn't)
|
||||
|
||||
`hiveops-module-packet-capture` (module name `packet-capture`, class `PacketCaptureModule`) is **not** a passive monitor — it is a **fleet-task executor**. It sits idle polling for work and only runs `tcpdump`/`tshark`/`pktmon`/`netsh trace` when a `PACKET_CAPTURE` fleet task is queued for that ATM from **Fleet → Tasks** (`fleet.bcos.cloud`). There is no continuous packet monitoring, no baseline, no alerting from this module — it captures for a bounded window, uploads the file, and stops.
|
||||
|
||||
Flow at a glance:
|
||||
1. Operator creates a `PACKET_CAPTURE` fleet task for one or more devices, with `duration` (seconds), `interface` (or "auto"), and an optional BPF `filter` (e.g. `tcp port 443`).
|
||||
2. The agent's `packet-capture` module polls every **30s** for a pending task of kind `PACKET_CAPTURE`.
|
||||
3. It runs the platform-appropriate capture tool for the requested duration (capped at **300s / 5 minutes** even if a larger value is requested).
|
||||
4. The resulting file (`.pcap` or `.etl`) is uploaded back to the fleet task in chunks, then deleted from the ATM's local disk.
|
||||
5. Fleet shows the task COMPLETED with a downloadable capture file (or FAILED with a reason).
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
**None.** Unlike most agent modules there is no `packet-capture.*` properties namespace. The module reuses whichever of these is already configured:
|
||||
|
||||
| Key | Used for |
|
||||
|-----|----------|
|
||||
| `agent.endpoint` | Preferred base URL for polling tasks / uploading the capture (module picks this prefix first) |
|
||||
| `incident.endpoint` | Fallback base URL if `agent.endpoint` is not set (older ATMs) |
|
||||
| `agent.auth.token` / `agent.institution.key` | Auth for the same HTTP calls (shared with every other module) |
|
||||
|
||||
There is no way to tune poll interval, max duration, or upload chunk size via properties — they are hardcoded (`POLL_INTERVAL_SECONDS = 30`, `MAX_DURATION = 300`, `SEGMENT_SIZE = 100_000` bytes) in `PacketCaptureModule` / `PacketCaptureHandler`. If Fleet requests more than 300s, the agent silently clamps it — the task will still say "300s" in its running status, not the requested value.
|
||||
|
||||
## Enabled / deployed to ATMs
|
||||
|
||||
- **Enable condition:** the module self-enables whenever `agent.endpoint` **or** `incident.endpoint` is set — i.e. it's on by default on any normally-configured ATM. To turn it off for a specific ATM/institution, push `modules.disabled=packet-capture` (comma-separated list) via a `CONFIG_UPDATE` fleet task.
|
||||
- **Shipped how:** it's one of the standard ATM module set built by `deployment/build-dist.sh` (module set `atm`, the default) — it ships in every standard ATM install/patch ZIP as `modules/hiveops-module-packet-capture.jar`, a drop-in JAR (not baked into the fat JAR). It is **excluded** from the `srv` module set (server-monitoring-only installs).
|
||||
- **Retrofitting an ATM that doesn't have it yet:** deliver via a fleet `INSTALL_MODULE` task (drops the JAR into `modules/` and restarts the agent), or via a full agent update that includes the current module set.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---|---|---|
|
||||
| Task stays PENDING forever | Agent's `packet-capture` module isn't running, or was disabled via `modules.disabled` | Confirm the module started in `logs/modules/packet-capture.log`; check `modules.disabled` on that ATM |
|
||||
| Task fails immediately: "Another capture is already in progress" | A second `PACKET_CAPTURE` task landed on the same ATM while one was still running | Only one capture runs per ATM at a time (`captureInProgress` guard). Wait for the first to finish or cancel it |
|
||||
| Task fails: "Capture produced no output — capture tool may require elevated privileges" | `tcpdump`/capture tool isn't allowed to open the interface (not running as root/Administrator, or interface doesn't exist) | Check the agent service's OS-level privileges; on Linux confirm `tcpdump` is installed (`apt-get install tcpdump`) |
|
||||
| Task fails: "tcpdump is not installed" | Linux ATM is missing `tcpdump` | Install it; there's no fallback tool on Linux (Windows has tshark → pktmon → netsh) |
|
||||
| Windows capture uses `.etl` instead of `.pcap` | `tshark` isn't installed/on PATH, so the agent fell back to `pktmon` or `netsh trace` (neither supports a BPF filter — captures everything) | Install Wireshark/tshark on the ATM if a filtered `.pcap` capture is required |
|
||||
| Upload never completes / times out | Network issue mid-upload, or capture file very large for a slow link | Handler retries a stalled chunk up to 5 times before failing; check ATM network health and capture duration/size |
|
||||
| Task shows COMPLETED but capture file missing locally | Expected — the local file is always deleted after upload (success or failure) to avoid leaving PCAPs (which may include unmasked traffic) on the ATM disk | Download the file from the Fleet task record, not the ATM |
|
||||
| A different fleet task type (e.g. `LOG_COLLECT`) seems to interfere | The legacy `command-processor` module explicitly releases any `PACKET_CAPTURE` task it sees back to `PENDING` (with message "Released to packet-capture module") so the dedicated module can claim it — this is expected coordination, not a bug | If capture tasks never move past PENDING, confirm the `packet-capture` module (not just `command-processor`) is actually running |
|
||||
|
||||
## Not applicable here
|
||||
|
||||
This module has no journal/incident event integration — it does not emit `IAT_*` or any `EventType` events. Its only "reporting" is fleet task status (`RUNNING` / `COMPLETED` / `FAILED`) via the same status-report endpoint every fleet task uses.
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
module: agent.remote
|
||||
title: Remote Control Module — how it's built
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Architect · dev view.** Grounded in `hiveops-module-remote` source (parent v4.4.3, module `pom` version 1.0.0) on 2026-07-01. See [technical.agent] for the shared module system this plugs into.
|
||||
|
||||
## Shape
|
||||
|
||||
A standard agent module (SPI `com.hiveops.core.module.AgentModule`, registered in `META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.remote.module.RemoteModule`). Lifecycle is the platform default: `getDefaultConfiguration()` → `isEnabled()` → `initialize()` → `start()` → `stop()`. Unlike most modules it is a **command executor**, not a monitor: there is no log tail, no journal parse, and no incident/journal event emission. It drives a request/response loop against a separate server.
|
||||
|
||||
## Classes and flow
|
||||
|
||||
```
|
||||
RemoteModule (SPI entry: config defaults, enable gate, wiring, lifecycle)
|
||||
└─ RemoteCommandProcessor (register → poll loop → dispatch → status/upload)
|
||||
├─ RemoteHttpClient (HttpURLConnection + Jackson; Bearer auth)
|
||||
├─ ScreenCaptureService (AWT Robot → JPEG)
|
||||
└─ FileSystemService (dir list / file open, guarded by PathSecurityUtils)
|
||||
common.dto.* (PollResponse, CommandType, CommandStatus, CommandStatusUpdate,
|
||||
AgentRegistrationRequest/Response, FileEntry, ...)
|
||||
common.security.PathSecurityUtils
|
||||
```
|
||||
|
||||
**`RemoteModule`** — `initialize()` reads `remote.server.url` (required; throws `ModuleInitializationException` if blank), derives `machineId` from `remote.machine.id` else the platform hostname (`PlatformService.getSystemInfoProvider().getHostname()`) else a random UUID, and constructs the HTTP client + services. Advertised `capabilities` are hardcoded to `["SCREENSHOT","FILE_LIST","FILE_DOWNLOAD"]`. `isEnabled()` gates on `remote.enabled`.
|
||||
|
||||
**`RemoteCommandProcessor`** — owns a 2-thread `ScheduledExecutorService`. On `start()`:
|
||||
1. `registerWithServer()` — builds `AgentRegistrationRequest` (name = `context.getAtmName()`, hostname, `os.name`, agent version from `hiveops.version`, module version `1.0.0-SNAPSHOT`, machineId, capabilities), `POST`s it, and stores the returned Bearer token. **Registration errors are caught and logged, not retried** — a failed register leaves the token null and every subsequent poll 401s.
|
||||
2. `scheduleWithFixedDelay(pollAndProcess, 0, remote.poll.interval.ms)`.
|
||||
|
||||
Each `pollAndProcess()` calls `GET /api/v1/agents/poll`; if `PollResponse.hasCommand`, it dispatches on `CommandType`:
|
||||
- `SCREENSHOT` → `ScreenCaptureService.capture(quality, displayIndex)` → `uploadBinaryResult`.
|
||||
- `FILE_LIST` → `FileSystemService.listDirectory(path)` (defaults to `user.home` if blank) → status `COMPLETED` with a JSON `{path, entries}` result (no binary).
|
||||
- `FILE_DOWNLOAD` → `FileSystemService.downloadFile(path)` → read all bytes → `uploadBinaryResult`.
|
||||
- default (incl. `SYSTEM_INFO`) → `FAILED: Unknown command type`.
|
||||
|
||||
Status transitions use `CommandStatusUpdate` factories (`inProgress()`, `completed(result)`, `failed(msg)`, `completedWithBinary(totalChunks)`) mapping to `CommandStatus` `IN_PROGRESS` / `COMPLETED` / `FAILED`. The processor sets `IN_PROGRESS` before running and reports `FAILED` on any exception (with a nested try so a failed status-report doesn't crash the loop).
|
||||
|
||||
**`uploadBinaryResult`** — splits `byte[]` into `remote.chunk.size.bytes` (1 MiB) chunks, computes a **SHA-256** hex checksum per chunk, and posts each as multipart `POST /api/v1/agents/commands/{id}/upload`. Empty results still send 1 chunk. No retry on chunk failure.
|
||||
|
||||
**`ScreenCaptureService`** — cross-platform via `java.awt.Robot`. Detects headless at construction; `capture()` throws `UnsupportedOperationException` in headless mode. JPEG quality is `quality/100f` or default `0.85`. `displayIndex` selects a `GraphicsDevice`; multi-monitor defaults to device 0. Alpha images are flattened to `TYPE_INT_RGB` before JPEG encode.
|
||||
|
||||
**`FileSystemService`** — enforces, in order: `PathSecurityUtils.validatePath` (null/empty, null-byte, `..` traversal), `validateWithinAllowedPaths` (only when `remote.allowed.paths` is set), and `isPathBlocked` against `remote.blocked.paths` (OS-specific defaults if unset, `*` wildcards). Then existence/type/readable checks; `downloadFile` also enforces `remote.max.file.size.bytes`. `FileEntry` carries name/size/mtime/type (`DIRECTORY`/`FILE`/`SYMLINK`/`OTHER`)/readable/writable and POSIX perm string (null on Windows). Symlinks are read `NOFOLLOW_LINKS`.
|
||||
|
||||
## How results reach the platform
|
||||
|
||||
This module does **not** use the standard agent → agent-proxy/incident journal-event path. It speaks a dedicated REST protocol to the **hiveops-remote** server (`RemoteHttpClient`, base `remote.server.url`, prod path `/api/remote/`, internal `:8090`):
|
||||
|
||||
| Call | Method + path | Auth |
|
||||
|---|---|---|
|
||||
| Register | `POST /api/v1/agents/register` | none (returns Bearer token) |
|
||||
| Poll for command | `GET /api/v1/agents/poll` | Bearer |
|
||||
| Report status | `POST /api/v1/agents/commands/{id}/status` | Bearer |
|
||||
| Upload binary chunk | `POST /api/v1/agents/commands/{id}/upload` (multipart: `chunkIndex`, `totalChunks`, `checksum`, `data`) | Bearer |
|
||||
| Heartbeat | `POST /api/v1/agents/heartbeat` | Bearer |
|
||||
|
||||
`RemoteHttpClient` is plain `HttpURLConnection` + Jackson (`FAIL_ON_UNKNOWN_PROPERTIES=false`, `JavaTimeModule`); any `>=400` throws `IOException` with the error body. Note: `heartbeat()` is implemented but **never invoked** by the processor — liveness is inferred from polling.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
Minimal: only `PlatformService.getSystemInfoProvider().getHostname()` for the machine-id fallback. Screen capture and file access use portable JVM APIs (AWT Robot, `java.nio.file`) rather than the agent's `SystemCommands`/`PathResolver`, so behavior differs by environment (headless, POSIX vs Windows perms) rather than by an explicit platform impl. Delivery/restart on `INSTALL_MODULE`, by contrast, does go through the platform layer (`ProcessInstallModule` → `SystemCommands.restartService`).
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
module: agent.remote
|
||||
title: Remote Control Module — Claude quick reference
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> Terse, act-without-guessing reference. Everything below is confirmed in `hiveops-module-remote` source (parent v4.4.3) unless marked `(unverified)`.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Module name (SPI `getName()` + JAR `Module-Name`):** `hiveops-remote`
|
||||
- **SPI class:** `com.hiveops.remote.module.RemoteModule`
|
||||
- **Maven artifact:** `com.hiveops:hiveops-module-remote` (pom version `1.0.0`)
|
||||
- **Deployed JAR:** `modules/hiveops-module-remote.jar` (drop-in; Jackson shaded in, hiveops-core `provided`)
|
||||
- **`getVersion()` returns:** `1.0.0-SNAPSHOT` (also the hardcoded `moduleVersion` sent at registration — mismatched with pom `1.0.0`)
|
||||
|
||||
## Config keys (all `remote.*`)
|
||||
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `remote.enabled` | `false` | Must be `true` or module won't load |
|
||||
| `remote.server.url` | `http://localhost:8090` | Required; init throws if blank. Prod = hiveops-remote (`/api/remote/`, `:8090`) |
|
||||
| `remote.poll.interval.ms` | `5000` | Poll delay |
|
||||
| `remote.poll.timeout.ms` | `15000` | Declared default but **unused in code** |
|
||||
| `remote.chunk.size.bytes` | `1048576` | 1 MiB upload chunks |
|
||||
| `remote.max.file.size.bytes` | `104857600` | 100 MiB cap on FILE_DOWNLOAD |
|
||||
| `remote.machine.id` | *(not in defaults)* → hostname → random UUID | Registration identity |
|
||||
| `remote.allowed.paths` | *(not in defaults; empty = allow all)* | Comma/semicolon list; allowlist base dirs |
|
||||
| `remote.blocked.paths` | *(not in defaults; OS default block list)* | Comma/semicolon list, `*` wildcard |
|
||||
|
||||
## Command types (`CommandType` enum)
|
||||
|
||||
- Advertised capabilities / handled: `SCREENSHOT`, `FILE_LIST`, `FILE_DOWNLOAD`
|
||||
- In enum but **NOT** advertised and **NOT** handled → replies `FAILED: Unknown command type`: `SYSTEM_INFO`
|
||||
|
||||
## Command status (`CommandStatus`)
|
||||
|
||||
- `IN_PROGRESS` (set before execution), `COMPLETED`, `FAILED`
|
||||
- Factory helpers: `CommandStatusUpdate.inProgress() / completed(result) / completedWithBinary(totalChunks) / failed(msg)`
|
||||
|
||||
## Server protocol (base = `remote.server.url`)
|
||||
|
||||
| Purpose | Method + path | Auth |
|
||||
|---|---|---|
|
||||
| Register | `POST /api/v1/agents/register` | none → returns Bearer token |
|
||||
| Poll | `GET /api/v1/agents/poll` | Bearer |
|
||||
| Status | `POST /api/v1/agents/commands/{id}/status` | Bearer |
|
||||
| Upload chunk | `POST /api/v1/agents/commands/{id}/upload` (multipart `chunkIndex`,`totalChunks`,`checksum`,`data`) | Bearer |
|
||||
| Heartbeat | `POST /api/v1/agents/heartbeat` | Bearer (**method exists, never called**) |
|
||||
|
||||
- Upload checksum = **SHA-256** hex, per chunk.
|
||||
- Screenshot = AWT Robot → JPEG (default quality `0.85`).
|
||||
|
||||
## Deploy / enable
|
||||
|
||||
- Enable: `CONFIG_UPDATE` fleet task → `remote.enabled=true` (+ `remote.server.url`, and `remote.allowed.paths` on locked-down ATMs).
|
||||
- Retrofit JAR: `INSTALL_MODULE` fleet task (filename must end `.jar`; drops into `modules/`, restarts agent).
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Emits **NO** incident/journal events — no `IAT_*`, no `EventType`. Reporting is only remote-command status/binary to hiveops-remote.
|
||||
- Talks to **hiveops-remote** (port 8090 / `/api/remote/`), **not** agent-proxy or hiveops-incident.
|
||||
- Registration failure is logged, **not retried** → token stays null → polls 401 until agent restart.
|
||||
- `heartbeat()` is dead code (never invoked).
|
||||
- `remote.poll.timeout.ms` has no effect (declared, unreferenced).
|
||||
- SCREENSHOT throws `UnsupportedOperationException` on headless JVMs.
|
||||
- Binary uploads have **no retry**; a failed chunk → command `FAILED`.
|
||||
- `getVersion()`/registration report `1.0.0-SNAPSHOT` while the pom is `1.0.0`.
|
||||
|
||||
## Do NOT
|
||||
|
||||
- Do NOT expect `remote.enabled` on by default — it is `false`.
|
||||
- Do NOT expect it to route through agent-proxy/incident or appear in the journal-event pipeline.
|
||||
- Do NOT assume `SYSTEM_INFO` works — it is unhandled.
|
||||
- Do NOT rely on `remote.allowed.paths`/`remote.machine.id`/`remote.blocked.paths` being set from defaults — they aren't in `getDefaultConfiguration()`; set them explicitly.
|
||||
- Do NOT invent event strings — this module has none.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
module: agent.remote
|
||||
title: Remote Control Module — on-demand screen capture & file access on ATMs
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-remote` source (`RemoteModule`, agent parent v4.4.3) on 2026-07-01.
|
||||
|
||||
## What it does (and doesn't)
|
||||
|
||||
`hiveops-module-remote` (module name `hiveops-remote`, class `com.hiveops.remote.module.RemoteModule`) is **not** a monitor and does **not** emit incident/journal events. It is an **on-demand remote-control executor**. It sits idle, registers with the **HiveIQ Remote server** (`hiveops-remote`, internal port 8090), long-polls it for a single queued command, executes it, and uploads the binary result back. There is no continuous monitoring, no baseline, and no `IAT_*` / `EventType` reporting from this module.
|
||||
|
||||
Supported commands (the module's advertised `capabilities`):
|
||||
|
||||
| Capability | What the agent does |
|
||||
|---|---|
|
||||
| `SCREENSHOT` | Captures the ATM's display via AWT `Robot`, encodes JPEG, uploads bytes in chunks |
|
||||
| `FILE_LIST` | Lists a directory (name, size, mtime, type, POSIX perms) and returns JSON |
|
||||
| `FILE_DOWNLOAD` | Streams a single file's bytes back in chunks |
|
||||
|
||||
`SYSTEM_INFO` exists in the `CommandType` enum but is **not** advertised as a capability and is **not** handled — if the server ever sends it the agent replies `FAILED: Unknown command type`.
|
||||
|
||||
Flow at a glance:
|
||||
1. On `start()`, the module `POST`s to `/api/v1/agents/register` on the remote server and stores the returned Bearer token.
|
||||
2. It polls `GET /api/v1/agents/poll` every **5s** (`remote.poll.interval.ms`).
|
||||
3. When `hasCommand=true`, it sets the command `IN_PROGRESS`, runs it, then reports `COMPLETED` (with result/binary) or `FAILED`.
|
||||
4. Binary results (screenshot JPEG, file bytes) are uploaded in **1 MiB** chunks with a per-chunk SHA-256 checksum to `POST /api/v1/agents/commands/{id}/upload`.
|
||||
|
||||
> This module talks to the **hiveops-remote** service (`/api/remote/`, port 8090) — **not** to agent-proxy or hiveops-incident. It is a separate path from the standard journal/event pipeline.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
Registered in `getDefaultConfiguration()` (namespace `remote.*`):
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `remote.enabled` | `false` | Master on/off. Module only loads when `true`. |
|
||||
| `remote.server.url` | `http://localhost:8090` | Remote-server base URL. **Required** — init throws if blank. |
|
||||
| `remote.poll.interval.ms` | `5000` | Delay between polls for a queued command. |
|
||||
| `remote.poll.timeout.ms` | `15000` | Registered default, but **not referenced anywhere in code** (no effect — unverified whether wired in a newer build). |
|
||||
| `remote.chunk.size.bytes` | `1048576` | Upload chunk size (1 MiB) for binary results. |
|
||||
| `remote.max.file.size.bytes` | `104857600` | Max file size (100 MiB) for `FILE_DOWNLOAD`; larger files fail. |
|
||||
|
||||
Also read at init but **not** in the defaults set (must be added explicitly to take effect):
|
||||
|
||||
| Key | Default if unset | Meaning |
|
||||
|---|---|---|
|
||||
| `remote.machine.id` | derived from hostname (then random UUID) | Stable machine identity sent at registration. |
|
||||
| `remote.allowed.paths` | *(empty = all paths allowed)* | Comma/semicolon list; if set, file ops are restricted to these base dirs. |
|
||||
| `remote.blocked.paths` | OS default block list | Comma/semicolon list of blocked path patterns (`*` wildcard). Defaults block `/etc/shadow`, `/etc/passwd`, `/etc/sudoers`, `/root/.ssh`, `/home/*/.ssh` (Linux) and `C:\Windows\System32\config`, `...\drivers`, credential stores (Windows). |
|
||||
|
||||
## Enabled / deployed to ATMs
|
||||
|
||||
- **Enable condition:** module self-enables only when `remote.enabled=true`. It is **off by default**. Turn it on for a device/institution with a `CONFIG_UPDATE` fleet task setting `remote.enabled=true` (and, on locked-down ATMs, `remote.allowed.paths`).
|
||||
- **Shipped how:** it's a drop-in JAR (`modules/hiveops-module-remote.jar`), not baked into the fat JAR. Jackson is shaded into the JAR (hiveops-core is `provided`).
|
||||
- **Retrofitting an ATM that doesn't have the JAR:** deliver via a fleet `INSTALL_MODULE` task — it drops the JAR into `modules/` and restarts the agent (`ProcessInstallModule`). The filename must end in `.jar`.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---|---|---|
|
||||
| Module never registers / no commands run | `remote.enabled` not `true`, or module JAR absent | Confirm `remote.enabled=true` and that `modules/hiveops-module-remote.jar` is present; check agent log for `Initializing remote module for server: ...` |
|
||||
| Init fails: `remote.server.url is required` | Property blank/missing | Set `remote.server.url` to the remote server (prod path is via `/api/remote/`, internal `:8090`). |
|
||||
| Registration logs `Failed to register with remote server` | Server unreachable / auth rejected | Registration errors are swallowed (no retry loop) — the module keeps polling with a null token, so polls 401. Verify network path to the remote server and that it accepts the registration payload. |
|
||||
| `SCREENSHOT` fails: `Screen capture not available in headless mode` | ATM/JVM is headless (no X display / service session) | Screen capture needs a real graphics environment + AWT `Robot`. Headless Linux boxes and some Windows service sessions can't capture. |
|
||||
| `FILE_DOWNLOAD` fails: `File exceeds maximum size limit` | File > `remote.max.file.size.bytes` (100 MiB default) | Raise the limit via config, or collect large files with a fleet `FILECOLLECT`/`LOG_COLLECT` task instead. |
|
||||
| File op fails: `Access to path is blocked` / `Path is not within allowed directories` | Hit a `remote.blocked.paths` pattern, or path is outside `remote.allowed.paths` | Expected security behavior — adjust the allow/block lists if the target should be reachable. |
|
||||
| File op fails: `Path contains directory traversal sequences` / `null bytes` | Path contained `..` or `\0` | Path-traversal guard (`PathSecurityUtils`) — send a normalized absolute path. |
|
||||
| Upload fails mid-transfer | Network drop, or server rejected a chunk (checksum/size) | Uploads are **not** retried by this module — the command goes `FAILED`. Re-issue the command. |
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
module: agent.srv-monitor
|
||||
title: Server Monitor — CPU/memory/disk/service health watcher for SRV devices
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · how it's built.** Confirmed against `hiveops-module-srv-monitor` source (`com.hiveops.srvmonitor`) 2026-07-01. See [technical.agent] for agent-wide module-system and event-pipeline context.
|
||||
|
||||
## Maven module
|
||||
|
||||
`hiveops-module-srv-monitor` (artifact `hiveops-module-srv-monitor`, own version `1.0.0`, parented by `hiveops-agent-parent` — currently `4.4.3`). Packaged as a standalone drop-in jar (`<finalName>${project.artifactId}</finalName>` → `hiveops-module-srv-monitor.jar`).
|
||||
|
||||
Dependencies (all **provided** scope — supplied by the running agent, not fat-packed):
|
||||
- `hiveops-core` — the `AgentModule` SPI, `ModuleContext`, `PlatformService`, HTTP helpers.
|
||||
- `hiveops-agent-journal` — the `IncidentEventClient` and `CreateJournalEventRequest` DTO (the shared journal-events pipeline).
|
||||
- `log4j-api`, `commons-lang3` — logging + `StringUtils`/`NumberUtils` parsing.
|
||||
|
||||
Registered via ServiceLoader SPI: `src/main/resources/META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.srvmonitor.ServerMonitorModule`.
|
||||
|
||||
## Classes and flow
|
||||
|
||||
Five source classes, no sub-packages:
|
||||
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `ServerMonitorModule` | `AgentModule` implementation — lifecycle, config, scheduler, transition/threshold logic, event build+send |
|
||||
| `ServerMonitorCollector` | Interface: `ServerMetrics collect(List<String> monitoredServices)` |
|
||||
| `LinuxServerMonitorCollector` | Linux implementation — `/proc/stat`, `/proc/meminfo`, `File.listRoots()`, `systemctl is-active` |
|
||||
| `WindowsServerMonitorCollector` | Windows implementation — PowerShell (`Get-Counter`, CIM `Win32_*`, `Get-Service`) |
|
||||
| `ServerMetrics` | Immutable value object: `cpuPercent`, `memoryPercent`, `List<DiskUsage>`, `Map<String,Boolean> serviceStatus` |
|
||||
|
||||
### Lifecycle (`AgentModule`)
|
||||
|
||||
`initialize()` → `isEnabled()` → `start()` → `stop()`, the standard module lifecycle from [technical.agent].
|
||||
|
||||
- **`initialize(ModuleContext)`** — reads `srv.monitor.enabled`; if false, logs and returns with `ready=false`. Resolves the endpoint (`agent.endpoint` preferred, else `incident.endpoint`); if neither, logs "no endpoint configured" and stays inactive. Builds `HttpClientSettings` via `HttpClientHelper.loadSettingsFromProperties(props, prefix)` where `prefix` is `"agent"` or `"incident"` to match the chosen endpoint. Parses thresholds/interval/samples, splits `srv.monitor.services` on commas. Constructs the `IncidentEventClient` (with `atmName` + `country` from the context) and selects the collector via **`context.getPlatformService().isWindows()`**. Sets `ready=true`.
|
||||
- **`isEnabled()`** returns `ready` — so a module that failed the enable gate or endpoint check reports disabled and is never started by the registry.
|
||||
- **`start()`** creates a single daemon-thread `ScheduledExecutorService` (thread name `srv-monitor`) and `scheduleAtFixedRate(this::poll, 0, intervalSeconds, SECONDS)` — first poll immediately, then every interval.
|
||||
- **`stop()`** `shutdownNow()` on the scheduler. **`isHealthy()`** = scheduler exists and isn't terminated.
|
||||
- **`getDependencies()`** is empty — no ordering constraints against other modules. **`getLogPackage()`** = `com.hiveops.srvmonitor` (routes to `logs/modules/srv-monitor.log`).
|
||||
|
||||
### Poll → detect → emit
|
||||
|
||||
`poll()` (the fixed-rate task) is a pure poll loop — no log tailing, no hooks:
|
||||
|
||||
1. `collector.collect(monitoredServices)` gathers one `ServerMetrics` snapshot.
|
||||
2. Four checkers each append zero or one `CreateJournalEventRequest` to a batch list, based on **state transition** against the checker's stored last-state:
|
||||
- `checkCpu` — increments `cpuHighCount` while at/above threshold; emits `SERVER_CPU_HIGH` only when `!cpuHighState && cpuHighCount >= sustainedSamples`. Dropping below threshold resets the counter and, if it was high, emits `SERVER_CPU_NORMAL`.
|
||||
- `checkMemory` — emits `SERVER_MEMORY_HIGH`/`SERVER_MEMORY_NORMAL` on the `memHighState` boolean flip (no debounce).
|
||||
- `checkDisks` — per-mount `Map<String,Boolean> diskHighState`; emits `SERVER_DISK_LOW`/`SERVER_DISK_NORMAL` per filesystem on flip.
|
||||
- `checkServices` — per-service `Map<String,Boolean> serviceDownState`; emits `SERVICE_DOWN`/`SERVICE_UP` on flip.
|
||||
3. If the batch is non-empty, `client.sendEvents(events)`.
|
||||
|
||||
All transition state (`cpuHighState`, `memHighState`, the two maps) is **in-memory only** — an agent restart clears it, so the module re-emits a HIGH on the next breaching poll after restart if the condition persists.
|
||||
|
||||
### Collectors (platform abstraction)
|
||||
|
||||
The module picks a collector at init via the agent's shared `PlatformService.isWindows()` — the same platform-detection facade the rest of the agent uses (see [technical.agent] "platform abstraction"). Beyond that choice each collector does its own low-level I/O; it does **not** route metric collection through `SystemCommands`/`PathResolver`.
|
||||
|
||||
- **Linux** — CPU: reads the `cpu ` line of `/proc/stat` twice, 1 s apart (`Thread.sleep(1000)` inside the poll thread), and diffs busy/total jiffies. Memory: `MemTotal`/`MemAvailable` from `/proc/meminfo` → `(total-available)/total`. Disk: `File.listRoots()` → `getTotalSpace()`/`getFreeSpace()` (root filesystems only). Services: `systemctl is-active <name>`, up iff first output line trims to `active`.
|
||||
- **Windows** — all via `powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command`. CPU: `Get-Counter '\Processor(_Total)\% Processor Time'`. Memory: `Win32_OperatingSystem` `TotalVisibleMemorySize`/`FreePhysicalMemory`. Disk: `Win32_LogicalDisk` (per drive letter). Services: `Get-Service -Name @(...)`, up iff `Status = Running`; any queried service not returned is marked down. A daemon stderr-drainer thread reads PowerShell stderr and a non-zero exit is logged.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
`ServerMonitorModule` builds each event with `CreateJournalEventRequest.builder()`:
|
||||
|
||||
```
|
||||
agentAtmId = atmName (from ModuleContext)
|
||||
deviceType = "SRV" (constant DEVICE_TYPE)
|
||||
eventType = <one of the 8 strings>
|
||||
eventDetails = human string (e.g. "CPU at 92.3% (threshold 85.0%, sustained 3 samples)")
|
||||
eventSource = "HIVEOPS_AGENT" (constant EVENT_SOURCE)
|
||||
```
|
||||
|
||||
and sends via the shared journal-events client:
|
||||
|
||||
```
|
||||
ServerMonitorModule
|
||||
│ IncidentEventClient.sendEvents(List<CreateJournalEventRequest>)
|
||||
│ (HttpClientSettings from agent.endpoint / incident.endpoint)
|
||||
▼
|
||||
POST {endpoint}/api/journal-events ← one HTTP POST per event
|
||||
│ (via agent-proxy, X-Institution-Key/auth on the shared client)
|
||||
▼
|
||||
hiveops-incident journal-events ingest → SRV device auto-registers, events stored
|
||||
```
|
||||
|
||||
Note: `sendEvents` is **not** a batch call — it loops and issues one `POST /api/journal-events` per event (the DTO comment states the backend "may not support batch operations"). This is the same endpoint and `IncidentEventClient` the `journal-events` module uses for ATM events; `srv-monitor` reuses it wholesale, only changing `deviceType` to `SRV`. In the current topology `agent.endpoint` points at **agent-proxy**, which forwards to incident's ingest — see [technical.agent] for the agent → agent-proxy → incident path.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
module: agent.srv-monitor
|
||||
title: Server Monitor — CPU/memory/disk/service health watcher for SRV devices
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse, act-without-guessing. Confirmed against `hiveops-module-srv-monitor` source 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Module name (`AgentModule.getName()`): **`srv-monitor`**
|
||||
- Maven artifact / drop-in jar: **`hiveops-module-srv-monitor.jar`** (own version `1.0.0`; parent `hiveops-agent-parent` currently `4.4.3`)
|
||||
- Entry class: `com.hiveops.srvmonitor.ServerMonitorModule` (SPI file: `META-INF/services/com.hiveops.core.module.AgentModule`)
|
||||
- Other classes: `ServerMonitorCollector` (iface), `LinuxServerMonitorCollector`, `WindowsServerMonitorCollector`, `ServerMetrics`
|
||||
- Log package (`getLogPackage()`): `com.hiveops.srvmonitor` → `logs/modules/srv-monitor.log`
|
||||
- Dependencies (`getDependencies()`): **none** (empty list)
|
||||
- Constants: `deviceType = "SRV"`, `eventSource = "HIVEOPS_AGENT"`
|
||||
|
||||
## Config properties (exact keys, from `hiveops.properties`)
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `srv.monitor.enabled` | `false` | master gate; module inactive unless `true` |
|
||||
| `srv.monitor.interval.seconds` | `60` | fixed-rate poll; initial delay 0 |
|
||||
| `srv.monitor.cpu.threshold` | `85` | CPU busy % |
|
||||
| `srv.monitor.cpu.sustained.samples` | `3` | consecutive breaching polls before CPU_HIGH |
|
||||
| `srv.monitor.memory.threshold` | `90` | mem used % |
|
||||
| `srv.monitor.disk.threshold` | `90` | per-filesystem used % |
|
||||
| `srv.monitor.services` | *(empty)* | comma list of service/unit names; empty = no service checks |
|
||||
- Shared (not srv-monitor-specific), required for events: `agent.endpoint` (preferred) **or** `incident.endpoint` (fallback). Neither set → module stays inactive even if enabled. HTTP client settings loaded with prefix `agent` or `incident` to match.
|
||||
- Config is read **once in `initialize()`** → change needs an agent restart to apply.
|
||||
- Disable via `srv.monitor.enabled=false` **or** add `srv-monitor` to `modules.disabled`.
|
||||
|
||||
## Event-type strings emitted (exact)
|
||||
- CPU: **`SERVER_CPU_HIGH`**, **`SERVER_CPU_NORMAL`**
|
||||
- Memory: **`SERVER_MEMORY_HIGH`**, **`SERVER_MEMORY_NORMAL`**
|
||||
- Disk: **`SERVER_DISK_LOW`**, **`SERVER_DISK_NORMAL`**
|
||||
- Service: **`SERVICE_DOWN`**, **`SERVICE_UP`**
|
||||
- Edge-triggered: fired on state **transition** only, not per poll. CPU is debounced by `sustained.samples`; memory/disk/service transition immediately.
|
||||
- Sent as `CreateJournalEventRequest` (`agentAtmId`, `deviceType=SRV`, `eventType`, `eventDetails`, `eventSource=HIVEOPS_AGENT`).
|
||||
|
||||
## Transport
|
||||
- `IncidentEventClient.sendEvents(...)` → **`POST {endpoint}/api/journal-events`**, **one POST per event** (not batched — loops `sendEvent`).
|
||||
- Same client/endpoint as the `journal-events` module; only `deviceType=SRV` differs. Routed via agent-proxy → incident ingest.
|
||||
|
||||
## Deployment (fleet)
|
||||
- Deliver jar via **`INSTALL_MODULE`** fleet task (`ProcessInstallModule`): artifact chunked-download **or** `configPatch.url` + `filename` (filename must end `.jar`). Drops into `modules/`, then restarts agent.
|
||||
- Then enable via `CONFIG_UPDATE` setting `srv.monitor.enabled=true`.
|
||||
|
||||
## Collectors
|
||||
- Linux: `/proc/stat` (CPU, 1 s sample), `/proc/meminfo` (MemTotal/MemAvailable), `File.listRoots()` (disk), `systemctl is-active <svc>` (up iff `active`).
|
||||
- Windows: PowerShell (`-NoProfile -NonInteractive -ExecutionPolicy Bypass`) — `Get-Counter`, `Win32_OperatingSystem`, `Win32_LogicalDisk`, `Get-Service` (up iff `Running`).
|
||||
- Chosen at init by `context.getPlatformService().isWindows()`.
|
||||
|
||||
## Gotchas
|
||||
- **Linux disk = roots only.** `File.listRoots()` is typically just `/`; separate mounts (e.g. `/var`) are not monitored per-partition. Windows lists each drive letter.
|
||||
- **CPU spike shorter than `sustained.samples` polls never fires** (default 3 × 60 s ≈ 3 min sustained).
|
||||
- **Transition state is in-memory only** — agent restart re-emits HIGH on next breaching poll if condition persists.
|
||||
- **Flapping at the threshold** — memory/disk/service have no debounce; a value hovering at the threshold alternates HIGH/NORMAL each poll.
|
||||
- **Windows collector returns 0.0/empty** if PowerShell is unavailable/locked down; non-zero exit + stderr logged.
|
||||
- **Service name mismatch → false `SERVICE_DOWN`** (needs exact systemd unit / Windows service name).
|
||||
- **Log-line cosmetic bug:** several info logs use `{:.1f}` placeholders (SLF4J-C style), which log4j2 does not format — the value is dropped in the log text. The `eventDetails` sent to the platform use `String.format("%.1f", …)` and are correct; only the local log strings are affected.
|
||||
|
||||
## Do NOT
|
||||
- Do **not** expect batched event POSTs — it's one HTTP request per event.
|
||||
- Do **not** assume enabling `srv.monitor.enabled` alone works — the jar must also be present in `modules/` and an endpoint configured.
|
||||
- Do **not** expect config changes to apply without an agent restart.
|
||||
- Do **not** point this at ATM incidents expecting ATM device type — it always emits `deviceType=SRV`.
|
||||
- Do **not** invent `srv-monitor.*` (dot-dash) keys — the real prefix is `srv.monitor.*` (all dots).
|
||||
- Do **not** rely on Linux per-mount disk alerts for non-root filesystems.
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
module: agent.srv-monitor
|
||||
title: Server Monitor — CPU/memory/disk/service health watcher for SRV devices
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-srv-monitor` source (`com.hiveops.srvmonitor`) 2026-07-01. HiveIQ agent drop-in module. Not customer-facing.
|
||||
|
||||
## What it does
|
||||
|
||||
The **Server Monitor** module (`srv-monitor`) turns a HiveIQ agent install into a lightweight server-health probe. On a fixed interval it samples four things and reports **only state transitions** (breach and recovery) to the platform as journal events on an `SRV`-type device:
|
||||
|
||||
| Signal | What it watches | Breach event | Recovery event |
|
||||
|---|---|---|---|
|
||||
| CPU | busy % (sustained N samples) | `SERVER_CPU_HIGH` | `SERVER_CPU_NORMAL` |
|
||||
| Memory | used % of physical RAM | `SERVER_MEMORY_HIGH` | `SERVER_MEMORY_NORMAL` |
|
||||
| Disk | used % per filesystem/volume | `SERVER_DISK_LOW` | `SERVER_DISK_NORMAL` |
|
||||
| Services | named services up/down | `SERVICE_DOWN` | `SERVICE_UP` |
|
||||
|
||||
It is edge-triggered: it fires an event when a metric crosses its threshold and again when it drops back, not on every poll. This keeps the incident stream quiet — one HIGH and one NORMAL per episode, not one per minute.
|
||||
|
||||
CPU has a debounce: it must stay at/above threshold for `srv.monitor.cpu.sustained.samples` consecutive polls before `SERVER_CPU_HIGH` fires (default 3). Memory, disk, and service transitions fire immediately on the first breaching poll.
|
||||
|
||||
The module is designed for the planned **SRV device type** (branch back-office / QDS servers, not ATMs) — events carry `deviceType=SRV` and `eventSource=HIVEOPS_AGENT`, so the device auto-registers as an SRV device on first event, the same way ATMs auto-register on first journal event.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
All read from the agent's main `hiveops.properties` in `initialize()`. Defaults are the code defaults:
|
||||
|
||||
| Property | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `srv.monitor.enabled` | `false` | Master gate. Module stays inactive unless this is `true`. |
|
||||
| `srv.monitor.interval.seconds` | `60` | Poll interval (fixed-rate). Also the initial delay is 0 — first poll on start. |
|
||||
| `srv.monitor.cpu.threshold` | `85` | CPU busy-% threshold. |
|
||||
| `srv.monitor.cpu.sustained.samples` | `3` | Consecutive breaching polls required before `SERVER_CPU_HIGH`. |
|
||||
| `srv.monitor.memory.threshold` | `90` | Memory used-% threshold. |
|
||||
| `srv.monitor.disk.threshold` | `90` | Per-filesystem used-% threshold. |
|
||||
| `srv.monitor.services` | *(empty)* | Comma-separated service names to watch (e.g. `hiveops-agent,qds,postgresql`). Empty = no service checks. |
|
||||
|
||||
It also **reuses** the shared agent endpoint/auth config (not srv-monitor-specific):
|
||||
|
||||
- `agent.endpoint` (preferred) or `incident.endpoint` (fallback) — the HTTP base URL events are POSTed to. If neither is set the module logs `no endpoint configured, module inactive` and stays down even when enabled.
|
||||
- The matching HTTP client settings (`agent.*` or `incident.*` — auth token, institution key, timeouts) are loaded via the same `HttpClientHelper` path every other event-emitting module uses.
|
||||
|
||||
## How it's enabled / deployed to ATMs
|
||||
|
||||
Two independent switches must both be right: the **JAR must be present** and `srv.monitor.enabled=true`.
|
||||
|
||||
1. **Ship the JAR.** `srv-monitor` is a drop-in module JAR (`hiveops-module-srv-monitor.jar`), not bundled in the core agent. It lives in the agent's `modules/` directory. The agent's `ModuleLoader` discovers it there on startup.
|
||||
2. **Fleet delivery — `INSTALL_MODULE` task.** Push the JAR fleet-wide with an `INSTALL_MODULE` fleet task. The agent's `ProcessInstallModule` handler downloads it (either from a fleet artifact via chunked download, or from a `configPatch.url`/`filename` pair — filename must end in `.jar`), drops it into `modules/`, then restarts the agent so the new module loads. See the [technical.agent] fleet-task docs for the task shape.
|
||||
3. **Turn it on.** Set `srv.monitor.enabled=true` (plus any threshold/service overrides) in `hiveops.properties`, typically via a `CONFIG_UPDATE` fleet task. The module reads config only at `initialize()`, so a config change needs an agent restart to take effect.
|
||||
4. **Disable** either by `srv.monitor.enabled=false` or by adding `srv-monitor` to the `modules.disabled` comma list.
|
||||
|
||||
## Failure modes / what to check
|
||||
|
||||
- **Enabled but no events ever.** Check `agent.endpoint`/`incident.endpoint` is set — with neither, the module logs `no endpoint configured, module inactive` and never schedules. Also confirm the JAR actually landed in `modules/` and the agent restarted after `INSTALL_MODULE`.
|
||||
- **No CPU events despite obvious load.** CPU is debounced by `srv.monitor.cpu.sustained.samples` (default 3) — a spike shorter than 3 poll intervals never fires. At the 60 s default that's a ~3-minute sustained requirement.
|
||||
- **No disk events for a full `/var` (Linux).** The Linux collector enumerates `File.listRoots()`, which on Linux is typically just `/`. Separate mount points that aren't reported as roots are **not** monitored per-partition. Windows reports each logical drive (`C:`, `D:` …) individually.
|
||||
- **`SERVICE_DOWN` for a service that's actually up.** Linux uses `systemctl is-active <name>` (expects exactly `active`); Windows uses `Get-Service -Name <name>` (expects `Running`). A wrong unit name, a non-systemd Linux host, or a service the query can't reach is reported **down**. Verify the exact service/unit name in `srv.monitor.services`.
|
||||
- **Nothing on Windows + PowerShell warnings in the log.** The Windows collector shells out to `powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass`. If PowerShell is locked down or missing, CPU/mem/disk read back as `0.0`/empty and a non-zero exit is logged with stderr.
|
||||
- **Event stream noisy / flapping.** A metric hovering right at the threshold will alternate HIGH/NORMAL each poll (memory/disk/service have no debounce). Raise the threshold or the interval.
|
||||
- **Where to look:** the module logs under its own package `com.hiveops.srvmonitor` → `logs/modules/srv-monitor.log`. Startup line `srv-monitor initialized (interval=…, cpu=…%, …)` confirms it armed with the config you expect.
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
module: agent.tcr-events
|
||||
title: TCR Events — cassette + sensor + process monitoring for Teller Cash Recyclers
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Architect · dev.** Written 2026-07-01 from the compiled `hiveops-module-tcr-events` artifact (v4.3.5); `.java` sources are absent from the working tree, so class/method flow below is reconstructed from bytecode in `target/classes`. Cross-reference [technical.agent] for the module system and event pipeline.
|
||||
|
||||
## Shape
|
||||
|
||||
A ServiceLoader `AgentModule` (`com.hiveops.tcrevents.TcrEventsModule`, registered in `META-INF/services/com.hiveops.core.module.AgentModule`) that owns one scheduler and three stateless log pollers. Package `com.hiveops.tcrevents`. Classes:
|
||||
|
||||
- `TcrEventsModule` — lifecycle, config, scheduler, wiring.
|
||||
- `TcrAccessLogMonitor` — process/shutdown events from `ACESSLog<MMdd>.txt`.
|
||||
- `CstSnsLogMonitor` — sensor/mismatch events from `CST_SNR_<MMdd>.log`.
|
||||
- `TcrCassetteMonitor` — per-slot inventory from `JsonLog_<MMdd>.json`.
|
||||
|
||||
It uses no platform-abstraction layer (`PlatformService`) — file access is portable JVM I/O (`java.nio.file.Paths`, `RandomAccessFile` opened `"r"`), even though the target log layout is the Windows LG/ATEC TCR software.
|
||||
|
||||
## Lifecycle & config (`TcrEventsModule`)
|
||||
|
||||
`getName()` → `tcr-events`; `getVersion()` → `1.0.0`.
|
||||
|
||||
`initialize(ModuleContext)` runs a short guard chain, then wires the pollers:
|
||||
|
||||
1. **Device gate** — reads `device.type` from `ctx.getMainProperties()` (default `ATM`), `trimToEmpty` + `upperCase`; unless it equals `ATEC_TCR` or `TCR`, logs `device.type is '{}', not ATEC_TCR — module disabled` and returns not-ready.
|
||||
2. **Enable gate** — `tcr.events.enabled` (default `true`); if not `equalsIgnoreCase("true")`, disabled.
|
||||
3. **Log dir** — `tcr.log.dir` via `trimToNull`; if null, disabled. Resolved with `Paths.get(...)`; if the path doesn't exist it only warns (`will retry each poll`) and continues — pollers re-check existence each tick.
|
||||
4. **Cadence** — `tcr.events.poll.interval.sec` via `NumberUtils.toInt(..., 30)` → `pollSec` field (default 30).
|
||||
5. **Transport** — if `agent.endpoint` is set, `HttpClientHelper.loadSettingsFromProperties(props, "agent")`; otherwise prefix `"incident"`. Builds one `IncidentEventClient(settings, ctx.getAtmName(), ctx.getCountry())`, shared by all three pollers.
|
||||
6. **Wire** — constructs `CstSnsLogMonitor`, `TcrAccessLogMonitor`, `TcrCassetteMonitor`, each `(Path logDir, String atmName, IncidentEventClient client)`; sets `ready = true`.
|
||||
|
||||
`isEnabled()` returns the `ready` flag. `start()` creates a `newSingleThreadScheduledExecutor` with a daemon thread named `tcr-events` and `scheduleAtFixedRate` at `pollSec` seconds; each tick calls `cstMonitor.poll()`, `accessMonitor.poll()`, `cassetteMonitor.poll()` inside a try/catch that logs `tcr-events: poll error: {}` (a poll failure never kills the schedule). `stop()` shuts the scheduler down.
|
||||
|
||||
## Poller pattern (tail → detect → emit)
|
||||
|
||||
All three share the same structure:
|
||||
|
||||
1. Compute today's filename from a `MMdd` `SimpleDateFormat` and resolve it under `tcr.log.dir`.
|
||||
2. Open with `RandomAccessFile(file, "r")`, seek to the retained per-file byte offset, read newly appended lines, and advance the offset. (In-memory offset state — no persistence, so an agent restart re-reads from 0 for the current file; the vendor writes append-only within a day.)
|
||||
3. Match lines and, on a hit, build a `CreateJournalEventRequest` via its fluent `builder()` and call `client.sendEvent(...)`; failures log `tcr-events: failed to send {}: {}` and are swallowed per-line.
|
||||
|
||||
### `TcrAccessLogMonitor`
|
||||
Plain substring/`contains` matching on `ACESSLog<MMdd>.txt`:
|
||||
- `"Process restarted from abnormal termination"` → `TCR_PROCESS_RESTART` (details "TCR process restarted after abnormal termination").
|
||||
- `"## Shutdown ##"` / `"## Shutdown for reboot ##"` → `TCR_SHUTDOWN`; details are "TCR shutdown for reboot" if the line contains `reboot`, else "TCR operator shutdown".
|
||||
- Builder fields set: `agentAtmId`, `eventType`, `eventDetails`, `eventSource=TCR_ACESS_LOG`.
|
||||
|
||||
### `CstSnsLogMonitor`
|
||||
Regex-driven parse of `CST_SNR_<MMdd>.log`. Compiled patterns cover block delimiters (`>>>>> hh:mm:ss.SSS - <TAG> START/END`), `[N_E]` lines, generic `(\w{2,3}):\s*(\w+)` key/value pairs, and the mismatch line `** CST <x> - SNR deposit count ... mismatch ... SNR count: - N - N`. State-code → event mapping:
|
||||
- `EMPT` → `CASSETTE_EMPTY` ("TCR cassette slot <n> is empty")
|
||||
- `SNRF` → `TCR_SENSOR_FAULT` ("... sensor failure")
|
||||
- `TMAN` → `TCR_MANUAL_REQUIRED` ("... requires manual intervention")
|
||||
- `NORM` → recovery only (logs `slot {} recovered (was {})`, no event)
|
||||
- deposit-count mismatch → `TCR_SENSOR_MISMATCH` ("Slot <n>: SNR mismatch — data=<> sensor=<>")
|
||||
- Builder fields set: `agentAtmId`, `eventType`, `eventDetails`, `eventSource=TCR_CST_SNR`, `cassettePosition` (slot).
|
||||
|
||||
### `TcrCassetteMonitor`
|
||||
Parses `JsonLog_<MMdd>.json`, filtering to lines whose JSON has `"Command":"RU"` (recycler-unit inventory snapshot; both `"Command":"RU"` and `"Command": "RU"` spacings are matched). For each entry in `Cassette_Info` it reads `Cassette_Type`, `Cassette_Status`, `Position`/`Sub_Position`, `Nation`(US)/currency `USD`, `Denomination`, `Denom`, `Count`, and emits one `CASSETTE_INVENTORY` event per slot ("TCR slot <pos>: <status> notes (<count>)"). Mappings:
|
||||
- Status `Available` / `Low` / `Empty` / `Full` → `cassetteFillLevel`.
|
||||
- Type `Recycle_Cassette`→`RECYCLE`, `Operation_Cassette`→`OPERATION`, `Reject_Cassette`→`REJECT` → `cassetteType`.
|
||||
- Builder fields set: `agentAtmId`, `eventType`, `eventSource=TCR_CASSETTE`, `eventDetails`, `cassettePosition`, `cassetteType`, `cassetteCurrency`, `cassetteDenomination`, `cassetteFillLevel`, `cassetteBillCount`. Logs `sent CASSETTE_INVENTORY for {} slots`.
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
`IncidentEventClient` (from `hiveops-agent-journal`, `com.hiveops.events`) serializes each `CreateJournalEventRequest` with Jackson (`@JsonInclude(NON_NULL)`) and does `POST {settings.endpoint}/api/journal-events`, accepting 200/201/202. Because the endpoint prefix is chosen from `agent.endpoint` vs `incident.endpoint`, on current fleets this targets the **agent-proxy** `/agent` path, which forwards into the incident pipeline; older configs post straight to incident. The ATM is auto-registered from `agentAtmId` — no `resolveAtmId()` call (deprecated). `agent.auth.token` is the bearer for every request.
|
||||
|
||||
Notable: none of the three pollers call `builder.deviceType(...)`, so `deviceType` is null on the wire and the backend applies its default (ATM) — the events are not tagged as TCR-device events even though the module only runs on TCR devices.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
None used. The module is pure JVM file-tailing + HTTP; it inherits ATM identity (`getAtmName()`, `getCountry()`) and shared properties from `ModuleContext`, and the HTTP layer from `HttpClientHelper`/`HttpClientSettings`. There is no `SystemCommands`/`PathResolver`/registry interaction, so behavior is identical across OSes given the same `tcr.log.dir` contents.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
module: agent.tcr-events
|
||||
title: TCR Events — cassette + sensor + process monitoring for Teller Cash Recyclers
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Claude · terse act-without-guessing reference.** From compiled `hiveops-module-tcr-events` v4.3.5 (`.java` not in tree; read from bytecode + core `IncidentEventClient`/`HttpClientHelper`).
|
||||
|
||||
## Identity
|
||||
|
||||
- Maven artifact / JAR: `hiveops-module-tcr-events` (groupId `com.hiveops`, built v`4.3.5`).
|
||||
- Agent module id (`getName()`): `tcr-events`. Module `getVersion()`: `1.0.0`.
|
||||
- SPI: `com.hiveops.core.module.AgentModule` → `com.hiveops.tcrevents.TcrEventsModule`.
|
||||
- Classes: `TcrEventsModule`, `TcrAccessLogMonitor`, `CstSnsLogMonitor`, `TcrCassetteMonitor` (pkg `com.hiveops.tcrevents`).
|
||||
- Shares module id `tcr-events` with `hiveops-module-atec-tcr-events`; **this is the superset** (adds cassette-JSON monitor). Never install both on one device.
|
||||
|
||||
## Config keys (from `hiveops.properties`)
|
||||
|
||||
| Key | Default | Notes |
|
||||
|---|---|---|
|
||||
| `device.type` | `ATM` | Must be `ATEC_TCR` or `TCR` (upper-cased) or module disables. |
|
||||
| `tcr.events.enabled` | `true` | Anything != `true` (ci) disables. |
|
||||
| `tcr.log.dir` | *(required)* | Dir of the 3 log files. Missing key → disabled; missing dir → warn + retry. |
|
||||
| `tcr.events.poll.interval.sec` | `30` | Scheduler period (seconds). |
|
||||
| `agent.endpoint` | *(none)* | Present → HTTP prefix `agent`; absent → prefix `incident`. |
|
||||
| `agent.auth.token` | — | Bearer for all requests (not module-specific). |
|
||||
|
||||
## Watched files (daily, `MMdd`)
|
||||
|
||||
- `ACESSLog<MMdd>.txt` — `TcrAccessLogMonitor` (note the misspelling "ACESS", it's the vendor's).
|
||||
- `CST_SNR_<MMdd>.log` — `CstSnsLogMonitor`.
|
||||
- `JsonLog_<MMdd>.json` — `TcrCassetteMonitor` (only `"Command":"RU"` lines).
|
||||
|
||||
## Event strings emitted
|
||||
|
||||
`eventType` (exact):
|
||||
- `TCR_PROCESS_RESTART`
|
||||
- `TCR_SHUTDOWN`
|
||||
- `CASSETTE_EMPTY`
|
||||
- `TCR_SENSOR_FAULT`
|
||||
- `TCR_MANUAL_REQUIRED`
|
||||
- `TCR_SENSOR_MISMATCH`
|
||||
- `CASSETTE_INVENTORY`
|
||||
|
||||
`eventSource` (exact): `TCR_ACESS_LOG`, `TCR_CST_SNR`, `TCR_CASSETTE`.
|
||||
|
||||
Sensor state codes → events: `EMPT`→`CASSETTE_EMPTY`, `SNRF`→`TCR_SENSOR_FAULT`, `TMAN`→`TCR_MANUAL_REQUIRED`, `NORM`→**recovery, no event**.
|
||||
|
||||
Cassette type map: `Recycle_Cassette`→`RECYCLE`, `Operation_Cassette`→`OPERATION`, `Reject_Cassette`→`REJECT`. Status values: `Available`/`Low`/`Empty`/`Full`.
|
||||
|
||||
## Transport
|
||||
|
||||
- `IncidentEventClient.sendEvent` → `POST {endpoint}/api/journal-events`, accepts 200/201/202.
|
||||
- Auto-register via `agentAtmId`; do NOT rely on `resolveAtmId()` (deprecated).
|
||||
- Endpoint from `agent.endpoint` (prefix `agent`, agent-proxy `/agent`) else `incident.endpoint`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Monitors set **no** `deviceType` on the request → backend records events as ATM default, not TCR.
|
||||
- No offset persistence — agent restart re-reads current day's file from 0 (append-only vendor logs mitigate dupes; midnight rollover switches filename automatically).
|
||||
- Poll errors are caught/logged (`tcr-events: poll error:`), never fatal.
|
||||
- No `PlatformService` — pure `java.nio` + `RandomAccessFile("r")`; identical across OSes given same log dir.
|
||||
- Deploy/retrofit: fleet `INSTALL_MODULE` task (drops JAR into `modules/`, restarts agent) or full standard-set agent update.
|
||||
|
||||
## Do NOT
|
||||
|
||||
- Do NOT expect events on a device unless `device.type=ATEC_TCR|TCR` AND `tcr.log.dir` is set.
|
||||
- Do NOT install alongside `hiveops-module-atec-tcr-events` (same module id).
|
||||
- Do NOT invent config keys — only the five above are read. No `tcr-events.*`-dotted keys exist; the real prefixes are `tcr.events.*` and `tcr.log.dir`.
|
||||
- Do NOT assume batch upload — events post one-by-one per `sendEvent`.
|
||||
- Do NOT trust `getVersion()`=`1.0.0` as the release version; the shipped artifact version is the agent build (`4.3.5` here).
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
module: agent.tcr-events
|
||||
title: TCR Events — cassette + sensor + process monitoring for Teller Cash Recyclers
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support.** Written 2026-07-01 from the compiled `hiveops-module-tcr-events` artifact (v4.3.5). The `.java` sources are not in the working tree — facts below were read from the module bytecode in `target/classes` plus the core `IncidentEventClient` / `HttpClientHelper` sources. Verify against code before relying on specifics.
|
||||
|
||||
## What this module is
|
||||
|
||||
`hiveops-module-tcr-events` (Maven artifact `hiveops-module-tcr-events`, agent module id **`tcr-events`**, reported module version `1.0.0`) is an **agent drop-in module**, not a backend service. It runs inside the HiveIQ Agent process on **TCR (Teller Cash Recycler) devices only**. It self-disables unless the ATM's `device.type` is `ATEC_TCR` (or the legacy alias `TCR`).
|
||||
|
||||
It tails **three** log files that the LG/ATEC TCR software already writes to disk and turns specific lines into HiveIQ journal events, which it POSTs into the incident pipeline. The module does **not** talk to the recycler hardware directly and does not reconcile cash totals — it only reads logs and forwards structured events.
|
||||
|
||||
> Note: this module shares the agent module id `tcr-events` with `hiveops-module-atec-tcr-events`. This variant is the superset — it adds the JSON cassette-inventory monitor (`TcrCassetteMonitor`) on top of the access-log and sensor-log monitors. Do not install both on the same device.
|
||||
|
||||
## What it monitors
|
||||
|
||||
Three independent pollers, all driven by one scheduler tick (default every 30s), each reading a daily-rotated file (`MMdd` = month+day) from `tcr.log.dir`:
|
||||
|
||||
| Poller class | File pattern | Watches for |
|
||||
|---|---|---|
|
||||
| `TcrAccessLogMonitor` | `ACESSLog<MMdd>.txt` *(sic — misspelled in the vendor logs)* | abnormal process restarts, operator/reboot shutdowns |
|
||||
| `CstSnsLogMonitor` | `CST_SNR_<MMdd>.log` | cassette sensor state (empty / fault / manual-required) and SNR deposit-count mismatches |
|
||||
| `TcrCassetteMonitor` | `JsonLog_<MMdd>.json` | per-slot cassette inventory (`"Command":"RU"` recycler-unit status lines) |
|
||||
|
||||
Each poller tracks a byte offset into its current file (opened read-only) and only processes newly appended lines each tick.
|
||||
|
||||
## Events it emits
|
||||
|
||||
All events are sent as HiveIQ journal events (`POST {endpoint}/api/journal-events`). Event types by source:
|
||||
|
||||
| `eventSource` | `eventType` | Trigger |
|
||||
|---|---|---|
|
||||
| `TCR_ACESS_LOG` | `TCR_PROCESS_RESTART` | log line "Process restarted from abnormal termination" |
|
||||
| `TCR_ACESS_LOG` | `TCR_SHUTDOWN` | "## Shutdown ##" / "## Shutdown for reboot ##" (reboot vs operator shutdown in details) |
|
||||
| `TCR_CST_SNR` | `CASSETTE_EMPTY` | sensor state `EMPT` |
|
||||
| `TCR_CST_SNR` | `TCR_SENSOR_FAULT` | sensor state `SNRF` |
|
||||
| `TCR_CST_SNR` | `TCR_MANUAL_REQUIRED` | sensor state `TMAN` |
|
||||
| `TCR_CST_SNR` | `TCR_SENSOR_MISMATCH` | SNR deposit-count mismatch line |
|
||||
| `TCR_CASSETTE` | `CASSETTE_INVENTORY` | one per slot from each `RU` status snapshot |
|
||||
|
||||
Sensor state `NORM` is treated as **recovery** — it is logged (`slot {} recovered`) but emits **no** event.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
Read from the agent's main properties (`hiveops.properties`) during `initialize()`:
|
||||
|
||||
| Property | Default | Effect |
|
||||
|---|---|---|
|
||||
| `device.type` | `ATM` | Gate — module runs only when this is `ATEC_TCR` or `TCR` (upper-cased). Anything else: module disabled. |
|
||||
| `tcr.events.enabled` | `true` | Master on/off. Any value other than `true` (case-insensitive): module disabled. |
|
||||
| `tcr.log.dir` | *(none — required)* | Directory holding the three TCR log files. If unset: module disabled. If set but missing on disk: warns and retries each poll. |
|
||||
| `tcr.events.poll.interval.sec` | `30` | Scheduler period in seconds. |
|
||||
| `agent.endpoint` | *(none)* | If present, HTTP settings load from the `agent.*` prefix (agent-proxy); if absent, they fall back to the `incident.*` prefix. |
|
||||
|
||||
Transport credentials are **not** module-specific: the bearer token always comes from `agent.auth.token`, and the institution key resolves `<{prefix}>.institution.key` → `agent.institution.key` → `incident.institution.key` (see `HttpClientHelper.loadSettingsFromProperties`).
|
||||
|
||||
## How it's enabled / deployed
|
||||
|
||||
1. Device must be flagged `device.type=ATEC_TCR` (or `TCR`) and have `tcr.log.dir` pointed at the vendor log directory.
|
||||
2. Deliver the module JAR like any other agent module: upload to the fleet artifact store, then push a fleet **`INSTALL_MODULE`** task, which drops the JAR into the agent's `modules/` directory and restarts the agent (`ProcessInstallModule` in `hiveops-file-collection`). It can also ship as part of a full standard-set agent update.
|
||||
3. Standard rollout order applies: lab TCR → pilot device → fleet.
|
||||
|
||||
On successful init the agent logs `tcr-events initialized (log.dir={}, poll={}s)` and, once started, `tcr-events started`.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
- **Module silently absent from a device.** Check `device.type` — if it is not `ATEC_TCR`/`TCR` the agent logs `tcr-events: device.type is '{}', not ATEC_TCR — module disabled` and never starts. This is the most common cause.
|
||||
- **No events at all.** Confirm `tcr.events.enabled` is not set to `false` (`tcr-events: disabled via tcr.events.enabled=false`) and `tcr.log.dir` is configured (`tcr.log.dir not configured — module disabled`).
|
||||
- **Log dir warning on every poll.** `log dir '{}' does not exist yet — will retry each poll` means the module initialized but the directory is missing/misnamed — the vendor software may not have created that day's folder yet.
|
||||
- **Events built but not arriving in incident.** Look for `tcr-events: failed to send {}: {}` (HTTP error from `/api/journal-events`) — usually an endpoint/token/institution-key problem, or the wrong prefix (`agent.endpoint` present but `agent.*` settings wrong).
|
||||
- **Read errors.** `tcr-events: failed to read {}` / `cassette monitor failed to read {}` — permissions or a locked/rotating vendor file.
|
||||
- **Wrong day / nothing new.** All three files are day-stamped (`MMdd`); at midnight rollover the pollers switch to the new filename automatically.
|
||||
|
||||
> Gotcha: emitted events do **not** set `deviceType` on the request, so the incident backend records them under its default device type (ATM), even though the module only runs on TCR devices. See [architect.tcr-events].
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
module: agent.win-app-events
|
||||
title: Windows App Event Log Monitor — forwards Windows Application-log errors as HiveIQ incidents
|
||||
tab: Architect
|
||||
order: 30
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · how it's built.** Confirmed against `hiveops-module-win-app-events` source (`com.hiveops.winevents`) 2026-07-01. See [technical.agent] for agent-wide module/event context.
|
||||
|
||||
## Maven module
|
||||
|
||||
`hiveops-module-win-app-events` (artifact `hiveops-module-win-app-events`, own version `1.0.1`, parented by `hiveops-agent-parent` `4.4.3`). Packaged as a standalone jar (`<finalName>${project.artifactId}</finalName>` → `hiveops-module-win-app-events.jar`). Dependencies are all `provided` (supplied by the fat JAR at runtime):
|
||||
|
||||
- `hiveops-core` — the `AgentModule` SPI, `ModuleContext`, `HttpClientHelper`/`HttpClientSettings`, `PatternReloadRegistry`.
|
||||
- `hiveops-agent-journal` (`1.0.0`) — `IncidentEventClient`, `EventType`, `CreateJournalEventRequest` (the same event path `journal-events` uses).
|
||||
- `log4j-api`, `commons-lang3`.
|
||||
|
||||
Registered via ServiceLoader SPI: `src/main/resources/META-INF/services/com.hiveops.core.module.AgentModule` → `com.hiveops.winevents.WinAppEventModule`.
|
||||
|
||||
## Classes and flow
|
||||
|
||||
Four classes, no sub-packages:
|
||||
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `WinAppEventModule` | `AgentModule` impl — OS/enable gating, config parse, scheduler, hot-reload registration |
|
||||
| `WinAppEventPoller` | `Runnable` — the anchor/poll loop; runs `wevtutil`, applies filters, emits events |
|
||||
| `WinAppEventParser` | Turns raw `wevtutil` XML into `WinAppEvent` records |
|
||||
| `WinAppEvent` | Immutable holder: `recordId, provider, eventId, level, timeCreated, message`; builds `matchText()` and `toDetails()` |
|
||||
|
||||
**Poll-based** cycle (not log-tail or ETW hook — it repeatedly shells `wevtutil` and diffs by RecordID):
|
||||
|
||||
1. **`initialize(ModuleContext)`** — bails (leaves `ready=false`) if not Windows, if `winevt.events.enabled != true`, or if no `agent.endpoint`/`incident.endpoint`. Otherwise builds an `IncidentEventClient` from `HttpClientHelper.loadSettingsFromProperties(props, prefix)` (prefix `"agent"` if `agent.endpoint` set, else `"incident"`), reads `levels`/`batch.size`/`sources`/pattern, constructs the `WinAppEventPoller`, resolves the `hiveops.properties` path for later reloads, and sets `ready=true`.
|
||||
2. **`isEnabled()`** returns `ready`.
|
||||
3. **`start()`** — creates a single daemon thread (`newSingleThreadScheduledExecutor`, thread name `win-app-events`) and `scheduleAtFixedRate(poller, 0, intervalSec, SECONDS)`. Then registers `this::reloadPatterns` with `PatternReloadRegistry`.
|
||||
4. **`WinAppEventPoller.run()`** — if `suspended`, skip. If `lastEventRecordId < 0`, call `anchor()` and return (the very first tick). Otherwise `poll()`.
|
||||
- **`anchor()`** runs `wevtutil qe Application /rd:true /f:XML /c:1 /q:*[System[Level>=1 and Level<=<maxLevel>]]`, sets `lastEventRecordId` to that newest RecordID (or `0` if the log is empty). This is why no history is replayed.
|
||||
- **`poll()`** runs `wevtutil qe Application /rd:false /f:XML /LNIFO:Message /c:<batchSize> /q:*[System[(Level>=1 and Level<=<maxLevel>) and (EventRecordID > <lastId>)]]`. `/rd:false` = oldest-first (in-order processing); `/LNIFO:Message` renders human-readable message text from the provider DLL.
|
||||
5. **Filtering** (in order, per event): the **level** filter is baked into the `wevtutil` XPath; then **source** filter (`sourceFilter` non-empty → Provider name must case-insensitively equal one entry); then **pattern** filter (`eventPattern != null` → regex `.find()` on `matchText()` = `"<provider> <eventId> <message>"`).
|
||||
6. **Bookmark advance** — `lastEventRecordId` is set to the max RecordID seen this poll **even if every event was filtered out**, so filtered RecordIDs are never re-queried.
|
||||
7. **Emit** — survivors become `CreateJournalEventRequest`s (`agentAtmId=atmName`, `eventType=WIN_APP_EVENT_ERROR`, `eventDetails=WinAppEvent.toDetails()`, `eventSource=HIVEOPS_AGENT_WINEVT`) and are sent via `client.sendEvents(list)`.
|
||||
|
||||
`WinAppEventParser` wraps the `wevtutil` output in a synthetic `<Events>…</Events>` root (handles both single- and multi-event output), DOM-parses each `<Event>`, and reads `EventRecordID`, `Provider@Name`, `Level` (mapped 1→Critical, 2→Error, 3→Warning, 4→Information), `EventID`, `TimeCreated@SystemTime`, and the message. Message extraction **prefers** `RenderingInfo/Message` (present because of `/LNIFO:Message`) and **falls back** to joining `EventData/Data` values with `" | "`. Unparseable individual events are skipped at DEBUG; a totally unparseable payload yields an empty list.
|
||||
|
||||
## Hot reload
|
||||
|
||||
`WinAppEventModule.reloadPatterns()` (registered with `PatternReloadRegistry`) re-reads `hiveops.properties` **from disk** (`startedFromDir/hiveops.properties`, cwd fallback) and calls `poller.setPattern(...)` and `poller.setSuspended(...)`. Both poller fields are `volatile`. `ProcessConfigUpdate` (the `CONFIG_UPDATE` fleet-task handler in `hiveops-file-collection`) writes the new properties then calls `PatternReloadRegistry.trigger()` — that's the entire hot-reload path. Level/sources/interval/batch are **not** re-read here (captured at `initialize()` time only).
|
||||
|
||||
## How events reach the platform
|
||||
|
||||
This module rides the **same journal-event path** as the `journal-events` module — there is no separate protocol:
|
||||
|
||||
```
|
||||
WinAppEventPoller
|
||||
│ IncidentEventClient.sendEvents(List<CreateJournalEventRequest>)
|
||||
│ (endpoint from agent.endpoint | incident.endpoint; auth via HttpClientHelper)
|
||||
▼
|
||||
POST {endpoint}/api/journal-events ← one HTTP POST per event (sendEvents loops sendEvent)
|
||||
▼
|
||||
agent-proxy (agent-facing gateway, X-Institution-Key / token auth)
|
||||
▼
|
||||
hiveops-incident → journal-event ingest → incident pipeline / device_summary
|
||||
```
|
||||
|
||||
Note: `sendEvents` is **not** a batch POST — it iterates and POSTs each event individually to `/api/journal-events`; `winevt.events.batch.size` only bounds how many events one `wevtutil` call pulls, not the HTTP shape.
|
||||
|
||||
## Platform abstraction
|
||||
|
||||
The module does **not** use `com.hiveops.platform.PlatformServiceFactory` / `PlatformService`. OS detection is done locally (`System.getProperty("os.name")…contains("windows")`) and it shells `wevtutil` directly via `ProcessBuilder`. The runner (`runWevtutil`) decodes stdout with the charset from the `file.encoding` system property (a comment notes `wevtutil` emits UTF-16LE and relies on the JVM's active code-page resolution), reads stdout fully, then `waitFor()`s; a non-zero exit is logged at DEBUG and yields an empty string (→ empty parse → no events that tick).
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
module: agent.win-app-events
|
||||
title: Windows App Event Log Monitor — forwards Windows Application-log errors as HiveIQ incidents
|
||||
tab: Claude
|
||||
order: 40
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · agent reference.** Terse. Confirmed against `hiveops-module-win-app-events` source (`com.hiveops.winevents`) 2026-07-01.
|
||||
|
||||
## Identity
|
||||
- Module name (`AgentModule.getName()`): **`win-app-events`**
|
||||
- Maven artifact: **`hiveops-module-win-app-events`** (own version `1.0.1`; parent `hiveops-agent-parent` `4.4.3`)
|
||||
- Entry class: `com.hiveops.winevents.WinAppEventModule` (SPI: `META-INF/services/com.hiveops.core.module.AgentModule`)
|
||||
- Other classes: `WinAppEventPoller`, `WinAppEventParser`, `WinAppEvent`
|
||||
- Log package (`getLogPackage()`): `com.hiveops.winevents`
|
||||
- Dependencies: `getDependencies()` → **empty** (no ordering deps)
|
||||
- Windows-only; Linux → `initialize()` no-ops (logs `not Windows, module inactive`)
|
||||
|
||||
## Config property keys (all `winevt.*`, from `hiveops.properties`)
|
||||
| Key | Code default | Notes |
|
||||
|---|---|---|
|
||||
| `winevt.events.enabled` | **`false`** | Must be `true` to run. Javadoc claims default true — **code default is `false`**. |
|
||||
| `winevt.events.levels` | `3` | Max level: 1=Critical, 2=Error, 3=Warning. Query is `Level>=1 and Level<=N`. |
|
||||
| `winevt.events.sources` | `""` | CSV of Provider names; exact case-insensitive match; empty = all. |
|
||||
| `winevt.events.poll.interval.sec` | `60` | Scheduler period (seconds). |
|
||||
| `winevt.events.batch.size` | `50` | `wevtutil /c:` cap per call (NOT an HTTP batch). |
|
||||
| `winevt.pattern.WIN_APP_EVENT_ERROR` | `""` | Regex, `.find()`, CASE_INSENSITIVE, on `"<provider> <eventId> <message>"`; empty/invalid = catch-all. |
|
||||
| `winevt.events.suspended` | `false` | Hot-reload-only pause; poller keeps ticking, does nothing. |
|
||||
|
||||
- Endpoint/auth (not owned by this module): `agent.endpoint` (preferred) else `incident.endpoint` (fallback); prefix drives `HttpClientHelper.loadSettingsFromProperties(props, prefix)`.
|
||||
- Disable via `modules.disabled=win-app-events` too.
|
||||
- **Hot-reloadable (via `CONFIG_UPDATE` → `PatternReloadRegistry.trigger()`):** ONLY `winevt.pattern.WIN_APP_EVENT_ERROR` + `winevt.events.suspended`. Everything else needs an **agent restart** (captured in `initialize()`).
|
||||
|
||||
## Event emitted
|
||||
- `EventType.WIN_APP_EVENT_ERROR` → `eventType = "WIN_APP_EVENT_ERROR"` (only event type this module emits)
|
||||
- `eventSource = "HIVEOPS_AGENT_WINEVT"` (constant `EVENT_SOURCE`)
|
||||
- `agentAtmId = <atmName>` (from `ModuleContext`)
|
||||
- `eventDetails` format: `[Source: <provider>] [EventID: <id>] [Level: <Critical|Error|Warning|...>] <message>`
|
||||
- No `IAT_*` events. No fleet-task status. Nothing else.
|
||||
|
||||
## Wire path
|
||||
- `IncidentEventClient.sendEvents(list)` → loops `sendEvent` → **one** `POST {endpoint}/api/journal-events` **per event** (not batched).
|
||||
- endpoint → `agent-proxy` → `hiveops-incident` journal-event ingest.
|
||||
|
||||
## wevtutil commands (exact)
|
||||
- Anchor (1st tick): `wevtutil qe Application /rd:true /f:XML /c:1 /q:*[System[Level>=1 and Level<=<maxLevel>]]`
|
||||
- Poll: `wevtutil qe Application /rd:false /f:XML /LNIFO:Message /c:<batchSize> /q:*[System[(Level>=1 and Level<=<maxLevel>) and (EventRecordID > <lastId>)]]`
|
||||
- Channel is **Application** only. `/LNIFO:Message` renders provider-DLL message text.
|
||||
|
||||
## Filter order (per event)
|
||||
1. Level — in the `wevtutil` XPath (`Level<=maxLevel`).
|
||||
2. Source — `sourceFilter` non-empty → Provider name must case-insensitively equal an entry.
|
||||
3. Pattern — `eventPattern != null` → regex `.find()` on `matchText()` = `"<provider> <eventId> <message>"`.
|
||||
- Bookmark `lastEventRecordId` advances to max RecordID seen **even if all filtered**.
|
||||
|
||||
## Deployment
|
||||
- Ships in the **`atm`** module set of `deployment/build-dist.sh` (default), NOT `srv`.
|
||||
- Drop-in JAR `modules/hiveops-module-win-app-events.jar` (not in fat JAR).
|
||||
- Retrofit: `INSTALL_MODULE` fleet task (drops JAR → restarts agent) or full standard-set update.
|
||||
|
||||
## Gotchas
|
||||
- `winevt.events.enabled` defaults to **`false`** in code — module is OFF unless explicitly enabled. Do not trust the class Javadoc's "default true."
|
||||
- First tick **anchors** to newest RecordID → **no historical replay**; only post-start events forward.
|
||||
- Changing filters does **not** replay already-skipped RecordIDs (bookmark already advanced).
|
||||
- `batch.size` bounds the `wevtutil` fetch, NOT the HTTP call — each event is a separate POST.
|
||||
- Uses `os.name` + raw `ProcessBuilder`; does **not** use `PlatformService`/`SystemCommands`.
|
||||
- Charset for `wevtutil` stdout = `file.encoding` system property; wrong code page → garbled messages.
|
||||
- Only pattern + suspended hot-reload; `levels`/`sources`/`interval`/`batch` require restart.
|
||||
|
||||
## Do NOT
|
||||
- Do NOT assume it's running by default — `winevt.events.enabled=true` is required (code default `false`).
|
||||
- Do NOT expect historical/backfilled events after enable or restart — it anchors forward-only.
|
||||
- Do NOT expect `IAT_*` or fleet-task-status output — it emits **only** `WIN_APP_EVENT_ERROR`.
|
||||
- Do NOT treat `winevt.events.batch.size` as an HTTP batch size — POSTs are one-per-event.
|
||||
- Do NOT expect `levels`/`sources`/`interval`/`batch` changes to hot-reload — only pattern + suspended do; restart otherwise.
|
||||
- Do NOT look for a `PlatformService` code path — OS detection and `wevtutil` invocation are local to this module.
|
||||
- Do NOT run it on Linux and expect activity — it silently no-ops off-Windows.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
module: agent.win-app-events
|
||||
title: Windows App Event Log Monitor — forwards Windows Application-log errors as HiveIQ incidents
|
||||
tab: Internal
|
||||
order: 20
|
||||
audience: dev
|
||||
---
|
||||
|
||||
> **Internal · ops/support view.** Confirmed against `hiveops-module-win-app-events` source (`com.hiveops.winevents`) 2026-07-01.
|
||||
|
||||
## What it monitors (and why)
|
||||
|
||||
`hiveops-module-win-app-events` (module name `win-app-events`, class `WinAppEventModule`) is a **passive poller** of the Windows **Application** event log. Every poll it shells out to `wevtutil` for new **Critical / Error / Warning** entries, applies filters, and forwards each survivor to the HiveIQ platform as a `WIN_APP_EVENT_ERROR` journal event (which lands in the incident pipeline like any other agent event).
|
||||
|
||||
The point: surface application-layer failures that never appear in the ATM's own `.jrn` journal — driver faults, .NET/service crashes, XFS middleware errors, disk/hardware warnings — so they become visible in HiveIQ without logging into the box.
|
||||
|
||||
- **Windows-only.** On Linux it logs `not Windows, module inactive` in `initialize()` and does nothing (silent no-op).
|
||||
- Reads the **Application** channel only (not System/Security).
|
||||
- Only levels **1 (Critical), 2 (Error), 3 (Warning)** are ever queried; level 4 (Information) and 0 are excluded by the `wevtutil` query itself.
|
||||
- On the **first tick after start** the module *anchors* to the newest existing RecordID and only forwards events that arrive **after** that — no historical flood on install/restart.
|
||||
|
||||
## Config properties it registers
|
||||
|
||||
All read from `hiveops.properties` (main properties). Namespace is `winevt.*`. Defaults are the code defaults (second arg to `getProperty`), which do **not** always match the class Javadoc — trust this table.
|
||||
|
||||
| Key | Default (in code) | Meaning |
|
||||
|---|---|---|
|
||||
| `winevt.events.enabled` | **`false`** | Master on/off. Module stays inactive unless this is explicitly `true`. (The Javadoc says default true — the **code default is `false`**.) |
|
||||
| `winevt.events.levels` | `3` | Max Windows level to capture: `1`=Critical only, `2`=Critical+Error, `3`=+Warning. |
|
||||
| `winevt.events.sources` | `""` (all) | Comma-separated Provider names to include; exact case-insensitive match. Empty = all providers. |
|
||||
| `winevt.events.poll.interval.sec` | `60` | Poll interval in seconds. |
|
||||
| `winevt.events.batch.size` | `50` | Max events pulled per `wevtutil` call (`/c:`). |
|
||||
| `winevt.pattern.WIN_APP_EVENT_ERROR` | `""` (catch-all) | Regex matched (`.find()`, case-insensitive) against `"<Provider> <EventID> <message>"`. Empty/absent = accept all. Invalid regex → logs a warning and falls back to catch-all. |
|
||||
| `winevt.events.suspended` | `false` | Hot-reload-only pause switch: when `true` the poller keeps ticking but skips all work. Only read on reload (see below), not at startup. |
|
||||
|
||||
It also reuses the shared endpoint/auth keys (not its own): `agent.endpoint` (preferred) or `incident.endpoint` (fallback), and whatever `agent.*`/`incident.*` auth settings `HttpClientHelper` loads for that prefix.
|
||||
|
||||
### Hot reload (no restart)
|
||||
|
||||
`winevt.pattern.WIN_APP_EVENT_ERROR` and `winevt.events.suspended` are **hot-reloadable**. A `CONFIG_UPDATE` fleet task rewrites `hiveops.properties` on disk and then fires `PatternReloadRegistry.trigger()`, which re-reads the file and pushes the new pattern + suspended flag into the running poller. **Everything else** (`enabled`, `levels`, `sources`, `poll.interval.sec`, `batch.size`) is captured once in `initialize()` and only changes on an **agent restart**.
|
||||
|
||||
## How it's enabled / deployed to ATMs
|
||||
|
||||
- **Enable condition:** module self-activates only when `winevt.events.enabled=true` **and** an `agent.endpoint`/`incident.endpoint` is configured **and** the OS is Windows. Missing any of these → inactive, logged at INFO.
|
||||
- **Shipped how:** part of the default **`atm`** module set in `deployment/build-dist.sh` — it ships in every standard ATM install/patch ZIP as a drop-in JAR `modules/hiveops-module-win-app-events.jar` (not baked into the fat JAR). It is **not** in the `srv` (server-monitor) set.
|
||||
- **Turn on for a fleet:** push `winevt.events.enabled=true` (plus any `winevt.*` tuning) via a `CONFIG_UPDATE` fleet task.
|
||||
- **Turn off:** set `winevt.events.enabled=false` (needs restart to take effect since `enabled` isn't hot-reloaded) **or** set `winevt.events.suspended=true` (immediate, hot-reload) **or** add `win-app-events` to `modules.disabled`.
|
||||
- **Retrofit an ATM missing the JAR:** deliver an `INSTALL_MODULE` fleet task (drops the JAR into `modules/` and restarts the agent), or a full standard-set agent update.
|
||||
|
||||
## Common failure modes / what to check
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---|---|---|
|
||||
| No `WIN_APP_EVENT_ERROR` events ever arrive | `winevt.events.enabled` is at its default `false`, or module not on this ATM | Confirm `winevt.events.enabled=true` in `hiveops.properties`; confirm the JAR is in `modules/`; check the module log for `win-app-events initialized`. |
|
||||
| Module log says `not Windows, module inactive` | Running on a Linux ATM | Expected — this module is Windows-only. |
|
||||
| Started but silent | No new qualifying events since anchor, or everything filtered out | Log shows `anchored at RecordID N`, then `no new events` / `all N event(s) filtered`. Loosen `levels`/`sources`/pattern. |
|
||||
| Historical errors never backfilled after enabling | By design — first tick anchors to newest RecordID; only new events forward | Not a bug. There is no historical replay. |
|
||||
| Too noisy (flooding incidents) | `levels=3` (Warnings) + no source/pattern filter | Tighten `winevt.events.levels` to `2`, set `winevt.events.sources`, or set a `winevt.pattern.WIN_APP_EVENT_ERROR` regex. |
|
||||
| Changed the pattern but old skipped events don't reappear | Bookmark always advances even when events are filtered out | Expected — filters apply going forward only; already-passed RecordIDs are not re-queried. |
|
||||
| Pattern/suspend change didn't take effect | Only pattern + suspended hot-reload; the change wasn't a `CONFIG_UPDATE` (which fires the reload) | Push the change via `CONFIG_UPDATE`; for `levels`/`sources`/`interval`/`batch` you must restart the agent. |
|
||||
| Garbled message text | `wevtutil` charset / code page mismatch on that box | The runner decodes with the JVM `file.encoding`; check the ATM's active code page if provider messages look corrupted. |
|
||||
| Events fail to POST | Endpoint/auth wrong, or agent-proxy/incident down | Module logs `failed to send events`; each event is a POST to `/api/journal-events` — verify `agent.endpoint` and token. |
|
||||
Reference in New Issue
Block a user