[session 1] docs(guide): agent + journal Gotchas tabs (Windows updater, heartbeat TZ, raw journal) (#33)

2026-07-17 17:55:05 -04:00
parent 6361d01e1b
commit 879e118579
2 changed files with 72 additions and 0 deletions
@@ -0,0 +1,42 @@
+---
+module: technical.agent
+title: HiveOps Agent — on-ATM Java monitoring agent
+tab: Gotchas
+order: 40
+audience: dev
+---
+
+# Agent gotchas & diagnostics
+
+## Windows JAR self-update needs an external updater process
+
+Windows holds a file lock on any JAR loaded by a running JVM, so a self-update **cannot** copy over
+its own JAR from inside the same process — no CMD-script workaround gets around the lock. The only
+clean pattern is a standalone external updater (`hiveops-agent-update.jar`) launched **outside** the
+JVM's Job Object:
+
+1. Write `update-config.properties` (source, dest, pid, restart params).
+2. Launch `hiveops-agent-update.jar` via PowerShell `Start-Process` (escapes the Job Object).
+3. `Runtime.halt(0)` in the agent — releases the file locks immediately.
+4. The updater polls for the agent PID to die, copies the JAR, then restarts via `schtasks`.
+
+Do not attempt CMD-script self-copy on Windows; it cannot work.
+
+## Heartbeat timestamps span three timezones — never diff them directly
+
+Three different timezones are layered across the telemetry stack. Comparing them directly makes a
+live ATM look hours-stale:
+
+- **Backend-stored timestamps** — `devices.lastHeartbeat`, fleet-task `createdAt`/`startedAt`/`completedAt`
+  — are in **Eastern (EDT UTC-4 / EST UTC-5)**, a fixed app display TZ (NOT the device's local TZ, NOT UTC).
+- **HTTP `date:` response header** is **UTC**.
+- **Agent log files** (inside a `LOG_COLLECT` zip) are in the **ATM's own local time** (e.g. an Alabama
+  ATM = Central, CDT UTC-5).
+
+Example — one reboot instant: agent log `14:04 CDT` = fleet DB `15:04 EDT` = `19:04 UTC`.
+
+**To judge liveness:** do NOT diff `lastHeartbeat` against the UTC `date` header. Either convert
+(add 4h in summer to reach UTC), or compare the device against the **newest `lastHeartbeat` across
+its institution's fleet** — if it matches the freshest cluster it's live. A whole institution
+clustered at "now minus 4h exactly" is the TZ artifact, not an outage. (The `agentConnectionStatus`
+mirror can also be stale — no reconciler.)
@@ -0,0 +1,30 @@
+---
+module: technical.journal
+title: Journal Service — ATM journal storage, parsing & gap detection
+tab: Gotchas
+order: 40
+audience: dev
+---
+
+# Journal gotchas & diagnostics
+
+## Diagnose service-state events against the RAW journal, not parsed data
+
+When checking whether an ATM actually *reported* a service-state event (IN_SERVICE / supervisor /
+etc.), **read the raw journal file — not the parsed `journal_events` table and not the agent parse
+log.** Both of those are downstream of the agent parser, and the parser can drop lines (e.g. a 30s
+dedup window suppressing a recovery). "0 events found" in the agent log means *parsed to nothing*
+(could be dedup/filter) — it never means the source line was absent.
+
+Raw journals live in the journal service:
+
+```
+hiveiq-journal:/app/uploads/journals/<INSTITUTION>/<ATM>/<YYYYMMDD>.txt      # services VM .250
+```
+
+- Stored as **UTF-8 with a per-line BOM** (`ef bb bf`) — NOT UTF-16LE despite the agent's read
+  format. Grep directly; strip the BOM with `sed 's/\xef\xbb\xbf//g'`.
+- Path/size recorded in `hiveiq_journal.journal_files`.
+
+**Never state "absent at source" from parsed data**, and never let a secondary-evidence conclusion
+propagate into a customer email or ticket before checking the primary (raw) source.