hiveops-guide/backend/src/main/resources/content/technical/agent/gotchas.md

module, title, tab, order, audience

module	title	tab	order	audience
technical.agent	HiveOps Agent — on-ATM Java monitoring agent	Gotchas	40	dev

Agent gotchas & diagnostics

Windows JAR self-update needs an external updater process

Windows holds a file lock on any JAR loaded by a running JVM, so a self-update cannot copy over its own JAR from inside the same process — no CMD-script workaround gets around the lock. The only clean pattern is a standalone external updater (hiveops-agent-update.jar) launched outside the JVM's Job Object:

1. Write update-config.properties (source, dest, pid, restart params).
2. Launch hiveops-agent-update.jar via PowerShell Start-Process (escapes the Job Object).
3. Runtime.halt(0) in the agent — releases the file locks immediately.
4. The updater polls for the agent PID to die, copies the JAR, then restarts via schtasks.

Do not attempt CMD-script self-copy on Windows; it cannot work.

Heartbeat timestamps span three timezones — never diff them directly

Three different timezones are layered across the telemetry stack. Comparing them directly makes a live ATM look hours-stale:

• Backend-stored timestamps — devices.lastHeartbeat, fleet-task createdAt/startedAt/completedAt — are in Eastern (EDT UTC-4 / EST UTC-5), a fixed app display TZ (NOT the device's local TZ, NOT UTC).
• HTTP date: response header is UTC.
• Agent log files (inside a LOG_COLLECT zip) are in the ATM's own local time (e.g. an Alabama ATM = Central, CDT UTC-5).

Example — one reboot instant: agent log 14:04 CDT = fleet DB 15:04 EDT = 19:04 UTC.

To judge liveness: do NOT diff lastHeartbeat against the UTC date header. Either convert (add 4h in summer to reach UTC), or compare the device against the newest lastHeartbeat across its institution's fleet — if it matches the freshest cluster it's live. A whole institution clustered at "now minus 4h exactly" is the TZ artifact, not an outage. (The agentConnectionStatus mirror can also be stale — no reconciler.)

Command Center REBOOT: what it executes, and the Hyosung gotcha

The fleet REBOOT task is executed agent-side by ProcessReboot (in hiveops-file-collection, driven by CommandCenterModule). The flow on a Windows ATM:

1. Go out of service — WindowsCommands.goOutOfService() runs applctrl checkupdate <timeout>. applctrl is Wincor/Diebold tooling; on a Hyosung ATM it does not exist (CreateProcess error=2, file not found). Since agent 4.5.1 this missing-applctrl case is treated as an acceptable no-op and the flow proceeds (mirrors the Linux non-Wincor path). Before 4.5.1 it returned goOutOfService=-1 and the reboot never ran on Hyosung. (agent#82) A real non-0/100 applctrl exit code still fails, so a Wincor box that legitimately refuses is not force-rebooted.
2. Restart — WindowsCommands.reboot() runs shutdown /r /f /t 0 (forced Windows restart, no delay).
3. The agent persists a beforeReboot marker, then Thread.sleeps waiting to be killed by the OS. On a graceful reboot the JVM shutdown hook interrupts that sleep — this is expected (swallowed since 4.5.1) and must NOT be reported as failure. Before 4.5.1 the interrupt was caught as FAILED and it deleted the resume marker, so a successful reboot showed FAILED. (agent#94)
4. On restart, CommandProcessor reads the persisted marker and resumeProcessReboot posts COMPLETED. (A hard-killed JVM instead leaves the task RUNNING → stale re-dispatch after 5 min; the graceful case is the one that needed the fix.)

Open: there is no proper Hyosung/Nextware out-of-service command wired in yet — REBOOT currently force-restarts without draining. The agent-out-of-service.cmd script hook is the intended place to put a site's OOS command if one exists.

Deploying an agent fix to a box (e.g. to verify a reboot change): a fleet UPDATE_AGENT task must use a fleet-hosted (direct-upload) artifact — a CDN-hosted import-cdn artifact 410s because the UPDATE_AGENT handler has no CDN-download path (unlike SOFTWARE/HOTFIX). The patch applies the JAR but does not auto-restart the JVM, so send a RESTART_AGENT before testing the new code.