From 6542c4f348d25d3ebf53a70922187d787df810db Mon Sep 17 00:00:00 2001 From: Johannes Date: Wed, 1 Jul 2026 09:01:58 -0400 Subject: [PATCH] =?UTF-8?q?content(guide):=20per-service=20internal=20know?= =?UTF-8?q?ledge=20base=20=E2=80=94=2019=20technical.=20guides=20(DRA?= =?UTF-8?q?FT,=20Refs=20#3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One code-grounded internal guide per backend service (Responsibility / Data / Kafka / API surface / Gotchas), authored by parallel per-service agents reading each service's own source, then an adversarial verify pass (DB names, topics, endpoints spot-checked against code — 0 hard errors). Several guides correctly document CODE over stale CLAUDE.md (remote = HTTP long-poll not WebSocket; analytics = has a DB, not stateless; devices = has its own backend). Unconfirmed facts are marked (unverified). audience: internal, all DRAFT. Co-Authored-By: Claude Opus 4.8 --- .../content/technical/agent-proxy/overview.md | 58 ++++++++++ .../content/technical/agent/overview.md | 101 +++++++++++++++++ .../content/technical/ai/overview.md | 81 ++++++++++++++ .../content/technical/analytics/overview.md | 82 ++++++++++++++ .../content/technical/aria/overview.md | 71 ++++++++++++ .../content/technical/auth/overview.md | 81 ++++++++++++++ .../content/technical/claims/overview.md | 71 ++++++++++++ .../content/technical/config/overview.md | 53 +++++++++ .../content/technical/devices/overview.md | 94 ++++++++++++++++ .../content/technical/fleet/overview.md | 89 +++++++++++++++ .../content/technical/incident/overview.md | 83 ++++++++++++++ .../content/technical/journal/overview.md | 102 ++++++++++++++++++ .../content/technical/messaging/overview.md | 85 +++++++++++++++ .../content/technical/mgmt/overview.md | 75 +++++++++++++ .../content/technical/recon/overview.md | 77 +++++++++++++ .../content/technical/remote/overview.md | 84 +++++++++++++++ .../content/technical/reports/overview.md | 77 +++++++++++++ .../technical/transactions/overview.md | 49 +++++++++ .../content/technical/vault/overview.md | 71 ++++++++++++ 19 files changed, 1484 insertions(+) create mode 100644 backend/src/main/resources/content/technical/agent-proxy/overview.md create mode 100644 backend/src/main/resources/content/technical/agent/overview.md create mode 100644 backend/src/main/resources/content/technical/ai/overview.md create mode 100644 backend/src/main/resources/content/technical/analytics/overview.md create mode 100644 backend/src/main/resources/content/technical/aria/overview.md create mode 100644 backend/src/main/resources/content/technical/auth/overview.md create mode 100644 backend/src/main/resources/content/technical/claims/overview.md create mode 100644 backend/src/main/resources/content/technical/config/overview.md create mode 100644 backend/src/main/resources/content/technical/devices/overview.md create mode 100644 backend/src/main/resources/content/technical/fleet/overview.md create mode 100644 backend/src/main/resources/content/technical/incident/overview.md create mode 100644 backend/src/main/resources/content/technical/journal/overview.md create mode 100644 backend/src/main/resources/content/technical/messaging/overview.md create mode 100644 backend/src/main/resources/content/technical/mgmt/overview.md create mode 100644 backend/src/main/resources/content/technical/recon/overview.md create mode 100644 backend/src/main/resources/content/technical/remote/overview.md create mode 100644 backend/src/main/resources/content/technical/reports/overview.md create mode 100644 backend/src/main/resources/content/technical/transactions/overview.md create mode 100644 backend/src/main/resources/content/technical/vault/overview.md diff --git a/backend/src/main/resources/content/technical/agent-proxy/overview.md b/backend/src/main/resources/content/technical/agent-proxy/overview.md new file mode 100644 index 0000000..6c84f92 --- /dev/null +++ b/backend/src/main/resources/content/technical/agent-proxy/overview.md @@ -0,0 +1,58 @@ +--- +module: technical.agent-proxy +title: Agent-Proxy — agent-facing API gateway +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-agent-proxy source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-agent-proxy` is the **sole entry point for all ATM agent traffic** — no agent request reaches `hiveops-incident` directly. It (1) authenticates agents (bearer `agtkn_*` tokens validated via mgmt, or legacy `X-Institution-Key` header), (2) tracks per-machine fingerprints in its own DB, and (3) forwards each agent call natively to incident's internal API (`/api/internal/**`) using the shared `X-Internal-Secret`. Runs on container port **8093** (prod host port 8015). See [platform.service-ownership]. + +## Data + +Own PostgreSQL DB **`hiveiq_agent`** (prod profile uses `DB_HOST`/`DB_PORT`/`DB_NAME`/`DB_USERNAME`/`DB_PASSWORD`; `DB_NAME` defaults to `hiveiq_agent`). Dev profile runs H2 in-memory. See [platform.data-stores]. + +| Table / Entity | Purpose | +|---|---| +| `agent_connections` (`AgentConnection`) | Machine fingerprint registry — `machine_id` (unique), `atm_id`, `institution_key`, `agent_version`, `hostname`, `first_seen`, `last_seen`. V2 adds `device_id` (dual-write with `atm_id` via DB trigger). | + +## Kafka + +No Kafka. This service is a synchronous HTTP proxy — no producers or `@KafkaListener` consumers exist in the source. + +## API surface + +Base path `/api/agent` (plus a few root-level paths). All paths require `ROLE_AGENT` (set by `AgentAuthFilter`). Each controller forwards to the matching incident `/api/internal/**` endpoint via `IncidentInternalService`. + +| Method + Path | Controller | Purpose | +|---|---|---| +| PUT `/api/agent/{country}/{atm}/heartbeat` | HeartbeatController | Agent heartbeat; upserts machine fingerprint + updates device connection state | +| GET `/api/agent/{country}/{atm}/task` | FleetTaskController | Poll for pending fleet task | +| GET `/api/agent/tasks/{tid}/download` | FleetTaskController | Download task artifact | +| POST `/api/agent/tasks/{tid}/status` | FleetTaskController | Report task status | +| POST `/api/agent/tasks/{tid}/verify` | FleetTaskController | Verify task completion | +| POST `/api/agent/tasks/{tid}/upload` | FleetTaskController | Upload task result/log | +| POST `/api/agent/{country}/{atm}/auto-update` | FleetTaskController | Agent auto-update check | +| POST `/api/journal-events` | JournalEventController | Submit journal events | +| POST `/api/agent/register` | AgentRegistrationController | Agent/ATM registration | +| POST `/api/agent/{country}/{atm}/log/errors` | AgentLogController | Push error log entries | +| POST `/api/agent/{country}/{atm}/log/upload` | AgentLogController | Upload log file (multipart, ≤100MB) | +| POST `/api/atms/hardware/sync` · POST `/api/atms/config/sync` | AtmSyncController | Hardware / config sync | +| POST `/api/agent/{country}/{atm}/config` | AtmSyncController | Config sync (injects atmName+country) | +| POST `/api/agent/{country}/{atm}/modules` · PUT `/api/agent/{country}/{atm}/transit` | AgentModuleTransitController | Module inventory / transit state | +| `/api/agent/**` (wildcard) | AgentProxyController | Fallback for un-migrated legacy agent paths | + +Not exhaustive — see the controller package for the full set. + +## Gotchas + +- **Auth role is `ROLE_AGENT`, not the admin pattern.** `SecurityConfig` requires `hasRole("AGENT")` on every request except `/actuator/health`, `/actuator/info`, `/error`. Do NOT expect `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` here — that is for admin-facing services. +- **Two auth mechanisms, checked in order** (`AgentAuthFilter`): (1) `Authorization: Bearer agtkn_*` → validated against mgmt `/api/internal/agent-tokens/validate`, cached 5 min (`AGENT_TOKEN_CACHE_SECONDS`, default 300); (2) legacy `X-Institution-Key` header → provisional `ROLE_AGENT`, uppercased, incident re-validates. Principal is `agent:{institutionKey}` in both cases. +- **`INTERNAL_SERVICE_SECRET` is required.** All outbound calls to incident carry it as `X-Internal-Secret`; incident's `InternalSecretFilter` grants `ROLE_INTERNAL`. Empty by default in config. +- **Uses `DB_HOST`/`DB_PORT`/`DB_NAME`, not `DB_URL`.** (Contrast with hiveops-journal, which expects a full `DB_URL`.) Dev profile has Flyway disabled and uses H2 `create-drop`; prod uses Flyway + `ddl-auto: validate`. +- **`atm_id` / `device_id` dual-write.** V2 migration keeps both columns in sync via a BEFORE INSERT/UPDATE trigger (device-rename Phase 1). `MachineFingerprintService` should accept both `atmId` and `deviceId` JSON field names from heartbeats. +- **Exact `@PutMapping`/`@PostMapping` paths win over the `/api/agent/**` wildcard** in `AgentProxyController` — the wildcard is only a fallback for paths not yet given a native controller. diff --git a/backend/src/main/resources/content/technical/agent/overview.md b/backend/src/main/resources/content/technical/agent/overview.md new file mode 100644 index 0000000..255a048 --- /dev/null +++ b/backend/src/main/resources/content/technical/agent/overview.md @@ -0,0 +1,101 @@ +--- +module: technical.agent +title: HiveOps Agent — on-ATM Java monitoring agent +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-agent source. Verify against code before relying on specifics. + +## Responsibility + +The HiveOps Agent is a **Java client that runs on ATMs / servers**, not a Spring microservice. It parses journal files locally, uploads journal data + transaction images, reports structured events, and executes remote fleet-management commands (reboot, patch, file-collect, agent/module self-update). It is a multi-module Maven build (fat JAR via `maven-shade-plugin`) with **no REST controllers, no database, and no Kafka** — all integration is outbound HTTP to the `agent-proxy` / `incident` / `journal` services. See [platform.service-ownership]. + +Entry point: `com.hiveops.AgentApplication`. Module discovery is `ServiceLoader`-based (`com.hiveops.core.module.AgentModule` SPI) with lifecycle `initialize() → isEnabled() → start() → stop()`, topologically ordered by declared dependencies (`ModuleLoader` / `ModuleRegistry`). + +## Data + +**None.** The agent has no database and no ORM entities. State it needs to survive restarts is persisted locally to disk (e.g. `PersistantStateAndLog` for task state, per-file byte offsets for journal/applog readers). Fleet/journal/incident data lives in the backend services' own stores — see [platform.data-stores]. (Backend `hiveops-journal` reads env var `DB_URL`; the agent never touches it.) + +## Kafka + +**No Kafka.** The agent communicates only over HTTP(S). See [platform.kafka]. + +## API surface + +The agent is an **HTTP client**, so this section lists the outbound endpoints it *calls* (base URLs from `agent.endpoint` / `journal.endpoint` / `incident.endpoint` in `hiveops.properties`). Paths are drawn from the HTTP client classes (`FileUploadClient`, `TransactionImageHttpClient`, `IncidentEventClient`, config/hardware sync + aria clients). Not exhaustive. + +| Method | Path (relative to service base) | Purpose | Module / client | +|--------|--------------------------------|---------|-----------------| +| POST | `/api/journal-events` | Report parsed journal / app-log events | `journal-events` (`IncidentEventClient`) | +| POST | `/api/v1/agents/register` | Auto-register ATM on first contact | agent lifecycle | +| POST | `/api/v1/agents/heartbeat` | Periodic health / connection status | heartbeat | +| GET | `/api/v1/agents/poll` | Poll for pending fleet tasks | `command-processor` | +| GET/POST | `/api/v1/agents/commands/{...}` | Fetch / ack remote commands | `command-processor` | +| GET/POST | `/api/agent/tasks/{...}` | Fleet task detail / status update | `command-processor` | +| POST | `/api/agent/image/upload?atmName=&atmCountry=&datetime=&filename=&category=&orientation=` | Transaction image upload | `image-upload` | +| POST | `/api/agent/image/cheque/metadata?...` | Cheque image metadata | image (Gaia) | +| PUT/POST | `/api/agent/{country}/{atm}/transit?groupId=` | Report ATM transit | config-sync | +| POST | `/api/atms/config/sync` | Push ATM configuration | `atm-config-sync` | +| POST | `/api/atms/hardware/sync` | Push hardware inventory | `hw-inventory` | +| GET | `/api/atms/search?query=` | Resolve ATM id | `AtmIdResolver` | +| POST | `/api/aria/ingest/events`, `/api/aria/posture/report` | ARIA threat signals / posture | `aria-signals` | +| GET | `https://cdn.bcos.cloud/downloads/agent/latest.json` | Auto-update manifest poll | `auto-update` | + +Legacy `/atm/image/*/v1` endpoints (`FileUploadClient`, `TransactionImageHttpClient`) remain for backward compatibility. + +## Modules + +Registered via `META-INF/services/com.hiveops.core.module.AgentModule`; disable through `modules.disabled` (comma-separated). + +| Module (class) | Purpose | +|----------------|---------| +| `CommandProcessorModule` / `CommandCenterModule` | Poll + dispatch fleet tasks (`REBOOT`, `UPDATE`, `WINPATCH`, `FILECOLLECT`, `CCSUPPORT`, `RESTART_AGENT`, `UPDATE_AGENT`) | +| `AutoUpdateModule` / `ModuleAutoUpdateModule` | Self-update agent JAR / drop-in modules from CDN | +| `JournalUploadModule` | Chunked journal file upload | +| `JournalEventModule` | Parse journals → structured events → incident | +| `AppLogEventModule` | Monitor app logs for ERROR entries | +| `GenericImageUploadModule` / `GaiaImageUploadModule` | Transaction / cheque image upload | +| `LogUploadModule` / `LogMonitorModule` / `LogCollectPathModule` | Agent log upload + collection | +| `AtmConfigSyncModule` / `HardwareInventoryModule` | Config + hardware inventory sync | +| `EJournalDbMonitorModule` | Warn on Hyosung EJournal `.mdb` nearing 2 GB limit | +| `WinAppEventModule` | Windows application event monitoring | +| `NhTcrJournalModule` / `NhTcrEventsModule` / `TcrEventsModule` | TCR (teller cash recycler) journal + events | +| `ServerMonitorModule` | SRV installs: CPU/mem/disk/service monitoring | +| `AriaSignalModule` | ARIA threat-intelligence signals | +| `PacketCaptureModule` / `RemoteModule` | Packet capture / remote command execution | + +## Platform abstraction + +`com.hiveops.platform` — `PlatformServiceFactory` detects OS at runtime and returns Windows or Linux implementations of `ConfigurationReader`, `SystemCommands`, `PathResolver`, `SystemInfoProvider`. Windows reads config from the registry and runs `.cmd` scripts; Linux reads `/etc/hiveops/atm-config.properties` + `HIVEOPS_*` env vars and runs `.sh` scripts. + +## Key config keys + +From `hiveops-app/src/main/resources/hiveops.properties`: + +| Key | Meaning | +|-----|---------| +| `agent.endpoint` | Base URL for fleet mgmt + heartbeat (default `https://api.bcos.cloud/incident`; replaces deprecated `server.endpoint`) | +| `journal.endpoint` | Journal upload/query base URL (separate service) | +| `incident.endpoint` | Deprecated fallback for events (older builds) | +| `agent.auth.token` | Bearer token (Incident → Settings → Agent Properties → Tokens); preferred auth | +| `agent.institution.key` | Legacy institution auth, used only if `agent.auth.token` unset | +| `agent.cdn.token` | Shared `X-CDN-Token` for `/downloads/agent/` + `/downloads/hotfix/` | +| `atm.id` / `country` / `atm.transit` | ATM identity (id resolution: property → `HIVEOPS_ATM_ID` → registry → hostname) | +| `heartbeat.interval` / `heartbeat.restart.threshold` | Heartbeat cadence; self-restart after N consecutive failures | +| `agent.update.enabled` / `agent.update.check.url` / `agent.update.window.start`/`end`/`timezone` | Auto-update toggle, manifest URL, after-hours window | +| `incident.events.enabled` / `.batch.size` / `.recheck.delay.msec` | Event reporting behavior | +| `applog.events.*` | App-log ERROR monitoring (dir, filename format, correlation) | +| `ejournal.db.monitor.*` | EJournal DB size warn/critical thresholds | +| `srv.monitor.*` | SRV-install CPU/mem/disk/service monitoring (ATMs leave `enabled=false`) | +| `modules.disabled` | Comma-separated modules to skip | + +## Gotchas + +- **Auth headers:** the agent sends `Authorization: Bearer ` when set, else legacy `X-Institution-Key`. Update downloads carry `X-CDN-Token`; there is also `X-ATM-Profile` and (internal) `X-Service-Secret`. Institution is auto-derived server-side from the token/key. +- **`agent.endpoint` vs `incident.endpoint`:** `agent.endpoint` is the current key; `incident.endpoint` and `server.endpoint` are deprecated fallbacks kept for older builds — don't rely on them for new work. +- **Token revocation kills module threads:** revoking `agent.auth.token` while ATMs still hold it kills their module threads and requires a manual JVM restart per machine. Push a fleet-wide `CONFIG_UPDATE` to clear the token and wait for tasks to drain *before* revoking. +- **ATM id resolution order matters:** property → `HIVEOPS_ATM_ID` env → Windows registry → hostname fallback; a mis-imaged box can silently take the wrong identity/institution. +- **`agent.update.enabled=false` customers** (e.g. ATEC Java 8 branch) get patches by hand via share.bcos.cloud, not CDN auto-update. +- **`hiveops-module-tcr-events` directory exists but is not a listed ``** in the root `pom.xml` (the built TCR modules are `atec-tcr-events` / `nh-tcr-events`); treat the bare `tcr-events` dir as legacy. (unverified intent) diff --git a/backend/src/main/resources/content/technical/ai/overview.md b/backend/src/main/resources/content/technical/ai/overview.md new file mode 100644 index 0000000..6ed664b --- /dev/null +++ b/backend/src/main/resources/content/technical/ai/overview.md @@ -0,0 +1,81 @@ +--- +module: technical.ai +title: Adoons (hiveops-ai) — Spring AI fleet assistant & incident/rule intelligence +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-ai source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-ai` (externally branded **Adoons**) is the Spring AI microservice on port **8085**. It owns the natural-language chat assistant (`ChatClient` over Anthropic Claude with `@Tool` function-calling into upstream services) plus a set of internal AI helpers for the incident domain: RAG incident recommendations, journal-line rule/closing-rule suggestions, ATM customer-journey analysis, and periodic analytics insight generation. It is read-only against fleet data — it queries other services on the caller's behalf and never mutates their domain. Cross-link [platform.service-ownership]. + +## Data + +Two separate Hikari datasources (Spring Boot `DataSourceAutoConfiguration` is **excluded** in `application.yml`; both are wired manually). + +| Datasource | Default DB | Config key | Tables | +|-----------|-----------|-----------|--------| +| `aiDataSource` (`AiDatabaseConfig`) | `hiveiq_ai` | `AI_DB_URL` / `hiveops.ai.db.url` | `ai_conversation_sessions`, `ai_chat_memory` | +| `vectorDataSource` (`VectorStoreConfig`) | `hiveops_incident` | `VECTOR_DB_URL` / `hiveops.ai.vectordb.url` | `incident_embeddings` (pgvector) | + +- **`ai_conversation_sessions`** and **`ai_chat_memory`** are auto-created via `@PostConstruct CREATE TABLE IF NOT EXISTS` in `ConversationSessionRepository` / `JdbcChatMemoryRepository` — no Flyway in this service. Chat history is persisted (not in-memory, despite older CLAUDE.md notes). +- **`incident_embeddings`** is a `PgVectorStore` (COSINE distance, **1024 dims** for `voyage-3`), `initializeSchema(false)` — its schema is owned by Flyway in **hiveops-incident**, not here. + +Cross-link [platform.data-stores]. (Journal note: several sibling services use env var `DB_URL`; hiveops-ai instead uses `AI_DB_URL` + `VECTOR_DB_URL`.) + +## Kafka + +Constants in `AiKafkaConfig`. Bootstrap `KAFKA_BOOTSTRAP_SERVERS` (default `kafka:29092`). + +**Consumed** +| Topic | Listener class | Group | +|-------|---------------|-------| +| `analytics.snapshot-ready` | `AnalyticsInsightJob.onSnapshotReady` | `ai-insight-group` | +| `adoons.rule-suggestion.requested` | `RuleSuggestionConsumer.onRequest` | `ai-adoons-group` | + +**Produced** +| Topic | Producer class | Notes | +|-------|---------------|-------| +| `adoons.rule-suggestion.ready` | `RuleSuggestionConsumer` (`ruleSuggestionKafkaTemplate`) | Async result of a rule-suggestion request; status `READY`/`FAILED` | + +Note: `AnalyticsInsightJob` reacts to `analytics.snapshot-ready` but publishes its insights back over **REST** to hiveops-analytics (`AnalyticsApiClient`), not Kafka. Consumer `auto-offset-reset: latest` — historical snapshots are skipped on redeploy. Cross-link [platform.kafka]. + +## API surface + +Two controllers. `/api/ai/**` requires a customer JWT; `/api/adoons/**` is internal (`permitAll` in Spring Security, gated inside the controller by the `X-Internal-Secret` header). + +**`AiChatController` — base `/api/ai`** (JWT auth, `@AuthenticationPrincipal String userEmail`) +| Method | Path | Purpose | +|--------|------|---------| +| POST | `/api/ai/chat` | Synchronous chat; returns `{message, conversationId}` | +| POST | `/api/ai/chat/stream` | SSE streaming chat (`text/event-stream`) with 20s heartbeat | +| GET | `/api/ai/conversations` | List current user's conversations | +| GET | `/api/ai/conversations/{id}/messages` | Full message history for a conversation | +| DELETE | `/api/ai/conversations/{id}` | Delete a conversation + its messages | + +**`IncidentAiController` — base `/api/adoons`** (X-Internal-Secret, called by hiveops-incident) +| Method | Path | Purpose | +|--------|------|---------| +| POST | `/api/adoons/incidents/index` | Upsert an incident into the pgvector store (on create/update) | +| POST | `/api/adoons/incident-recommend` | RAG recommendation from similar resolved incidents | +| POST | `/api/adoons/suggest-rule` | Suggest a processing rule from a raw log line (sync) | +| POST | `/api/adoons/suggest-closing-rule` | Suggest a closing rule (sync) | +| POST | `/api/adoons/analyze-journey` | Analyze a raw ATM journal block → JSON customer-journey narrative | +| POST | `/api/adoons/journal/index-now` | Trigger journal embedding run immediately (bypass 02:00 cron) | + +The `suggest-rule`/`suggest-closing-rule` endpoints have an **async Kafka twin**: `adoons.rule-suggestion.requested` → `RuleSuggestionConsumer` → `adoons.rule-suggestion.ready`. Both the sync REST path and the async Kafka path share `RuleSuggestionService` / `ClosingRuleSuggestionService`. + +## Gotchas + +- **No role checks.** `/api/ai/**` only requires an authenticated caller (`anyRequest().authenticated()`) — any valid customer JWT (issued by **hiveops-auth**, HS512) works; there is no `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. Upstream tool calls forward the caller's raw token (`RequestTokenHolder`, `@RequestScope`) so downstream services enforce their own permissions. +- **`JWT_SECRET` must match hiveops-auth**, not the mgmt internal secret. +- **`X-Internal-Secret`** (`INTERNAL_SERVICE_SECRET` / `hiveops.internal.service-secret`) guards all `/api/adoons/**` endpoints in the controller — if it is blank/unset, every internal call is rejected with 403 (`isValidSecret` requires a non-blank configured secret). +- **Streaming token capture:** `streamChat` reads the JWT from `RequestTokenHolder` on the servlet thread *before* entering the Reactor chain, because `@RequestScope` is not visible on scheduler threads where tool calls run. `shouldFilterAllDispatcherTypes(false)` skips re-auth on ASYNC SSE dispatches. +- **Analytics insight throttle:** even when `analytics.snapshot-ready` events arrive frequently, `AnalyticsInsightJob` only regenerates once per `INSIGHT_INTERVAL_MINUTES` (default 30). +- **Journal indexing cron:** `JournalIndexingJob` runs `0 0 2 * * *` (02:00 UTC daily); `/api/adoons/journal/index-now` forces an immediate run. +- **Model/provider swap:** default is Anthropic `claude-sonnet-4-6` + `voyage-3` embeddings; activating the `ollama` profile (`SPRING_PROFILES_ACTIVE=prod,ollama`) disables Anthropic and uses local Ollama chat + `mxbai-embed-large` embeddings instead. `spring.ai.retry.max-attempts: 1` (no retries). +- **DB-name inconsistency:** the vector datasource default URL points at `hiveops_incident` while most platform DBs are named `hiveiq_*` — confirm the actual DB name per environment before relying on it. +- **Stale CLAUDE.md:** the repo's `CLAUDE.md` still describes a stateless service with `InMemoryChatMemory` and only Fleet/Incident/Mgmt tools; the code now persists chat to Postgres and has more tool classes (`ClaimsTools`, `FleetTools`, `IncidentTools`, `JournalTools`, `MgmtTools`, `PatternTools`, `RagTools`). diff --git a/backend/src/main/resources/content/technical/analytics/overview.md b/backend/src/main/resources/content/technical/analytics/overview.md new file mode 100644 index 0000000..2c626be --- /dev/null +++ b/backend/src/main/resources/content/technical/analytics/overview.md @@ -0,0 +1,82 @@ +--- +module: technical.analytics +title: Analytics — fleet dashboard aggregation & snapshots +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-analytics source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-analytics` computes fleet dashboard analytics by aggregating data from **hiveops-incident** and **hiveops-journal** (and **hiveops-mgmt**). It exposes read-oriented dashboard endpoints (fleet health, uptime, incident trends, cash intelligence, journal coverage, agent insights) and persists periodic **snapshots** plus **AI insights** ("Adoons"-style callouts) in its own database. Runs on **port 8089** (`PORT` env override). Cross-link [platform.service-ownership]. + +Note: the in-repo `CLAUDE.md` still describes the service as "stateless — no database," but the code has since gained its own DB (`analytics_snapshots`, `ai_insights`), a snapshot scheduler, and Kafka consumers. Treat it as stateful with an aggregation cache. + +## Data + +Database: **`hiveiq_analytics`** (PostgreSQL in `prod`, H2 in-memory in `dev`). Cross-link [platform.data-stores]. + +Prod connection uses split env vars **`DB_HOST` / `DB_PORT` / `DB_NAME` / `DB_USERNAME` / `DB_PASSWORD`** (JDBC URL assembled in `application.yml`), not a single `DB_URL`. `DB_NAME` defaults to `hiveiq_analytics`. + +| Table | Purpose | +|-------|---------| +| `analytics_snapshots` | Point-in-time fleet metrics: atm counts (total/connected/disconnected/never-connected), incidents by severity (open/critical/high/medium/low), tasks (pending/in-progress/completed-today), transactions today + failed, plus `opened_incidents_today` / `closed_incidents_today` (V2). Unique on `(snapshot_date, snapshot_hour)`. | +| `ai_insights` | Generated insight cards: `category`, `severity`, `title`, `body`, optional `snapshot_id` FK, `expires_at`, `acknowledged` / `acknowledged_at` / `acknowledged_by_user_id`. | + +Flyway migrations in `src/main/resources/db/migration/` (V1 init, V2 open/close incident columns, V3 snapshot_hour type fix). Cache TTL for aggregations via `ANALYTICS_CACHE_TTL` (default 300s). + +## Kafka + +Topic constants in `kafka/AnalyticsKafkaConfig.java`. Cross-link [platform.kafka]. + +**Produced** + +| Topic | Producer class | Notes | +|-------|----------------|-------| +| `analytics.snapshot-ready` | `SnapshotEventProducer` | Emitted (key `"snapshot"`, `SnapshotReadyEvent{snapshotDate, snapshotHour}`) after a snapshot is collected by `SnapshotCollectionService`. | + +**Consumed** + +| Topic | Listener class | +|-------|----------------| +| `hiveops.incidents.created` | `IncidentEventConsumer` | +| `hiveops.incidents.updated` | `IncidentEventConsumer` | +| `hiveops.incidents.resolved` | `IncidentEventConsumer` | +| `journal.file-parsed` | `JournalFileParsedConsumer` | + +Consumer group `hiveops-analytics`, `auto-offset-reset: latest`. Incident/journal events trigger real-time snapshot `collect()`; a `@Scheduled(cron = "0 0 */6 * * *")` job in `SnapshotCollectionService` is the 6-hour fallback. + +## API surface + +Most controllers sit under base **`/api/analytics`**; snapshots and insights have their own sub-paths; there is one unauthenticated public path. + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/api/analytics/dashboard/overview` | Fleet overview: atm counts, incidents, tasks, transactions | +| GET | `/api/analytics/fleet-health` | Fleet health summary | +| GET | `/api/analytics/uptime` | Uptime metrics | +| GET | `/api/analytics/incident-trends` | Incident trends over a period | +| GET | `/api/analytics/cash-intelligence` | Cash intelligence metrics | +| GET | `/api/analytics/journal-coverage` | Journal coverage/reporting metrics | +| GET | `/api/analytics/agents` | Agent insights | +| GET | `/api/analytics/snapshots/latest` | Most recent snapshot | +| GET | `/api/analytics/snapshots` | List snapshots | +| GET | `/api/analytics/snapshots/range` | Snapshots for a date range | +| GET | `/api/analytics/insights` | All AI insights | +| GET | `/api/analytics/insights/active` | Active (unacknowledged, unexpired) insights | +| POST | `/api/analytics/insights` | Create an insight — `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` | +| GET | `/api/public/stats` | Public stats (no auth) | + +Not exhaustive. + +## Gotchas + +- **`/api/public/**` and `/actuator/health` are `permitAll()`** in `SecurityConfig`; everything else requires an authenticated JWT. `/api/public/stats` is intentionally unauthenticated — do not put sensitive fleet data there. +- **Only `POST /api/analytics/insights` is role-gated** (`hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`); the many GET endpoints are `.anyRequest().authenticated()` with no role restriction. +- **Service-to-service auth forwards the caller's JWT** to incident/journal/mgmt (`WebClientConfig`); there is no separate service account. Downstream calls fail if the user token lacks access. +- **Downstream URLs are env-configured** (`INCIDENT_SERVICE_URL`, `JOURNAL_SERVICE_URL`, `MGMT_SERVICE_URL`), defaulting to in-cluster hostnames (`hiveops-incident-backend:8080`, `hiveops-journal:8087`, `hiveops-mgmt:8080`). +- **Snapshots are event-driven with a 6h cron fallback**; if incident/journal Kafka events stop flowing, dashboard freshness degrades to the 6-hour cadence. +- **Prod uses split DB_HOST/DB_PORT/DB_NAME env vars** (unlike hiveops-journal, which expects a single `DB_URL`). +- **`dev` profile uses H2 with `ddl-auto: create-drop` and Flyway disabled**; `prod` uses PostgreSQL with `ddl-auto: validate` and Flyway enabled — schema drift surfaces only in prod validation. diff --git a/backend/src/main/resources/content/technical/aria/overview.md b/backend/src/main/resources/content/technical/aria/overview.md new file mode 100644 index 0000000..2a04a98 --- /dev/null +++ b/backend/src/main/resources/content/technical/aria/overview.md @@ -0,0 +1,71 @@ +--- +module: technical.aria +title: ARIA — ATM threat intelligence & precursor-sequence detection +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-aria source. Verify against code before relying on specifics. + +## Responsibility + +ARIA (`hiveops-aria`, `spring.application.name=hiveops-aria`, port **8095**) is the threat-intelligence service. It owns the IOC (indicator-of-compromise) catalog seeded from FBI FLASH alerts and refreshed from external feeds (OTX, MalwareBazaar, ThreatFox), ingests device threat events from agents, classifies/matches them against IOCs, and detects multi-phase attack "precursor sequences" (e.g. ATM jackpotting) within a time window. It also tracks per-device security posture. Cross-link [platform.service-ownership]. + +## Data + +- **Database:** `hiveiq_aria` (PostgreSQL). Datasource via `DB_URL` env var (`spring.datasource.url=${DB_URL:jdbc:postgresql://localhost:5432/hiveiq_aria}`), `DB_USERNAME`/`DB_PASSWORD`. `ddl-auto=validate`; schema managed by **Flyway** (`db/migration`, V1–V8). Cross-link [platform.data-stores]. + +| Table | Purpose | +|-------|---------| +| `threat_ioc` | IOC catalog (ioc_type, value, source, source_ref, confidence, active, expires_at). Seeded in V2 from FBI FLASH `FLASH-20260219-001` (Ploutus jackpotting). | +| `threat_event` | Ingested/classified device events (device_agent_id, signal_key, event_type, severity, matched_ioc_id, attack_phase, sequence_id, raw_payload JSONB). | +| `precursor_sequence_alert` | Detected attack sequences (sequence_type, phases_detected[], confidence, auto_response_taken, resolved_at/by, sequence_id UUID). | +| `device_security_posture` | Per-device posture (gold_image_hash vs current_image_hash, disk_encryption_enabled, os_eol_status, posture_score). | +| `ioc_feed_run` | External feed refresh run history (added V6). | + +## Kafka + +**Produced** (both defined in `AriaKafkaConfig`, JSON serializer): + +| Topic | Producer class | Trigger | +|-------|---------------|---------| +| `hiveops.aria.threats` | `AriaThreatProducer.publishThreatEvent` | called from `ThreatEventIngestionService` (key = deviceAgentId) | +| `hiveops.aria.sequences` | `AriaSequenceProducer.publishSequenceAlert` | called from `SequenceDetectionService` when a precursor sequence trips | + +**Consumed:** No Kafka consumers — no `@KafkaListener` in the codebase. `spring.kafka.admin.fail-fast=false`. Cross-link [platform.kafka]. + +## API surface + +Base paths under `/api/aria`. Most read/admin endpoints require `hasRole('BCOS_ADMIN')`. + +| Method | Path | Purpose | +|--------|------|---------| +| POST | `/api/aria/ingest/events` | Agent/internal event ingestion (X-Service-Secret header, permitAll) | +| POST | `/api/aria/posture/report` | Device posture report submission (permitAll) | +| GET | `/api/aria/posture` | List device postures | +| GET | `/api/aria/posture/{deviceId}` | Single device posture | +| GET | `/api/aria/events` | List threat events | +| GET | `/api/aria/stats` | Threat/dashboard stats | +| GET | `/api/aria/ioc` | List IOCs | +| GET | `/api/aria/ioc/active` | Active IOCs (authenticated, no BCOS_ADMIN gate) | +| POST | `/api/aria/ioc` | Create IOC | +| PUT | `/api/aria/ioc/{id}` | Update IOC | +| DELETE | `/api/aria/ioc/{id}` | Delete IOC | +| GET | `/api/aria/sequences` | List precursor sequence alerts | +| GET | `/api/aria/sequences/{id}/events` | Events composing a sequence | +| POST | `/api/aria/sequences/{id}/resolve` | Resolve a sequence alert | +| GET | `/api/aria/feeds/status` | Feed status | +| GET | `/api/aria/feeds/runs` | Feed run history | +| POST | `/api/aria/feeds/refresh` `/refresh/{feedName}` | Trigger feed refresh | + +## Gotchas + +- **Auth model:** stateless JWT (`JwtAuthenticationFilter`) + `@EnableMethodSecurity`. Almost every admin/read endpoint uses `hasRole('BCOS_ADMIN')` — note it is **BCOS_ADMIN only**, not the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` seen elsewhere. +- **Public endpoints:** `/api/aria/ingest/**` and `/api/aria/posture/report` are `permitAll()` in `SecurityConfig`. Ingestion is instead gated by the `X-Service-Secret` header compared to `aria.internal.service-secret` (`ARIA_SERVICE_SECRET`) — **but only if that secret is non-blank**; if unset, the endpoint accepts any request. +- `GET /api/aria/ioc/active` has no `@PreAuthorize`, so any authenticated caller can read active IOCs (unlike the BCOS_ADMIN-gated `/ioc`). +- **Sequence detection window:** `aria.sequence.window.minutes` (`ARIA_SEQUENCE_WINDOW_MINUTES`, default 15). +- **Feed refresh:** scheduled via cron `aria.feed.refresh.cron` (default `0 0 3 * * *`), toggle `ARIA_FEED_ENABLED` (default true). Optional API keys for OTX / MalwareBazaar / ThreatFox. +- **Auto-response is off by default:** `aria.autoresponse.shutdown.enabled=false`. Auto-response fleet tasks would call hiveops-incident (`INCIDENT_SERVICE_URL`, default `http://hiveiq-incident:8080`). +- Swagger disabled unless `SWAGGER_ENABLED=true`. `spring.security` logging is at DEBUG. +- Enum DB types were converted to VARCHAR in V3 (`convert_enums_to_varchar`) despite V1 defining PG enum types. diff --git a/backend/src/main/resources/content/technical/auth/overview.md b/backend/src/main/resources/content/technical/auth/overview.md new file mode 100644 index 0000000..ba6dfd4 --- /dev/null +++ b/backend/src/main/resources/content/technical/auth/overview.md @@ -0,0 +1,81 @@ +--- +module: technical.auth +title: hiveops-auth — Customer authentication + TOTP MFA + JWT issuer +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-auth source. Verify against code before relying on specifics. + +## Responsibility + +Owns **customer** authentication only — internal/admin auth lives in `hiveops-mgmt`. Handles login, TOTP/backup-code MFA, JWT issuance + refresh, self-service registration, password reset, and account lockout. It also issues the JWTs that other microservices validate via the shared `JWT_SECRET` (see `hiveops-security-common`). Serves both a JSON API (`/auth/api/**`) and server-rendered Thymeleaf login UI (`/auth/**`). Cross-link [platform.service-ownership]. + +- Spring Boot 3.4.1, Java 21, port **8082** (`${PORT:8082}`). +- JWT generation resolves institution keys by calling `hiveops-mgmt` (`mgmt-service.url`, default `http://hiveiq-mgmt:8080`). +- MSP domain (`auth.msp-domain`, default `bcos.cloud`) is granted `MSP_ADMIN` role in the JWT. + +## Data + +**Database:** `hiveiq_auth` (PostgreSQL in prod, H2 in-memory in dev). Note: the prod JDBC URL is built from `DB_HOST`/`DB_PORT`/`DB_NAME` (env `DB_NAME` default in code is `hiveops_auth`, but the deployed DB is `hiveiq_auth`) — not a full `DB_URL`. Cross-link [platform.data-stores]. + +Key tables/entities (Flyway `db/migration`, prod only; dev uses `ddl-auto: create-drop`): + +| Table | Purpose | +|-------|---------| +| `users` | Customer accounts, password hash, role, MFA secret (AES-encrypted), lockout/attempt counters | +| `mfa_backup_codes` | 10 BCrypt-hashed single-use backup codes per user | +| `auth_sessions` | Short-lived (5 min) intermediate MFA sessions between login step 1 and 2 | +| `refresh_tokens` | Issued refresh tokens | +| `auth_audit_logs` | Auth event audit trail (queried by AuditAdminController) | +| `customer_domains` | Email-domain → institution key/name mapping | +| `password_reset_tokens` | 1-hour password reset tokens | +| `SPRING_SESSION` | Spring Session JDBC store (server-rendered UI sessions, auto-created) | + +## Kafka + +No Kafka. No producers, listeners, `KafkaTemplate`, or `StreamBridge` in the service. Cross-link [platform.kafka]. + +## API surface + +Base paths: `/auth/api` (JSON), `/auth` (Thymeleaf UI). Most important JSON endpoints: + +| Method Path | Purpose | +|-------------|---------| +| POST `/auth/api/login` | Step 1 — credentials (field is **`email`**, not username); returns token or `mfaRequired` + sessionToken | +| POST `/auth/api/verify-mfa` | Step 2a — verify TOTP code, return access + refresh token | +| POST `/auth/api/verify-backup-code` | Step 2b — verify backup code | +| POST `/auth/api/refresh` | Refresh access token | +| POST `/auth/api/logout` | Invalidate session/token | +| POST `/auth/api/change-password` | Change password (authenticated) | +| POST `/auth/api/register` | Self-service registration | +| GET `/auth/api/mfa/status` | MFA enabled? backup codes remaining? | +| POST `/auth/api/mfa/setup` | Begin MFA setup → QR + secret | +| POST `/auth/api/mfa/confirm` | Confirm TOTP to activate MFA | +| POST `/auth/api/mfa/disable` | Disable MFA | +| POST `/auth/api/mfa/regenerate-backup-codes` | Issue new backup codes | +| GET/PUT `/auth/api/users/me` | Current user profile; `/preferences` sub-paths | +| GET `/auth/api/docs-validate` | Docs-site JWT validation (DocsAuthController) | + +Admin / service-to-service (not exhaustive): + +| Method Path | Guard | +|-------------|-------| +| GET `/auth/api/msp/users`, PUT/POST `/auth/api/msp/users/{id}/...` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` | +| GET `/auth/api/admin/audit/logs`, `/summary` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` | +| `/auth/api/admin/rate-limit/**`, `/auth/api/admin/users/**` | `hasRole('ADMIN')` | +| `/auth/api/internal/users/**`, `/domains/**` | `hasRole('SERVICE')` (shared `AUTH_SERVICE_TOKEN`) | + +## Gotchas + +- **Login field is `email`, not `username`** — wrong field returns 403. +- **Two role vocabularies:** MSP/audit controllers use `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`; rate-limit and `/admin/users` controllers use `hasRole('ADMIN')`; internal endpoints use `hasRole('SERVICE')`. They are not interchangeable. +- **Fail-fast secrets:** `JWT_SECRET` and `MFA_SECRET_ENCRYPTION_KEY` have no defaults — the app will not start without them. `JWT_SECRET` must match `hiveops-mgmt` exactly (shared token validation). +- **Refresh expiration is 1 day**, despite older docs mentioning 7 days — `jwt.refresh-expiration` and `short-refresh-expiration` are both `86400000` (24h); "remember me" now only saves the email, not a long session. +- **MFA:** SHA1, 6 digits, 30s period; secret AES-encrypted at rest; 10 BCrypt-hashed single-use backup codes. +- **Lockout:** 5 failed attempts → 15-minute lockout (`auth.max-login-attempts` / `auth.lockout-duration`). Unlock via SQL: `UPDATE users SET failed_login_attempts=0, locked_until=NULL WHERE email=...`. +- **Rate limiting** is in-process (bucket4j-style config under `rate-limit`): IP bucket 20/min, user bucket 5/15min. +- **CORS_ALLOWED_ORIGINS is required** — service refuses to start without it; new frontend origins must be added here AND in `hiveops-mgmt`. +- **Password expiry / inactivity:** forced password change after 90 days; account auto-disabled after 90 days inactivity (`auth.password-expiry-days` / `inactivity-disable-days`). +- **Dev profile** uses H2 in-memory with Flyway disabled and seeds `db/h2/data.sql`; Swagger + H2 console only enabled via `SWAGGER_ENABLED` / dev profile. diff --git a/backend/src/main/resources/content/technical/claims/overview.md b/backend/src/main/resources/content/technical/claims/overview.md new file mode 100644 index 0000000..feab569 --- /dev/null +++ b/backend/src/main/resources/content/technical/claims/overview.md @@ -0,0 +1,71 @@ +--- +module: technical.claims +title: hiveops-claims — ATM cash-shortage loss recovery service +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-claims source. Verify against code before relying on specifics. + +## Responsibility + +Spring Boot backend (port 8092, image `hiveiq-claims`) that tracks ATM cash-shortage loss-recovery **claims** and their multi-party payment splits between the institution, the CIT (cash-in-transit) vendor, and the SP (service provider). Owns claims, vendors, claim messages/notes/attachments, payment splits, and per-institution email templates. All claim/vendor data is scoped by `institutionKey` from the JWT principal. Frontend is a separate Svelte SPA (`claims.bcos.cloud`, image `hiveiq-claims-frontend`). See [platform.service-ownership]. + +## Data + +- **Database:** `hiveiq_claims` (PostgreSQL, `prod` profile). `dev` profile uses H2 in-memory (`create-drop`, Flyway disabled). See [platform.data-stores]. +- **Connection env vars:** split form `DB_HOST` / `DB_PORT` (default 5432) / `DB_NAME` (default `hiveiq_claims`) / `DB_USERNAME` / `DB_PASSWORD` — note: NOT the single `DB_URL` form used by hiveops-journal. +- **Key tables** (Flyway `db/migration`, V1–V11): + +| Table | Purpose | +|-------|---------| +| `claims` | Shortage claim per terminal — shortage amount, loss period, status, claim type, CIT/SP vendor FKs, `submitted_at` | +| `claim_payments` | Per-party owed amount (`INSTITUTION` / `CIT` / `SP`), one row per involved party | +| `claim_messages` | Threaded institution↔vendor messages (V3 adds additional email) | +| `claim_notes` | Internal institution-only notes, never shown to vendors | +| `claim_attachments` | Uploaded documents (stored under `uploads.dir`, default `/tmp/claims-uploads`) | +| `vendors` | CIT/SP vendor registry, scoped by `institutionKey` | +| `email_templates` | Per-institution email templates (V2) | +| `device_summary` | Read-model mirror of devices, kept in sync via Kafka (V9/V11) | + +Claim status enum (entity `Claim.ClaimStatus`, updated by V5/V8): `OPEN`, `WAITING_CIT`, `WAITING_SP`, `WAITING_INSTITUTION`, `PENDING_PAYMENT`, `RESOLVED_PAID`, `RESOLVED_OFFAGE_FOUND`, `RESOLVED_DENIED`. (The `OPEN→REVIEW→PENDING→RESOLVED` set in CLAUDE.md is stale.) + +## Kafka + +- **Consumed:** `hiveops.device.events` — `DeviceEventConsumer` (`@KafkaListener`, group `hiveops-claims-device-sync`, factory `deviceEventKafkaListenerContainerFactory`; topic/group constants in `DeviceEventKafkaConfig`). Upserts/deletes rows in `device_summary`; `DEVICE_DELETED` events remove the record; missing `institutionKey` defaults to `"BCOS"`. +- **Produced:** None. + +See [platform.kafka]. This service is one of the `device_summary` mirror consumers (incident/fleet/reports/recon/claims). + +## API surface + +All endpoints require a JWT Bearer token except health/swagger/h2-console/error and `/api/internal/**` (see Gotchas). Base path `/api/claims`; externally routed as `api.bcos.cloud/claims/`. + +| METHOD | Path | Purpose | +|--------|------|---------| +| GET | `/api/claims` | List claims (filter `?status=`, `?search=`) | +| GET | `/api/claims/stats` | Dashboard stats — exposure, recovered YTD, totals | +| POST | `/api/claims` | File claim — creates split payments across involved parties | +| GET | `/api/claims/{uuid}` | Claim detail with payments | +| PUT | `/api/claims/{uuid}` | Update status / claim type / assign-clear CIT & SP vendors | +| GET | `/api/claims/atms` | ATM list (from `device_summary`) for claim creation | +| GET/POST | `/api/claims/vendors` | List (`?type=CIT`/`SP`) / register vendors | +| GET/POST | `/api/claims/{claimUuid}/messages` | Vendor thread (`?vendorUuid=` filter) / send message | +| GET/POST | `/api/claims/{claimUuid}/notes` | Internal notes | +| GET/POST | `/api/claims/{claimUuid}/attachments` | List / upload attachments | +| GET | `/api/claims/{claimUuid}/attachments/{attachmentUuid}/download` | Download an attachment | +| GET | `/api/claims/{claimUuid}/payments` | Payment breakdown by party | +| PUT | `/api/claims/{claimUuid}/payments/{party}` | Update one party's owed amount (`party`: INSTITUTION/CIT/SP) | +| GET/POST/PUT/DELETE | `/api/claims/settings/email-templates` | Manage per-institution email templates (`/effective` = resolved) | +| GET | `/api/users/me` | Current principal info | +| GET/POST/PUT/DELETE | `/api/internal/email-templates` | Internal template management (secret-gated) | + +## Gotchas + +- **No `hasAnyRole` checks.** Security is JWT-authentication-only (`SecurityConfig` + `JwtAuthenticationFilter`); every authenticated user passes `anyRequest().authenticated()`. Data isolation comes from `institutionKey` scoping in services, not role guards — unlike most HiveOps services that use `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. +- **`/api/internal/**` is `permitAll()` at the security layer** but gated inside `InternalEmailTemplateController` by an `X-Internal-Secret` header compared against `INTERNAL_SERVICE_SECRET` (default empty). +- **Split is NOT always a flat 3-way.** `createSplitPayments` always adds `INSTITUTION`, and adds `CIT`/`SP` only when a vendor of that type is assigned; the shortage is divided equally with `RoundingMode.FLOOR` and the remainder assigned to the last party. Later CIT/SP vendor assignment via `PUT /{uuid}` creates that party's payment row at `amountOwed = 0` (not re-split). (CLAUDE.md's "auto-created as equal thirds" is only true when both vendors are set at filing.) +- **Dev profile is H2** with `ddl-auto: create-drop` and Flyway disabled; prod uses `validate` + Flyway. Schema drift only surfaces in prod. +- **Attachment storage is a local dir** (`UPLOADS_DIR`, default `/tmp/claims-uploads`) — not object storage; multipart cap is 50MB file / 52MB request. +- **`device_summary` has no reconciler** — if the devices service fails to publish a `hiveops.device.events` change, the claims mirror goes stale silently. diff --git a/backend/src/main/resources/content/technical/config/overview.md b/backend/src/main/resources/content/technical/config/overview.md new file mode 100644 index 0000000..51577a2 --- /dev/null +++ b/backend/src/main/resources/content/technical/config/overview.md @@ -0,0 +1,53 @@ +--- +module: technical.config +title: hiveops-config — Spring Cloud Config Server (centralized configuration) +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-config source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-config` is the Spring Cloud Config Server (`@EnableConfigServer`) that serves centralized, git-backed configuration to all other HiveOps microservices. It owns no business domain of its own — it exposes profile/label-resolved config files (`hiveops-[-].yml`) over HTTP Basic auth. Clients bootstrap against it and can hot-reload via `@RefreshScope` + their own `/actuator/refresh`. Cross-link [platform.service-ownership]. + +## Data + +- **No database.** This service has no `@Entity` classes, no datasource, and no Flyway migrations. +- Backing store is a **git repository** of YAML config files (`config-repo/`). Dev uses a local repo (`file://${user.dir}/config-repo`, `default-label: main`); prod pulls a remote repo via `GIT_REPO_URL` / `GIT_USERNAME` / `GIT_PASSWORD` (`application-prod.yml`). +- Config files present in-repo: `application.yml` (shared defaults) plus `hiveops-{auth,mgmt,incident}[-dev|-prod].yml`. + +Cross-link [platform.data-stores]. + +## Kafka + +No Kafka. No producers, `@KafkaListener`s, or topics in this service. + +## API surface + +No custom `@RestController`s. The endpoints below are provided by Spring Cloud Config Server and Actuator (auth + exposure verified in `SecurityConfig.java` / `application.yml`). Base URL is the service root on port **8083** (NGINX path `/api/config/`). + +| METHOD path | Purpose | +|---|---| +| GET `/{application}/{profile}` | Resolved config as JSON (e.g. `/hiveops-mgmt/prod`) | +| GET `/{application}/{profile}/{label}` | Config for a specific git label/branch | +| GET `/{application}-{profile}.yml` | Resolved config as YAML | +| GET `/{application}-{profile}.properties` | Resolved config as properties | +| GET `/{label}/{application}-{profile}.yml` | YAML from a specific git label | +| GET `/actuator/health` | Health incl. `configServer` component (public, no auth) | +| GET `/actuator/info` | Build/info (public, no auth) | +| GET `/actuator/env` | Environment (auth required) | +| POST `/actuator/refresh` | Refresh server's own config (auth required) | + +All paths except `/actuator/health` and `/actuator/info` require HTTP Basic auth. + +## Gotchas + +- **No default credentials.** `CONFIG_USERNAME` and `CONFIG_PASSWORD` are required env vars with no fallback (`application.yml`); the app fails to start if unset. Single in-memory user with role `CONFIG_CLIENT` (`SecurityConfig.java`), not the usual `MSP_ADMIN`/`BCOS_ADMIN` role model. +- **Auth is HTTP Basic, not JWT.** This service does not use the shared JWT model of other HiveOps services. +- **CSRF disabled**; only `/actuator/health` and `/actuator/info` are `permitAll` — everything else is `authenticated()`. +- **Git `force-pull: true` + `clone-on-start: true`** — server force-pulls the config repo; local edits to the working copy are discarded on refresh/restart. +- **Prod vs dev git source differs by profile:** dev serves from a local `config-repo` dir; prod requires `GIT_REPO_URL`/`GIT_USERNAME`/`GIT_PASSWORD` (activate with `SPRING_PROFILES_ACTIVE=prod`). +- **Actuator exposure is limited** to `health,info,env,refresh` — other actuator endpoints are not exposed. +- **Client-side refresh is separate:** hitting this server's `/actuator/refresh` reloads the server; consuming services reload via their own `/actuator/refresh` (and must have `@RefreshScope`). diff --git a/backend/src/main/resources/content/technical/devices/overview.md b/backend/src/main/resources/content/technical/devices/overview.md new file mode 100644 index 0000000..8978276 --- /dev/null +++ b/backend/src/main/resources/content/technical/devices/overview.md @@ -0,0 +1,94 @@ +--- +module: technical.devices +title: Devices — device management & source of truth for device data +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-devices source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-devices` (Spring Boot backend, `backend/`, artifact `hiveops-devices`, port **8096**) is the system of record for everything device/ATM-related: device inventory, groups, makes/models, custom field definitions, cassette configurations, agent tokens, agent profiles/scripts, per-institution agent properties, and config/hardware change tracking. It also serves the agent-facing internal endpoints (registration, heartbeat, config/hardware sync, log & module upload). It publishes device changes on Kafka so mirror consumers (incident/fleet/reports/recon) stay in sync. See [platform.service-ownership]. + +> Note: the repo `CLAUDE.md` is stale on this point — it claims "no dedicated devices microservice / calls hiveops-incident." The backend under `backend/` is real and owns these domains. The frontend (port 5177, `devices.bcos.cloud`) is the SPA layer. + +## Data + +- **Database:** `hiveiq_devices` (`spring.datasource.url` default `jdbc:postgresql://localhost:5432/hiveiq_devices`, overridable via env **`DB_URL`**). `ddl-auto=validate`; schema managed by **Flyway** (`V1`–`V16`). See [platform.data-stores]. +- **Key tables / entities:** + +| Table | Entity | Purpose | +|-------|--------|---------| +| `atms` | `Atm` | Core device inventory (source of truth) | +| `atm_properties` | `AtmProperties` | Per-device properties | +| `atm_groups` | `AtmGroup` | Device groupings | +| `atm_make` / `atm_model` | `AtmMake` / `AtmModel` | Make/model catalog | +| `atm_field_definitions` | `AtmFieldDefinition` | Custom field defs | +| `atm_config_changes` / `atm_hardware_changes` | `AtmConfigChange` / `AtmHardwareChange` | Config/hardware change audit | +| `atm_module_status` | `AtmModuleStatus` | Per-device module state | +| `atm_log_collects` | `AtmLogCollect` | Log collection records | +| `cassette_configurations` / `cassette_slots` | `CassetteConfiguration` / `CassetteSlot` | Cassette templates | +| `device_cassette_config` | `DeviceCassetteConfig` | Per-device cassette config | +| `agent_tokens` | `AgentToken` | Agent auth tokens | +| `agent_profiles` | `AgentProfile` | Agent config profiles | +| `agent_scripts` | `AgentScript` | Pushable agent scripts | +| `institution_keys` | `InstitutionKey` | Institution key registry | +| `institution_agent_properties` | `InstitutionAgentProperties` | Per-institution agent props | +| `institution_config_audit` | `InstitutionConfigAudit` | Institution config audit trail | +| `device_fleet_tasks` | `DeviceFleetTask` | Fleet task mirror for devices | +| `atm_service_schedule` | `AtmServiceSchedule` | Service schedule | + +## Kafka + +Bootstrap: `KAFKA_BOOTSTRAP_SERVERS` (default `kafka:29092`). Trusted deserializer packages: `com.hiveops.devices.kafka`, `com.hiveops.journal.kafka`. See [platform.kafka]. + +**Produced:** + +| Topic | Producer class | Notes | +|-------|----------------|-------| +| `hiveops.device.events` | `DeviceEventPublisher` (constant `KafkaProducerConfig.DEVICE_EVENTS_TOPIC`) | keyed by atmId; `DeviceEventType` = `DEVICE_CREATED`, `DEVICE_UPDATED`, `DEVICE_DELETED` | + +**Consumed:** + +| Topic | Listener class | +|-------|----------------| +| `hiveops.fleet.task.events` (`FleetTaskKafkaConfig.TOPIC`) | `FleetTaskEventConsumer` | +| `journal.replenishments` (`ReplenishmentKafkaConfig.TOPIC`) | `ReplenishmentConsumer` | + +## API surface + +Admin controllers require JWT with role `MSP_ADMIN` or `BCOS_ADMIN` on mutating endpoints. Many device paths carry a Phase-2 `/api/devices*` alias alongside the legacy `/api/atms*` path. + +Base paths: `/api/atms` + `/api/devices`, `/api/atm-groups`, `/api/atm-makes`, `/api/atm-models`, `/api/atm-field-definitions`, `/api/cassette-configurations`, `/api/institution-keys`, `/api/agent-tokens`, `/api/agent-profiles`, `/api/fleet/scripts`, `/api/internal/*`. + +| Method + Path | Purpose | +|---------------|---------| +| GET `/api/atms` (alias `/api/devices`) | List devices | +| GET `/api/atms/paginated` | Paginated/filterable device list | +| GET `/api/atms/{id}` | Device detail | +| GET `/api/atms/dashboard/stats` | Fleet dashboard stats | +| GET `/api/atms/summary` | Device summary | +| POST `/api/atms` | Create device (admin) | +| PATCH `/api/atms/{id}/institution` | Reassign institution (admin) | +| POST `/api/atms/config/sync` · `/api/atms/hardware/sync` | Config/hardware sync ingest | +| GET `/api/atm-groups` · POST `/api/atm-groups/{id}/atms/{atmId}` | Groups + membership | +| GET/POST `/api/cassette-configurations` | Cassette templates | +| GET `/api/devices/cassette-config` | Resolve device cassette config | +| GET/POST `/api/institution-keys` · GET `/api/institution-keys/{key}/agent-properties` | Institution keys + agent props | +| GET/POST `/api/agent-tokens` · POST `/api/agent-tokens/{id}/revoke` | Agent token lifecycle | +| GET/POST/PUT/DELETE `/api/agent-profiles` · GET `/api/agent-profiles/modules-manifest` | Agent profiles + CDN modules manifest proxy | +| POST `/api/internal/atms/register` | Agent auto-registration (INTERNAL) | +| POST `/api/internal/atms/agent/{country}/{atm}/heartbeat` | Agent heartbeat (INTERNAL) | +| POST `/api/internal/agent-tokens/validate` | Token validation (INTERNAL) | + +## Gotchas + +- **Stale CLAUDE.md.** `hiveops-devices/CLAUDE.md` says the app has no backend and calls `hiveops-incident`; that's outdated — the `backend/` service exists and owns these domains. +- **DB uses `DB_URL`** (full JDBC URL) via `spring.datasource.url=${DB_URL:...}`, not split host/port/name vars. +- **Two auth realms.** Admin controllers use `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`; internal/agent controllers (`/api/internal/*`) use `hasRole('INTERNAL')` (e.g. `InternalAtmController`, `AgentTokenInternalController`, `InternalLogController`). +- **Phase-2 path aliases.** `/api/atms`, `/api/atm-groups`, `/api/atm-makes`, `/api/atm-models`, `/api/atm-field-definitions` each also respond on a `/api/device*` alias for the ongoing ATM→device rename; both must stay routable in nginx. +- **CDN token.** `agent.cdn.token` (env `CDN_TOKEN`) has a hardcoded fallback in `application.properties` used by the `/api/agent-profiles/modules-manifest` proxy. +- **Publish-on-every-change.** Device mirrors in incident/fleet/reports/recon depend on `hiveops.device.events`; any write path that skips `DeviceEventPublisher` leaves those mirrors stale (no reconciler). +- **Error message hiding.** `server.error.include-message=never` and `management.endpoint.health.show-details=never` — expect terse errors in prod. diff --git a/backend/src/main/resources/content/technical/fleet/overview.md b/backend/src/main/resources/content/technical/fleet/overview.md new file mode 100644 index 0000000..d64c65b --- /dev/null +++ b/backend/src/main/resources/content/technical/fleet/overview.md @@ -0,0 +1,89 @@ +--- +module: technical.fleet +title: Fleet — fleet task orchestration & artifact delivery +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-fleet source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-fleet` owns fleet tasks (download/install/software/config/capture jobs targeting devices), the uploadable/CDN-imported artifacts they deliver, patch (maintenance) windows, and network capture jobs. Agents pull work and report status through internal (`X-Internal-Secret`) endpoints; agent-proxy is the caller. Fleet keeps its own local device snapshot (`device_summary`) synced from Kafka rather than calling devices/incident over HTTP. Task terminal history is archived locally and completion is published to Kafka for other services (devices stores the durable device record). Cross-link [platform.service-ownership]. + +## Data + +- **Database:** `hiveiq_fleet` on PostgreSQL (Spring `spring.datasource.url`, overridable via env `DB_URL`; port 8097). `ddl-auto=validate`, Flyway-managed. +- **Key tables** (V1__init_fleet.sql): + - `fleet_tasks` — active tasks (PENDING/QUEUED/RUNNING lifecycle) + - `fleet_task_history` — archived terminal tasks (COMPLETED/FAILED) + - `fleet_artifacts` — uploadable/CDN-imported artifacts (agents, patches, scripts); `cdn_url` added V7 + - `patch_windows` — scheduled maintenance windows (+ `patch_window_devices`, V4) + - `atm_captures` — network capture jobs + - `device_summary` — local device-state snapshot synced via Kafka (not authoritative) + +Cross-link [platform.data-stores]. + +## Kafka + +Cross-link [platform.kafka]. + +**Produced:** + +| Topic | Producer class | Notes | +|-------|----------------|-------| +| `hiveops.fleet.task.events` | `FleetTaskEventPublisher` (via `FleetTaskService`) | Emits `created` / status-update / `archived` `FleetTaskEvent`s; topic constant `FleetTaskKafkaConfig.FLEET_TASK_EVENTS_TOPIC`, 3 partitions, idempotent acks=all producer. | + +**Consumed:** + +| Topic | Listener class | Notes | +|-------|----------------|-------| +| `hiveops.device.events` | `DeviceEventConsumer` (`@KafkaListener`, groupId `hiveops-fleet-device-sync`) | Upserts `device_summary`; trusts packages `com.hiveops.devices.kafka`, `com.hiveops.fleet.kafka`; `auto-offset-reset=earliest`. | + +## API surface + +Base paths: `/api/fleet` (authenticated), `/api/fleet/patch-windows`, `/api/internal/fleet` (agent-proxy only, `hasRole('INTERNAL')`), `/api/users`. + +**`FleetApiController` (`/api/fleet`)** — selected: + +| Method Path | Purpose | +|-------------|---------| +| GET `/api/fleet/tasks` (+ `/tasks/paginated`) | List fleet tasks | +| POST `/api/fleet/tasks` | Create a fleet task | +| POST `/api/fleet/tasks/{id}/cancel` | Cancel a task | +| POST `/api/fleet/tasks/{id}/retry` | Retry a task | +| GET `/api/fleet/tasks/pending-approval` | Tasks awaiting approver action | +| POST `/api/fleet/tasks/{id}/approve` · `/reject` | Approver decision | +| GET `/api/fleet/tasks/history` (+ `/history/recent`) | Archived terminal tasks | +| GET `/api/fleet/stats` (+ `/tasks/stats`) | Task/fleet stats | +| GET `/api/fleet/artifacts` · GET `/artifacts/{id}` | List / fetch artifacts | +| POST `/api/fleet/artifacts` (multipart) | Upload an artifact | +| GET `/api/fleet/cdn-manifest/{type}` · POST `/artifacts/import-cdn` | Browse CDN manifest / import artifact from CDN | +| POST `/api/fleet/artifacts/{id}/toggle-enabled` | Enable/disable an artifact | +| GET `/api/fleet/devices` (+ `/devices/paginated`) | Local device summary list | +| GET `/api/fleet/captures` | Network capture jobs | +| GET `/api/fleet/modules/summary` | Agent module summary | + +**`PatchWindowController` (`/api/fleet/patch-windows`)**: GET `/`, GET `/{id}`, POST `/`, PUT `/{id}`, DELETE `/{id}`, GET `/{id}/atms`, PUT/DELETE `/{windowId}/atms/{atmId}`. + +**`InternalFleetController` (`/api/internal/fleet`, ROLE_INTERNAL)** — the agent work loop: + +| Method Path | Purpose | +|-------------|---------| +| GET `/api/internal/fleet/tasks/next` | Agent polls for next task | +| GET `/api/internal/fleet/tasks/{taskId}/download` | Download task artifact/payload | +| POST `/api/internal/fleet/tasks/{taskId}/status` | Report task progress/status | +| POST `/api/internal/fleet/tasks/{taskId}/verify` | Verify step | +| POST `/api/internal/fleet/tasks/{taskId}/upload` | Upload task result (e.g. capture/logs) | + +## Gotchas + +- **Two auth planes.** User/admin endpoints use JWT (`JWT_SECRET`) with role checks; most reads allow `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')`, mutations require `ADMIN`/`MSP_ADMIN`/`BCOS_ADMIN`. Internal endpoints require `hasRole('INTERNAL')`, granted via `X-Internal-Secret` (`INTERNAL_SECRET` env, default `dev-internal-secret`) — agent-proxy is the only intended caller. +- **Approver gate.** Approve/reject require `hasAuthority('FLEET_APPROVER') or hasRole('BCOS_ADMIN')`; `FLEET_APPROVER` comes from the `fleetTaskApprover` boolean JWT claim. +- **No long-term task store here.** Terminal tasks move to `fleet_task_history`; the durable device-facing record lives in devices. Fleet publishes `archived`/status events on `hiveops.fleet.task.events` for downstream consumers. +- **device_summary is a mirror, not source of truth.** Populated only from `hiveops.device.events`; stale if devices stops publishing (no reconciler). `auto-offset-reset=earliest`. +- **DB_URL env var** overrides the datasource URL (default `jdbc:postgresql://localhost:5432/hiveiq_fleet`); production points it at `hiveiq-postgres`. +- **Artifact CDN import** (`/artifacts/import-cdn`, `cdn.token`/`CDN_TOKEN`) is preferred over local multipart uploads for software tasks — CDN-imported artifacts carry `cdn_url` so agents download directly from the CDN. +- **Script signing:** `SCRIPT_SIGNING_PRIVATE_KEY` (base64 PKCS8 RSA) signs agent scripts (V3 migration `add_agent_scripts`). +- Multipart upload limit 100MB; `management` exposes only `health,info`. diff --git a/backend/src/main/resources/content/technical/incident/overview.md b/backend/src/main/resources/content/technical/incident/overview.md new file mode 100644 index 0000000..1191f99 --- /dev/null +++ b/backend/src/main/resources/content/technical/incident/overview.md @@ -0,0 +1,83 @@ +--- +module: technical.incident +title: hiveops-incident — ATM incident management & journal-event store +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-incident source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-incident` owns the **incidents domain** and the **journal-event store**: creating/assigning/transitioning incidents, storing ATM inventory (`atms`), and ingesting per-event journal data from ATMs. Incidents are auto-created from device events, ARIA threat events, and journal activity. See [platform.service-ownership]. + +Agents never call this service directly — all agent traffic goes through **hiveops-agent-proxy**, which calls incident's `/api/internal/**` endpoints (secured by `X-Internal-Secret` → `ROLE_INTERNAL` via `InternalSecretFilter`). Fleet-task endpoints have moved out to **hiveops-fleet** (`InternalFleetController` no longer here). + +## Data + +- **Database:** `hiveiq_incident` (Postgres). Configured via env var **`DB_URL`** (full JDBC URL) — the code default is `jdbc:postgresql://192.168.200.103:5432/hiveops_incident`, overridden in prod. Schema managed by Flyway (`ddl-auto=validate`). See [platform.data-stores]. +- **Key tables/entities:** + - `atms` — ATM inventory (`Atm`: location, status, model, `atmId`) + - `incidents` — issues (type, severity, status, assignment); plus `incident_events`, `incident_notes`, `workflow_transitions` + - `journal_events` — individual ATM journal events (`JournalEvent`) + - `technicians`, `helpdesk_persons` — assignment targets + - `device_summary` — Kafka-fed mirror of device data from hiveops-devices + - `event_type`, `incident_type`, `incident_severity`, `incident_status` — lookup tables + - `ticket_processing_rules`, `incident_closing_rules`, `rule_suggestions`, `rule_drafts` — auto-processing / Adoons rule suggestions + - `institution_keys`, `institution_agent_properties`, `institution_config_audit` — per-institution config + - `incident_embeddings` — AI (Adoons) incident indexing + +## Kafka + +Bootstrap via `KAFKA_BOOTSTRAP_SERVERS`. See [platform.kafka]. + +**Produced:** + +| Topic | Producer class | When | +|-------|---------------|------| +| `hiveops.incidents.created` | `IncidentLifecycleProducer.publishCreated` | incident created | +| `hiveops.incidents.updated` | `IncidentLifecycleProducer.publishUpdated` | incident updated (non-resolved) | +| `hiveops.incidents.resolved` | `IncidentLifecycleProducer.publishUpdated` | status RESOLVED/CLOSED | +| `adoons.rule-suggestion.requested` | `RuleSuggestionProducer` | request Adoons/AI rule suggestion | + +**Consumed:** + +| Topic | Listener class | Purpose | +|-------|---------------|---------| +| `hiveops.device.events` | `DeviceEventConsumer` (group `hiveops-incident-device-sync`) | sync `device_summary`; auto-create incidents via `IncidentAutoCreationService` | +| `hiveops.aria.threats` | `AriaThreatEventConsumer` (group `hiveops-incident-aria`) | create incidents from ARIA threat events | +| `adoons.rule-suggestion.ready` | `RuleSuggestionResultConsumer` (group `hiveops-incident-adoons`) | ingest completed AI rule suggestions | + +## API surface + +Base paths: `/api/incidents`, `/api/incident-management`, `/api/journal-events`, `/api/atms`, `/api/internal/**`. + +| Method + path | Purpose | +|---------------|---------| +| `POST /api/incidents` | Create incident | +| `GET /api/incidents` | List incidents | +| `GET /api/incidents/paginated` | Paginated list | +| `GET /api/incidents/status/open` | Open incidents | +| `GET /api/incidents/severity/critical` | Critical incidents | +| `GET /api/incidents/{atm,device}/{atmId}` | Incidents for a device (both aliases) | +| `PUT /api/incidents/{id}` | Update incident | +| `POST /api/incident-management/{incidentId}/assign` | Assign incident (admin) | +| `POST /api/incident-management/{incidentId}/transition` | Workflow transition (admin) | +| `POST /api/incident-management/bulk/update` | Bulk update (admin) | +| `POST /api/internal/journal-events` | **Journal-event ingest — internal only** (agent-proxy) | +| `POST /api/internal/agent/{country}/{atm}/heartbeat` | Agent heartbeat (internal) | +| `POST /api/internal/agent/{country}/{atm}/auto-update` | Agent auto-update check (internal) | +| `GET /api/journal-events/{atm,device}/{atmId}/recent` | Recent events for a device | +| `GET /api/atms/dashboard/stats` | Dashboard statistics | + +Not exhaustive — the service has ~26 controllers (ATM groups, cassette config, event types/exceptions, settings, technicians, geocoding, Adoons suggestions, auth-audit proxy, etc.). + +## Gotchas + +- **`DB_URL` is a single full JDBC URL**, not split `DB_HOST`/`DB_PORT`/`DB_NAME`. The hardcoded default points at a stale `hiveops_incident` DB on `192.168.200.103`; production overrides to `hiveiq_incident`. +- **Admin mutations require `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`** — `/api/incident-management` assign/unassign/transition/notes/reassign/bulk all gate on this. +- **Agents cannot reach this service directly.** `AgentApiController` was removed; agent-facing routes live in hiveops-agent-proxy, which authenticates to `/api/internal/**` via `X-Internal-Secret` (`INTERNAL_SERVICE_SECRET`). Fleet-task internal endpoints moved to hiveops-fleet (port 8097). +- **`/atm/...` vs `/device/...` dual mappings** exist on incident and journal-event controllers (Phase 2 device-rename aliases) — both resolve to the same handler. +- `JWT_SECRET` is shared with hiveops-auth and hiveops-mgmt; `mgmt.service.url` is used for agent token validation. +- Server port is **8080** (NGINX path `/api/incident/`); multipart uploads allow up to 500MB (agent log files). diff --git a/backend/src/main/resources/content/technical/journal/overview.md b/backend/src/main/resources/content/technical/journal/overview.md new file mode 100644 index 0000000..6de9b9a --- /dev/null +++ b/backend/src/main/resources/content/technical/journal/overview.md @@ -0,0 +1,102 @@ +--- +module: technical.journal +title: Journal Service — ATM journal storage, parsing & gap detection +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-journal source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-journal` receives raw ATM journal files from agents (chunked upload), stores the originals on disk, and parses them into structured transactions using YAML rules (VC journal format + TCR rules). It also detects missing journal dates per ATM (gaps), archives old transactions (90-day hot/cold split), and exposes a REST API consumed by the incident/devices frontends. It replaces the legacy SmartJournal dependency. See [platform.service-ownership]. + +Runs as **one image, two roles** driven by `JOURNAL_ROLE`: +- `upload` (default) — HTTP controllers, scheduled jobs, Kafka **producer**. +- `worker` — parse listener only; one worker container **per institution** (`JOURNAL_INSTITUTION`), Kafka **consumer**. + +Server port **8087** (`server.port`, context-path `/`). + +## Data + +- **Database:** `hiveiq_journal` — connection via full JDBC URL in env var **`DB_URL`** (`spring.datasource.url=${DB_URL:...}`), not split host/port/name. `ddl-auto=none`; schema managed by **Flyway** (migrations `V1`–`V15`). +- **Secondary read-only datasource:** incident DB via `INCIDENT_DB_URL` — used to sync ATM→institution mapping. + +Key entities (`com.hiveops.journal.entity`): + +| Entity | Table | Purpose | +|--------|-------|---------| +| `JournalFile` | `journal_files` | Uploaded file metadata; `parseStatus` = PENDING/PARSING/PARSED/ERROR | +| `JournalTransaction` | `journal_transactions` | Parsed transactions (hot, < 90 days) | +| `JournalTransactionArchive` | `journal_transactions_archive` | Cold archive (no FK to files) | +| `JournalGap` | `journal_gaps` | Detected missing journal dates per ATM | +| `AtmInstitutionMap` | `atm_institution_map` | ATM→institution routing map (V14) | +| `ChequeImage` | cheque image table (V7/V8) | Cheque/deposit images from agents | + +Notable migrations: `V10` archive table, `V14` ATM institution map, `V15` parse dedup fields. See [platform.data-stores]. + +## Kafka + +Topic constants defined in `kafka/JournalKafkaConfig.java`. See [platform.kafka]. + +**Produced** (upload role): + +| Topic | Producer class | +|-------|----------------| +| `journal.parse-requests.{INSTITUTION}` (prefix `journal.parse-requests.`, or `...UNASSIGNED`) | `JournalParsingService` (routes per institution) | +| `journal.replenishments` | `ReplenishmentEventProducer` | +| `journal.file-parsed` | `FileParsedEventProducer` | +| `hiveops.journal.transactions.recorded` | `JournalTransactionEventProducer` | + +Static topic `journal.parse-requests` is also declared. Per-institution topics are created at startup from `JOURNAL_INSTITUTIONS` env plus an always-present `...UNASSIGNED`. + +**Consumed:** + +| Topic | Listener class | groupId | +|-------|----------------|---------| +| `journal.parse-requests.{INSTITUTION}` (SpEL `#{@parseTopicName}`) | `JournalParseConsumer` (worker role) | `#{@parseGroupId}` (derived from `JOURNAL_INSTITUTION`) | +| `journal.file-parsed` | `JournalGapDetectionService` | `journal-gap-detection` | +| `journal.file-parsed` | `CassetteInventoryService` | `journal-cassette-inventory` (offset reset = latest) | + +## API surface + +Agent-facing (no JWT): +- `GET /atm/journal/query` — resume offset for a file (legacy `FileUploadClient` protocol) — `AgentUploadController` +- `POST /atm/journal/upload` — chunked gzip upload of a journal file — `AgentUploadController` +- `GET|POST /api/agent/journal/query|upload` — newer agent journal path — `AgentJournalApiController` +- `POST /api/agent/image/upload`, `POST /api/agent/image/cheque/metadata` — cheque images — `AgentImageController` + +Protected (`/api/journal`, `JournalApiController`, `@PreAuthorize("isAuthenticated()")`): +- `GET /api/journal/transactions` — list transactions (atmId, from, to, type, page, size) +- `GET /api/journal/transactions/{id}` — single transaction +- `GET /api/journal/transactions/summary` — aggregate summary +- `GET /api/journal/transactions/top-atms` — top ATMs by volume +- `PATCH /api/journal/transactions/{id}/adoons-analysis` — store AI (Adoons) analysis +- `GET /api/journal/files` — list uploaded files +- `GET /api/journal/files/{id}/raw` — stream raw file content (handles `.gz` transparently) +- `POST /api/journal/files/{id}/reparse` — re-queue file for parsing +- `GET /api/journal/files/download-zip` — bulk file download +- `GET /api/journal/gaps` — list detected gaps +- `PUT /api/journal/gaps/{id}/request-fetch` — mark gap FETCH_REQUESTED +- `GET /api/journal/cassette-config` — cassette configuration + +Admin (`/api/journal/admin`, `AdminController`, all `@PreAuthorize("hasAnyRole('MSP_ADMIN', 'BCOS_ADMIN')")`): +- `POST /reload-rules`, `GET|PUT /rules`, `GET|PUT /rules/structured`, `GET /types` +- `POST /reparse-all`, `POST /parse-sync`, `POST /archive-now` +- `POST /reconcile` — re-queue stranded PENDING/PARSING files (**run after every deploy**) +- `POST /migrate-storage`, `POST /sync-cassette-inventory` +- `GET|PUT /tcr-rules`, `POST /tcr-rules/reload`, `GET|PUT /tcr-rules/raw` + +Also: `/api/journal/cheques/...` (`JournalImageQueryController`), `/api/simulator/transactions` (`SimulatorController`). + +## Gotchas + +- **`DB_URL` is a full JDBC string** — unlike sibling services that use split `DB_HOST/DB_PORT/DB_NAME`. +- **Admin endpoints require `hasAnyRole('MSP_ADMIN', 'BCOS_ADMIN')`**; read/query endpoints only require `isAuthenticated()`; agent upload routes require **no JWT** (agent-facing). +- **Post-deploy reconcile is mandatory** — `POST /api/journal/admin/reconcile` re-queues files stranded in PENDING/PARSING when the instance restarted; skipping it leaves files stuck forever. A scheduled recovery also re-queues files stuck in PARSING > 30 min (`JournalParsingService`). +- **Per-institution routing** — parse requests go to `journal.parse-requests.{INSTITUTION}` (uppercase); unknown/unmapped ATMs fall back to `...UNASSIGNED` so no data is lost before a dedicated worker exists. +- **Dedup happens at two layers**: upload-time guard skips files already `PENDING`/`PARSING` in flight (`JournalUploadService`); parse-time `deduplicateSubTransactions` merges two-phase records that share a SEQ# (`JournalParser`). +- **SEQ# `0` is not a real host sequence** (deposits/settlements/no-card) — parser stores it as null, frontend renders `-`. +- **Compression:** agent uploads gzip → stored as `.txt`; after `journal.archive.days` (90) archival re-compresses to `.txt.gz` and updates `storage_path`. Reader/raw-view/reparse handle `.gz` transparently. +- **Workers use a smaller Hikari pool** (`JOURNAL_DB_POOL_SIZE`, default 30 for upload; workers set to ~5). Kafka producer fails fast (`max.block.ms=3000`) to avoid blocking request threads. diff --git a/backend/src/main/resources/content/technical/messaging/overview.md b/backend/src/main/resources/content/technical/messaging/overview.md new file mode 100644 index 0000000..ccff76e --- /dev/null +++ b/backend/src/main/resources/content/technical/messaging/overview.md @@ -0,0 +1,85 @@ +--- +module: technical.messaging +title: Messaging — user messages, broadcasts & system alerts +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-messaging source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-messaging` owns in-app user messaging for the platform: direct messages between users, admin BROADCAST messages, and ACTION alerts injected by other services (e.g. incident) over Kafka. It also owns per-user read/unread state, per-user soft-delete, and MSP-managed **contact groups** that subscribe to event types for fan-out notifications. Message content and delivery state live here — no other service writes it. Cross-link [platform.service-ownership]. Runs on port **8086** (`PORT` env override). + +## Data + +- **Database:** `hiveiq_messaging` (PostgreSQL, prod profile). Dev profile uses H2 in-memory (`create-drop`). Connection via split `DB_HOST` / `DB_PORT` / `DB_NAME` / `DB_USERNAME` / `DB_PASSWORD` env vars (not `DB_URL`). Flyway migrations in `db/migration` (prod only). Cross-link [platform.data-stores]. +- **Key tables** (from V1–V5 migrations): + +| Table | Purpose | +|-------|---------| +| `messages` | Core record. `type` = DIRECT \| BROADCAST \| ACTION; `sender_user_id`/`sender_email`; `recipient_user_id` (NULL for BROADCAST); `subject`, `body`, `metadata` (JSONB); `institution_key` (V3, for institution-scoped broadcasts); `uuid`, `created_at` | +| `message_reads` | Per-user read state; UNIQUE (message_id, user_id) — drives BROADCAST read tracking | +| `message_deletes` | Per-user soft-delete; PK (message_id, user_id) | +| `contact_groups` | MSP-managed groups (V4); `name` UNIQUE | +| `contact_group_members` | Group membership; PK (group_id, user_id) | +| `group_subscriptions` | Event types a group is subscribed to (V5); UNIQUE (group_id, event_type) | + +## Kafka + +Cross-link [platform.kafka]. Bootstrap via `KAFKA_BOOTSTRAP_SERVERS` (default `localhost:9092`). Topic constants in `config/KafkaConfig.java`. + +**Produced:** + +| Topic | Producer | Notes | +|-------|----------|-------| +| `hiveops.messages` | `kafka/MessageProducer.publishMessage` | Fired after every REST-initiated send (`MessageService.sendMessage`). Key = sender userId or `"system"`. Payload = `MessageEvent` (carries `sourceService`) | + +**Consumed:** + +| Topic | Listener (group) | Behavior | +|-------|------------------|----------| +| `hiveops.messages` | `kafka/MessageConsumer.onMessageEvent` (group `hiveops-messaging`) | Persists events from external services; **skips events where `sourceService == "hiveops-messaging"`** to avoid re-persisting its own REST sends | +| `hiveops.reports.completed` | `kafka/ReportCompletedConsumer.onReportCompleted` (group `hiveops-messaging-reports`) | Deserializes `ReportCompletedEvent` as raw String via a separate `reportKafkaListenerContainerFactory`; creates one DIRECT message per `recipientUserIds` entry | + +## API surface + +Two controllers. All endpoints require a JWT Bearer token (parsed by `JwtAuthenticationFilter` into `MessagingPrincipal`). + +**`/api/messaging/messages`** (`MessageController`): + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/unread/count` | Unread count — polled by Electron app every 30s | +| GET | `/unread` | All unread messages | +| GET | `/` | Paged inbox (Spring `Pageable`) | +| GET | `/sent` | Messages sent by current user | +| POST | `/` | Send a message (BROADCAST requires admin) | +| PATCH | `/{messageId}/read` | Mark one message read (204) | +| PATCH | `/read-all` | Mark all read (204) | +| DELETE | `/{messageId}` | Per-user soft-delete (204) | + +**`/api/messaging/groups`** (`ContactGroupController`): + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/` | List contact groups | +| POST | `/` | Create group (admin) | +| GET | `/{groupId}` | Group detail (members + subscriptions) | +| PUT / DELETE | `/{groupId}` | Update / delete group (admin) | +| POST | `/{groupId}/members` | Add members (admin) | +| DELETE | `/{groupId}/members/{userId}` | Remove member (admin) | +| GET/POST | `/{groupId}/subscriptions` | List / add event-type subscription | +| DELETE | `/{groupId}/subscriptions/{eventType}` | Remove subscription (admin) | +| POST | `/notify` | Fan out to all groups subscribed to an event type (admin) | + +## Gotchas + +- **Base path is `/api/messaging/...`, not `/api/v1/messages`.** The repo's `CLAUDE.md` still documents the old `/api/v1/messages` paths — the code (`@RequestMapping`) is the source of truth. +- **Admin check is role-string based, not `@PreAuthorize`.** `MessagingPrincipal.isAdmin()` returns true for role `ADMIN`, `MSP_ADMIN`, or `BCOS_ADMIN` (case-insensitive). BROADCAST send and group mutations throw `AccessDeniedException` when not admin — enforced in controller code, not annotations. +- **Self-consume guard.** REST sends both persist locally *and* publish to `hiveops.messages`; `MessageConsumer` drops any event whose `sourceService` is `hiveops-messaging`, so the service does not double-persist its own messages. +- **Two consumer factories.** `hiveops.messages` uses the default `JsonDeserializer` bound to `MessageEvent` (`spring.json.value.default.type`); `hiveops.reports.completed` needs a dedicated `StringDeserializer` factory (`reportKafkaListenerContainerFactory`) and manual Jackson parsing to avoid a type conflict with that default. +- **`JWT_SECRET` has no default** — the app fails fast if unset (shared secret with hiveops-auth/mgmt). +- **No dead-letter topic.** `MessageConsumer` logs and swallows persistence failures (a TODO note calls for a `KafkaListenerErrorHandler` DLT in production). +- **Institution scoping.** Unread/inbox queries take `principal.institutionKey()`; `messages.institution_key` (V3) supports institution-scoped broadcasts. diff --git a/backend/src/main/resources/content/technical/mgmt/overview.md b/backend/src/main/resources/content/technical/mgmt/overview.md new file mode 100644 index 0000000..c34dd63 --- /dev/null +++ b/backend/src/main/resources/content/technical/mgmt/overview.md @@ -0,0 +1,75 @@ +--- +module: technical.mgmt +title: hiveops-mgmt — Management, licensing & internal identity service +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-mgmt source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-mgmt` is the Spring Boot 3.4.1 / Java 21 management backend. It owns **internal/admin users**, **license keys + activations**, **agent tokens**, **app modules and their per-user / per-institution allocations**, **global settings**, **billing (QuickBooks Online)**, and the server-rendered **MSP portal** (Thymeleaf). Customer-facing auth lives in `hiveops-auth`; mgmt handles the internal-admin side and shares JWT secret material with it. See [platform.service-ownership]. + +## Data + +Database: PostgreSQL, name supplied via env var `DB_NAME` in the `prod` profile (JDBC URL built from `DB_HOST`/`DB_PORT`/`DB_NAME`; conventionally `hiveiq_mgmt`). Dev profile uses an in-memory H2 DB. Schema managed by Flyway (`db/migration`, baseline v3). See [platform.data-stores]. + +Key tables/entities (`@Table` names): + +| Table | Purpose | +|-------|---------| +| `users` | Internal/admin user accounts (JWT auth, MFA fields added V11) | +| `refresh_tokens` | Refresh-token store for admin sessions | +| `licenses` / `license_activations` | License keys and per-device activation records | +| `agent_tokens` | Agent auth tokens (validated for ATM agents) | +| `app_modules` | Installable app modules (has `is_core` flag, V21) | +| `user_module_allocations` | Per-user module grants | +| `institution_module_allocations` | Per-institution module grants (V18) | +| `global_settings` | Key/value global config | +| `audit_logs` | Compliance audit trail | +| `customers` / `customer_institution_access` | Customer records + institution access (V16) | +| `api_keys` | API keys | +| `billing_rates` / `billing_invoices` / `qb_credentials` | Billing + QuickBooks Online integration (V22) | + +## Kafka + +No Kafka. No `@KafkaListener`, `KafkaTemplate`, or `StreamBridge` usage found in `src/main`. + +## API surface + +REST base path `/api/v1` (plus `/api/internal/*` service-to-service, and Thymeleaf `/portal/*` UI). Selected endpoints: + +| Method + Path | Purpose | +|---------------|---------| +| POST `/api/v1/auth/login` | Internal admin login (returns JWT) | +| POST `/api/v1/auth/refresh` | Refresh JWT | +| POST `/api/v1/auth/logout` | Invalidate session | +| POST `/api/v1/licenses/activate` | Activate a license key for a device | +| POST `/api/v1/licenses/validate` | Validate an active license | +| POST `/api/v1/licenses/deactivate` | Deactivate a license | +| GET `/api/v1/licenses/{licenseKey}/status` | License status | +| GET/PUT `/api/v1/settings` , GET/PUT `/api/v1/settings/{key}` | Global settings | +| GET/POST `/api/v1/admin/modules` , PUT/DELETE `/api/v1/admin/modules/{id}` | Manage app modules | +| GET/PUT `/api/v1/admin/modules/users/{authUserId}` | Get/set a user's module allocations | +| GET `/api/v1/users/me/modules` | Current user's allocated modules | +| GET `/api/v1/msp/institutions` , PUT `/api/v1/msp/institutions/{institutionKey}/contact` | MSP institution list / contact | +| GET/PUT `/api/v1/msp/modules/users/{authUserId}` | MSP-scoped module allocations | +| POST/GET `/api/v1/agent-tokens` , POST `/api/v1/agent-tokens/{id}/revoke` | Manage agent tokens | +| POST `/api/internal/agent-tokens/validate` | Internal: validate an agent token | +| GET `/api/internal/customers/domain/{domain}/institution-keys` | Internal: resolve institution keys by domain | +| GET/PUT `/api/v1/billing/rates` , POST `/api/v1/billing/invoices/generate` | Billing rates + invoice generation | +| GET `/api/v1/billing/qb/connect` , GET `/api/v1/billing/qb/callback` | QuickBooks OAuth connect/callback | + +Not exhaustive — see the `controller/` and `controller/portal/` packages. + +## Gotchas + +- **Two JWT secrets.** `jwt.secret` (`JWT_SECRET`) signs internal-admin tokens; a separate `customer-auth.jwt-secret` (`CUSTOMER_JWT_SECRET`) must match `hiveops-auth` for customer tokens. Both, plus `LICENSE_SECRET`, are **required with no defaults** — the app fails to start if unset. +- **Authorization roles are inconsistent.** Most endpoints use `hasRole('ADMIN')`, but MSP/module paths use `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` (and a few mix `ADMIN`, `MSP_ADMIN`, `BCOS_ADMIN`, `CUSTOMER`). Don't assume a single canonical role check across controllers. +- **DB name is env-driven, not literal.** Prod JDBC URL is assembled from `DB_HOST`/`DB_PORT`/`DB_NAME`; there is no hardcoded `hiveiq_mgmt` in `application.yml`. +- **Bootstrap requires the config server.** `bootstrap.yml` sets `fail-fast: true` with `CONFIG_USERNAME`/`CONFIG_PASSWORD`; mgmt will not start if `hiveops-config` is unreachable. +- **CORS is mandatory in prod.** `CORS_ALLOWED_ORIGINS` must list every frontend origin (and be mirrored in `hiveops-auth`); missing it blocks the SPAs. +- **Internal service secret is shared.** `INTERNAL_SERVICE_SECRET` is reused for both the claims service call and the incident service-to-service trust. +- **Swagger and H2 console are disabled by default** (`SWAGGER_ENABLED=false`, H2 console off in prod) — security hardening, don't expect them live. diff --git a/backend/src/main/resources/content/technical/recon/overview.md b/backend/src/main/resources/content/technical/recon/overview.md new file mode 100644 index 0000000..093cbae --- /dev/null +++ b/backend/src/main/resources/content/technical/recon/overview.md @@ -0,0 +1,77 @@ +--- +module: technical.recon +title: Recon — ATM cash reconciliation service +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-recon source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-recon` is the ATM cash reconciliation microservice. Custodians record replenishments; the service aggregates cash movements from `hiveops-journal` per **reconciliation cycle** (period between two replenishments) and computes expected vs. actual closing balances, flagging variances. It also tracks per-cassette configuration, cash positions, and short-term cash forecasts, and maintains a local `device_summary` mirror (via Kafka) so it can scope data by device/institution without calling `hiveops-devices`. Owns the recon domain exclusively — see [platform.service-ownership]. + +Backend port **8091** (NGINX `/api/recon/`, prod external `8013`); frontend `recon.bcos.cloud`. + +## Data + +- **Database:** `hiveiq_recon` (Postgres, prod profile). Prod builds the JDBC URL from `DB_HOST` / `DB_PORT` / `DB_NAME` (default `hiveiq_recon`) — **not** a single `DB_URL` var. Dev profile uses in-memory H2 (`ddl-auto: create-drop`, Flyway disabled). See [platform.data-stores]. +- **Key tables/entities** (Flyway `db/migration` + `@Table`): + +| Table | Purpose | +|---|---| +| `replenishment_sessions` | Custodian-recorded replenishment events | +| `replenishment_cassette_entries` | Per-cassette/denomination detail for a session | +| `reconciliation_cycles` | Open/closed cycles with opening/closing balances + variance | +| `atm_cash_positions` | Current cash position per ATM | +| `atm_cassette_configs` | Per-ATM cassette layout/denomination config | +| `cash_forecasts` | Projected cash levels / forecast output | +| `recon_device_rules` | Rules driving auto device selection into recon | +| `recon_device_selection` | Which devices are enrolled in recon | +| `device_summary` | Local mirror of device data, fed by Kafka | + +## Kafka + +**Consumer-only — no producers found** (no `KafkaTemplate` / `StreamBridge` sends in source). + +| Direction | Topic | Listener class | Group | +|---|---|---|---| +| Consumed | `hiveops.device.events` | `DeviceEventConsumer` | `hiveops-recon-device-sync` | +| Consumed | `journal.replenishments` | `ReplenishmentEventConsumer` | `recon-replenishment-group` | +| Consumed | `hiveops.journal.transactions.recorded` | `JournalTransactionConsumer` | `recon-transaction-group` | + +Topic constants: `hiveops.device.events` in `config/DeviceEventKafkaConfig`; the two journal topics in `kafka/ReconKafkaConfig`. The `device_summary` mirror is populated from `hiveops.device.events`. See [platform.kafka]. + +## API surface + +Base path `/api/recon` (admin sub-paths under `/api/recon/admin`, `/api/recon/device-rules`, `/api/recon/device-selection`). Not exhaustive: + +| Method + path | Purpose | +|---|---| +| GET `/atm/{atmId}/summary` | Recon summary for one ATM | +| GET `/fleet/summary` | Fleet-wide recon summary | +| GET `/fleet/pending` | ATMs pending reconciliation | +| GET `/groups` | Grouped fleet summary | +| GET `/atm/{atmId}/cycles` · `/cycles/current` | List cycles / current open cycle | +| GET `/cycles/{id}` · `/cycles/{id}/startup-snapshots` | Cycle detail / startup snapshots | +| POST `/atm/{atmId}/sync` | Pull journal txns, recalc open cycle | +| GET `/atm/{atmId}/replenishments` · POST same | List / create replenishment session | +| GET `/atm/{atmId}/replenishments/prefill` | Auto-parse journal into a prefilled form | +| POST `/replenishments/{id}/lock` · PUT `/replenishments/{id}` | Lock / edit a session | +| GET/PUT `/atm/{atmId}/cassettes` · POST `/atm/{atmId}/cassettes/sync` | Cassette config read/update/sync | +| GET `/fleet/forecast` · GET `/atm/{atmId}/forecast` | Cash forecasts | +| GET/PUT `/device-selection` · POST `/device-selection/enroll/{atmId}` | Manage enrolled devices | +| GET/POST/PUT/DELETE `/device-rules[/{id}]` | Recon device rules CRUD | +| GET/PUT `/admin/rules` (text/plain) · POST `/admin/reload-rules` | recon-rules.yml view/edit/reload | +| POST `/admin/backfill[/{atmId}]` · `/admin/reparse/{atmId}` · `/admin/ingest-replenishments` | Admin backfill/reparse tooling | + +## Gotchas + +- **Write/admin endpoints are role-gated** with `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")` (POST/PUT/DELETE across replenishments, cycles sync, cassettes, device-rules, device-selection, forecast recompute, all `/admin/*`). Read GETs (summaries, cycles, forecasts, prefill) are not annotated at method level. +- **Prod DB config uses `DB_HOST`/`DB_PORT`/`DB_NAME`**, unlike `hiveops-journal` which uses a single `DB_URL`. +- **Journal dependency:** prefill/sync require `hiveops-journal` to define the `REPLENISHMENT` transaction type (OPCode `AA`, `jpr_contains: "STANDARD REPLENISHMENT"`) in journal-rules; recon parses the `STANDARD REPLENISHMENT` block (BEG/USE/CUR per denomination). +- **Rules editing:** `/admin/rules` GET/PUT are `text/plain` (raw recon-rules.yml); changes need `/admin/reload-rules` to take effect. Rules path overridable via `RECON_RULES_PATH`. +- **Variance math** (from CLAUDE.md): `expected = opening + added - removed - withdrawals + deposits - reversals`; `variance = actual − expected`. +- **Consumer-only Kafka**: recon reacts to device + journal events but publishes nothing; the `device_summary` mirror has no reconciler, so it goes stale if upstream stops emitting `hiveops.device.events`. +- Swagger disabled by default (`SWAGGER_ENABLED=false`); actuator exposes only `health,info`. diff --git a/backend/src/main/resources/content/technical/remote/overview.md b/backend/src/main/resources/content/technical/remote/overview.md new file mode 100644 index 0000000..3d0b2c7 --- /dev/null +++ b/backend/src/main/resources/content/technical/remote/overview.md @@ -0,0 +1,84 @@ +--- +module: technical.remote +title: Remote — Remote screen capture, file access & module deployment for ATMs +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-remote source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-remote` is the central server for remote operations on ATM devices: on-demand screenshots, remote directory listing/file download, and distribution of the agent-side "remote" module JAR. Maven multi-module — `hiveops-remote-common` (shared DTOs/enums), `hiveops-remote-module` (agent-side client running on the ATM: screen capture, filesystem, HTTP client), and `hiveops-remote-server` (Spring Boot 3.4.1, Java 21, this service). Communication is HTTP long-poll + multipart chunk upload, **not** WebSocket: agents register, then long-poll `/api/v1/agents/poll` for pending commands and upload binary results in chunks. Owns its agent registry, command queue, and module version catalog exclusively. Cross-link [platform.service-ownership]. + +## Data + +Database `hiveiq_remote` (PostgreSQL). Note: `application.yml` builds the JDBC URL from `DB_HOST`/`DB_PORT`/`DB_NAME`/`DB_USER`/`DB_PASSWORD` env vars (default `DB_NAME` in yml is `hiveops_remote`; production uses `hiveiq_remote`). Tables are created by Hibernate `ddl-auto: update`; `schema.sql` is reference only. Cross-link [platform.data-stores]. + +| Table | Entity | Holds | +|-------|--------|-------| +| `remote_agents` | `RemoteAgent` | Registered agents: `agent_id` (UUID), name, hostname, platform, agent/module version, status, `last_heartbeat_at`, unique `machine_id` | +| `remote_agent_capabilities` | (element collection) | Per-agent capability strings | +| `remote_commands` | `RemoteCommand` | Command queue: `command_id` (UUID), type, status, `parameters`/`result` (JSONB), `has_binary_result` | +| `binary_chunks` | `BinaryChunk` | Chunked binary results (BYTEA) keyed by `(command_id, chunk_index)`, with checksum | +| `module_versions` | `ModuleVersion` | Remote module JAR catalog: version, filename, checksum, size, `is_latest` | + +## Kafka + +Produces only — no consumers (`@KafkaListener`) exist in this service. + +| Direction | Topic | Class | +|-----------|-------|-------| +| Produced | `hiveops.agent.online` | `AgentConnectionEventProducer.publishOnline` (called from `AgentService` on register), constant `RemoteKafkaConfig.TOPIC_AGENT_ONLINE` | +| Produced | `hiveops.agent.offline` | `AgentConnectionEventProducer.publishOffline` (called from `AgentService` for stale agents), constant `RemoteKafkaConfig.TOPIC_AGENT_OFFLINE` | + +Messages are JSON `AgentConnectionEvent`, keyed by `machineId`. Both topics declared as `NewTopic` with 3 partitions / 1 replica. Cross-link [platform.kafka]. + +## API surface + +Three controllers, all under `/api/v1`. + +**Agent-facing** — `AgentController` @ `/api/v1/agents` (called by the ATM-side RemoteModule): + +| Method Path | Purpose | +|-------------|---------| +| POST `/api/v1/agents/register` | Agent self-registration (permitAll) — returns JWT | +| GET `/api/v1/agents/poll` | Long-poll for pending commands (JWT-authenticated) | +| POST `/api/v1/agents/heartbeat` | Keep-alive, updates `last_heartbeat_at` | +| POST `/api/v1/agents/commands/{commandId}/status` | Report command execution status | +| POST `/api/v1/agents/commands/{commandId}/upload` | Upload binary result chunk (multipart: chunkIndex, totalChunks, checksum, data) | + +**User-facing** — `RemoteController` @ `/api/v1`: + +| Method Path | Purpose | +|-------------|---------| +| GET `/api/v1/remote/agents` | List registered agents | +| GET `/api/v1/remote/agents/{agentId}` | Get agent details | +| DELETE `/api/v1/remote/agents/{agentId}` | Unregister agent | +| POST `/api/v1/commands/screenshot` | Request a screenshot from an agent | +| POST `/api/v1/commands/file-list` | List a directory on an agent | +| POST `/api/v1/commands/file-download` | Download a file from an agent | +| GET `/api/v1/commands/{commandId}` | Get command status | +| GET `/api/v1/commands/{commandId}/result` | Download binary result (SCREENSHOT→`image/jpeg` `screenshot.jpg`; FILE_DOWNLOAD→`application/octet-stream`) | + +**Module distribution** — `ModuleController` @ `/api/v1/modules`: + +| Method Path | Purpose | +|-------------|---------| +| GET `/api/v1/modules/latest` | Latest module version info | +| GET `/api/v1/modules/{version}` | Specific version info | +| GET `/api/v1/modules/{version}/download` | Download module JAR for a version | +| GET `/api/v1/modules/latest/download` | Download latest module JAR | + +Port **8090**; NGINX path `/api/remote/`. + +## Gotchas + +- **Not WebSocket.** Despite the "remote command execution" framing, transport is HTTP long-poll (`remote.poll-timeout-ms: 15000`) plus chunked multipart upload (`chunk-size-bytes: 1048576`, `max-file-size-bytes: 104857600`). +- **Three separate security filter chains** (`SecurityConfig`, ordered): `/api/v1/agents/**` uses a custom `AgentAuthenticationFilter` (JWT via `AgentPrincipal`), with only `/register` permitAll; `/api/v1/modules/**` is **fully permitAll** (module JARs are unauthenticated downloads); everything else (user-facing commands, actuator, swagger) falls to the default chain using **HTTP Basic** (`ADMIN_USER`/`ADMIN_PASSWORD`). No `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` here — user-facing endpoints are Basic-auth gated, not JWT-role gated. +- **Fails to start without secrets.** `JWT_SECRET`, `ADMIN_USER`, and `ADMIN_PASSWORD` have no defaults by design. +- **Swagger disabled by default** — enable only in dev with `SWAGGER_ENABLED=true`. +- **Agent liveness is timer-driven.** Offline events (`hiveops.agent.offline`) are emitted when a sweep marks stale agents (`agent-timeout-ms: 60000`, checked every `agent-check-interval-ms: 30000`), not on an explicit disconnect. +- **Module binaries live on a volume** at `MODULE_STORAGE_PATH` (`/app/modules`), not in the DB (only chunked command results use `binary_chunks` BYTEA). +- **`kafka.admin.fail-fast: false`** — the service still boots if Kafka is unavailable. diff --git a/backend/src/main/resources/content/technical/reports/overview.md b/backend/src/main/resources/content/technical/reports/overview.md new file mode 100644 index 0000000..1fb1d5e --- /dev/null +++ b/backend/src/main/resources/content/technical/reports/overview.md @@ -0,0 +1,77 @@ +--- +module: technical.reports +title: hiveops-reports — Custom report builder, scheduling & distribution +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-reports source. Verify against code before relying on specifics. + +## Responsibility + +Spring Boot microservice that owns the **custom report builder**: users define templates (data source + columns + filters + optional groupBy), runs execute asynchronously against other services' APIs, and results export as JSON preview or CSV. Also handles **cron scheduling** and **in-app distribution** of completed reports via hiveops-messaging. It does not own the underlying fleet data — it fetches from incident, journal, and fleet services at run time. See [platform.service-ownership]. + +## Data + +- **DB name:** `hiveiq_reports` (PostgreSQL prod; H2 in-memory `dev` profile). Connection built from split `DB_HOST` / `DB_PORT` / `DB_NAME` / `DB_USERNAME` / `DB_PASSWORD` env vars (not `DB_URL`). Flyway migrations in `src/main/resources/db/migration/`. See [platform.data-stores]. +- **Key tables** (from V1/V3/V8 migrations): + +| Table | Purpose | +|---|---| +| `report_templates` | User-defined report definition (data source, columns, filters, schedule cron, distribution config) | +| `report_runs` | Execution instance — status PENDING → RUNNING → COMPLETED/FAILED, row count, CSV path | +| `report_distribution_log` | Per-recipient delivery record | +| `report_queries` | Named saved queries attached to a template (V3) | +| `device_summary` | Local mirror of device data kept in sync from Kafka (V8), used by `/api/devices/atms` | + +## Kafka + +See [platform.kafka]. + +**Produced** + +| Topic | Producer class | Notes | +|---|---|---| +| `hiveops.reports.completed` | `kafka.ReportCompletedEventProducer` (constant in `ReportsKafkaConfig`) | Published from `ReportDistributorService`; message key = `institutionKey` or `"global"`; 3 partitions | + +**Consumed** + +| Topic | Listener class | Notes | +|---|---|---| +| `hiveops.device.events` | `kafka.DeviceEventConsumer` (constant in `DeviceEventKafkaConfig`) | Group `hiveops-reports-device-sync`, offset reset `earliest`; upserts/deletes `device_summary`. Handles `DEVICE_DELETED` event type; defaults null `institutionKey` to `"BCOS"` | + +## API surface + +Base paths under `/api/reports/*` (plus `/api/devices`, `/api/users`). All require a JWT Bearer token. + +| Method | Path | Purpose | +|---|---|---| +| GET | `/api/reports/schema` | All data-source schemas (fields + supported filters) | +| GET | `/api/reports/schema/{dataSource}` | Single data-source schema | +| GET | `/api/reports/templates` | List caller's templates | +| POST | `/api/reports/templates` | Create a template | +| PUT | `/api/reports/templates/{id}` | Update a template | +| POST | `/api/reports/templates/{id}/run` | Trigger async run (returns run id) | +| GET | `/api/reports/templates/shared` | List templates shared with caller | +| POST | `/api/reports/templates/{id}/share` | Share a template | +| GET | `/api/reports/templates/{templateId}/queries` | List saved queries for a template | +| POST | `/api/reports/templates/{templateId}/queries/{queryId}/set-scheduled` | Mark a query as the scheduled one | +| GET | `/api/reports/runs` | List runs (pageable) | +| GET | `/api/reports/runs/{id}/preview` | JSON preview (first rows) | +| GET | `/api/reports/runs/{id}/download` | Download CSV | +| POST | `/api/reports/runs/{id}/distribute` | Manually distribute a completed run | +| GET | `/api/devices/atms` | List devices from local `device_summary` mirror | +| GET | `/api/users/me` | Current principal info | + +Data sources (from `SchemaRegistry`, fetchers): `INCIDENTS`, `ATM_HEALTH`, `TRANSACTIONS`, `FLEET_TASKS`, `AGENT_VERSIONS`, `AGENT_MODULES` — each fetches from incident/journal/fleet APIs at run time. + +## Gotchas + +- **Scheduled runs use a self-minted JWT.** `InternalTokenGenerator` issues a short-lived ADMIN token signed with `JWT_SECRET` via `Keys.hmacShaKeyFor(secret.getBytes(UTF_8))` (same scheme as validation). User-triggered runs instead forward the caller's `Authorization` header to downstream services. +- **`/api/devices/atms`** is guarded by `@PreAuthorize("hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')")` — broader than the typical `MSP_ADMIN`/`BCOS_ADMIN`-only pattern; other endpoints rely on the JWT filter rather than method-level `@PreAuthorize`. +- **`device_summary` is a Kafka-fed mirror** with no reconciler — if `hiveops.device.events` stops flowing, device data in reports goes stale (see the platform-wide device_summary mirror note). +- **Reports fetches, doesn't store, fleet data.** A report run makes live paginated HTTP calls to incident/journal/fleet; downstream outages surface as failed runs, not empty data. +- **CSV files** are written under `REPORT_UPLOAD_DIR` (default `/app/uploads/reports`; bind-mounted to `/data/hiveiq/reports/` in OpenMetal), not stored in the DB. +- **Ports:** internal `8088` (`PORT` env), external `8011` in OpenMetal; NGINX route `api.bcos.cloud/reports/` → `hiveiq-reports:8088/api/reports/`. +- **Async pool** (`AsyncConfig`) is small (core=2, max=5); the scheduler polls due templates on a fixed delay. diff --git a/backend/src/main/resources/content/technical/transactions/overview.md b/backend/src/main/resources/content/technical/transactions/overview.md new file mode 100644 index 0000000..e5b5107 --- /dev/null +++ b/backend/src/main/resources/content/technical/transactions/overview.md @@ -0,0 +1,49 @@ +--- +module: technical.transactions +title: Transactions — stateless proxy gateway for the transaction browser SPA +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-transactions source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-transactions` is a thin, stateless **backend-for-frontend proxy** for the transaction-browser / customer-journey SPA (`transactions.bcos.cloud`, port 5178). It owns no domain data of its own: it authenticates the caller's JWT, then forwards `/api/**` requests to the services that actually own the data — journal, devices, and incident/Adoons. It exposes one first-party endpoint, `/api/users/me`, derived from the JWT claims. See [platform.service-ownership]. + +## Data + +**No database.** The service has no `@Entity`, no JPA config, and no Flyway migrations — it holds no persistent state. Transaction data lives in the journal service DB; ATM/device data in the devices service DB. See [platform.data-stores]. + +## Kafka + +No Kafka. No producers, consumers, or `@KafkaListener` in the codebase. + +## API surface + +Server port **8098** (`application.properties`). All routes require a valid JWT (`hasAnyRole(USER, CUSTOMER, ADMIN, MSP_ADMIN, QDS_ADMIN, BCOS_ADMIN)`) except `/actuator/health` and `/error`. + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/api/users/me` | First-party: current user (email, role, institutionKey(s), fleetTaskApprover) from JWT — `UserController` | +| ANY | `/api/journal`, `/api/journal/**` | Proxy → journal service (`services.journal.url`) — `JournalProxyController` | +| ANY | `/api/atms`, `/api/atms/**` | Proxy → devices service — `DevicesProxyController` | +| ANY | `/api/atm-groups`, `/api/atm-groups/**` | Proxy → devices service — `DevicesProxyController` | +| ANY | `/api/adoons`, `/api/adoons/**` | Proxy → AI/Adoons endpoints (`services.ai.url`) — `AdoonsProxyController` | + +The three proxy controllers accept any HTTP method (`@RequestMapping` without a `method` restriction) and forward the original path + query string verbatim via `ProxyService.forward()`. + +**Upstream targets** (`AppProperties` / `application.properties`): +- `services.journal.url` → `JOURNAL_URL` (default `http://hiveiq-journal:8087`) +- `services.devices.url` → `DEVICES_URL` (default `http://hiveiq-devices-backend:8096`) +- `services.ai.url` → `AI_URL` (default `http://hiveiq-incident:8081`) + +## Gotchas + +- **`services.ai.url` points at incident, not the AI service.** Default is `http://hiveiq-incident:8081`; incident owns the `/api/adoons/**` endpoints (per the comment in `AdoonsProxyController`). Don't assume Adoons traffic hits hiveops-ai directly. +- **Role check is broad.** `SecurityConfig` gates `/api/**` with `hasAnyRole("USER","CUSTOMER","ADMIN","MSP_ADMIN","QDS_ADMIN","BCOS_ADMIN")` — not the `MSP_ADMIN`/`BCOS_ADMIN`-only pattern used elsewhere. Authorization beyond role is delegated to the upstream services. +- **Only `Authorization` and `Content-Type` headers are forwarded.** `ProxyService` rebuilds a fresh `HttpHeaders` — other inbound headers (custom `X-*`, cookies) are dropped. The `transfer-encoding` response header is stripped from replies. +- **CORS is disabled at the app** (`.cors(AbstractHttpConfigurer::disable)`); nginx is the CORS authority. Allowed origins config key (`cors.allowed-origins`, default `http://localhost:5178`) exists but is not wired into the disabled security chain. +- **JWT secret is required** — `jwt.secret=${JWT_SECRET}` has no default; the service will fail to start without it. +- `/api/users/me` reads the institution key(s) from the auth credentials: a single-element list → `institutionKey`, multiple → `institutionKeys`; presence of a `FLEET_APPROVER` authority sets `fleetTaskApprover`. diff --git a/backend/src/main/resources/content/technical/vault/overview.md b/backend/src/main/resources/content/technical/vault/overview.md new file mode 100644 index 0000000..d2e56bc --- /dev/null +++ b/backend/src/main/resources/content/technical/vault/overview.md @@ -0,0 +1,71 @@ +--- +module: technical.vault +title: Vault — CIT visit tracking & cash count service +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 from hiveops-vault source. Verify against code before relying on specifics. + +## Responsibility + +`hiveops-vault` records CIT (Cash in Transit) service visits on ATMs — what the CIT vendor finds, adds, and removes during a replenishment. It is the CIT-side data source for cash reconciliation: per-cassette counts (opening/added/removed/closing), non-cash removals (checks, deposits, rejected notes), and CSV bulk import of vendor files. It maintains a local read-only mirror of device metadata (`device_summary`) fed by Kafka. See [platform.service-ownership]. + +Combined repo: Spring Boot backend (`backend/`, port 8094) + Svelte SPA (`frontend/`, port 5185, `vault.bcos.cloud`). + +## Data + +PostgreSQL database **`hiveiq_vault`** (prod profile). See [platform.data-stores]. + +- Env vars are the split form `DB_HOST` / `DB_PORT` (default 5432) / `DB_NAME` (default `hiveiq_vault`) / `DB_USERNAME` / `DB_PASSWORD` — **not** the `DB_URL` form some other services use. +- Dev profile runs H2 in-memory (`create-drop`, Flyway disabled); prod runs PostgreSQL with `ddl-auto: validate` + Flyway migrations. + +Key tables (Flyway `V1__create_vault_schema.sql`): + +| Table | Entity | Purpose | +|-------|--------|---------| +| `cit_visits` | `CitVisit` | One CIT service call on an ATM; status `PENDING → CONFIRMED / DISPUTED` | +| `cit_cassette_counts` | `CitCassetteCount` | Per-slot counts (denomination, `cassette_type` DISPENSER/RECYCLER/RETRACT/MIX, opening/added/removed/closing) | +| `cit_removals` | `CitRemoval` | Non-cash items removed: `CHECK`, `DEPOSIT`, `REJECTED_NOTE` | +| `cit_imports` | `CitImport` | CSV import audit log; status `PROCESSING → COMPLETED / FAILED` | +| `device_summary` | `DeviceSummary` | Device metadata mirror populated from Kafka (added V3/V5 migrations) | + +All rows are scoped by `institution_key`. + +## Kafka + +**Consumed:** `hiveops.device.events` — listener `DeviceEventConsumer` (config `DeviceEventKafkaConfig`, consumer group `hiveops-vault-device-sync`, offset reset `earliest`). Upserts/deletes rows in `device_summary`; `DEVICE_DELETED` events remove the record. Payload type `DeviceEventMessage`. + +**Produced:** None (no `KafkaTemplate`/`StreamBridge` in the codebase). + +See [platform.kafka]. + +## API surface + +Base path **`/api/vault`**. All endpoints require a JWT Bearer token (validated by `JwtAuthenticationFilter` → `VaultPrincipal`). Results are scoped to the caller's `institutionKey`. + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/api/vault/visits` | List visits, paginated + filters (`atmId`, `vendor`, `status`, `from`, `to`) | +| GET | `/api/vault/visits/{id}` | Visit detail (cassette counts + removals) | +| POST | `/api/vault/visits` | Create a visit manually (201) | +| PUT | `/api/vault/visits/{id}` | Update status / notes / vendor | +| DELETE | `/api/vault/visits/{id}` | Delete a visit (204) | +| GET | `/api/vault/visits/recent` | Recent visits for dashboard | +| GET | `/api/vault/atm/{atmId}/visits` | All visits for one ATM | +| GET | `/api/vault/atms` | Device list (from `device_summary` mirror) for the institution | +| POST | `/api/vault/imports` | Upload CSV multipart `file` → async processing (202 Accepted) | +| GET | `/api/vault/imports` | Import history (paginated) | +| GET | `/api/vault/imports/{id}` | Single import status (poll while PROCESSING) | +| GET | `/api/vault/stats` | Dashboard aggregate stats | +| GET | `/api/vault/users/me` | Current user info decoded from JWT | + +## Gotchas + +- **No role checks.** `SecurityConfig` only requires `authenticated()` on `anyRequest()` — there is no `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` anywhere. Authorization is purely institution-scoping via `VaultPrincipal.institutionKey()`, not role gating. +- **Spring-level CORS is disabled** (`.cors(...::disable)`); CORS is handled upstream by NGINX. NGINX maps `api.bcos.cloud/vault/` → `hiveiq-vault:8094/api/vault/`. +- **MSP_ADMIN create path:** on `POST /visits`, when the principal has no `institutionKey` (multi-institution admin), the institution is derived from the visited ATM via `deviceSummaryRepository.findByAtmId(...)`. +- **device_summary is a Kafka-fed mirror, not owned data** — vault never writes device metadata except from the `hiveops.device.events` consumer. New devices with no `deviceId` in the event are skipped on insert; missing `institutionKey` defaults to `"BCOS"`. If the mirror is stale, it's an upstream (devices) publish gap, not a vault bug. +- **CSV upload validation** is content-type based (`csv`/`text` substring) and processing is `@Async` — the POST returns 202 immediately; clients must poll `/imports/{id}` for terminal `COMPLETED`/`FAILED`. +- **Uploads dir** defaults to `/tmp/vault-uploads` (`UPLOADS_DIR`); multipart max file size 20MB.