hiveiq-src/hiveops-guide
4
0
Fork 0
Code Issues 10 Pull Requests Actions 2 Packages Projects Releases Wiki Activity

content(guide): role-tabs (Internal/Architect/Claude) for all 63 customer modules (DRAFT, Refs #1 #3)

Browse Source 
Full role-tab rollout: every customer module now has Customer/Internal/
Architect/Claude tabs, grounded per-module in real code. Backend serves 296
docs / 96 modules. Grounding surfaced ~20 real bugs across services (filed
separately after verification).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Johannes
2026-07-01 12:14:54 -04:00
parent 916095f43b
commit 2f94d62769
189 changed files with 12123 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/main/resources/content/analytics/adoons/architect.md
+71
View File
@@ -0,0 +1,71 @@
---
module: analytics.adoons
title: Adoons Insights
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Written 2026-07-01 from `hiveops-analytics` source. Verify against code before relying on specifics.
## Shape of the feature
Adoons Insights is the simplest slice of `hiveops-analytics`: a **single-table CRUD store** (`ai_insights`) exposed over REST, rendered as a card list. Unlike the dashboard/snapshot side of analytics, the insight path is **not** event-driven and does **not** use the executor→Kafka→record-store pattern. It's a straight `Controller → Service → JpaRepository → Postgres` flow.
```
AdoonsTab.svelte ──GET /insights/active──▶ InsightController ──▶ AiInsightService ──▶ AiInsightRepository ──▶ ai_insights (hiveiq_analytics)
publisher (admin JWT) ──POST /insights──▶ InsightController (@PreAuthorize MSP_ADMIN/BCOS_ADMIN) ──▶ save()
```
## Services involved
- **hiveops-analytics** — owns everything for this feature: controller, service, entity, table. Port **8089** (prod host **8012**). Cross-link [platform.service-ownership].
- **Publisher of insights** — *external to this repo.* Analytics has **no insight generator, no Kafka consumer that creates insights, and no scheduled producer** for them. Insight rows arrive only via authenticated `POST`. The intended author is Adoons (`hiveops-ai`), but that producer is **not present in hiveops-analytics** (unverified where it lives / whether it's live).
## Data flow
There is no Kafka in the insight read/write path. Cross-link [platform.kafka] for the surrounding analytics topics (`analytics.snapshot-ready` produced by `SnapshotEventProducer`; `hiveops.incidents.*` and `journal.file-parsed` consumed for snapshots) — **none of these touch `ai_insights`.**
The only coupling to the rest of analytics is a **nullable FK** `ai_insights.snapshot_id → analytics_snapshots.id`. On publish:
- If `snapshotId` is supplied, that snapshot is linked.
- Else `AiInsightService.publish()` links the **latest** snapshot (`findTopByOrderByCreatedAtDesc()`), or `null` if none exist.
So an insight can *reference* the fleet metrics snapshot it was derived from, but the reference is optional and never enforced.
## DB tables
| Table | R/W here | Columns of interest |
|-------|----------|---------------------|
| `ai_insights` | read + write | `id`, `category`, `severity`, `title`, `body`, `snapshot_id` (FK), `published_at`, `expires_at`, `acknowledged`, `acknowledged_at`, `acknowledged_by_user_id` |
| `analytics_snapshots` | read-only (FK lookup on publish) | `id`, `created_at` (used by `findTop...`) |
DB **`hiveiq_analytics`** (PostgreSQL prod / H2 dev). Cross-link [platform.data-architecture]. Flyway manages schema; `ai_insights` is created in the analytics migration set (V1 init + later columns).
**Enums (JPA `EnumType.STRING`):**
- `Category` = `ANOMALY | TREND | RECOMMENDATION | SUMMARY`
- `Severity` = `INFO | WARNING | CRITICAL` (entity default `INFO`)
## Key API endpoints (`InsightController`, base `/api/analytics/insights`)
| Method | Path | Behaviour | Auth |
|--------|------|-----------|------|
| GET | `/api/analytics/insights` | Paged list, `findAllByOrderByPublishedAtDesc` → `Page<AiInsightResponse>` | authenticated |
| GET | `/api/analytics/insights/active` | `findActive(now)`: unack'd + unexpired, newest first | authenticated |
| POST | `/api/analytics/insights` | `publish(req)`, returns `201` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
| PATCH | `/api/analytics/insights/{id}/acknowledge` | `acknowledge(id, principal.userId())`; `404` if id unknown; returns `204` | authenticated |
| DELETE | `/api/analytics/insights/active` | `acknowledgeAll(now)`; returns `204` | authenticated |
## Lifecycle / retention
- `published_at` set on `@PrePersist` if unset.
- Active = `acknowledged = false AND (expires_at IS NULL OR expires_at > now)`.
- `acknowledge` is an atomic JPQL `UPDATE`; `acknowledgeAll` clears every unacknowledged row.
- **Nightly purge:** `AiInsightService.purgeOldAcknowledged()` — `@Scheduled(cron = "0 0 3 * * *")`, deletes rows where `acknowledged = true AND acknowledged_at < now-30d`.
## Security model
- `SecurityConfig`: stateless, CSRF disabled; `permitAll` only for `/actuator/health` + `/api/public/**`; `anyRequest().authenticated()`.
- `JwtAuthenticationFilter` reads the token from the **`Authorization: Bearer` header only** (no cookie fallback) and builds an `AnalyticsPrincipal(userId, email, role, institutionKey)`.
- Only publish is role-restricted; read + acknowledge are open to any authenticated principal (note: no institution scoping on insights — they are fleet-global).
Cross-links: [technical.analytics], [platform.data-architecture], [platform.kafka], [platform.service-ownership].
backend/src/main/resources/content/analytics/adoons/claude.md
+68
View File
@@ -0,0 +1,68 @@
---
module: analytics.adoons
title: Adoons Insights
tab: Claude
order: 40
audience: internal
---
> **Internal · agent reference.** Grounded in `hiveops-analytics` @ 2026-07-01. Only confirmed facts below.
## Ownership
- **Service:** `hiveops-analytics` owns this feature end-to-end (controller, service, entity, table).
- **Port:** 8089 (prod host 8012). **NGINX:** `/api/analytics/`. External base e.g. `https://api.bcos.cloud/analytics/api/analytics/insights`.
- **No AI generation in-repo.** Insights exist only because someone `POST`ed them. Do not assume an in-service generator/Kafka consumer creates them.
## Endpoints (`InsightController`, base `/api/analytics/insights`)
| Method | Path | Auth |
|--------|------|------|
| GET | `/api/analytics/insights` | authenticated (paged) |
| GET | `/api/analytics/insights/active` | authenticated |
| POST | `/api/analytics/insights` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
| PATCH | `/api/analytics/insights/{id}/acknowledge` | authenticated |
| DELETE | `/api/analytics/insights/active` | authenticated (ack all) |
Frontend calls: `getActiveInsights` → `GET /insights/active`; `acknowledgeInsight(id)` → `PATCH /insights/{id}/acknowledge`.
## Auth
- Header only: `Authorization: Bearer <jwt>`. No cookie fallback.
- Open paths: `/actuator/health`, `/api/public/**`. Everything else authenticated.
- Principal: `AnalyticsPrincipal(userId, email, role, institutionKey)`.
## Data
- **DB:** `hiveiq_analytics` (Postgres prod on 10.10.10.188 via ProxyJump; H2 dev).
- **Table:** `ai_insights` (cols: `id, category, severity, title, body, snapshot_id, published_at, expires_at, acknowledged, acknowledged_at, acknowledged_by_user_id`).
- **FK:** `ai_insights.snapshot_id → analytics_snapshots.id` (nullable).
- **Enums:** `category ∈ {ANOMALY,TREND,RECOMMENDATION,SUMMARY}`; `severity ∈ {INFO,WARNING,CRITICAL}`.
## POST body (`PublishInsightRequest`)
```json
{"category":"TREND","severity":"WARNING","title":"...","body":"...","snapshotId":123,"expiresAt":"2026-08-01T00:00:00Z"}
```
- `category,severity,title,body` = `@NotBlank`. `snapshotId,expiresAt` optional.
- Omit `snapshotId` → links latest snapshot (or null if none).
## Kafka
- Insight path uses **no Kafka**. (Analytics produces `analytics.snapshot-ready`; consumes `hiveops.incidents.created/updated/resolved`, `journal.file-parsed` — snapshots only, NOT insights.)
## Active-list rule
- `/insights/active` = `acknowledged=false AND (expires_at IS NULL OR expires_at > now)`, ORDER BY `published_at DESC`.
- Nightly purge: cron `0 0 3 * * *` deletes acknowledged rows older than 30d.
## Gotchas
- Severity enum is `INFO/WARNING/CRITICAL`, but frontend colour map keys `CRITICAL/HIGH/MEDIUM/LOW/INFO` → `WARNING` renders default blue; `HIGH/MEDIUM/LOW` can never come from this backend.
- Customer `overview.md` lists CRITICAL/HIGH/MEDIUM/LOW/INFO — does NOT match backend enum. Trust the enum.
- Frontend `api.ts` sets NO Authorization header / no axios interceptor, yet backend requires JWT on all `/api/analytics/**`. Token injection is not in frontend code (mechanism unverified — likely NGINX/host shell).
- `AdoonsTab.dismiss()` swallows PATCH errors silently → failed acknowledge = card silently stays.
- `acknowledge` / `acknowledgeAll` are NOT role-gated — any authenticated user can dismiss fleet-global insights.
- POST with bad enum string → 400 (`valueOf` throws). PATCH unknown id → 404.
## Do NOT
- Do NOT expect insights to auto-appear — none generate inside analytics; only `POST` creates them.
- Do NOT use `Authorization` bearer header assumptions for the frontend; it doesn't set one.
- Do NOT invent per-institution scoping — insights are fleet-global, no `institution_key` on the table.
- Do NOT route insight writes through Kafka; it's plain REST+JPA.
- Do NOT publish `HIGH/MEDIUM/LOW` severities — not valid enum values.
- Do NOT put insight reads under `/api/public/**` — they require auth by design.
Cross-links: [technical.analytics], [platform.service-ownership].
backend/src/main/resources/content/analytics/adoons/internal.md
+73
View File
@@ -0,0 +1,73 @@
---
module: analytics.adoons
title: Adoons Insights
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support/admin.** Written 2026-07-01 from `hiveops-analytics` source. Verify against code before relying on specifics.
## What this screen actually is
The **Adoons Insights** (a.k.a. "Fleet Insights") tab is a thin card list over a plain REST + JPA store. Each card is one row in the `ai_insights` table, served by **hiveops-analytics** (port **8089**, prod host port **8012**, NGINX `/api/analytics/`). There is **no AI/ML generation inside analytics** — insights only exist because something `POST`ed them.
- **Owner service:** `hiveops-analytics`
- **Table:** `ai_insights` in DB `hiveiq_analytics`
- **Frontend:** `AdoonsTab.svelte` → `analyticsAPI.getActiveInsights()` on load and on **Refresh**.
## Roles required
| Action | Endpoint | Auth |
|--------|----------|------|
| View active insights | `GET /api/analytics/insights/active` | Any authenticated JWT |
| View all (paged) | `GET /api/analytics/insights` | Any authenticated JWT |
| **Publish an insight** | `POST /api/analytics/insights` | **`hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`** |
| Dismiss one (Refresh ✕) | `PATCH /api/analytics/insights/{id}/acknowledge` | Any authenticated JWT |
| Dismiss all | `DELETE /api/analytics/insights/active` | Any authenticated JWT |
**Admin-only action = publishing.** Only MSP_ADMIN / BCOS_ADMIN can create insight cards. Everyone else can read and dismiss.
## How a card gets on the screen
1. A publisher (Adoons / an operator with an admin JWT) calls `POST /api/analytics/insights` with `{category, severity, title, body, [snapshotId], [expiresAt]}`.
2. Row is written to `ai_insights`; `published_at` defaults to now.
3. It shows in `/insights/active` until it is **acknowledged** or its `expires_at` passes.
4. Acknowledged rows older than 30 days are purged nightly (`@Scheduled cron = "0 0 3 * * *"`).
`GET /insights/active` returns only rows where `acknowledged = false AND (expires_at IS NULL OR expires_at > now)`, newest first.
## First things to check when it misbehaves
**"No active fleet insights" but there should be some**
- Confirm rows exist: query `ai_insights` in `hiveiq_analytics` (Postgres on **10.10.10.188** via ProxyJump).
- Check they aren't all `acknowledged = true` or past `expires_at` — the active query hides both.
- Remember: analytics does **not** self-generate insights. If nothing is `POST`ing them, the tab is *correctly* empty. Check whoever owns publishing (external Adoons/AI job — not in this repo).
**Cards won't load / error bar shows**
- The SPA calls `/api/analytics/insights/active`. Hit it directly with a real JWT and expect `200` + a JSON array.
- All `/api/analytics/**` require an `Authorization: Bearer <jwt>` header (only `/api/public/**` and `/actuator/health` are open). A `401/403` here = missing/expired token or wrong `role`.
- Downstream summary endpoints (incident/journal/mgmt) do **not** feed this tab — insights are self-contained. A red insights panel is an analytics-only problem.
**Publish returns 403**
- Caller's JWT `role` is not `MSP_ADMIN` or `BCOS_ADMIN`. Only publish is role-gated.
**Publish returns 400**
- `category` must be one of `ANOMALY | TREND | RECOMMENDATION | SUMMARY`; `severity` must be one of `INFO | WARNING | CRITICAL`. Any other value fails `valueOf(...)`. `title`/`body`/`category`/`severity` are all `@NotBlank`.
**Dismiss (✕) does nothing / card reappears on refresh**
- `AdoonsTab.dismiss()` **swallows errors silently** (empty `catch`). If the `PATCH .../acknowledge` fails (e.g. 404 for a stale id, or auth), the card stays and no error is shown. Test the PATCH directly to see the real status.
**Severity colour looks wrong (card renders blue instead of orange)**
- Known mismatch: backend severities are `INFO | WARNING | CRITICAL`, but the frontend colour map only keys `CRITICAL/HIGH/MEDIUM/LOW/INFO`. A `WARNING` insight has **no colour entry** and falls back to the INFO blue. If you need a visible non-critical warning, that's a frontend gap, not bad data.
## Manual publish (support/testing)
```bash
curl -s -X POST https://api.bcos.cloud/analytics/api/analytics/insights \
-H "Authorization: Bearer $JWT" \
-H "Content-Type: application/json" \
-d '{"category":"TREND","severity":"WARNING","title":"Test insight","body":"Manual test.","expiresAt":"2026-08-01T00:00:00Z"}'
```
Cross-links: [technical.analytics], [platform.service-ownership].
backend/src/main/resources/content/analytics/agents/architect.md
+82
View File
@@ -0,0 +1,82 @@
---
module: analytics.agents
title: Agents
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Grounded in hiveops-analytics + hiveops-devices source (2026-07-01). Cross-link [platform.data-architecture], [platform.kafka], [platform.service-ownership], [technical.analytics].
## Shape of the feature
The Agents tab is a **thin read-through aggregation**. No dedicated persistence, no Kafka, no snapshot. On each page load the analytics backend fans out **one** HTTP call to the devices backend, computes distributions in-memory, and returns a single DTO. This is the exception to the rest of the analytics app, which is snapshot/Kafka-driven (see [technical.analytics]).
## Services involved
| Service | Role for this feature | Port |
|---------|-----------------------|------|
| hiveops-analytics | Serves `GET /api/analytics/agents`; computes version distribution, connection counts, latest-semver, outdated list | 8089 |
| hiveops-devices | **Source of truth** for ATM/agent data; serves `GET /api/atms/paginated` | 8096 |
Analytics reaches devices via a `WebClient` bean named **`incidentWebClient`** (`WebClientConfig`), base URL from `services.incident.url` (`INCIDENT_SERVICE_URL`, default `http://hiveops-incident-backend:8080`). Historically `/api/atms/*` lived in incident; those endpoints were **extracted into hiveops-devices ~2026-06-18** (`AtmController`, `@RequestMapping({"/api/atms","/api/devices"})`). The analytics bean name and default URL still say "incident" — in a correct deploy that URL must resolve to the backend actually serving `/api/atms/paginated` (devices). See [platform.service-ownership].
## Data flow
```
Browser (AgentsTab.svelte)
│ GET /api/analytics/agents (Authorization: Bearer <userJWT>)
▼
hiveops-analytics
AgentInsightController.agents()
→ extracts raw token from Authorization header
AgentInsightService.buildAgentInsight(token)
→ incidentWebClient GET /api/atms/paginated?size=500&page=0
(forwards "Bearer <userJWT>")
▼
hiveops-devices
AtmController.getAllAtmsPaginated(...)
→ AtmService.getAllAtmsPaginated(...) → Spring Data Page<AtmDTO>
→ reads atms + atm_properties (agentVersion, lastHeartbeat)
→ agentConnectionStatus derived by calculateConnectionStatus(lastHeartbeat)
◀ Page JSON: { content: [AtmDTO...], page: {...} }
▼
hiveops-analytics (in-memory aggregation)
- versionDistribution = groupingBy(agentVersion), desc by count
- latestVersion = findLatestSemver(distinct versions) [highest x.y.z present]
- connected/disconnected/neverConnectedCount = filter on agentConnectionStatus
- outdatedAtms = every ATM where version blank or != latestVersion
◀ AgentInsightResponse
▼
Browser renders bars / status counts / outdated table
```
### Fields consumed from each `AtmDTO`
`atmId`, `agentVersion`, `lastHeartbeat` (ISO string after JSON serialization of `LocalDateTime`), `agentConnectionStatus` (`CONNECTED` | `DISCONNECTED` | `NEVER_CONNECTED`). Analytics reads them as loosely-typed `Map<String,Object>` and defensively coerces to String.
## Kafka
**None for this feature.** The Agents tab is a synchronous read-through — it does **not** consume or produce any topic, and does **not** read the analytics `analytics_snapshots` table. Contrast with the rest of analytics, which consumes `hiveops.incidents.*` / `journal.file-parsed` and produces `analytics.snapshot-ready` (see [technical.analytics], [platform.kafka]). Device changes propagate to *other* services via devices' `hiveops.device.events`, but analytics' Agents tab bypasses that and hits devices' REST API live.
## Databases
| DB | Owner | Read/written by this feature |
|----|-------|------------------------------|
| `hiveiq_devices` → `atms`, `atm_properties` | hiveops-devices | **Read** (indirectly, via devices REST). `agentVersion`/`lastHeartbeat` live on `atm_properties`. |
| `hiveiq_analytics` | hiveops-analytics | **Not touched** by the Agents tab. |
Connection status is computed, not stored: `calculateConnectionStatus(lastHeartbeat)` → `NEVER_CONNECTED` (null), `CONNECTED` (≤15 min), else `DISCONNECTED`.
## Key endpoints
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/agents` | hiveops-analytics | authenticated JWT (no role gate) |
| GET | `/api/atms/paginated?size=500&page=0` | hiveops-devices | authenticated JWT (no role gate); JWT forwarded from analytics |
## Design notes & consequences
- **Read-through, not executor→Kafka→record-store.** Unlike the snapshot pipeline in [technical.analytics], this tab has no scheduler, no producer, no record store — it trades freshness-guarantees for simplicity and is always live-consistent with devices at request time.
- **Fail-soft aggregation:** `fetchAtmsPaginated` catches everything and returns `List.of()`, so upstream failures surface as an empty (not errored) tab.
- **Single page, size 500:** the aggregation is bounded to the first 500 ATMs — a scaling limit as fleets grow.
- **"Latest" is fleet-relative:** `findLatestSemver` = max semver present among devices, independent of the CDN `latest.json`. Strict `\d+\.\d+\.\d+` regex means suffixed versions collapse "latest" to `"unknown"`.
backend/src/main/resources/content/analytics/agents/claude.md
+55
View File
@@ -0,0 +1,55 @@
---
module: analytics.agents
title: Agents
tab: Claude
order: 40
audience: internal
---
> **Internal · AI agent quick-ref.** Confirmed against source 2026-07-01. Act on confirmed facts only.
## Ownership
- **Tab served by:** `hiveops-analytics` (port 8089), `AgentInsightController` + `AgentInsightService`.
- **Data owned by:** `hiveops-devices` (port 8096) — source of truth for ATM/agent fields.
- **DB behind the tab:** `hiveiq_devices` only. `hiveiq_analytics` is **not** used by this tab.
## Endpoints (METHOD path)
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/agents` | hiveops-analytics | authenticated JWT, **no role** |
| GET | `/api/atms/paginated?size=500&page=0` | hiveops-devices | authenticated JWT, **no role**; caller JWT forwarded by analytics |
## Response DTO (`AgentInsightResponse`)
- `versionDistribution: Map<version,count>`
- `latestVersion: String` — highest `x.y.z` present in fleet, else `"unknown"`
- `connectedCount` / `disconnectedCount` / `neverConnectedCount: int`
- `outdatedAtms: [{ atmId, agentVersion, lastHeartbeat, connectionStatus }]`
## Tables
- `atms`, `atm_properties` (in `hiveiq_devices`) — `atm_properties` holds `agentVersion`, `lastHeartbeat`.
## Kafka topics
- **None** produced or consumed by this tab. (Analytics-wide topics `hiveops.incidents.*`, `journal.file-parsed`, `analytics.snapshot-ready` are unrelated to Agents. Devices emits `hiveops.device.events` but this tab reads devices via REST, not that topic.)
## Connection status rule (computed in devices, not stored)
- `lastHeartbeat == null` → `NEVER_CONNECTED`
- heartbeat ≤ **15 min** → `CONNECTED`
- else → `DISCONNECTED`
## Wiring gotcha
- Analytics calls devices through a bean named `incidentWebClient`, base URL `services.incident.url` / `INCIDENT_SERVICE_URL` (default `http://hiveops-incident-backend:8080`). `/api/atms/paginated` is a **devices** endpoint (moved out of incident ~2026-06-18). That env var must resolve to the backend serving `/api/atms/paginated`, or the tab goes empty.
## Gotchas
- Empty tab = upstream failure. `fetchAtmsPaginated` **catches all exceptions → returns `[]`**; logs `AgentInsight: failed to fetch ATMs paginated` (Loki, service `hiveiq-analytics`).
- Only **first 500 ATMs** (`size=500,page=0`) are aggregated. Larger fleets under-count.
- `latestVersion` = max semver **present in fleet**, NOT the CDN release. Regex is strict `\d+\.\d+\.\d+`; suffixed/prefixed versions (`v4.4.3`, `4.4.3-patch`) → `"unknown"` → every ATM flagged outdated.
- `lastHeartbeat` serialized from `LocalDateTime` → ISO string; frontend renders relative ("2h ago").
## Do NOT
- Do NOT tell users the Agents tab pushes/schedules updates — it is **read-only**.
- Do NOT look for an analytics DB table for this tab — there isn't one; it's a live read-through.
- Do NOT claim "latest" reflects the newest released agent — it's fleet-relative.
- Do NOT expect a Kafka topic for this feature.
- Do NOT trust totals on fleets > 500 devices.
- Do NOT require an admin role to view — no `@PreAuthorize` on either endpoint.
- Do NOT hit incident for this data — devices owns `/api/atms/paginated`.
backend/src/main/resources/content/analytics/agents/internal.md
+62
View File
@@ -0,0 +1,62 @@
---
module: analytics.agents
title: Agents
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support/admin.** Grounded in hiveops-analytics + hiveops-devices source (2026-07-01). This is a **read-only** dashboard tab — there is nothing to push, edit, or approve here.
## What this tab actually is
The **Agents** tab (analytics SPA, `analytics.bcos.cloud`) is a single GET against the analytics backend that aggregates agent-software health across the fleet. It shows:
- **Version Distribution** — count of ATMs per `agentVersion`.
- **Connection Status** — Connected / Disconnected / Never Connected counts + percentages.
- **Outdated ATMs** — every ATM not on the computed "latest" version.
All three panels are derived from **one** upstream call. There is no analytics DB table behind this tab — it is computed live on each request.
## Access / roles
- **Viewing:** any authenticated JWT. The analytics `/api/analytics/agents` endpoint is **not** role-gated — analytics `SecurityConfig` only requires `.authenticated()` (permitAll is limited to `/actuator/health` and `/api/public/**`).
- **Upstream `GET /api/atms/paginated`** (hiveops-devices) is likewise authenticated-only, **no** `@PreAuthorize` — the `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` gates on that controller apply to *mutating* device endpoints, not the paginated list.
- There are **no admin-only actions on this tab.** (The admin-gated action in the analytics app is `POST /api/analytics/insights`, which belongs to the Adoons Insights panel, not Agents.)
## Data path (one line)
`Agents tab → GET /api/analytics/agents (hiveops-analytics :8089) → GET /api/atms/paginated?size=500&page=0 (hiveops-devices :8096) → atms/atm_properties tables`
The analytics service **forwards the caller's Bearer token** downstream (no service account). If the caller's token can't read devices, the tab degrades silently (see below).
## First things to check when it misbehaves
**Tab is empty / "No version data" / everything shows "unknown":**
1. `AgentInsightService.fetchAtmsPaginated()` **swallows all exceptions and returns an empty list** (`log.warn("AgentInsight: failed to fetch ATMs paginated: ...")`). So an empty tab = the upstream call failed or returned nothing. Check analytics logs (Loki, service `hiveiq-analytics`) for that warn line.
2. **Verify `INCIDENT_SERVICE_URL` points at a backend that serves `/api/atms/paginated`.** This endpoint is owned by **hiveops-devices** (extracted from incident ~2026-06-18). Analytics calls it through its `incidentWebClient` (default `http://hiveops-incident-backend:8080`). If that URL still points at plain incident (which no longer serves `/api/atms/paginated`), the call 404s → empty tab. This is the #1 thing to confirm on a fresh/mis-wired deploy.
3. Confirm the caller's JWT is valid and reaches devices (401/403 downstream also yields an empty list, not an error to the user).
**"Latest" version looks wrong / all ATMs flagged outdated:**
- "Latest" is **not** the CDN release version. `findLatestSemver()` picks the **highest semver already present in the fleet**. If no ATM is on the newest release yet, "latest" lags reality.
- The regex is strict: `\d+\.\d+\.\d+` only. Any version with a suffix/prefix (`v4.4.3`, `4.4.3-patch`, `4.4.3-standard`) **fails to parse** → `latestVersion` becomes `"unknown"` → **every** ATM is flagged outdated. If the whole fleet suddenly shows outdated, check whether agent version strings picked up a suffix.
**Counts look short on a large fleet:**
- The upstream call is hard-coded to `size=500, page=0`. **Only the first 500 ATMs are considered.** Fleets over 500 devices under-report version distribution, connection counts, and outdated lists. This is a code limitation, not a config knob.
**Connection status disagrees with the Devices app:**
- Status is computed in **hiveops-devices** (`calculateConnectionStatus`): `NEVER_CONNECTED` if `lastHeartbeat` is null; `CONNECTED` if last heartbeat ≤ **15 minutes** ago; else `DISCONNECTED`. (Note: older docs say "5 min" — the code uses **15**.) The Agents tab just reflects whatever devices returns, so reconcile at the Devices source, not here.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Empty tab, `AgentInsight: failed to fetch` in logs | `INCIDENT_SERVICE_URL` not pointing at a backend serving `/api/atms/paginated` (devices), or downstream 401/403 | Repoint env to devices backend; verify JWT reaches devices |
| All ATMs "outdated", latest = `unknown` | Agent version strings not bare `x.y.z` semver | Confirm agent version format; the semver regex is strict |
| Version/connection counts too low | Fleet > 500 devices; only page 0 fetched | Known limitation (see uncertainties) — do not trust totals on large fleets |
| Status differs from Devices app | Different heartbeat window / stale mirror | Devices is source of truth; this tab is downstream-only |
## Related
- Customer-facing description: [analytics.agents] Overview tab.
- Service internals: [technical.analytics].
backend/src/main/resources/content/analytics/cash/architect.md
+85
View File
@@ -0,0 +1,85 @@
---
module: analytics.cash
title: Cash Intelligence
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Grounded in `hiveops-analytics` + `hiveops-recon` source, 2026-07-01. Key point: Cash Intelligence is a **synchronous fan-out**, not the usual executor→Kafka→record-store pattern. Analytics stores nothing for this feature and consumes no Kafka topic to build it.
## Services involved
| Service | Role in this feature |
|---------|----------------------|
| **hiveops-analytics** (port 8089, `analytics.bcos.cloud`) | Serves the tab. Owns the frontend + a thin transform. **No cash data of its own.** |
| **hiveops-recon** (port 8091, prod ext 8013, `recon.bcos.cloud`) | Source of truth for all cash/reconciliation data. Owns DB `hiveiq_recon`. |
| **hiveops-incident** | Called *by recon* (not analytics) to enumerate all fleet ATM IDs for admin/full-fleet views. |
| **hiveops-journal** | Upstream of recon's cash math (withdrawals/deposits/replenishments); not touched at cash-intelligence request time. |
## Data flow (request time, fully synchronous)
```
Browser (CashIntelligenceTab.svelte)
│ GET /api/analytics/cash-intelligence (Bearer JWT)
▼
hiveops-analytics · CashIntelligenceController
│ extracts raw bearer token from Authorization header
▼
CashIntelligenceService.buildCashIntelligence(token)
│ reconWebClient (baseUrl services.recon.url = http://hiveiq-recon:8091)
│ GET /api/recon/fleet/summary (forwards caller's "Bearer <token>")
▼
hiveops-recon · SummaryController.fleetSummary()
│ InstitutionContext.resolveScope() → institution list or null (full fleet)
│ reads atm_cash_positions (scoped) + reconciliation_cycles (per position)
│ admin/full-fleet only: incidentFetchService.fetchAllAtmIds() → merge all ATMs
│ applies recon device-selection filter (recon_device_selection)
│ counts replenishment_sessions in PENDING_CONFIRMATION
▼ FleetSummaryResponse{totalAtms, balancedCycles, varianceCycles, openCycles,
│ pendingConfirmations, atms[AtmSummaryResponse]}
▲
CashIntelligenceService (transform, in-memory, no persistence)
│ near-empty = atms where projectedBalanceCents < 1_000_000, asc
│ variance = atms where abs(varianceCents) > 0, desc by |variance|
▼ CashIntelligenceResponse
Browser renders 4 stat cards + 2 tables
```
## No Kafka on this path
- The **cash-intelligence request itself uses no Kafka topic and no consumer.** Do not look for an `analytics.*` cash topic — there isn't one.
- The broader analytics service *does* run Kafka (snapshot/insight pipeline: `analytics.snapshot-ready`, consumers of `hiveops.incidents.*` / `journal.file-parsed`), but **Cash Intelligence bypasses all of it** and reads recon live. See [technical.analytics] for that separate machinery. Cross-link [platform.kafka].
- Recon's own cash figures are built asynchronously (journal sync + replenishment cycles) *before* this request; by read time they are already materialised in `hiveiq_recon`.
## DB tables
Analytics reads/writes **no table** for this feature. All reads happen in recon against **`hiveiq_recon`**:
| Table | Read here for | Key columns used |
|-------|---------------|------------------|
| `atm_cash_positions` | per-ATM balances + cycle pointer | `atm_id`, `institution`, `current_cycle_id`, `projected_balance_cents`, `last_replenishment_at`, `journal_format` |
| `reconciliation_cycles` | cycle status + variance | `status` (OPEN / PENDING_RECONCILIATION / BALANCED / VARIANCE / UNRECONCILED), `variance_cents`, `synced_at`, `cycle_start` |
| `replenishment_sessions` | pending-confirmation count | `status` = `PENDING_CONFIRMATION` |
Only `BALANCED`, `VARIANCE`, `OPEN` cycle statuses contribute to the three cycle counts; other statuses fall through `default -> {}`. Cross-link [platform.data-architecture].
## Key API endpoints
| Method | Path | Service | Auth | Notes |
|--------|------|---------|------|-------|
| GET | `/api/analytics/cash-intelligence` | analytics | authenticated JWT (no role gate) | The tab's only call |
| GET | `/api/recon/fleet/summary` | recon | authenticated JWT | Upstream source; institution-scoped internally |
| GET | `/api/recon/fleet/pending` | recon | authenticated JWT | Related, not used by this tab |
| GET | `/api/recon/atm/{atmId}/summary` | recon | authenticated JWT | Per-ATM detail, not used by this tab |
## Auth & scoping model
- Analytics has **no service account**: it forwards the **caller's** bearer JWT verbatim to recon (`Authorization: Bearer <token>`). If the user's token is invalid/expired at recon, the tab empties.
- Scoping is entirely a **recon** concern via `InstitutionContext.resolveScope()`:
- `ROLE_CUSTOMER` / `ROLE_MSP_ADMIN` → scoped to JWT credentials (their institution ATMs only, existing recon positions only).
- anything else (e.g. `BCOS_ADMIN`) → `null` → full fleet, with all ATM IDs merged from incident.
- Cross-link [platform.service-ownership]: cash reconciliation is recon's domain; analytics is a presentation layer that must not persist or mutate cash state.
## Configuration surface
| Property / env | Default | Effect |
|----------------|---------|--------|
| `services.recon.url` (Spring; override via `SERVICES_RECON_URL`) | `http://hiveiq-recon:8091` | recon base URL. **Not wired to `RECON_SERVICE_URL`** in analytics `application.yml`. |
| WebClient `maxInMemorySize` | 16 MiB | caps the recon response the analytics WebClient will buffer |
## Failure semantics (by design)
- Recon error / timeout → `CashIntelligenceService` catches, logs a warn, returns `Map.of()` → **degrades to all-zero response** rather than surfacing an error to the UI. Freshness/health of this tab therefore depends entirely on recon being reachable and the JWT being accepted.
backend/src/main/resources/content/analytics/cash/claude.md
+65
View File
@@ -0,0 +1,65 @@
---
module: analytics.cash
title: Cash Intelligence
tab: Claude
order: 40
audience: internal
---
> Terse agent reference. Cash Intelligence = **read-only pass-through**: analytics re-shapes one recon call. Analytics owns NO cash data.
## Ownership
- **UI + endpoint owner:** `hiveops-analytics` (port 8089, `analytics.bcos.cloud`).
- **Data owner (source of truth):** `hiveops-recon` (port 8091, prod ext 8013, DB `hiveiq_recon`).
- Analytics persists nothing and consumes no Kafka for this feature.
## Endpoints (METHOD + path)
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/cash-intelligence` | analytics | JWT authenticated, **no role gate** |
| GET | `/api/recon/fleet/summary` | recon (upstream) | JWT authenticated; institution-scoped internally |
- Frontend base: `window.__APP_CONFIG__.apiUrl` else `http://localhost:8089/api/analytics`.
- Analytics → recon base URL: property `services.recon.url`, default `http://hiveiq-recon:8091`. Caller JWT forwarded verbatim.
## Response shape
- Analytics `CashIntelligenceResponse`: `totalAtms, balancedCount, varianceCount, openCount, pendingConfirmationsCount, atmsNearEmpty[], atmsWithVariance[]`.
- `atmsNearEmpty` item: `{atmId, projectedBalanceCents, lastReplenishmentAt}`.
- `atmsWithVariance` item: `{atmId, varianceCents, cycleStatus}`.
- Recon `FleetSummaryResponse`: `totalAtms, balancedCycles, varianceCycles, openCycles, pendingConfirmations, atms[]`.
## Transform rules (in analytics)
- Near-empty filter: `projectedBalanceCents < 1_000_000` (< $10,000), sort asc.
- Variance filter: `abs(varianceCents) > 0`, sort desc by `|variance|`.
- `cycleStatus` maps from recon field `currentCycleStatus`.
- Frontend red-row highlight (visual only): `< 300_000` cents (< $3,000).
## Tables (all in `hiveiq_recon`, read by recon only)
| Table | Purpose |
|-------|---------|
| `atm_cash_positions` | balances, `current_cycle_id`, `institution`, `projected_balance_cents`, `last_replenishment_at` |
| `reconciliation_cycles` | `status`, `variance_cents`, `synced_at` |
| `replenishment_sessions` | pending count (`status = PENDING_CONFIRMATION`) |
- Cycle `Status` enum: `OPEN, PENDING_RECONCILIATION, BALANCED, VARIANCE, UNRECONCILED`. Only `BALANCED/VARIANCE/OPEN` counted.
## Topics
- **None** for cash-intelligence. (Analytics' snapshot/insight Kafka is unrelated — do not attribute it to this feature.)
## Scoping (in recon `InstitutionContext.resolveScope()`)
- `ROLE_CUSTOMER` / `ROLE_MSP_ADMIN` → scoped to JWT institutions; only ATMs with existing recon positions.
- other roles (e.g. `BCOS_ADMIN`) → null → full fleet; recon merges all ATM IDs from hiveops-incident.
- recon device-selection filter (`recon_device_selection`) can further narrow results.
## Gotchas
- Recon fetch failure → analytics catches, logs `CashIntelligence: failed to fetch fleet summary from recon:`, returns empty map → **all-zero tab, no error surfaced**. Empty tab == recon problem, not analytics.
- recon URL override is `SERVICES_RECON_URL` (Spring relaxed binding), **NOT** `RECON_SERVICE_URL` (that env is unmapped in analytics `application.yml`).
- Null `projectedBalanceCents` coerced to `0` → ATM appears in Near Empty at `$0.00` (admin full-fleet view includes position-less ATMs).
- Cents are integers; frontend divides by 100. Negative variance = short cash.
## Do NOT
- Do NOT add cash endpoints/tables to analytics — cash domain belongs to recon ([platform.service-ownership]).
- Do NOT look for a Kafka topic feeding this tab; it's a synchronous recon call.
- Do NOT assume a role gate on `/api/analytics/cash-intelligence` — any authenticated JWT reaches it (scoping happens downstream in recon).
- Do NOT set `RECON_SERVICE_URL` expecting it to change the recon target; use `SERVICES_RECON_URL`.
- Do NOT expect data for ATMs never onboarded in Recon (scoped users won't see them).
backend/src/main/resources/content/analytics/cash/internal.md
+62
View File
@@ -0,0 +1,62 @@
---
module: analytics.cash
title: Cash Intelligence
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support.** Written 2026-07-01, grounded in `hiveops-analytics` + `hiveops-recon` source. The Cash Intelligence tab is a **read-only pass-through**: analytics owns no cash data — it re-shapes one call to **hiveops-recon**. When it looks wrong, the fix is almost always in recon or the analytics→recon link, not in analytics itself.
## What it actually is
- Frontend `CashIntelligenceTab.svelte` (in `hiveops-analytics/frontend`) calls **`GET /api/analytics/cash-intelligence`**.
- `CashIntelligenceService` forwards the caller's JWT and calls **`GET /api/recon/fleet/summary`** on **hiveops-recon**.
- The four counts (Balanced / Variance / Open Cycle / Pending Conf.) and both tables (Near Empty, With Variance) are derived entirely from recon's `FleetSummaryResponse`. No analytics DB, no Kafka on this path.
## Roles required
- **Cash Intelligence endpoint (`/api/analytics/cash-intelligence`): authenticated only — no role gate.** `SecurityConfig` permits `/actuator/health` + `/api/public/**`, everything else just needs a valid JWT.
- **Recon `/api/recon/fleet/summary`: authenticated only** as well.
- **Institution scoping is enforced in recon, by role:**
- `CUSTOMER` and `MSP_ADMIN` → **scoped** to the institutions in their JWT credentials. They see only ATMs that already have a recon cash position in their institution(s).
- Non-scoped roles (e.g. `BCOS_ADMIN`) → **full fleet**: recon merges the complete ATM list from hiveops-incident so every ATM appears, even those with no recon position yet.
- There is **no admin-only mutating action** on this tab — it is view-only. Cash data is created/edited in the **Recon** app (custodian replenishments), not here.
## First things to check when it misbehaves
**Tab is all zeros / both tables empty**
1. This is the signature of a **recon fetch failure**. `CashIntelligenceService.fetchFleetSummary()` swallows any error and returns an empty map → every count is 0 and both lists empty.
2. Check analytics logs for: `CashIntelligence: failed to fetch fleet summary from recon:` (Loki, service `hiveiq-analytics`). The message tail is the real cause.
3. Common causes: recon container down, wrong recon URL (see below), or the caller's JWT rejected by recon.
4. Confirm recon directly: `GET /api/recon/fleet/summary` with the same JWT should return `{totalAtms, balancedCycles, varianceCycles, openCycles, pendingConfirmations, atms[]}`.
**Wrong recon hostname (silent empty tab)**
- Analytics resolves recon via property `services.recon.url`, default **`http://hiveiq-recon:8091`**.
- The analytics `application.yml` `services:` block only wires `incident`, `journal`, `mgmt` — **recon is NOT mapped to an env var there.** So a `RECON_SERVICE_URL` env var does nothing; only Spring relaxed binding **`SERVICES_RECON_URL`** overrides it.
- If recon runs under a different container name than `hiveiq-recon:8091`, the tab silently empties. Verify container name and this property first.
**A customer/MSP admin sees fewer ATMs than expected**
- Expected: scoped users only see ATMs with an existing recon position in their institution. An ATM that has never had a replenishment recorded in Recon will not appear for them.
- A **device selection filter** in recon further narrows the list: if a non-empty saved selection exists for that scope, only selected ATMs are returned.
**An ATM shows in "Near Empty" at $0.00 that shouldn't**
- For full-fleet (admin) views, recon includes ATMs with **no cash position** (`null`). Analytics coerces a null `projectedBalanceCents` to `0`, so such ATMs land in Near Empty at `$0.00`. This is expected given the current code, not a data-loss bug — it means "no recon cycle has ever run for this ATM." (See uncertainties.)
**Counts don't match the tables**
- Counts come straight from recon (`balancedCycles`, etc.); the tables are re-filtered/re-sorted in analytics. Thresholds differ intentionally:
- **Near Empty** list = ATMs with projected balance **< $10,000** (`< 1,000,000` cents), lowest first.
- Frontend **red-row** highlight = **< $3,000** (`< 300,000` cents) — a subset, purely visual.
- **With Variance** list = any ATM with a non-zero variance, largest absolute first.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|--------|--------------|-----|
| All counts 0, both tables empty | recon down / unreachable / auth-rejected | Check analytics log warn line; restart/verify `hiveiq-recon`; test recon endpoint with the JWT |
| Empty tab only in prod, fine in dev | recon URL not resolving to the prod container | Set `SERVICES_RECON_URL` (not `RECON_SERVICE_URL`) or fix container name to `hiveiq-recon:8091` |
| Scoped user sees too few ATMs | no recon position for their ATMs, or device selection filter active | Confirm replenishments exist in Recon; clear the recon device selection |
| Ghost `$0.00` ATMs in Near Empty | fleet ATMs without a recon cycle (admin view) | Expected; onboard those ATMs into Recon or ignore |
| Variance/Balanced numbers stale | recon cycle not synced | Trigger recon `POST /api/recon/atm/{atmId}/sync`; cash positions update on replenishment/sync only |
## Cross-links
- Recon service internals: see `hiveops-recon/CLAUDE.md`.
- Analytics service overview: [technical.analytics].
- [platform.service-ownership] — cash reconciliation domain is owned by **recon**, surfaced by **analytics**.
backend/src/main/resources/content/analytics/fleet-health/architect.md
+81
View File
@@ -0,0 +1,81 @@
---
module: analytics.fleet-health
title: Fleet Health
tab: Architect
order: 30
audience: internal
---
> **How it's built.** Fleet Health is a **synchronous fan-out aggregation** — not a snapshot/Kafka read. Grounded in `FleetHealthController` + `FleetHealthService` (hiveops-analytics) and the frontend `FleetHealthTab.svelte`. Cross-link [platform.data-architecture], [platform.kafka], [platform.service-ownership].
## Services involved
| Service | Role in this feature |
|---------|----------------------|
| **hiveops-analytics** (owner, port 8089) | Serves `GET /api/analytics/fleet-health`; orchestrates + aggregates. Holds **no state** for this tab. |
| **hiveops-incident** | Source of ATM connectivity summary, recent activity events, and open incidents. |
| **hiveops-journal** (port 8087) | Source of open journal gaps. |
Cross-link [platform.service-ownership]: analytics owns the *aggregation shape*; incident/journal own the underlying records. Analytics reads no incident/journal DB tables directly — only their HTTP APIs.
## Data flow (request-time, no Kafka)
```
Browser (FleetHealthTab.svelte)
└─ GET /api/analytics/fleet-health (Bearer JWT)
└─ FleetHealthService.buildFleetHealth(callerToken) [forwards the SAME JWT]
├─ incidentClient GET /api/atms/summary → {total,connected,disconnected,neverConnected}
├─ incidentClient GET /api/journal-events/activity?hoursBack=24 → [{atmId,eventType,...}]
├─ incidentClient GET /api/incidents/status/open → [ ...open incidents ]
└─ journalClient GET /api/journal/gaps?status=OPEN → [{atmId,...}]
└─ aggregate → FleetHealthResponse (JSON)
```
- **No executor→Kafka→record-store pattern here.** That pattern belongs to the analytics *snapshot* pipeline (`SnapshotCollectionService` → `analytics.snapshot-ready` → `analytics_snapshots`), which powers other tabs (Overview/Trends), **not** Fleet Health. This tab bypasses the DB entirely. Cross-link [platform.kafka].
- Each downstream call is independently try/catch-wrapped; a failure logs a warning and substitutes an empty result, so the response degrades gracefully to partial/zero data.
## Aggregation logic (in `FleetHealthService`)
- **Connectivity**: pulls `total/connected/disconnected/neverConnected` from `/api/atms/summary`; `connectivityPct = connected*100.0/total` (0 when total=0).
- **Event Breakdown**: activity events (last **24h**) grouped by category via `classifyEventType(eventType)`. Category map is **seeded** with `HARDWARE, CASH, NETWORK, POWER, AUTH, SYSTEM, OTHER` (all 0), then incremented.
- Notable: there is **no `OPERATIONS` category in the backend** — the frontend `categoryColors` defines one, but `classifyEventType` never emits it; unmatched types fall to `OTHER`.
- **Top Problematic ATMs**: groups the same 24h events by `atmId`, counts, sorts desc, `limit(5)` → `ProblematicAtm{atmId,eventCount}`.
- **Open Incidents**: `openIncidents.size()` of `/api/incidents/status/open`.
- **ATMs with Journal Gaps**: distinct `atmId` count from `/api/journal/gaps?status=OPEN`.
### Event-type → category mapping (source of truth: `classifyEventType`)
| Category | Event types |
|----------|-------------|
| POWER | `POWER_OFF` |
| HARDWARE | `CARD_READER_FAIL`, `DISPENSER_JAM`, `SELF_TEST_FAILED`, `APPLICATION_ERROR_CARD`, `APPLICATION_ERROR_DISPENSER`, `APPLICATION_ERROR_ENCRYPTION`, `APPLICATION_ERROR_PRINTER`, `APPLICATION_ERROR_PINPAD`, `MIXED_MEDIA_ERROR`, `CASH_ACCEPTOR_ERROR`, `ENCRYPTOR_PINPAD_ERROR` |
| CASH | `CASSETTE_LOW`, `CASSETTE_EMPTY`, `CASSETTE_ABNORMAL` |
| NETWORK | `NETWORK_DISCONNECTED`, `APPLICATION_ERROR_NETWORK` |
| AUTH | `AUTHENTICATION_FAILURE` |
| SYSTEM | `ATM_OUT_OF_SERVICE`, `APPLICATION_ERROR`, `APPLICATION_ERROR_JOURNAL`, `AGENT_LOG_ERROR`, `EJOURNAL_DB_WARNING`, `EJOURNAL_DB_CRITICAL` |
| OTHER | anything unmatched / null |
## DB tables read/written
- **This feature: none.** No reads or writes to `hiveiq_analytics`. (Snapshot tables `analytics_snapshots` / `ai_insights` exist for other tabs — see [technical.analytics].)
- Underlying records live in incident and journal databases and are reached only via HTTP.
## Key API endpoints
| Method | Path | Service | Used for |
|--------|------|---------|----------|
| GET | `/api/analytics/fleet-health` | analytics | The tab's single data call |
| GET | `/api/atms/summary` | incident | Connectivity donut |
| GET | `/api/journal-events/activity?hoursBack=24` | incident | Event breakdown + top problematic |
| GET | `/api/incidents/status/open` | incident | Open incidents count |
| GET | `/api/journal/gaps?status=OPEN` | journal | ATMs with journal gaps |
## Frontend
- `FleetHealthTab.svelte` renders four cards (Connectivity donut, Event Breakdown bars, Top Problematic ATMs, Alerts & Gaps) from `FleetHealthData`.
- API base resolves from `window.__APP_CONFIG__.apiUrl` (fallback `http://localhost:8089/api/analytics`); call is `analyticsAPI.getFleetHealth()` → `GET {base}/fleet-health`.
- `FleetHealthData.topProblematicAtms` carries a `deviceId` "Phase 2 alias" alongside `atmId` — part of the ongoing ATM→device rename. Cross-link [platform.service-ownership].
## Auth
- JWT-authenticated; **no role restriction** (`SecurityConfig`: `.anyRequest().authenticated()`). The caller's token is forwarded verbatim to incident/journal (`WebClientConfig` clients, no service account). Cross-link [platform.auth].
backend/src/main/resources/content/analytics/fleet-health/claude.md
+61
View File
@@ -0,0 +1,61 @@
---
module: analytics.fleet-health
title: Fleet Health
tab: Claude
order: 40
audience: internal
---
> Act-without-guessing reference. All facts from `FleetHealthController`/`FleetHealthService`/`SecurityConfig` (hiveops-analytics) + `FleetHealthTab.svelte`/`api.ts`.
## Owner
- **hiveops-analytics** owns `GET /api/analytics/fleet-health` (port **8089**, DB `hiveiq_analytics` but **not used by this endpoint**).
- Live synchronous aggregation. No snapshot, no cache, no Kafka, no DB read/write for this tab.
## Endpoints
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/fleet-health` | analytics | JWT, any authenticated role |
| GET | `/api/atms/summary` | incident | JWT (forwarded) |
| GET | `/api/journal-events/activity?hoursBack=24` | incident | JWT (forwarded) |
| GET | `/api/incidents/status/open` | incident | JWT (forwarded) |
| GET | `/api/journal/gaps?status=OPEN` | journal | JWT (forwarded) |
Frontend call: `analyticsAPI.getFleetHealth()` → `GET {apiBase}/fleet-health`; `apiBase` = `window.__APP_CONFIG__.apiUrl` || `http://localhost:8089/api/analytics`.
## Roles
- Endpoint gate: `.anyRequest().authenticated()` — **no `hasAnyRole`**. Not admin-only.
- `permitAll()`: only `/actuator/health`, `/api/public/**`.
- Role-gated in analytics = **only** `POST /api/analytics/insights` (`hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`) — a different feature.
## Tables / topics
- Tables read/written by this feature: **none**.
- Kafka topics in this feature's path: **none**.
- (Context only, other tabs: DB `analytics_snapshots`, `ai_insights`; topic `analytics.snapshot-ready`.)
## Response fields (`FleetHealthResponse`)
- `totalAtms, connectedAtms, disconnectedAtms, neverConnectedAtms, connectivityPct`
- `openIncidents` (= size of open-incidents list)
- `eventsByCategory: Map<String,Long>` (24h window, always seeded)
- `topProblematicAtms: [{atmId, eventCount}]` (top 5 by 24h event count; `deviceId` alias in frontend)
- `atmsWithOpenGaps` (distinct atmId count, status=OPEN)
## Categories (from `classifyEventType`)
- Emitted set: `HARDWARE, CASH, NETWORK, POWER, AUTH, SYSTEM, OTHER`.
- `POWER_OFF`→POWER · cassette*→CASH · network*→NETWORK · `AUTHENTICATION_FAILURE`→AUTH · reader/dispenser/app-error-hw→HARDWARE · out-of-service/app-error/ejournal-db→SYSTEM · unmatched→OTHER.
## Gotchas
- Downstream failures are **swallowed** (try/catch → log `FleetHealth: failed to fetch …` + empty). **200 with all-zeros = broken dependency, not healthy fleet.**
- Event breakdown + top problematic are a **fixed 24h window** (`hoursBack=24`, hardcoded). Not configurable via request.
- `eventsByCategory` is **always seeded with 7 keys at 0**, so the map is never empty → frontend "No events recorded" empty-state effectively never triggers.
- **No `OPERATIONS` backend category** despite the frontend color map / customer copy mentioning "Operations"; such events land in `OTHER`.
- Journal gaps come from **hiveops-journal** (`/api/journal/gaps`), everything else from **hiveops-incident**.
- Downstream hosts are env-driven: `INCIDENT_SERVICE_URL` (def `http://hiveops-incident-backend:8080`), `JOURNAL_SERVICE_URL` (def `http://hiveops-journal:8087`).
## Do NOT
- Do NOT claim this endpoint needs MSP_ADMIN/BCOS_ADMIN — it only needs a valid JWT.
- Do NOT say Fleet Health reads snapshots / Kafka / the analytics DB — it calls incident+journal live.
- Do NOT add an incident/gap/device write here — analytics owns aggregation only; incident/journal own the records ([platform.service-ownership]).
- Do NOT treat a 200 all-zeros response as "fleet is fine" — check analytics logs first.
- Do NOT assume events older than 24h are counted — they are not.
- Do NOT invent an `OPERATIONS` category in backend logic; unmatched → `OTHER`.
backend/src/main/resources/content/analytics/fleet-health/internal.md
+56
View File
@@ -0,0 +1,56 @@
---
module: analytics.fleet-health
title: Fleet Health
tab: Internal
order: 20
audience: internal
---
> **Ops / Support / Admin view.** How to keep the Fleet Health screen honest and what to check when it lies. Grounded in `hiveops-analytics` source (`FleetHealthController`, `FleetHealthService`, `SecurityConfig`).
## What actually serves this screen
- Endpoint: **`GET /api/analytics/fleet-health`** on **hiveops-analytics** (port **8089**).
- It is **computed live on every request** — no analytics DB, no cache, no snapshot for this tab. Each load fans out to hiveops-incident and hiveops-journal and aggregates in-memory.
- The user's **JWT is forwarded** to those downstream services. If the caller's token can't see a device/incident, neither can this screen. Institution scoping is inherited from downstream, not enforced here.
## Roles required
- **Any authenticated JWT** can read fleet-health. `SecurityConfig` only has `permitAll()` on `/actuator/health` + `/api/public/**`; everything else is `.anyRequest().authenticated()`.
- There is **no `@PreAuthorize` / role gate** on this endpoint — it is *not* admin-only. Do not tell a customer "you need MSP_ADMIN to see it." (Role gates in analytics apply only to `POST /api/analytics/insights`.)
## First things to check when it misbehaves
1. **Whole screen empty / zeros everywhere** → the caller's JWT likely can't reach downstream, or a downstream service is down. Every fetch in `FleetHealthService` is wrapped in try/catch that **logs a warning and returns empty** — so a dead dependency produces silent zeros, not an error. Check analytics logs for:
- `FleetHealth: failed to fetch ATM summary`
- `FleetHealth: failed to fetch activity events`
- `FleetHealth: failed to fetch open incidents`
- `FleetHealth: failed to fetch open gaps`
2. **Connectivity donut wrong / total = 0** → `GET /api/atms/summary` on hiveops-incident is failing or returning `{}`. Curl it directly with the same JWT.
3. **Event Breakdown / Top Problematic empty** → `GET /api/journal-events/activity?hoursBack=24` on hiveops-incident. Note the **fixed 24-hour window** — anything older than 24h never appears here, by design.
4. **"ATMs with Journal Gaps" = 0 but you expect gaps** → `GET /api/journal/gaps?status=OPEN` on hiveops-journal. Only gaps with status `OPEN` are counted; distinct `atmId` values.
5. **Open Incidents count off** → `GET /api/incidents/status/open` on hiveops-incident. The number shown is simply `openIncidents.size()` of that list.
## Common failure modes → fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| All cards zero, no error toast | A downstream call threw and was swallowed | Check analytics logs for the four `FleetHealth: failed to fetch…` warnings; restart/repair the named service |
| Donut total looks stale | incident `/api/atms/summary` cached/stale upstream | Investigate hiveops-incident, not analytics — analytics holds no state here |
| Event breakdown shows categories with **0** events | Backend seeds HARDWARE/CASH/NETWORK/POWER/AUTH/SYSTEM/OTHER at 0 (see architect tab) | Expected; not a data loss. The "No events recorded" empty-state effectively never fires |
| An event type shows under **OTHER** unexpectedly | `classifyEventType` switch has no case for it | Add the mapping in `FleetHealthService.classifyEventType` (see architect tab) |
| 401/403 on the endpoint | Missing/expired JWT | Re-auth; confirm `Authorization: Bearer` reaches analytics through NGINX |
| Downstream URL wrong after infra change | `INCIDENT_SERVICE_URL` / `JOURNAL_SERVICE_URL` env misconfigured | Fix env on the analytics container; defaults are `http://hiveops-incident-backend:8080` and `http://hiveops-journal:8087` |
## Admin actions
- There are **no admin mutation actions** on this screen. It is read-only. Nothing to "reset," acknowledge, or re-run — a browser refresh re-computes from live sources.
- To change what counts as Hardware/Cash/Network/etc., that is a **code change** in `classifyEventType`, not a config toggle.
## Smoke test
```bash
curl -s https://analytics.bcos.cloud/api/analytics/fleet-health \
-H "Authorization: Bearer $JWT" | jq '{totalAtms,connectedAtms,openIncidents,atmsWithOpenGaps,eventsByCategory}'
```
Expect HTTP 200 and a parseable body. Zeros with 200 = a swallowed downstream failure, not success — check logs.
backend/src/main/resources/content/analytics/incidents/architect.md
+72
View File
@@ -0,0 +1,72 @@
---
module: analytics.incidents
title: Incident Trends
tab: Architect
order: 30
audience: internal
---
> How the **Incident Trends** tab is built. Grounded in `hiveops-analytics` + `hiveops-incident` source, 2026-07-01. Cross-link [platform.data-architecture], [platform.kafka], [platform.service-ownership].
## Services involved
| Service | Role for this feature |
|---------|-----------------------|
| `hiveops-analytics` (port 8089) | Owns the tab endpoint; aggregates/shapes the 4 charts in-memory. No DB write for this feature. |
| `hiveops-incident` (port 8080) | Owns the data (`journal_events`) and applies institution scoping. |
| Frontend (`hiveops-analytics/frontend`) | Svelte SPA renders `IncidentTrendsTab.svelte` from one JSON payload. |
Ownership boundary (see [platform.service-ownership]): analytics **computes**, incident **owns the journal-event data**. Analytics holds no incident/journal tables of its own for this tab.
## Data flow (synchronous, per request)
```
IncidentTrendsTab.svelte
└─ analyticsAPI.getIncidentTrends()
└─ GET {apiBase}/incident-trends → IncidentTrendsController (analytics)
└─ IncidentTrendsService.buildIncidentTrends(token)
└─ incidentWebClient.get()
GET /api/journal-events/activity?hoursBack=24 → JournalEventController (incident)
└─ JournalEventService.getActivityEvents(24, InstitutionContext.resolveScope())
└─ JournalEventRepository.findActivityEvents / findActivityEventsByInstitutionIn
└─ SELECT from journal_events (joined to atms)
```
- **Auth propagation:** the caller's `Authorization: Bearer <jwt>` is extracted in the controller and re-sent on the WebClient call. No service account; downstream scoping keys off that token.
- **`incidentWebClient`** is the `@Qualifier("incidentWebClient")` bean; base URL from `INCIDENT_SERVICE_URL` (default `http://hiveops-incident-backend:8080`).
## No Kafka / no snapshot store on this path
This is the key architectural fact: **Incident Trends bypasses the analytics executor→Kafka→record-store pattern entirely.** It does *not* read `analytics_snapshots` or `ai_insights`, and it does *not* rely on the `IncidentEventConsumer` (`hiveops.incidents.created/updated/resolved`) or `JournalFileParsedConsumer` (`journal.file-parsed`) that feed the snapshot scheduler. Those exist in the service for the dashboard **Overview** snapshots, but the trends tab is a fresh live pull each time. Cross-link [platform.kafka] for the topic catalog used elsewhere in analytics.
## DB tables
| Table | Service | Access |
|-------|---------|--------|
| `journal_events` | hiveops-incident (`hiveiq_incident`) | **read** — `findActivityEvents(ACTIVITY_TYPES, since, institution)` filtered to a curated `ACTIVITY_TYPES` set and `event_time >= now-24h` |
| `atms` | hiveops-incident | **read** — joined for `atmName`/`location`, and drives institution scoping |
Analytics writes nothing for this tab. (`analytics_snapshots`, `ai_insights` in `hiveiq_analytics` are unused here.) See [platform.data-architecture].
## In-memory aggregation (analytics)
From the single `List<JournalEventDTO>`:
- **dailyCounts** — bucketed by date; seeded with **today only** (single bucket).
- **byCategory** — `FleetHealthService.classifyEventType(eventType)` → `HARDWARE | CASH | NETWORK | POWER | AUTH | SYSTEM | OTHER`.
- **recurringFailures** — group by `atmId|eventType` over last 24h, keep `count ≥ 3`, sort desc, top 20.
- **peakHours** — `int[7][24]`, Mon=row 0, by `eventTime` day-of-week × hour (UTC).
## Key API endpoints
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/incident-trends` | analytics | authenticated JWT (no role gate) |
| GET | `/api/journal-events/activity?hoursBack=24` | incident | authenticated JWT; institution-scoped |
## Config knobs
- `INCIDENT_SERVICE_URL` (analytics) — downstream base URL.
- `JWT_SECRET` — must match across services for the forwarded token to validate.
- `hoursBack` is **hard-coded to 24** in `IncidentTrendsService`; not env-configurable.
See the Internal tab for failure modes and the Overview tab for the customer-facing description.
backend/src/main/resources/content/analytics/incidents/claude.md
+60
View File
@@ -0,0 +1,60 @@
---
module: analytics.incidents
title: Incident Trends
tab: Claude
order: 40
audience: internal
---
> Terse agent reference. All facts confirmed in source 2026-07-01. `hiveops-ai` = "Adoons".
## Ownership
- **Endpoint owner:** `hiveops-analytics` (port 8089).
- **Data owner:** `hiveops-incident` (port 8080), table `journal_events`.
- This tab is a **live synchronous pull**, not a snapshot/Kafka read.
## Endpoints (METHOD + path)
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/incident-trends` | analytics | JWT, **no role gate** |
| GET | `/api/journal-events/activity?hoursBack=24` | incident | JWT, institution-scoped |
- Frontend call: `analyticsAPI.getIncidentTrends()` → `GET {apiBase}/incident-trends`.
- Analytics forwards caller `Bearer` token verbatim to incident. No service account.
## Tables
- `journal_events` (read) — `hiveiq_incident` DB.
- `atms` (read, join + institution scope) — `hiveiq_incident` DB.
- **Not used by this tab:** `analytics_snapshots`, `ai_insights` (`hiveiq_analytics`).
## Kafka
- **None on this path.** (`hiveops.incidents.*`, `journal.file-parsed`, `analytics.snapshot-ready` feed the snapshot tab, not this one.)
## Response shape (`IncidentTrendsResponse`)
- `dailyCounts: [{date, count}]` — seeded today-only.
- `byCategory: {HARDWARE,CASH,NETWORK,POWER,AUTH,SYSTEM,OTHER: n}`.
- `recurringFailures: [{atmId, eventType, count}]` — 24h, count≥3, top 20.
- `peakHours: int[7][24]` — Mon=0, UTC hour.
## Roles / scoping
- `CUSTOMER` / `MSP_ADMIN` → scoped to granted institution keys (downstream `InstitutionContext.resolveScope()`).
- `BCOS_ADMIN` / `ADMIN` / `QDS_ADMIN` / `USER` → unrestricted.
## Config
- `INCIDENT_SERVICE_URL` default `http://hiveops-incident-backend:8080`.
- `hoursBack` hard-coded = 24 (not env-tunable).
## Gotchas (confirmed)
- Window is **24h**, despite chart titled "30-DAY INCIDENT TREND".
- `dailyCounts` always length 1 → frontend renders "Not enough data" (needs ≥2).
- `recurringFailures` always empty: service reads `atmId` as String but DTO serializes it as numeric `Long` → skipped.
- No "OPERATIONS" category emitted; operational types → `OTHER`.
- Downstream failure is swallowed → whole tab silently empties (grep analytics log `IncidentTrends: failed to fetch activity events`).
## Do NOT
- Do NOT query `analytics_snapshots`/`ai_insights` for this tab — wrong source.
- Do NOT expect this tab to read Kafka — it's a per-request HTTP pull.
- Do NOT assume role-gating on `/api/analytics/incident-trends` — any authenticated JWT passes.
- Do NOT add incident/journal endpoints to analytics — analytics computes, incident owns the data ([platform.service-ownership]).
- Do NOT trust "0 recurring failures" as fleet health — it's the atmId-type bug.
- Do NOT hand analytics a service token expecting broader data — scope follows the caller's JWT.
backend/src/main/resources/content/analytics/incidents/internal.md
+55
View File
@@ -0,0 +1,55 @@
---
module: analytics.incidents
title: Incident Trends
tab: Internal
order: 20
audience: internal
---
> Ops/support view for the **Incident Trends** tab (analytics.incidents). Grounded in `hiveops-analytics` + `hiveops-incident` source, 2026-07-01.
## What actually powers this screen
The tab is **not** backed by the analytics snapshot store. It is a **live, synchronous pass-through**:
```
Browser → GET /api/analytics/incident-trends (hiveops-analytics)
→ GET /api/journal-events/activity?hoursBack=24 (hiveops-incident)
→ journal_events table
```
`IncidentTrendsService` forwards the **caller's own JWT** downstream and computes the 4 charts in-memory on every request. No caching, no DB write, no Kafka for this feature. If the incident service is down or slow, this tab is empty — nothing here reads analytics-local tables.
## Roles / access
- The endpoint has **no `@PreAuthorize`** — any authenticated JWT can call it. There is no MSP_ADMIN/BCOS_ADMIN gate on Incident Trends.
- **Data is institution-scoped downstream.** `hiveops-incident` calls `InstitutionContext.resolveScope()` on the forwarded token:
- `ROLE_CUSTOMER` / `ROLE_MSP_ADMIN` → scoped to their granted institution keys (sees only their ATMs).
- `BCOS_ADMIN` / `ADMIN` / `QDS_ADMIN` / `USER` → unrestricted (all institutions).
- So a support user seeing "less data than expected" is usually a **scoped role**, not a bug.
## First things to check when it misbehaves
1. **Whole tab empty / "No data" everywhere** → the downstream call failed. `IncidentTrendsService.fetchActivityEvents` swallows exceptions and returns an empty list, logging `IncidentTrends: failed to fetch activity events: ...`. Check `hiveops-analytics` logs (Loki, service VM) for that line, then check `hiveops-incident` health.
2. **Analytics can't reach incident** → verify `INCIDENT_SERVICE_URL` (default `http://hiveops-incident-backend:8080`). Wrong/stale hostname = every trends call empties out.
3. **401/403 from downstream** → the forwarded JWT lacks incident access, or is expired. The user's token is passed verbatim; there is no service account.
4. **Customer sees "Not enough data" on the line chart** → see Known quirks; this is expected with current code, not an outage.
5. **Only recent stuff shows** → the window is **hard-coded to the last 24 hours** (`hoursBack=24`), despite the "30-Day" chart title. Older events are simply not fetched.
## Known quirks (expected behavior of current build)
- **30-Day Incident Trend line chart always shows "Not enough data."** `dailyCounts` is seeded with **today only**, so it always has exactly one point; the frontend requires ≥2 points to draw a line. See uncertainties — this is a real defect, not a config issue.
- **Recurring Failures table is effectively always empty.** The service reads `atmId` as a String, but the incident DTO serializes `atmId` as a numeric `Long`, so every event is skipped. Do not tell users "no recurring failures = healthy fleet" from this tab. (real bug — noted for engineering.)
- **Event Category Breakdown never shows an "Operations" bar.** Operational event types (`OPERATOR_ACTION`, etc.) classify to **OTHER**, not OPERATIONS, even though the customer copy lists Operations. Categories the backend actually emits: `HARDWARE`, `CASH`, `NETWORK`, `POWER`, `AUTH`, `SYSTEM`, `OTHER`.
- **Peak Hours Heatmap and Category bars are the reliable panels** — they only depend on `eventTime`/`eventType` (both strings) and work as intended over the 24h window.
## Where to look
| Concern | Location |
|---------|----------|
| Trends endpoint | `hiveops-analytics` → `controller/IncidentTrendsController.java`, `service/IncidentTrendsService.java` |
| Upstream data | `hiveops-incident` → `controller/JournalEventController.java` (`/activity`), `service/JournalEventService.java#getActivityEvents` |
| Category mapping | `hiveops-analytics` → `service/FleetHealthService.classifyEventType` |
| Frontend | `hiveops-analytics/frontend/src/components/tabs/IncidentTrendsTab.svelte` |
See also [analytics.incidents] Overview (customer copy) and the Architect tab for data-flow detail.
backend/src/main/resources/content/analytics/transactions/architect.md
+76
View File
@@ -0,0 +1,76 @@
---
module: analytics.transactions
title: Transactions
tab: Architect
order: 30
audience: internal
---
> How the **Transactions** tab is built. It is a **synchronous pull-through aggregation**, not an event-driven or snapshot-backed view. Cross-link [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Services involved
| Service | Role for this feature |
|---------|----------------------|
| `hiveops-analytics-frontend` | Serves the dashboard SPA (`analytics.bcos.cloud`, port 5174). The Transactions tab = `JournalCoverageTab.svelte`. |
| `hiveops-analytics` (port 8089) | Aggregator. Exposes `GET /api/analytics/journal-coverage`, forwards the caller JWT to journal, reshapes 3 journal responses into one `JournalCoverageResponse`. **Owns no data for this tab.** |
| `hiveops-journal` (port 8087) | **Source of truth.** Owns `journal_transactions` and `journal_gaps` in DB `hiveiq_journal`. |
Ownership boundary: transaction + gap data belongs exclusively to **hiveops-journal**; analytics is a read-only consumer. See [platform.service-ownership].
## Data flow (request path)
```
Browser (Transactions tab)
→ GET /api/analytics/journal-coverage [hiveops-analytics : JournalCoverageController]
JournalCoverageService.buildJournalCoverage(callerToken)
via journalWebClient (JOURNAL_SERVICE_URL), forwarding "Bearer <callerToken>":
├─ GET /api/journal/transactions/summary?dateFrom=today&dateTo=today → {total, successful, failed}
├─ GET /api/journal/gaps?status=OPEN → [gaps...]
└─ GET /api/journal/transactions/top-atms?dateFrom=today&dateTo=today&limit=10 → [{atmId, transactionCount}]
← JournalCoverageResponse { dailyVolume[today], openGapsTotal, atmsWithGaps[≤20], parseSuccessRateToday, topAtmsByVolume }
```
Notes on the aggregation logic (`JournalCoverageService`):
- `dailyVolume` is built with **today only** (single call; the multi-bar chart therefore renders one bar).
- `parseSuccessRateToday = successful * 100.0 / total` (0 when `total == 0`).
- `atmsWithGaps` = open gaps grouped by `atmId`, counted, sorted desc, capped at 20 (UI slices to 10).
- `topAtmsByVolume` = journal's top-ATMs list (limit 10; journal hard-caps page size at 50).
## Kafka — NOT in this request path
This tab is a live synchronous pull; **no Kafka topic is read or written to serve `journal-coverage`.** For context on the surrounding platform (relevant to *other* analytics tabs, not this one):
- hiveops-analytics consumes `journal.file-parsed` (`JournalFileParsedConsumer`), `hiveops.incidents.created/updated/resolved` (`IncidentEventConsumer`) to drive **snapshots** (`analytics_snapshots`) and produces `analytics.snapshot-ready`.
- hiveops-journal uses `journal.parse-requests.{INSTITUTION}` (worker parse) and produces `journal.file-parsed`.
- The Transactions tab does **not** read snapshots (`analytics_snapshots`) — it queries journal live every load. See [platform.kafka].
## DB tables
Read (in `hiveiq_journal`, owned by hiveops-journal):
| Table | Used for | Via |
|-------|----------|-----|
| `journal_transactions` | today's total / APPROVED / DECLINED counts; top ATMs by tx count | `JournalTransactionRepository.countByDateRange...`, `findTopAtmsByTransactionCount` |
| `journal_gaps` | open gap count + gaps-per-ATM | `/api/journal/gaps?status=OPEN` |
Written: **none by this feature.** `journal_gaps` rows are created by journal's own scheduled **gap detection** job (daily 06:00 UTC): for each ATM, any date between its first journal and yesterday with no `journal_files` row becomes an OPEN gap row. `analytics_snapshots` / `ai_insights` in `hiveiq_analytics` are unrelated to this tab.
See [platform.data-architecture] for the hot/cold split (`journal_transactions` vs `journal_transactions_archive`, 90-day boundary) — the today-only queries here always hit the hot table.
## Key endpoints
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/journal-coverage` | analytics | authenticated JWT (no role gate) |
| GET | `/api/journal/transactions/summary?dateFrom=&dateTo=` | journal | JWT; institution-scoped |
| GET | `/api/journal/gaps?status=OPEN` | journal | `isAuthenticated()`; institution-scoped |
| GET | `/api/journal/transactions/top-atms?dateFrom=&dateTo=&limit=` | journal | `isAuthenticated()`; page size capped 50 |
| PUT | `/api/journal/gaps/{id}/request-fetch` | journal | `isAuthenticated()` — marks a gap `FETCH_REQUESTED` (agent re-upload) |
## Design characteristics
- **No executor→Kafka→record-store pattern here.** That pattern applies to snapshot/insight generation, not to this pull-through tab.
- **Auth is caller-forwarded**, not a service account: analytics extracts the `Authorization: Bearer` header and re-sends it to journal (`WebClientConfig` / `journalWebClient`). Downstream scope = caller scope.
- **Fault tolerance is silent degradation**: each downstream call is wrapped in try/catch that logs a warning and returns empty on failure, so the tab renders zeros rather than an error.
- **Caching**: journal's summary is `@Cacheable("txSummary")` keyed `institutions:dateFrom:dateTo`; gaps and top-ATMs are uncached.
backend/src/main/resources/content/analytics/transactions/claude.md
+72
View File
@@ -0,0 +1,72 @@
---
module: analytics.transactions
title: Transactions
tab: Claude
order: 40
audience: internal
---
> Act-without-guessing reference. Transactions tab = analytics UI tab `id: 'journal', label: 'Transactions'` → `JournalCoverageTab.svelte` → `GET /api/analytics/journal-coverage`. Data source of truth is **hiveops-journal**, not analytics.
## Ownership
- **hiveops-journal** OWNS the data (`journal_transactions`, `journal_gaps`, DB `hiveiq_journal`).
- **hiveops-analytics** (port 8089) is a read-only aggregator/proxy; forwards caller JWT to journal.
- Frontend: `hiveops-analytics/frontend`, `analytics.bcos.cloud` (port 5174).
## Endpoints (METHOD path → service)
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/analytics/journal-coverage` | analytics | authenticated JWT, **no role gate** |
| GET | `/api/journal/transactions/summary?dateFrom=&dateTo=` | journal | JWT (no method-level `@PreAuthorize`; filter chain enforces) |
| GET | `/api/journal/gaps?status=OPEN` | journal | `@PreAuthorize("isAuthenticated()")` |
| GET | `/api/journal/transactions/top-atms?dateFrom=&dateTo=&limit=10` | journal | `@PreAuthorize("isAuthenticated()")`, size cap 50 |
| PUT | `/api/journal/gaps/{id}/request-fetch` | journal | `isAuthenticated()` (marks gap `FETCH_REQUESTED`) |
- Analytics frontend `apiBase` default (dev): `http://localhost:8089/api/analytics`; prod from `window.__APP_CONFIG__.apiUrl`. NGINX path `/api/analytics/`.
- Analytics `journalWebClient` base = `JOURNAL_SERVICE_URL` (default `http://hiveops-journal:8087`).
## Tables (DB `hiveiq_journal`)
| Table | Use |
|-------|-----|
| `journal_transactions` | total / APPROVED / DECLINED counts, top-ATMs |
| `journal_gaps` | open gaps (status `OPEN`) |
- No table is WRITTEN to serve this tab.
- `analytics_snapshots` / `ai_insights` (DB `hiveiq_analytics`) are NOT used by this tab.
## Kafka
- **None in this request path.** `journal-coverage` is a synchronous WebClient pull.
- (Context only, other tabs) analytics consumes `journal.file-parsed`, `hiveops.incidents.{created,updated,resolved}`; produces `analytics.snapshot-ready`. Journal produces `journal.file-parsed`, consumes `journal.parse-requests.{INSTITUTION}`.
## Response shape (`JournalCoverageResponse`)
- `dailyVolume: [{date, total, successful, failed}]` — **today only, single element**.
- `parseSuccessRateToday = successful*100/total` (0 if total 0).
- `openGapsTotal`, `atmsWithGaps: [{atmId, gapCount}]` (≤20, UI shows 10).
- `topAtmsByVolume: [{atmId, transactionCount}]` (≤10).
## Field semantics (critical)
- journal `successful` = `COUNT(host_result='APPROVED')`; `failed` = `COUNT(host_result='DECLINED')`; `total` = all rows.
- Therefore "Parse Success Rate" and the "Successful/Failed" bars = transaction **approval** stats, NOT journal-parse health.
- `transactions/summary` is `@Cacheable("txSummary")` key `institutions:dateFrom:dateTo` → today's numbers can be stale.
- All scoped by `InstitutionContext.resolveScope()` in journal — results are per caller's institution.
## Gotchas
- Downstream failures are caught and return empty → tab shows **0 / "No data"** with no user error. Grep analytics logs: `JournalCoverage: failed to fetch`.
- Everything except open-gaps is `LocalDate.now()` (today) — early-day emptiness is expected.
- Gap rows only created by journal's daily 06:00 UTC gap-detection job.
## Do NOT
- Do NOT claim analytics stores transaction/gap data — it does not; query journal.
- Do NOT treat "Parse Success Rate" as an ingest/parse metric — it is approval rate.
- Do NOT expect multi-day history on this tab — `dailyVolume` is today-only.
- Do NOT assume `MSP_ADMIN`/`BCOS_ADMIN` is required — `journal-coverage` is any-authenticated.
- Do NOT look for Kafka in the read path — it is a synchronous pull.
- Do NOT add write endpoints or resolve gaps from analytics — gap fetch is `PUT /api/journal/gaps/{id}/request-fetch` on journal.
backend/src/main/resources/content/analytics/transactions/internal.md
+50
View File
@@ -0,0 +1,50 @@
---
module: analytics.transactions
title: Transactions
tab: Internal
order: 20
audience: internal
---
> Ops / support / admin view for the **Transactions** tab (analytics dashboard, tab id `journal`, label "Transactions"). This tab is a **read-only aggregation** — there are no write actions or admin buttons on it.
## What this tab actually is
- UI: `hiveops-analytics/frontend` → `App.svelte` tab `{ id: 'journal', label: 'Transactions' }` → renders `JournalCoverageTab.svelte`.
- Data: one call, `analyticsAPI.getJournalCoverage()` → `GET /api/analytics/journal-coverage` on **hiveops-analytics** (port 8089).
- Source of truth is **hiveops-journal**, not analytics. Analytics is a pass-through aggregator that forwards the caller's JWT to journal and reshapes the result.
## Roles required
- **`GET /api/analytics/journal-coverage`** — authenticated JWT only. No role gate: analytics `SecurityConfig` is `.requestMatchers("/actuator/health","/api/public/**").permitAll().anyRequest().authenticated()`. There is **no** `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` on this endpoint.
- Downstream journal endpoints (`/api/journal/transactions/summary`, `/gaps`, `/transactions/top-atms`) require an authenticated JWT (`@PreAuthorize("isAuthenticated()")`; the summary endpoint has no method annotation and relies on the journal filter chain).
- Results are **institution-scoped** in journal via `InstitutionContext.resolveScope()` — a user sees only their own institution's transactions/gaps. MSP/BCOS admins with multi-institution scope see aggregated counts.
## First things to check when it misbehaves
1. **All tiles show 0 / empty ("No data available", "No ATMs with open gaps").**
- `JournalCoverageService` **swallows downstream errors** and returns empty maps/lists (→ zeros). There is no user-facing error. Check analytics logs for the warnings it logs instead:
- `JournalCoverage: failed to fetch summary for {date}: ...`
- `JournalCoverage: failed to fetch open gaps: ...`
- `JournalCoverage: failed to fetch top ATMs: ...`
- Loki (services VM, `localhost:3100`) or `docker compose logs hiveiq-analytics`.
2. **Is hiveops-journal up and reachable?** Analytics calls it via `journalWebClient` (`JOURNAL_SERVICE_URL`, default `http://hiveops-journal:8087`). If journal is down or the URL is wrong, every tile zeros out.
3. **JWT / scope.** Analytics forwards the *caller's* Bearer token to journal — there is no service account. If the user's token is expired or lacks institution scope, journal returns empty/filtered data and the tab looks broken for that user only.
4. **No data for today is normal early in the day.** Everything on this tab is **today-only** (`LocalDate.now()`): daily volume, parse rate, and top-ATMs all use `dateFrom=dateTo=today`. Open gaps is the only cumulative number. Empty early-morning numbers usually mean "no journals ingested yet," not a fault.
5. **Numbers look stale.** Journal caches the summary: `JournalSummaryService.getTransactionSummary` is `@Cacheable("txSummary")` keyed on `institutions:dateFrom:dateTo`. Today's total/approved/declined can lag until the cache entry turns over.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| Whole tab zeros, warnings in analytics log | journal down / wrong `JOURNAL_SERVICE_URL` / journal 5xx | Restore journal, verify env, re-check `docker compose ps hiveiq-journal` |
| Tab zeros for one user only | that user's JWT expired or institution scope empty | Re-login; confirm the account has an institution and journal data exists for it |
| "Today's Journal Volume" shows a single bar | **by design** — service builds `dailyVolume` with today only ("Single call — today only") | Not a bug; multi-day history is not wired for this tab |
| Parse Success Rate looks like an approval rate | it *is* — see gotcha below | Interpret accordingly |
| Open gaps stuck high | gap detection runs daily 06:00 UTC; gaps clear only when the missing `journal_files` row appears | Use per-ATM "Request Fetch" (`PUT /api/journal/gaps/{id}/request-fetch`) to prompt the agent to re-upload |
## Gotchas that bite support
- **"Parse Success Rate" is not a parse rate.** It is computed as `successful / total * 100`, where journal defines `successful = COUNT(host_result='APPROVED')` and `total = all transactions`. So it reflects **transaction approval**, not journal-parse health. The green/red "Successful/Failed" bars are likewise APPROVED vs DECLINED host results — declines are normal customer behavior, not ingest failures. (See uncertainties — this labeling is misleading.)
- **"Other" transactions** (visible on the separate overview `TransactionCard`, not this tab) = `total − approved − declined` (PIN verifications, timeouts, etc.).
- Analytics never writes anything for this tab. There is nothing to reset in `hiveiq_analytics` for a Transactions problem — always look at **hiveops-journal**.
backend/src/main/resources/content/analytics/uptime/architect.md
+99
View File
@@ -0,0 +1,99 @@
---
module: analytics.uptime
title: Uptime
tab: Architect
order: 30
audience: internal
---
> **How Uptime is built.** Grounded in `hiveops-analytics` source (2026-07-01). Cross-link [platform.data-architecture], [platform.kafka], [platform.service-ownership].
## Shape of the feature
Uptime is a **read-over-snapshots + one live overlay** feature. `hiveops-analytics` owns the endpoint and the persisted history; the "currently down" list is a live pass-through to the incident service. Nothing about Uptime is computed on the ATM or by any other frontend.
```
UptimeTab.svelte ──GET /api/analytics/uptime──▶ UptimeController ──▶ UptimeService
│
┌──────────────────────────────────────────┤
▼ ▼
analytics_snapshots (DB, today) incident: GET /api/atms/paginated
→ NOW / 7d / 30d / trend ?agentStatus=DISCONNECTED,NEVER_CONNECTED
→ currentlyDown + neverConnected + table
```
## Services involved
| Service | Role for Uptime |
|---|---|
| **hiveops-analytics** (8089) | Owns `GET /api/analytics/uptime`, owns `analytics_snapshots`, runs the snapshot collector. |
| **hiveops-incident** (8080) | Serves the **live** down/never-connected ATM list via `/api/atms/paginated`. Token forwarded from caller. |
| **hiveops-devices** (8096) | Source of ATM health **counts** written into snapshots (via `/api/atms/summary`). |
| **hiveops-fleet** (8097) | Task counts in the same snapshot (not shown on Uptime, collected together). |
| **hiveops-journal** (8087) | Transaction counts in the same snapshot; `journal.file-parsed` also triggers snapshot collection. |
Downstream base URLs are env-configured in `WebClientConfig` (`services.incident.url`, `services.devices.url`, `services.fleet.url`, `services.journal.url`, `services.mgmt.url`) with in-cluster defaults.
## Data flow — write path (snapshots)
`SnapshotCollectionService.collect()` is the executor. It runs on two triggers:
- **Scheduled fallback:** `@Scheduled(cron = "0 0 */6 * * *")` — every 6 hours.
- **Event-driven (real-time):** Kafka consumers call `collect()` on each relevant event.
On each run it:
1. Mints an **internal service JWT** (`InternalTokenGenerator` — `sub=0`, `role=ADMIN`, 300 s TTL).
2. Calls `FleetOverviewService.buildOverview(token, "internal")`, which fans out to devices `/api/atms/summary`, incident `/api/incident-management/stats/summary`, fleet `/api/fleet/tasks/stats`, journal `/api/journal/transactions/summary` (each failure is caught → zeros for that section; `@Cacheable("fleetOverview")`).
3. **Upserts** one row into `analytics_snapshots` keyed by `(snapshot_date, snapshot_hour)` — same hour replaces, else insert.
4. Emits `analytics.snapshot-ready` via `SnapshotEventProducer`.
This is the platform **executor → Kafka → record-store** pattern: events fan in to trigger a collect, the collect persists a record, and a "ready" event fans back out. See [platform.kafka].
## Data flow — read path (the Uptime endpoint)
`UptimeService.buildUptime(callerToken)`:
1. Loads `analytics_snapshots` for **today** (`findBySnapshotDateBetween...`, `from = to = today`).
2. `currentUptimePct` = latest row's `connectedAtms/totalAtms`.
3. `last24Hours` = last 24 snapshot rows (hourly points).
4. `dailyTrend` = today's rows averaged into a single daily point.
5. `sevenDayAvgPct` and `thirtyDayAvgPct` = `average(dailyTrend)` (see Caveats — both are today's average today).
6. Overlays the **live** incident call `/api/atms/paginated?size=500&agentStatus=DISCONNECTED,NEVER_CONNECTED` (caller's token) for `currentlyDownCount`, `neverConnectedCount`, and the `atmsCurrentlyDown` table (sorted longest-down-first, capped at 50).
## Kafka
Cross-link [platform.kafka].
| Direction | Topic | Class / group |
|---|---|---|
| Consume | `hiveops.incidents.created`, `hiveops.incidents.updated`, `hiveops.incidents.resolved` | `IncidentEventConsumer`, group `hiveops-analytics-incidents` → `collect()` |
| Consume | `journal.file-parsed` | `JournalFileParsedConsumer`, group `hiveops-analytics-journal` → `collect()` |
| Produce | `analytics.snapshot-ready` | `SnapshotEventProducer` (`SnapshotReadyEvent{snapshotDate, snapshotHour}`) |
Uptime freshness therefore degrades gracefully: if incident/journal events stop flowing, the NOW/trend data falls back to the 6-hour cron cadence; the live down-list is unaffected (it doesn't read snapshots).
## DB tables
Cross-link [platform.data-architecture].
| Table | R/W for Uptime | Notes |
|---|---|---|
| `analytics_snapshots` | **Read** by `UptimeService`; **written** by `SnapshotCollectionService` | Unique `(snapshot_date, snapshot_hour)`; `snapshot_hour` INT (V3 fix). Fields used: `total_atms`, `connected_atms`, `disconnected_atms`, `never_connected_atms`. |
No Uptime-specific table; it reuses the shared fleet snapshot. `ai_insights` is unrelated to Uptime.
## Key endpoints
| Method | Path | Service | Purpose |
|---|---|---|---|
| GET | `/api/analytics/uptime` | analytics | The Uptime screen payload |
| GET | `/api/atms/paginated?size=500&agentStatus=DISCONNECTED,NEVER_CONNECTED` | incident | Live down/never-connected list |
| GET | `/api/atms/summary` | devices | ATM counts written into snapshots |
| GET | `/api/incident-management/stats/summary` | incident | Incident counts in snapshot |
| GET | `/api/fleet/tasks/stats` | fleet | Task counts in snapshot |
| GET | `/api/journal/transactions/summary?dateFrom=&dateTo=` | journal | Transaction counts in snapshot |
## Caveats / architectural notes
- **Snapshot vs live split:** NOW/7d/30d/trend are snapshot-derived (≤6 h stale possible); the down-list is live. Intentional but can look inconsistent.
- **Today-only window:** the read path only loads today's snapshots, so 7-day and 30-day averages resolve to the same value and the trend chart has a single point. See [analytics.uptime] Internal tab and the module's overview for the intended (advertised) behaviour vs. what ships today.
- **Cross-service ownership drift:** snapshot ATM **counts** come from devices, but the live down-list still calls **incident** `/api/atms/paginated`. If ATM detail ownership fully moves to devices, this call is the thing to re-point. See [platform.service-ownership].
backend/src/main/resources/content/analytics/uptime/claude.md
+68
View File
@@ -0,0 +1,68 @@
---
module: analytics.uptime
title: Uptime
tab: Claude
order: 40
audience: internal
---
> Terse agent reference for `analytics.uptime`. Only confirmed facts. Act, don't guess.
## Owner
- **Service:** `hiveops-analytics` — port **8089**, DB **`hiveiq_analytics`**.
- **Endpoint class:** `UptimeController` → `UptimeService`.
- **Frontend:** `hiveops-analytics/frontend/src/components/tabs/UptimeTab.svelte`, `api.getUptime()`.
## Endpoints (METHOD + path)
| Method | Path | Service | Auth |
|---|---|---|---|
| GET | `/api/analytics/uptime` | analytics | JWT (any authenticated; **no role gate**) |
| GET | `/api/atms/paginated?size=500&agentStatus=DISCONNECTED,NEVER_CONNECTED` | incident | caller's Bearer forwarded |
| GET | `/api/atms/summary` | devices | snapshot collector (internal token) |
| GET | `/api/incident-management/stats/summary` | incident | snapshot collector |
| GET | `/api/fleet/tasks/stats` | fleet | snapshot collector |
| GET | `/api/journal/transactions/summary?dateFrom=&dateTo=` | journal | snapshot collector |
- Frontend `apiBase` default: `http://localhost:8089/api/analytics`; prod via `window.__APP_CONFIG__.apiUrl`.
## Tables
| Table | DB | Access |
|---|---|---|
| `analytics_snapshots` | `hiveiq_analytics` | read by `UptimeService`, written by `SnapshotCollectionService`; unique `(snapshot_date, snapshot_hour)` |
## Kafka topics
| Direction | Topic | Group |
|---|---|---|
| consume | `hiveops.incidents.created` / `.updated` / `.resolved` | `hiveops-analytics-incidents` |
| consume | `journal.file-parsed` | `hiveops-analytics-journal` |
| produce | `analytics.snapshot-ready` | — |
- Any consumed event → `SnapshotCollectionService.collect()`. Cron fallback: `0 0 */6 * * *`.
## Roles
- `SecurityConfig`: `permitAll` on `/actuator/health` + `/api/public/**`; everything else `.anyRequest().authenticated()`.
- **No `@PreAuthorize`, no `hasAnyRole(...)` on Uptime.** MSP_ADMIN/BCOS_ADMIN not required to read.
- Snapshot collector uses `InternalTokenGenerator` (`sub=0`, `role=ADMIN`, 300s).
## Gotchas
- `UptimeService` queries **today only** (`from = to = LocalDate.now()`). Therefore `sevenDayAvgPct == thirtyDayAvgPct == today's avg`, and `dailyTrend` has **1 point** → frontend shows **"Not enough data"** (`trend.length < 2`). Not corruption — code limitation.
- **NOW/7d/30d/trend = snapshot data** (≤6 h stale). **CURRENTLY DOWN count/table = live incident call.** They can disagree.
- Down-list failure is swallowed (`catch → log.warn("failed to fetch down ATMs")`) → empty list, count 0. Check analytics logs, not an error response.
- Down-list still calls **incident** (`services.incident.url`, default `http://hiveops-incident-backend:8080`) even though ATM counts come from **devices**.
- Prod DB uses split `DB_HOST/DB_PORT/DB_NAME/DB_USERNAME/DB_PASSWORD`, **not** `DB_URL`.
- Snapshot upsert is per `(date, hour)` — a re-run in the same hour overwrites, not appends.
## Do NOT
- Do NOT assume a role check exists — Uptime is readable by any authenticated JWT.
- Do NOT expect the 30-day trend / distinct 7d vs 30d numbers to render — code only loads today.
- Do NOT look for Uptime data on the ATM, devices, or incident DB — history lives in `analytics_snapshots` (`hiveiq_analytics`).
- Do NOT trust the NOW ring as live truth; only the "Currently Down" list is live.
- Do NOT invent an admin/write endpoint for Uptime — it is read-only (`GET /api/analytics/uptime`).
- Do NOT confuse consumer groups: incidents = `hiveops-analytics-incidents`, journal = `hiveops-analytics-journal`.
backend/src/main/resources/content/analytics/uptime/internal.md
+66
View File
@@ -0,0 +1,66 @@
---
module: analytics.uptime
title: Uptime
tab: Internal
order: 20
audience: internal
---
> **Internal ops/support.** Grounded in `hiveops-analytics` source (2026-07-01). The Uptime screen is **read-only** — there are no admin buttons, no writes, no fleet actions. When it looks wrong, you are almost always debugging **stale snapshots** or a **failing downstream call**, not user input.
## Who owns it
- **Service:** `hiveops-analytics` (port **8089**, DB **`hiveiq_analytics`**).
- **Endpoint the screen hits:** `GET /api/analytics/uptime` (`UptimeController`).
- **Frontend:** `UptimeTab.svelte`, called via `api.getUptime()` → `${apiBase}/uptime`.
## Roles / access
- **No role gate.** `SecurityConfig` = `.anyRequest().authenticated()`; only `/actuator/health` and `/api/public/**` are `permitAll()`. Any valid JWT can read Uptime — MSP_ADMIN is **not** required.
- The endpoint **forwards the caller's Bearer token** downstream to the incident service. If the caller's token can't read ATMs, the "Currently Down" list comes back empty (the call is caught and logged, not surfaced).
## What the numbers actually come from
| Card / panel | Real source |
|---|---|
| **NOW** ring | Latest `analytics_snapshots` row for **today** (`connectedAtms / totalAtms`). Can be up to **6 h stale**. |
| **7-DAY AVG** / **30-DAY AVG** | Both computed from **today's snapshots only** (see caveat below). |
| **30-Day Trend** chart | `dailyTrend` — currently **today only** (1 point). |
| **CURRENTLY DOWN** count + table + **never connected** badge | **Live** call to incident: `GET /api/atms/paginated?size=500&agentStatus=DISCONNECTED,NEVER_CONNECTED`. Capped at 50 rows, sorted longest-down first. |
So the top-left ring is **snapshot-derived (may lag)** while the down-list is **live**. They can legitimately disagree for a few hours.
## First things to check when it misbehaves
1. **"Not enough data" on the trend / flat 7-day = 30-day numbers** → expected with current code (only today's snapshots are queried). Not a data-loss bug per se — see Common failure modes.
2. **All zeros / NOW ring empty** → no `analytics_snapshots` row for today. Check the snapshot pipeline:
- `docker compose logs hiveiq-analytics | grep -i snapshot` — look for `Analytics snapshot saved for <date>/<hour>`.
- Snapshots are written by `SnapshotCollectionService.collect()`: cron **every 6 h** (`0 0 */6 * * *`) **plus** Kafka triggers (incident + journal events). If Kafka is quiet and the cron hasn't fired, today may have no snapshot yet.
- Force one indirectly by generating an incident/journal event, or wait for the 6-hour cron.
3. **CURRENTLY DOWN empty but you know ATMs are offline** → the live incident call failed or returned empty. `grep "failed to fetch down ATMs"` in analytics logs. Verify incident-backend is up and still serves `/api/atms/paginated`.
4. **Numbers look right in analytics but the snapshot is old** → the collector calls devices/incident/fleet/journal with an **internal service token** (`InternalTokenGenerator`, role `ADMIN`). If a downstream is down, `FleetOverviewService` logs `Failed to fetch ATM summary` and the snapshot stores zeros for that section.
5. **Wrong DB / no rows at all** → confirm prod split env vars `DB_HOST/DB_PORT/DB_NAME/DB_USERNAME/DB_PASSWORD` (analytics does **not** use a single `DB_URL`). Query:
`SELECT snapshot_date, snapshot_hour, total_atms, connected_atms FROM analytics_snapshots ORDER BY snapshot_date DESC, snapshot_hour DESC LIMIT 5;`
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Trend chart always says "Not enough data" | `UptimeService` only loads **today's** snapshots → `dailyTrend` has 1 point; frontend needs ≥2 | Product/code issue (see Architect); not a config fix |
| 7-day and 30-day rings show the **same** number | Both = today's average in code | Same — code limitation, not corruption |
| NOW ring stale vs. down-list | NOW from snapshot (≤6 h old), down-list is live | Expected; force a fresh snapshot via a Kafka event if needed |
| Down-list empty / count 0 despite offline ATMs | Incident `/api/atms/paginated` failed or caller token lacks ATM read | Check analytics logs + incident-backend health |
| Whole screen zeros | No snapshot row for today | Check snapshot collector logs, Kafka consumers, cron |
| 401/403 loading tab | Caller JWT expired/invalid (no role issue — any auth works) | Re-login / refresh token |
## Handy commands
```bash
# analytics health + recent snapshot activity
docker compose logs --tail=200 hiveiq-analytics | grep -iE "snapshot|down ATMs|Failed to fetch"
# latest snapshots (DB VM via ProxyJump)
ssh -i ~/.ssh/id_ed25519_hiveiq -J bcosadmin@173.231.195.250 bcosadmin@10.10.10.188 \
"docker exec hiveiq-postgres psql -U hiveiq -d hiveiq_analytics -c \
'SELECT snapshot_date,snapshot_hour,total_atms,connected_atms,disconnected_atms FROM analytics_snapshots ORDER BY snapshot_date DESC, snapshot_hour DESC LIMIT 10;'"
```
backend/src/main/resources/content/aria/dashboard/architect.md
+72
View File
@@ -0,0 +1,72 @@
---
module: aria.dashboard
title: Threat Dashboard
tab: Architect
order: 30
audience: internal
---
How the ARIA Threat Dashboard is built. The dashboard is a **thin read-only aggregation view** — it does not own data, run detection, or consume Kafka. It renders one snapshot from `hiveops-aria`'s tables. All the interesting machinery (ingestion, IOC matching, sequence detection) runs upstream and only *populates* the tables this view reads.
## Services involved
- **`hiveops-aria`** (Spring Boot, `spring.application.name=hiveops-aria`, port **8095**, DB `hiveiq_aria`) — owns everything on this screen. Serves the stats, owns all four tables. See [platform.service-ownership].
- **`hiveops-auth`** — only for the header's `GET /auth/api/users/me` name/role lookup; not part of the stats path.
- No other service participates in the *dashboard read path*. (Ingestion into the tables comes from agents via the ARIA ingest endpoints; auto-response — disabled by default — would reach out to `hiveops-incident`.)
## Data flow for THIS feature
```
Browser (Dashboard.svelte, onMount)
→ GET /api/aria/stats [JWT Bearer, ROLE_BCOS_ADMIN]
→ ThreatEventController.getStats()
├─ eventRepository.count() → threat_event
├─ eventRepository.countBySeverityAndDetectedAtAfter(CRIT,-24h) → threat_event
├─ eventRepository.countBySeverityAndDetectedAtAfter(HIGH,-24h) → threat_event
├─ iocRepository.findByActiveTrue().size() → threat_ioc
├─ sequenceAlertRepository.countByResolvedAtIsNull() → precursor_sequence_alert
├─ postureRepository.countByPostureScoreLessThan(50) → device_security_posture
└─ postureRepository.countByOsEolStatus(EOL) → device_security_posture
← Map<String,Object> of 7 counts
→ renders 8 cards (Non-Critical Events computed client-side)
```
Clicking a card is pure **client-side navigation** (Svelte `createEventDispatcher` → `App.svelte handleDashboardNav`) to the Events / Sequences / IOC / Posture views. No stats-specific endpoint backs the click.
## Tables read (never written by this view)
| Table | Read for | Written by (upstream) |
|-------|----------|------------------------|
| `threat_event` | Total / Critical / High counts | event ingestion service (`POST /api/aria/ingest/events`) |
| `threat_ioc` | Active IOC count | Flyway seed (FBI FLASH) + IOC feed refresh + manual CRUD |
| `precursor_sequence_alert` | Active (unresolved) sequence count | sequence-detection service |
| `device_security_posture` | Low-posture + EOL device counts | posture report ingestion (`POST /api/aria/posture/report`) |
The dashboard issues **zero writes**. See [platform.data-architecture].
## Kafka role (indirect only)
The dashboard has **no Kafka consumer and no producer** — it is a synchronous DB read. Kafka appears one hop upstream, populating the tables it later reads:
| Topic | Direction (in aria) | Producer | Note |
|-------|--------------------|----------|------|
| `hiveops.aria.threats` | produced | `AriaThreatProducer` from `ThreatEventIngestionService` | fan-out of ingested threat events; not consumed within aria |
| `hiveops.aria.sequences` | produced | `AriaSequenceProducer` from `SequenceDetectionService` | fan-out when a precursor sequence trips |
There are **no `@KafkaListener`s** in `hiveops-aria`. This means the executor→Kafka→record-store pattern used elsewhere on the platform does **not** drive this dashboard: writes land directly in Postgres via the ingest/detection services, and the dashboard reads Postgres. Kafka here is purely an outbound notification channel for other consumers. See [platform.kafka].
## Key API endpoint
| Method | Path | Auth | Returns |
|--------|------|------|---------|
| GET | `/api/aria/stats` | `hasRole('BCOS_ADMIN')` | `{ totalEvents, criticalCount, highCount, activeIocCount, activeSequenceCount, lowPostureCount, eolDeviceCount }` |
Auth chain: `SecurityConfig` (stateless, `@EnableMethodSecurity`) → `JwtAuthenticationFilter` validates the Bearer JWT via shared `JwtTokenValidator`, maps the single `role` claim to `ROLE_<role>`. `/stats` is **not** in the `permitAll` list, so it requires an authenticated BCOS_ADMIN.
## Design notes / semantics
- **Windowing is mixed on purpose (or by oversight):** `criticalCount`/`highCount` are **24h** rolling counts (`...DetectedAtAfter(now-24h)`), while `totalEvents` and everything else are **point-in-time totals**. The client's "Non-Critical Events = Total − Critical − High" therefore subtracts a 24h slice from an all-time total. Worth confirming this is intended before building anything else on those numbers.
- **No caching / no auto-poll:** the stat map is recomputed on every `GET /stats`, and the SPA calls it exactly once on mount. Live dashboards would need a poll loop or push.
- **Counting shape:** `activeIocCount` materializes the active-IOC list and takes `.size()` rather than a `count(*)` query — fine at current scale, a candidate for a dedicated count query later.
Related: [aria.dashboard.internal], [technical.aria], [platform.data-architecture], [platform.kafka], [platform.service-ownership].
backend/src/main/resources/content/aria/dashboard/claude.md
+67
View File
@@ -0,0 +1,67 @@
---
module: aria.dashboard
title: Threat Dashboard
tab: Claude
order: 40
audience: internal
---
Terse act-without-guessing reference. Owner service: **`hiveops-aria`** (port **8095**, DB **`hiveiq_aria`**). Frontend: `hiveops-aria/frontend/src/components/Dashboard.svelte`.
## Endpoints (all this view uses)
| Method | Path | Auth | Notes |
|--------|------|------|-------|
| GET | `/api/aria/stats` | `hasRole('BCOS_ADMIN')` | THE only data call. Returns 7 counts. |
| GET | `/auth/api/users/me` | authenticated | header name/role only; not aria; via gateway URL |
Response keys of `/api/aria/stats`: `totalEvents`, `criticalCount`, `highCount`, `activeIocCount`, `activeSequenceCount`, `lowPostureCount`, `eolDeviceCount`.
Card-click navigation is client-side only (no endpoints): Critical→events(sev=CRITICAL), High→events(sev=HIGH), Active Sequences→sequences, Active IOCs→ioc, Low Posture/EOL→posture.
## Count sources (exact)
| Key | Query | Table | Window |
|-----|-------|-------|--------|
| totalEvents | `count()` | `threat_event` | all-time |
| criticalCount | `countBySeverityAndDetectedAtAfter(CRITICAL, now-24h)` | `threat_event` | **24h** |
| highCount | `countBySeverityAndDetectedAtAfter(HIGH, now-24h)` | `threat_event` | **24h** |
| activeIocCount | `findByActiveTrue().size()` | `threat_ioc` | current |
| activeSequenceCount | `countByResolvedAtIsNull()` | `precursor_sequence_alert` | current |
| lowPostureCount | `countByPostureScoreLessThan(50)` | `device_security_posture` | current |
| eolDeviceCount | `countByOsEolStatus(EOL)` | `device_security_posture` | current |
## Tables (owned by hiveops-aria, schema `hiveiq_aria`)
- `threat_event` · `threat_ioc` · `precursor_sequence_alert` · `device_security_posture`
- Dashboard = **read-only**. Zero writes from this view.
## Kafka
- Topics produced by aria (NOT by the dashboard): `hiveops.aria.threats`, `hiveops.aria.sequences`.
- **No `@KafkaListener` in hiveops-aria.** No consumer. Dashboard reads Postgres directly.
## Auth facts
- JWT Bearer, header `Authorization: Bearer <jwt>`. Single `role` claim → `ROLE_<role>`.
- `/stats` requires role **exactly `BCOS_ADMIN`**. `MSP_ADMIN` → 403.
- `permitAll` in SecurityConfig: `/api/aria/ingest/**`, `/api/aria/posture/report`, actuator health/info. `/stats` is NOT public.
## Gotchas
- Critical/High cards are **24h counts**, Total is all-time → "Non-Critical Events" (`total − crit24h − high24h`, computed in browser) mixes windows. Don't treat it as a clean category.
- Dashboard loads once on mount — **no auto-poll, no refresh button**. Stale = reload.
- Total Events + Non-Critical Events cards are non-clickable by design.
- `activeIocCount` loads full active-IOC list then `.size()` (not a count query).
- `frontend/src/lib/api.ts` dev fallback `apiUrl` is `http://localhost:8017`; real port is 8095 via injected `window.__APP_CONFIG__.apiUrl`. Don't hardcode 8017.
## Do NOT
- Do NOT assume `MSP_ADMIN` can load this — it's `BCOS_ADMIN` only.
- Do NOT expect Kafka to drive the dashboard — it's a synchronous DB read.
- Do NOT add write/mutation logic here — mutations live in IOC Management / Precursor Alerts views.
- Do NOT read `criticalCount`/`highCount` as all-time totals — they are 24h.
- Do NOT invent per-card endpoints — only `/api/aria/stats` backs this page.
- Do NOT route stats through incident/journal/devices — hiveops-aria owns all of it.
Related: [aria.dashboard.internal], [aria.dashboard.architect], [technical.aria].
backend/src/main/resources/content/aria/dashboard/internal.md
+60
View File
@@ -0,0 +1,60 @@
---
module: aria.dashboard
title: Threat Dashboard
tab: Internal
order: 20
audience: internal
---
Ops/support view for the ARIA Threat Dashboard — the landing screen of the ARIA (`hiveops-aria`) SPA. The whole page is driven by **one call**: `GET /api/aria/stats`. If a card is wrong or blank, that endpoint (or its underlying tables) is where to look — there is no per-card fetch.
## Who can see it
- **Role required: `BCOS_ADMIN` only.** `/api/aria/stats` (and `/api/aria/events`) is gated by `@PreAuthorize("hasRole('BCOS_ADMIN')")`.
- This is **not** the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` pattern. An `MSP_ADMIN` JWT gets **403** and the dashboard shows the "Could not load ARIA stats" toast with empty cards.
- Auth is a stateless JWT Bearer token. The filter reads the single `role` claim and maps it to `ROLE_<role>`, so the token's `role` must be exactly `BCOS_ADMIN`.
- The sidebar/header also calls `GET /auth/api/users/me` (via the gateway URL) purely to show the user's name/role — a failure there is silent and does not block the dashboard.
## What each card actually counts
| Card | Source (repo query) | Window |
|------|--------------------|--------|
| Total Events | `threat_event` row count | **all-time** |
| Critical | `threat_event` where severity=CRITICAL | **last 24h only** |
| High | `threat_event` where severity=HIGH | **last 24h only** |
| Active Sequences | `precursor_sequence_alert` where `resolved_at IS NULL` | current |
| Active IOCs | `threat_ioc` where `active = true` | current |
| Low Posture | `device_security_posture` where `posture_score < 50` | current |
| EOL Devices | `device_security_posture` where `os_eol_status = EOL` | current |
| Non-Critical Events | `Total − Critical − High` (computed in the browser) | mixed |
> **Gotcha — Critical/High are 24-hour rolling counts, not all-time.** The card label just says "Critical"/"High" (matching the customer Overview), but the backend uses `countBySeverityAndDetectedAtAfter(..., now-24h)`. "Non-Critical Events" subtracts 24h counts from an all-time total, so it is effectively "all events minus the last 24h of critical/high" — not a clean category. If someone asks "why does Critical show 0 when there are old critical events?", this is why.
## First things to check when it misbehaves
1. **All cards blank + "Load Failed" toast** → the `/api/aria/stats` call failed. Almost always a **403 (wrong role — not BCOS_ADMIN)** or a **401 (missing/expired JWT)**. Check the browser network tab for the status on `/api/aria/stats`.
2. **Page loads but numbers are all 0** → the tables are genuinely empty. ARIA data arrives only via event ingestion (`POST /api/aria/ingest/**`) and posture reports (`POST /api/aria/posture/report`). No ingestion = no events, IOCs still seeded from Flyway.
3. **Critical/High are 0 but you expect events** → confirm the events are within the last 24h (`detected_at`). Older events count toward Total but not Critical/High.
4. **Active IOCs unexpectedly low/zero** → `threat_ioc` seed (FBI FLASH, Flyway V2) may not have run, or IOCs were deactivated. Check `active = true` rows.
5. **502/timeout on the SPA** → `hiveops-aria` service (port 8095) is down or the gateway/nginx route to it is stale. Verify `curl localhost:8095/actuator/health`.
6. **A card doesn't navigate** → Total Events and Non-Critical Events are **intentionally not clickable**; the rest dispatch client-side navigation to Events/Sequences/IOC/Posture views (no extra API calls on click).
## Admin-only actions reachable from here
The dashboard itself is **read-only** (no buttons, no refresh trigger — it loads once on mount). The clickable cards jump to admin views that do have actions:
- **Active IOCs → IOC Management** — create/update/deactivate IOCs (`POST/PUT/DELETE /api/aria/ioc`).
- **Active Sequences → Precursor Alerts** — resolve a sequence (`POST /api/aria/sequences/{id}/resolve`).
- **Low Posture / EOL → Device Posture** — read-only posture inspection.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| 403 on `/api/aria/stats` | JWT role ≠ BCOS_ADMIN | Re-issue token with `role: BCOS_ADMIN` |
| 401 on `/api/aria/stats` | Expired/absent Bearer token | Re-login to the SPA |
| All zeros | No ingested events/postures yet | Confirm agents are posting to `/api/aria/ingest/**` |
| Stats slow | `activeIocCount` loads all active IOC rows into memory to size the list | Cosmetic; only bites with very large IOC tables |
| Card counts stale | Dashboard loads once per page open — no auto-poll | Reload the page |
Related: [aria.dashboard.architect], [technical.aria].
backend/src/main/resources/content/aria/device-posture/architect.md
+90
View File
@@ -0,0 +1,90 @@
---
module: aria.device-posture
title: Device Posture
tab: Architect
order: 30
audience: internal
---
> **How Device Posture is built.** Grounded in `hiveops-aria` backend + `hiveops-agent/hiveops-module-aria-signals`. See also [technical.aria] for the wider ARIA service. Cross-links: [platform.data-architecture], [platform.kafka], [platform.service-ownership].
## Services involved
Device Posture is entirely **`hiveops-aria`** on the backend + the ATM agent on the collection side. It does **not** touch incident/journal/devices/mgmt for this feature. Cross-link [platform.service-ownership] — ARIA owns the `device_security_posture` table exclusively.
| Layer | Component | Role |
|-------|-----------|------|
| Agent (ATM) | `hiveops-module-aria-signals` → `AriaPostureReporter` + `AriaHttpClient` | Collects posture facts on-box and POSTs them |
| Backend API | `DevicePostureController` (`/api/aria/posture`) | Ingest (report) + admin read endpoints |
| Backend logic | `DevicePostureService` | Upsert + score calculation |
| Persistence | `DeviceSecurityPostureRepository` → `device_security_posture` | Stores one row per device (keyed by agent id) |
| Frontend | `DevicePosture.svelte` (`aria.bcos.dev`, `ariaApi.getPostures`) | Institution-grouped table + detail panel |
## Data flow (no Kafka on this path)
Unlike ARIA's threat pipeline, posture is a **synchronous HTTP upsert**, not an executor→Kafka→record-store pattern. Kafka is used elsewhere in ARIA (`hiveops.aria.threats`, `hiveops.aria.sequences`) for threat events/sequences only — **posture is not published or consumed on Kafka**. Cross-link [platform.kafka].
```
[ATM agent: aria-signals]
AriaPostureReporter.run() (scheduleAtFixedRate, immediate + every 86400s default)
→ collects: OS ver/EOL (all OS); BitLocker/auditpol/Solidcore (Windows only);
SHA-256 image hash vs gold (only if aria.gold.image.hash + aria.image.path set)
→ AriaHttpClient POST /api/aria/posture/report
header: X-Service-Secret: <aria.service.secret>
body: PostureReportRequest (JSON)
│
▼
[hiveops-aria: DevicePostureController.report()]
permitAll at filter level; controller verifies X-Service-Secret == aria.internal.service-secret
(rejects 401 if header mismatches OR secret is blank)
│
▼
[DevicePostureService.report()] @Transactional
upsert by deviceAgentId (findByDeviceAgentId → else build new)
partial merge (only non-null fields overwrite)
set lastPostureCheckAt = now; postureScore = calculateScore(); updatedAt = now
→ repository.save() → device_security_posture
│
(read side) ▼
[DevicePosture.svelte] GET /api/aria/posture?page&size&scoreFilter|osEolStatus
BCOS_ADMIN JWT → grouped by institutionKey, avg score, <80 flags
```
## Scoring model (`DevicePostureService.calculateScore`)
Server-authoritative. Starts at **100**, floored at **0**:
| Condition | Penalty |
|-----------|---------|
| `diskEncryptionEnabled == false` | −25 |
| `auditPolicyCompliant == false` | −20 |
| `osEolStatus == EOL` | −20 |
| `softwareWhitelistEnabled == false` | −10 |
| `goldImageHash != currentImageHash` (both non-null) | −20 |
| `lastPostureCheckAt` older than 7 days | −5 per extra day, capped at −15 |
Null booleans are **not** penalized (a missing check ≠ a failed check). The frontend detail panel renders the first five penalties but not the staleness penalty.
## DB tables
| Table | R/W | Notes |
|-------|-----|-------|
| `device_security_posture` | read + write | One row per device. PK `id`; business key `device_agent_id`. Columns: `device_id`, `institution_key`, `disk_encryption_enabled`, `audit_policy_compliant`, `software_whitelist_enabled`, `gold_image_hash`, `current_image_hash`, `gold_image_validated_at`, `os_version`, `os_eol_status` (enum `SUPPORTED\|EOL\|UNKNOWN`), `last_posture_check_at`, `posture_score`, `updated_at`. |
Schema is Flyway-managed in `hiveiq_aria` with `ddl-auto=validate`. Cross-link [platform.data-architecture]. No other service mirrors this table (contrast with `device_summary` mirrors used by incident/fleet/reports/recon).
## Key API endpoints
| Method | Path | Auth | Purpose |
|--------|------|------|---------|
| POST | `/api/aria/posture/report` | `X-Service-Secret` header (permitAll at filter) | Agent/internal ingest — upsert + rescore |
| GET | `/api/aria/posture` | `hasRole('BCOS_ADMIN')` | Paged list; `osEolStatus` OR `scoreFilter` (low<50/medium<80); `page`/`size` (default 25) |
| GET | `/api/aria/posture/{deviceId}` | `hasRole('BCOS_ADMIN')` | Single record by `device_id` column |
Repository queries always sort **`OrderByPostureScoreAsc`** (worst devices first). `osEolStatus` filter takes precedence over `scoreFilter` in the controller.
## Agent collection detail
- Module: `hiveops-module-aria-signals`, active only when `aria.endpoint` is set. Config keys: `aria.endpoint`, `aria.service.secret`, `aria.posture.report.interval.sec` (default 86400).
- Windows checks shell out to `manage-bde -status C:`, `auditpol /get /subcategory:"Audit Policy Change"`, `sadmin status` (McAfee/Trellix Solidcore) with AppLocker registry fallback.
- OS EOL is classified **client-side** (Win7/XP/Vista/8 → EOL; other Windows/Linux → SUPPORTED). The backend stores whatever the agent sends.
backend/src/main/resources/content/aria/device-posture/claude.md
+55
View File
@@ -0,0 +1,55 @@
---
module: aria.device-posture
title: Device Posture
tab: Claude
order: 40
audience: internal
---
> Terse agent reference. Every item below confirmed in `hiveops-aria` + `hiveops-agent/hiveops-module-aria-signals`. Act without guessing.
## Ownership
- **Service:** `hiveops-aria` — port **8095**, DB **`hiveiq_aria`**, `spring.application.name=hiveops-aria`.
- **Frontend:** `hiveops-aria/frontend` → `DevicePosture.svelte`, host `aria.bcos.dev` / `aria.bcos.cloud`.
- No incident/journal/devices/mgmt involvement. No Kafka on the posture path.
## Endpoints (METHOD path → auth)
| Method | Path | Auth |
|--------|------|------|
| GET | `/api/aria/posture` | `hasRole('BCOS_ADMIN')` |
| GET | `/api/aria/posture/{deviceId}` | `hasRole('BCOS_ADMIN')` — matches **`device_id`** column, not PK |
| POST | `/api/aria/posture/report` | header `X-Service-Secret` == `aria.internal.service-secret` |
- `GET /api/aria/posture` params: `osEolStatus` (`SUPPORTED\|EOL\|UNKNOWN`), `scoreFilter` (`low`=<50, `medium`=<80), `page` (0), `size` (25; UI uses 500). `osEolStatus` wins if both set. Always sorted score ASC.
- Response: Spring `Page<DeviceSecurityPosture>` (`content[]`, `page.totalElements`).
## Storage
- **Table:** `device_security_posture` (only table for this feature). PK `id`; business key `device_agent_id`.
- Key columns: `device_id`, `institution_key`, `disk_encryption_enabled`, `audit_policy_compliant`, `software_whitelist_enabled`, `gold_image_hash`, `current_image_hash`, `os_version`, `os_eol_status`, `last_posture_check_at`, `posture_score`, `updated_at`.
- Flyway-managed, `ddl-auto=validate`.
## Kafka
- **None for posture.** (ARIA's `hiveops.aria.threats` / `hiveops.aria.sequences` are threat-event topics only — do not attribute posture to them.)
## Scoring (server-side, `DevicePostureService.calculateScore`)
- base 100, floor 0. diskEnc=false −25 · audit=false −20 · osEol=EOL −20 · whitelist=false −10 · gold≠current −20 · stale >7d −5/day cap −15. Null booleans not penalized.
## Ingest source
- Agent module `aria-signals` (`AriaPostureReporter`), `POST /api/aria/posture/report`, immediate on start + every `aria.posture.report.interval.sec` (default **86400s / 24h**).
- Windows-only booleans (BitLocker/auditpol/Solidcore). Linux → OS fields only.
## Gotchas
- Role gate is **BCOS_ADMIN only**, not `MSP_ADMIN,BCOS_ADMIN`. MSP admin → 403.
- If `aria.internal.service-secret` (`ARIA_SERVICE_SECRET`) is **blank**, `report()` returns **401 for every request** (`serviceSecret.isBlank() ||` short-circuit) → table silently stays empty. (Ingest events endpoint behaves differently — do not assume symmetry.)
- Upsert key is `deviceAgentId`; re-imaged device w/ new agent id = new row, old one orphaned.
- Blank `institutionKey` groups under `"Unknown"` in UI.
- Detail panel penalties omit the staleness penalty → displayed penalties may not sum to the score.
- `report()` merges only non-null fields; a report omitting a field keeps the prior stored value.
## Do NOT
- Do NOT expect `MSP_ADMIN` to read these endpoints.
- Do NOT add a Kafka producer/consumer for posture — it is synchronous HTTP.
- Do NOT look for posture data in incident/journal/devices/mgmt DBs — only `hiveiq_aria.device_security_posture`.
- Do NOT treat null booleans as failed checks (they are unreported, not non-compliant).
- Do NOT assume `/api/aria/posture/{deviceId}` takes the PK — it queries the `device_id` column.
- Do NOT deploy `hiveops-aria` without a non-blank `ARIA_SERVICE_SECRET` or ingest breaks.
backend/src/main/resources/content/aria/device-posture/internal.md
+57
View File
@@ -0,0 +1,57 @@
---
module: aria.device-posture
title: Device Posture
tab: Internal
order: 20
audience: internal
---
> **Internal ops/support view.** Grounded in `hiveops-aria` (`DevicePostureController`, `DevicePostureService`) and the agent `hiveops-module-aria-signals`. Verify against code before acting on production.
## Who owns it
- **Service:** `hiveops-aria` (port **8095**, DB `hiveiq_aria`, table `device_security_posture`). This screen reads/writes ARIA only — no incident/journal/devices involvement.
- **Data source:** the ATM agent's **`aria-signals`** module POSTs a posture report on a slow schedule (default every **24h**, fires once immediately on module start). There is no Kafka in this path — it is a direct HTTP upsert.
## Roles required
- **Viewing the screen** (`GET /api/aria/posture`, `GET /api/aria/posture/{deviceId}`) requires **`hasRole('BCOS_ADMIN')`**.
- ⚠ Note this is **BCOS_ADMIN only** — NOT the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` you see elsewhere. An `MSP_ADMIN` who can see other HiveIQ screens will get **403** here. This is by design in ARIA, not a bug — do not "fix" it without a ticket.
- **Posture ingest** (`POST /api/aria/posture/report`) is `permitAll()` at the security-filter level but gated inside the controller by the **`X-Service-Secret`** header matching `aria.internal.service-secret` (`ARIA_SERVICE_SECRET`).
## First things to check when it misbehaves
**"Empty table / No posture data yet"**
1. Confirm the caller is a **BCOS_ADMIN** — an MSP admin gets 403 and the UI shows a load error, not empty data.
2. Confirm **`ARIA_SERVICE_SECRET` is set (non-blank)** on the `hiveops-aria` service. If it is blank, **every** posture report is rejected `401` (see failure modes) — the table stays empty forever with no obvious error.
3. Confirm agents actually run `aria-signals`: the module is inactive unless `aria.endpoint` is configured in `hiveops.properties`, and it needs `aria.service.secret` to match the backend secret.
4. Remember the report cadence is **24h by default** (`aria.posture.report.interval.sec=86400`). A freshly deployed fleet can take a day to populate. It does fire once on agent start, so a restart is a fast way to force a report.
**"Device shows no Disk Enc / Audit / Whitelist (all `—`)"**
- Those three checks are **Windows-only** (BitLocker `manage-bde`, `auditpol`, McAfee Solidcore `sadmin` / AppLocker fallback). Linux ATMs report only `osVersion` + `osEolStatus`; the boolean columns stay null and render as `—`. Expected.
**"Image always shows `—` (never Match/Drift)"**
- Image drift only populates when the agent has **both** `aria.gold.image.hash` and `aria.image.path` set. Without them, `goldImageHash` is null and the Image column is `—`.
**"Score looks lower than the penalties shown in the detail panel"**
- The detail panel lists penalties for encryption (−25), audit (−20), whitelist (−10), OS EOL (−20) and image drift (−20). The backend **also** subtracts a **staleness penalty** (−5/day beyond 7 days stale, capped at −15) that the panel does **not** display. A device that stopped reporting will keep dropping in score without a visible reason on the panel. See uncertainties.
**"Device I expect is missing / duplicated"**
- Records are keyed by **`deviceAgentId`** (upsert via `findByDeviceAgentId`). A device re-imaged with a different agent ID creates a **new** posture row; the old one lingers until manually cleaned. Institution grouping uses `institutionKey` from the report — a blank key groups under **"Unknown"**.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| Table empty, no agents landing | `ARIA_SERVICE_SECRET` blank on backend → all reports `401` | Set the secret to a non-blank value on `hiveops-aria`, redeploy, and set matching `aria.service.secret` on agents |
| Reports `401` from one ATM only | Agent's `aria.service.secret` mismatches backend | Push a `CONFIG_UPDATE` with the correct secret |
| MSP admin sees load error | BCOS_ADMIN-only gate | Expected — use a BCOS_ADMIN JWT |
| Bool checks all `—` on many devices | Linux fleet (checks are Windows-only) | Expected, not a defect |
| Stale `Last Check` timestamps fleet-wide | `aria-signals` disabled or `aria.endpoint` unset | Verify agent config; restart forces an immediate report |
| Score drifting down with no failing check | Hidden staleness penalty for stopped reporters | Investigate why the device stopped reporting |
## Manual data checks (BCOS_ADMIN JWT)
- List: `GET /api/aria/posture?page=0&size=500` (UI pulls up to 500 at once).
- Filters: `scoreFilter=low` (<50) or `scoreFilter=medium` (<80); `osEolStatus=EOL|SUPPORTED|UNKNOWN`. `osEolStatus` takes precedence over `scoreFilter` in the controller (only one is applied).
- Single device: `GET /api/aria/posture/{deviceId}` — note `{deviceId}` is matched against the **`device_id`** column, not the row primary key.
backend/src/main/resources/content/aria/ioc-feeds/architect.md
+76
View File
@@ -0,0 +1,76 @@
---
module: aria.ioc-feeds
title: IOC Feeds
tab: Architect
order: 30
audience: internal
---
How the **IOC Feeds** feature is built. Everything for this feature lives inside **`hiveops-aria`** — it does not call any other HiveIQ service. Cross-link [platform.service-ownership].
## Component map
| Layer | Class | Role |
|-------|-------|------|
| Frontend | `frontend/src/components/IocFeeds.svelte` | cards + run-history table; polls `/status` every 30s |
| Frontend API | `frontend/src/lib/api.ts` (`ariaApi`) | `getFeedStatus`, `getFeedRuns`, `refreshAllFeeds`, `refreshFeed` |
| Controller | `controller/IocFeedController.java` | `/api/aria/feeds/*`; spawns background threads for refresh |
| Service | `service/IocFeedService.java` | orchestrates fetch → upsert → run record; owns `@Scheduled` cron |
| Feed clients | `feed/OtxFeedClient`, `feed/MalwareBazaarFeedClient`, `feed/ThreatFoxFeedClient` | call external APIs, map raw indicators → `ThreatIoc` |
| Repos | `IocFeedRunRepository`, `ThreatIocRepository` | run history + IOC catalog |
| Entities | `entity/IocFeedRun`, `entity/ThreatIoc` | `ioc_feed_run`, `threat_ioc` tables |
## Data flow (one feed run)
```
[BCOS_ADMIN] Run Feed / Run All [cron 0 0 3 * * *]
│ POST /api/aria/feeds/refresh(/{feed}) │ scheduledRefresh()
▼ ▼
IocFeedController ── new Thread() ──► IocFeedService.refreshFeed(feedName)
(202 accepted) │
├─ 1. countRunning guard (skip if RUNNING)
├─ 2. save ioc_feed_run row (status=RUNNING, started_at)
├─ 3. fetchFromFeed → <Feed>Client.fetch() ──► external HTTPS
├─ 4. upsertIocs → threat_ioc (add / update)
└─ 5. run row → SUCCESS|ERROR, counts, completed_at
```
- **Controller offloads to a raw `Thread`** (named `ioc-feed-manual-all` / `ioc-feed-manual-<FEED>`) with an uncaught-exception handler that logs, and returns `202` immediately. There is **no executor/queue and no Kafka** in this path — it's a synchronous fetch running on a plain thread.
- **Upsert** (`ThreatIocRepository.findByActiveTrueAndIocTypeAndValueIgnoreCase`): match on active + `ioc_type` + case-insensitive `value`. Hit → update `description/source_ref/confidence/reported_at/expires_at` (**updated++**); miss → insert (**added++**). Blank values are skipped.
## External sources (feed clients)
| Client | Endpoint | Auth header | Query strategy | IOC types mapped | expires_at |
|--------|----------|-------------|----------------|------------------|-----------|
| OTX | `https://otx.alienvault.com/api/v1` | `X-OTX-API-KEY` | subscribed pulses (8d) + searches `atm+malware`, `jackpotting`, `atm+skimming` | MD5, FILE_PATH, IP, REGISTRY_KEY | +90d |
| MalwareBazaar | `https://mb-api.abuse.ch/api/v1/` | `Auth-Key` | `get_taginfo` per tag: ATM, jackpotting, NCR, Diebold, GreenDispenser, Tyupkin | MD5, FILENAME | +365d |
| ThreatFox | `https://threatfox-api.abuse.ch/api/v1/` | `Auth-Key` | `iocs_by_malware_family` (ATMii, GreenDispenser, Tyupkin, Ploutus, SUCEFUL, Ripper) + `get_iocs` 7d filtered for ATM relevance | MD5, IP (port stripped), FILE_PATH, FILENAME | +180d |
- All three short-circuit to an empty list when their API key is blank (`isConfigured()` false) — that becomes a SUCCESS run with 0/0, not an error.
- Each client swallows per-request errors internally (`log.warn`) and returns whatever it collected; a run only flips to ERROR if the whole `fetchFromFeed`/`upsertIocs` throws.
## DB tables ([platform.data-architecture])
Database **`hiveiq_aria`** (PostgreSQL), Flyway-managed, `ddl-auto=validate`.
| Table | Access here | Notes |
|-------|-------------|-------|
| `ioc_feed_run` | read (status, runs) + write (each run) | added in Flyway **V6**; columns: `feed_name`, `started_at`, `completed_at`, `status`, `iocs_added`, `iocs_updated`, `error_message`; indexes on `feed_name` and `started_at DESC` |
| `threat_ioc` | read (dedup lookup) + write (upsert) | IOC catalog; feed rows carry `source` = `OTX`/`MALWARE_BAZAAR`/`THREAT_FOX` |
- `/status` = for each feed name, `findTopByFeedNameOrderByStartedAtDesc` → latest run summary + `configured` flag.
- `/runs` = `findAllByOrderByStartedAtDesc` as a Spring `Page` (default size 20 from the UI), so the frontend gets `content` + page metadata.
## Kafka ([platform.kafka])
- **This feature produces and consumes nothing.** ARIA's Kafka producers (`hiveops.aria.threats`, `hiveops.aria.sequences`) belong to the event-ingestion / sequence-detection paths, not IOC feeds. The catalog these feeds populate is later read synchronously by the classifier when matching device events — no topic sits between the feed and the catalog.
## Scheduling & config
- `@Scheduled(cron = "${aria.feed.refresh.cron:0 0 3 * * *}")` on `IocFeedService.scheduledRefresh()` → `refreshAll()` (iterates the three feeds sequentially).
- Keys: `aria.feed.otx.api-key`, `aria.feed.malware-bazaar.api-key`, `aria.feed.threat-fox.api-key` (env `ARIA_FEED_*_API_KEY`).
- Note: `aria.feed.enabled` is declared but **unused** in code (see Internal tab / uncertainties).
## Security
- `SecurityConfig`: stateless, `@EnableMethodSecurity`. `/api/aria/feeds/**` is **not** in the `permitAll` list, so it falls under `anyRequest().authenticated()` **plus** each method's `@PreAuthorize("hasRole('BCOS_ADMIN')")`. Cross-link [platform.service-ownership].
backend/src/main/resources/content/aria/ioc-feeds/claude.md
+55
View File
@@ -0,0 +1,55 @@
---
module: aria.ioc-feeds
title: IOC Feeds
tab: Claude
order: 40
audience: internal
---
Terse agent reference. Everything below is confirmed in `hiveops-aria` source.
## Ownership
- **Owner service:** `hiveops-aria` (port **8095**, `spring.application.name=hiveops-aria`). Self-contained — no other service involved for this feature.
- **DB:** `hiveiq_aria` (PostgreSQL, Flyway `ddl-auto=validate`).
## Endpoints (all `@PreAuthorize("hasRole('BCOS_ADMIN')")`)
| Method | Path | Returns |
|--------|------|---------|
| GET | `/api/aria/feeds/status` | `List<Map>` one per feed: `feedName`, `configured`, `enabled`(always true), `lastRunAt`, `lastRunStatus`, `lastIocsAdded`, `lastIocsUpdated` |
| GET | `/api/aria/feeds/runs?page=&size=` | Spring `Page<IocFeedRun>` (UI size=20) |
| POST | `/api/aria/feeds/refresh` | `202 {"status":"refresh_started"}` — all feeds, background thread |
| POST | `/api/aria/feeds/refresh/{feedName}` | `202 {"status":"refresh_started","feed":<UPPER>}`; `400` if feedName unknown |
## Feed names (exact, case-sensitive in service map)
- `OTX`, `MALWARE_BAZAAR`, `THREAT_FOX`
- `refresh/{feedName}` upper-cases input before validating.
## Tables
- `ioc_feed_run` — run history (`feed_name`, `started_at`, `completed_at`, `status`, `iocs_added`, `iocs_updated`, `error_message`). Added in Flyway **V6**.
- `threat_ioc` — IOC catalog; feeds upsert here. Match key = active + `ioc_type` + case-insensitive `value`.
## Status values
- `RUNNING` (default on insert), `SUCCESS`, `ERROR`. Frontend also renders "Never run" when no row exists.
## Config / env
- API keys: `ARIA_FEED_OTX_API_KEY`, `ARIA_FEED_MALWARE_BAZAAR_API_KEY`, `ARIA_FEED_THREAT_FOX_API_KEY`. Blank key ⇒ `configured=false`, run skips, SUCCESS with 0/0.
- Cron: `ARIA_FEED_REFRESH_CRON` (default `0 0 3 * * *`).
## Kafka
- **None** for this feature — no producer, no consumer. (ARIA topics `hiveops.aria.threats` / `hiveops.aria.sequences` are unrelated paths.)
## Gotchas
- Refresh is **async fire-and-forget** on a raw `Thread`; `202` ≠ success. Read `/status` or `/runs` for the real result.
- Role is **`BCOS_ADMIN` only** — NOT `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. `MSP_ADMIN` ⇒ 403.
- `enabled` in `/status` is **hardcoded true**; `ARIA_FEED_ENABLED` env is declared but **unused** — does not gate anything.
- `countRunning` concurrency guard is soft: single-feed refresh is `@Transactional` (RUNNING row uncommitted until done) → double-click can double-run; `refreshAll`/cron reach `refreshFeed` via self-invocation → **`@Transactional` bypassed**.
- Feed clients swallow per-request errors (`log.warn`); a run only goes ERROR if `fetchFromFeed`/`upsertIocs` throws.
- Both abuse.ch clients hard-filter for ATM relevance ⇒ 0 results is normal, not a bug.
## Do NOT
- Do NOT treat a `202` as "feed succeeded."
- Do NOT expect `MSP_ADMIN` to work — feeds are BCOS_ADMIN-gated.
- Do NOT set `ARIA_FEED_ENABLED=false` to stop auto-runs (no effect); change the cron or remove keys.
- Do NOT look for a Kafka topic between feed and catalog — the upsert is a direct DB write.
- Do NOT invent other feed names — only `OTX`, `MALWARE_BAZAAR`, `THREAT_FOX` exist.
- Do NOT expect any other service (incident/journal/mgmt/devices) to serve this data.
backend/src/main/resources/content/aria/ioc-feeds/internal.md
+65
View File
@@ -0,0 +1,65 @@
---
module: aria.ioc-feeds
title: IOC Feeds
tab: Internal
order: 20
audience: internal
---
Ops/support view for the **IOC Feeds** screen in ARIA (HiveIQ threat-intelligence service, `hiveops-aria`, port **8095**). This screen pulls known-bad indicators (IOCs) from three external sources and upserts them into ARIA's IOC catalog.
## Who can use it
- **Every feed endpoint requires `hasRole('BCOS_ADMIN')`** — not the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. A plain `MSP_ADMIN` JWT gets **403** on status, run history, and every refresh button.
- Auth is stateless JWT (`JwtAuthenticationFilter`), shared `JWT_SECRET`. No session.
- The screen self-polls the **status** endpoint every 30s. It never auto-triggers a refresh; refreshes are always manual (buttons) or the nightly cron.
## What "Run" actually does
- **Run All Feeds** → `POST /api/aria/feeds/refresh` → controller spawns a background thread `ioc-feed-manual-all` and returns `202 accepted` immediately with `{"status":"refresh_started"}`. The UI does not wait; it reloads status/history after ~3s.
- **Run Feed** (single card) → `POST /api/aria/feeds/refresh/{feedName}` → thread `ioc-feed-manual-<FEED>`, same 202.
- Because it's fire-and-forget, a **202 does not mean the feed succeeded**. Always confirm the result in the run-history table or the card's status badge, not in the button response.
- The single-card **Run Feed** button is disabled when the feed is not `configured` (no API key) — the card shows the amber *API key required* tag.
## Feeds & what makes them "configured"
| Card | feedName | Configured when set |
|------|----------|---------------------|
| AlienVault OTX | `OTX` | `ARIA_FEED_OTX_API_KEY` non-blank |
| MalwareBazaar | `MALWARE_BAZAAR` | `ARIA_FEED_MALWARE_BAZAAR_API_KEY` non-blank |
| ThreatFox | `THREAT_FOX` | `ARIA_FEED_THREAT_FOX_API_KEY` non-blank |
All three require an API key. MalwareBazaar and ThreatFox keys come from `https://auth.abuse.ch/`. With no key, a run is **not** an error — the client logs "no API key configured, skipping" and returns 0 IOCs (a SUCCESS run with 0 added / 0 updated).
## First things to check when it misbehaves
1. **Buttons/tables return 403** → the caller's JWT is not `BCOS_ADMIN`. Confirm the role claim, not just that the user is logged in.
2. **Card stuck on "API key required" / Run Feed greyed out** → the matching `ARIA_FEED_*_API_KEY` env var is unset or blank on the ARIA container. Check the deployed `.env`, redeploy.
3. **Run finishes SUCCESS but 0 added / 0 updated** → almost always a missing/blank API key (feed skipped silently) OR the source returned no ATM-relevant indicators. Both feeds filter hard for ATM relevance (see below), so 0 is common and not automatically a fault.
4. **Status ERROR with a message in the Error column** → hover the cell for the full text. It's the exception message from the fetch/upsert. Usually an external-source outage, bad/expired API key, or an HTTP/parse error. Retry with the single-card **Run Feed**.
5. **Card stuck on "Running"** → a prior run row was left with `status='RUNNING'` (never completed — e.g. container killed mid-run). New runs of that same feed will short-circuit and just return the last row (see concurrency note). Inspect `ioc_feed_run` for that `feed_name` and clear/complete the stale RUNNING row.
6. **Nothing ever runs on its own** → the scheduled refresh cron is `ARIA_FEED_REFRESH_CRON` (default `0 0 3 * * *`, 03:00 daily). Check ARIA logs at that time for `IOC feed scheduled refresh starting`.
7. **No data at all / screen empty** → check ARIA is up (`/actuator/health`) and DB `hiveiq_aria` reachable; the table reads `ioc_feed_run`, cards read the same via `/status`.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| 403 on every action | non-`BCOS_ADMIN` JWT | grant BCOS_ADMIN / use an admin token |
| "API key required" tag, Run disabled | `ARIA_FEED_*_API_KEY` blank | set env var, redeploy ARIA |
| SUCCESS, 0/0 | key missing (skipped) or no ATM-relevant hits | verify key present; 0 hits can be normal |
| ERROR row | source outage / bad key / parse error | read Error column, retry single feed |
| Card wedged "Running" | stale RUNNING row after a crash | fix the `ioc_feed_run` row for that feed |
| Feeds never auto-run | cron disabled/changed, or ARIA down at 03:00 | check `ARIA_FEED_REFRESH_CRON`, logs |
## Concurrency gotcha (know before you spam the button)
`refreshFeed` guards against overlap with `countRunning(feedName)` (counts rows where `status='RUNNING'`). If a run is already RUNNING it skips and returns the existing row. Two caveats:
- A single manual **Run Feed** runs inside one `@Transactional` method, so the RUNNING row is **not committed until the whole fetch finishes** — a second concurrent click may not see it yet and can double-run. Don't rely on the guard as a hard lock.
- **Run All** and the nightly cron call `refreshFeed` via internal self-invocation, which **bypasses the `@Transactional` proxy** entirely (only the single-card path is transactional). Behaviour differs between "Run Feed" and "Run All" — flagged in uncertainties.
## `ARIA_FEED_ENABLED` does nothing
`aria.feed.enabled` / `ARIA_FEED_ENABLED` (default true) is defined in `application.properties` but **not referenced anywhere in Java** and the `/status` endpoint hardcodes `enabled: true`. Setting it to false will **not** stop the scheduled refresh. To actually stop automatic runs, change the cron or remove the API keys. Flagged in uncertainties.
Cross-links: the customer-facing feature is on the **Overview** tab; build/data-flow detail is on the **Architect** tab.
backend/src/main/resources/content/aria/ioc-management/architect.md
+101
View File
@@ -0,0 +1,101 @@
---
module: aria.ioc-management
title: IOC Management
tab: Architect
order: 30
audience: internal
---
How the IOC watchlist is built and how it plugs into ARIA's threat pipeline. Cross-link [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Ownership
- **Single service: `hiveops-aria`** (`spring.application.name=hiveops-aria`, port **8095**, DB `hiveiq_aria`). The IOC catalog, its CRUD API, the external-feed importer, and the classifier that consumes it all live in the same service. No cross-service call is involved in IOC management itself.
- Frontend: `hiveops-aria/frontend` — `IocManagement.svelte`, calling `ariaApi` (`src/lib/api.ts`) at base `/api/aria`.
## Components for this feature
| Layer | Class | Role |
|-------|-------|------|
| Controller | `ThreatIocController` (`/api/aria/ioc`) | CRUD over the watchlist. |
| Repository | `ThreatIocRepository` (Spring Data JPA) | `findByActiveTrue`, `findByActiveTrueAndIocType`, `findByActiveTrueAndIocTypeAndValueIgnoreCase`, `findActiveExecutableIocs`. |
| Entity | `ThreatIoc` → table `threat_ioc` | The IOC record. |
| Feed importer | `IocFeedService` (+ `ioc_feed_run`) | Populates/refreshes IOCs from external feeds. |
| Consumer | `ThreatEventClassifier` | Reads active IOCs to match incoming device events. |
## Data model
Table **`threat_ioc`** (Flyway-managed, `db/migration` V1–V8; `ddl-auto=validate`):
| Column | Notes |
|--------|-------|
| `id` | PK, identity. |
| `ioc_type` | enum `FILENAME, MD5, REGISTRY_KEY, DIRECTORY, SERVICE_NAME, IP, FILE_PATH`. Immutable after create. |
| `value` | the thing being watched for (required). |
| `description` | free text. |
| `source` | enum `FBI_FLASH, FS_ISAC, MANUAL, OTX, MALWARE_BAZAAR, THREAT_FOX`. CRUD-created rows are always `MANUAL`. |
| `source_ref` | e.g. FBI FLASH number. Widened to TEXT in V8. |
| `confidence` | enum `HIGH, MEDIUM, LOW`. |
| `reported_at` | added V7; when the source reported it. |
| `added_at` | insert time (server default `Instant.now()`). |
| `expires_at` | optional expiry. |
| `active` | soft-delete flag; deactivate flips this to false. |
> Note: V1 defined Postgres enum types (`ioc_type`, `ioc_source`, `confidence_level`) but V3 (`convert_enums_to_varchar`) converted them to VARCHAR; the entity still declares `columnDefinition = "ioc_type"` etc.
## Data flow
**Write path (managing the watchlist):**
```
IocManagement.svelte ──HTTP──▶ ThreatIocController ──▶ ThreatIocRepository ──▶ threat_ioc
(INSERT / UPDATE / active=false)
```
- No Kafka on the IOC CRUD path — it is synchronous JPA.
- Parallel write path: `IocFeedService` runs on a schedule (cron `aria.feed.refresh.cron`, default `0 0 3 * * *`) pulling OTX / MalwareBazaar / ThreatFox, upserting into `threat_ioc` (dedup via `findByActiveTrueAndIocTypeAndValueIgnoreCase`) and recording each run in `ioc_feed_run`.
**Read path (how the watchlist is used):** the catalog is the matching table for ARIA's detection pipeline.
```
agent event ─▶ POST /api/aria/ingest/events ─▶ ThreatEventClassifier
│ (permitAll + X-Service-Secret)
│
├─ classifier looks up active IOCs by type+value (case-insensitive):
│ FILENAME / SERVICE_NAME / DIRECTORY / FILE_PATH / REGISTRY_KEY / IP
│
├─ match ─▶ threat_event (matched_ioc_id, severity, attack_phase) ─▶ Kafka hiveops.aria.threats
│
└─ SequenceDetectionService correlates phases within a window
─▶ precursor_sequence_alert ─▶ Kafka hiveops.aria.sequences
```
- IOC matching is **exact value, case-insensitive**, per `findByActiveTrueAndIocTypeAndValueIgnoreCase`. No wildcard/substring matching. `findActiveExecutableIocs` narrows to `FILENAME, MD5, FILE_PATH`.
- Matching applies only to **newly ingested** events; editing the watchlist is not retroactive.
## Kafka (context, not on the CRUD path)
Both topics are produced by ARIA (`AriaKafkaConfig`, JSON), downstream of an IOC match — see [platform.kafka]:
| Topic | Producer | Trigger |
|-------|----------|---------|
| `hiveops.aria.threats` | `AriaThreatProducer` | classified/matched threat event (key = deviceAgentId). |
| `hiveops.aria.sequences` | `AriaSequenceProducer` | a precursor sequence trips. |
ARIA has **no `@KafkaListener`** — it only produces. Auto-response fleet actions are off by default (`aria.autoresponse.shutdown.enabled=false`); when enabled they call `hiveops-incident`.
## Key API endpoints (this feature)
| Method | Path | Auth | Purpose |
|--------|------|------|---------|
| GET | `/api/aria/ioc?type=&activeOnly=` | `BCOS_ADMIN` | List (default `activeOnly=true`). |
| GET | `/api/aria/ioc/active?type=` | authenticated | Active IOCs, no role gate. |
| POST | `/api/aria/ioc` | `BCOS_ADMIN` | Create (`source` forced MANUAL). |
| PUT | `/api/aria/ioc/{id}` | `BCOS_ADMIN` | Update value/description/confidence/expires only. |
| DELETE | `/api/aria/ioc/{id}` | `BCOS_ADMIN` | Soft-deactivate (`active=false`). |
## Design notes / caveats
- **Soft-delete by design** — HTTP DELETE flips `active`, preserving audit/history; the classifier only reads `active = true`.
- **BCOS_ADMIN-only** across the board here — divergent from the platform-wide `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. See [platform.service-ownership].
- **PUT drops `sourceRef`** — the update handler never persists `request.getSourceRef()` (create does). Flagged as a real bug; noted in uncertainties.
backend/src/main/resources/content/aria/ioc-management/claude.md
+59
View File
@@ -0,0 +1,59 @@
---
module: aria.ioc-management
title: IOC Management
tab: Claude
order: 40
audience: internal
---
Terse act-without-guessing reference. Owning service: **`hiveops-aria`** (port 8095, DB `hiveiq_aria`). Frontend: `hiveops-aria/frontend/src/components/IocManagement.svelte` via `ariaApi` → base `/api/aria`.
## Endpoints (`ThreatIocController`)
| METHOD | Path | Auth | Notes |
|--------|------|------|-------|
| GET | `/api/aria/ioc` | `hasRole('BCOS_ADMIN')` | params `type` (enum), `activeOnly` (default `true`). |
| GET | `/api/aria/ioc/active` | authenticated (no role) | param `type`; only `active=true`. |
| POST | `/api/aria/ioc` | `hasRole('BCOS_ADMIN')` | body `CreateIocRequest`; `source` forced `MANUAL`. |
| PUT | `/api/aria/ioc/{id}` | `hasRole('BCOS_ADMIN')` | updates value/description/confidence/expiresAt ONLY. 404 if id missing. |
| DELETE | `/api/aria/ioc/{id}` | `hasRole('BCOS_ADMIN')` | soft-deactivate (`active=false`), returns 204. NOT a hard delete. |
`CreateIocRequest`: `iocType`*(NotNull), `value`*(NotBlank), `description`, `sourceRef`, `confidence`, `reportedAt`, `expiresAt`. (* = required)
## Storage
- **Table:** `threat_ioc` (only table for this feature).
- **Columns:** `id, ioc_type, value, description, source, source_ref, confidence, reported_at, added_at, expires_at, active`.
- **Enums:** `ioc_type` = FILENAME, MD5, REGISTRY_KEY, DIRECTORY, SERVICE_NAME, IP, FILE_PATH · `source` = FBI_FLASH, FS_ISAC, MANUAL, OTX, MALWARE_BAZAAR, THREAT_FOX · `confidence` = HIGH, MEDIUM, LOW.
- Flyway V1–V8; `ddl-auto=validate`.
## Kafka (NOT on CRUD path — downstream of IOC match only)
- Produces `hiveops.aria.threats` (`AriaThreatProducer`), `hiveops.aria.sequences` (`AriaSequenceProducer`).
- No consumers in ARIA (no `@KafkaListener`).
## Consumers of the catalog
- `ThreatEventClassifier` — matches ingested events to active IOCs via `findByActiveTrueAndIocTypeAndValueIgnoreCase` (exact value, case-insensitive) for FILENAME/SERVICE_NAME/DIRECTORY/FILE_PATH/REGISTRY_KEY/IP.
- `IocFeedService` — writes feed-sourced IOCs; dedups by active type+value; logs to `ioc_feed_run`.
## Gotchas
- Role is `BCOS_ADMIN` ONLY here — NOT `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. MSP_ADMIN → 403.
- DELETE = soft deactivate; no hard-delete endpoint exists.
- `ioc_type` immutable after create (UI disables it; backend update ignores it).
- **BUG:** `update()` never reads `request.getSourceRef()` → editing Source Reference is a silent no-op. Create persists it; PUT does not.
- POST always sets `source=MANUAL` regardless of payload.
- `confidence` defaults to `MEDIUM` server-side if null on create (UI sends `HIGH`).
- Matching is exact + case-insensitive; no wildcard/substring. Not retroactive to past events.
- Frontend `getIocs` filters by type client-side; it sends only `activeOnly` to the API.
- DB is on `10.10.10.188` (`hiveiq_aria`), not the services VM. Health: `GET /actuator/health` on 8095.
## Do NOT
- Do NOT expect DELETE to remove a row — it only sets `active=false`.
- Do NOT expect PUT to change `sourceRef` or `iocType`.
- Do NOT assume MSP_ADMIN can access — BCOS_ADMIN required.
- Do NOT add Kafka handling for IOC CRUD — it is synchronous JPA.
- Do NOT hand-write IOC rows into feed sources (`FBI_FLASH`/`FS_ISAC`/`OTX`/`MALWARE_BAZAAR`/`THREAT_FOX`) via this API — POST forces `MANUAL`; feed rows come from `IocFeedService`/seed migrations.
- Do NOT invent other IOC endpoints/tables/topics — the above is the complete surface for this module.
backend/src/main/resources/content/aria/ioc-management/internal.md
+49
View File
@@ -0,0 +1,49 @@
---
module: aria.ioc-management
title: IOC Management
tab: Internal
order: 20
audience: internal
---
Ops/support/admin view of the ARIA IOC watchlist. The screen is a thin CRUD over the `threat_ioc` table in `hiveiq_aria`, served by `hiveops-aria` (port 8095) via `ThreatIocController` (`/api/aria/ioc`). Editing the watchlist directly changes what the threat-classification pipeline matches device events against.
## Who can use it
- **`BCOS_ADMIN` only.** Every list/create/update/deactivate endpoint is gated `@PreAuthorize("hasRole('BCOS_ADMIN')")`. This is **not** the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` pattern — an MSP_ADMIN JWT will get **403** on this screen. If a customer/admin reports "IOC page is empty / won't load," check their role first.
- `GET /api/aria/ioc/active` is the one exception — no role gate, any authenticated caller can read active IOCs. It is used by internal consumers, not this UI.
## What the buttons actually do
| UI action | Endpoint | Effect |
|-----------|----------|--------|
| **+ Add IOC** | `POST /api/aria/ioc` | Inserts a row with `source = MANUAL` (always forced server-side, regardless of what you type). |
| **Edit → Update** | `PUT /api/aria/ioc/{id}` | Updates **value, description, confidence, expires_at only**. |
| **Deactivate** | `DELETE /api/aria/ioc/{id}` | **Soft** delete — sets `active = false`, returns 204. Row is never removed. |
- **Deactivate is not delete.** There is no hard-delete endpoint. Deactivated IOCs stay in the table and reappear under **Show inactive**. To "remove" a bad entry, deactivate it.
- IOC **Type cannot be changed** after creation — the form disables the select on edit, and the backend `update` ignores type. If the type is wrong, deactivate and re-add.
## First things to check when it misbehaves
1. **403 / blank list** → caller lacks `BCOS_ADMIN`. Decode the JWT `role` claim.
2. **"Save Failed" toast on add/edit** → validation: `iocType` (`@NotNull`) and `value` (`@NotBlank`) are required. The UI also blocks empty type/value client-side, so a server-side failure usually means a bad enum value for `iocType`/`confidence` or the service being down.
3. **Edited "Source Reference" didn't stick** → **known bug** (see below), not user error.
4. **New IOC not triggering detections** → confirm it is `active = true` and the `value` matches exactly (matching is case-insensitive but otherwise exact — see [architect]). Matching only happens for incoming events processed by `ThreatEventClassifier`; it is not retroactive to past events.
5. **Service reachability** → `curl http://localhost:8095/actuator/health` on the services VM; DB is on `10.10.10.188` (`hiveiq_aria`), not the services VM.
## Common failure modes + fixes
- **Duplicate-looking IOCs** — the CRUD endpoints do **not** de-duplicate. Only the external-feed importer (`IocFeedService`) checks for an existing active IOC of the same type+value. Two identical MANUAL entries are allowed; deactivate the extra.
- **Wrong confidence default** — if `confidence` is omitted on create the server defaults it to `MEDIUM`; the UI always sends `HIGH`. If IOCs show up as MEDIUM unexpectedly, something is calling the API directly without the field.
- **Feed-sourced IOCs you can't explain** — rows with `source` of `FBI_FLASH`, `FS_ISAC`, `OTX`, `MALWARE_BAZAAR`, or `THREAT_FOX` came from the seed migration or the scheduled feed refresh, not a person. Manage feed behavior on the **IOC Feeds** screen, not here.
- **Seed data** — `FILENAME` IOCs like the Ploutus jackpotting set were seeded by Flyway `V2` (`FLASH-20260219-001`). Deactivating them is safe and reversible.
## Known code bug (grounded, unfixed)
- `ThreatIocController.update()` sets value/description/confidence/expiresAt but **never reads `request.getSourceRef()`**. The Edit panel exposes a **Source Reference** field and the frontend sends it, but PUT silently ignores it. Editing Source Reference on an existing IOC is a no-op. Workaround: deactivate and re-create with the correct source reference. (Create honors `sourceRef`; update does not.)
## Related internal screens
- **IOC Feeds** — external feed status/refresh (`IocFeedService`, `ioc_feed_run` table).
- See [architect] for how the watchlist feeds the classification + precursor-sequence pipeline, and [claude] for the terse endpoint/table reference.
backend/src/main/resources/content/aria/precursor-alerts/architect.md
+71
View File
@@ -0,0 +1,71 @@
---
module: aria.precursor-alerts
title: Precursor Alerts
tab: Architect
order: 30
audience: internal
---
How the **Precursor Alerts** feature is built. Everything lives in **hiveops-aria** (port 8095, DB `hiveiq_aria`). ARIA owns detection end-to-end — it does not read another service's tables to build alerts. See [platform.service-ownership].
## Services involved
- **hiveops-aria** — the only service in the write path. Ingests threat events, classifies them, detects multi-phase sequences, persists alerts, serves this screen, and emits a Kafka event downstream.
- **Agents / internal callers** — POST threat events into ARIA ingestion (out of scope of this screen but the source of all data).
- **Downstream consumers** (potential) — anything subscribing to `hiveops.aria.sequences`. ARIA itself has **no Kafka consumers** for this feature; the topic is fire-and-forget.
## Data flow
```
agent/internal → POST /api/aria/ingest/events
→ ThreatEventIngestionService.ingest() [@Transactional]
→ ThreatEventClassifier.classify() (severity + attack_phase + IOC match)
→ save ThreatEvent (threat_event)
→ if severity HIGH/CRITICAL: AriaThreatProducer → topic hiveops.aria.threats
→ SequenceDetectionService.detectAndAlert(event)
→ query recent events for device within window
→ detect JACKPOTTING / SKIMMING / (null)
→ dedup guard (one active alert per device)
→ stamp events with sequence UUID, save
→ save PrecursorSequenceAlert (precursor_sequence_alert)
→ AriaSequenceProducer → topic hiveops.aria.sequences
```
This is the **ingest → executor(detect) → persist → Kafka** pattern: the ingestion service is the executor, detection persists the record, then publishes an event to the record-store topic for downstream fan-out. See [platform.kafka], [platform.data-architecture].
### Detection logic (`SequenceDetectionService`)
- Window: `aria.sequence.window.minutes` (`ARIA_SEQUENCE_WINDOW_MINUTES`, default **15**). Pulls all `threat_event` rows for the device with `detected_at` after `now - window`.
- Requires **≥2 recent events** and a **`PHYSICAL_ACCESS`** attack phase, else returns null (no alert).
- **JACKPOTTING** if `MALWARE_STAGING` or `MALWARE_EXECUTION` phase present. Confidence = weighted sum of phases (`PHYSICAL_ACCESS` .25, `MALWARE_STAGING` .30, `MALWARE_EXECUTION` .25, `PERSISTENCE` .10, `CLEANUP` .05) + .10 if any CRITICAL event + .05 if any IOC match, capped at 1.0, 3-dp.
- **SKIMMING** if signal `CARD_READER_TAMPER` or `USB_INSERTION` present. Base .35 + .40 (tamper) + .20 (USB) + .05 (critical), capped 1.0.
- **Dedup:** `existsByDeviceAgentIdAndResolvedAtIsNull` — a device with an unresolved alert gets no new alert until the existing one is resolved.
- On detect: all correlated events get the same `sequence_id` (UUID) written back, then the alert row is saved with `phasesDetected[]`, `confidence`, `institutionKey`, `sequenceId`.
## Kafka
| Topic | Direction | Producer | Key | When |
|-------|-----------|----------|-----|------|
| `hiveops.aria.sequences` | produced | `AriaSequenceProducer.publishSequenceAlert` | `deviceAgentId` | on every raised sequence alert |
| `hiveops.aria.threats` | produced | `AriaThreatProducer.publishThreatEvent` | `deviceAgentId` | per HIGH/CRITICAL ingested event (feeds context, not alerts) |
No `@KafkaListener` consumes these in ARIA. Payloads are JSON (`AriaSequenceEvent`, `AriaThreatEvent`). `AriaSequenceEvent.alertId` is a freshly generated UUID (not the DB row id). See [platform.kafka].
## Database (`hiveiq_aria`, Flyway-managed, `ddl-auto=validate`)
| Table | Role in this feature | Read | Write |
|-------|----------------------|------|-------|
| `threat_event` | source signals; also linked to a sequence via `sequence_id` | detection query; events panel | `sequence_id` stamped on correlate |
| `precursor_sequence_alert` | the alert record shown on this screen | list / resolve | created on detect; `resolved_at`/`resolved_by` on resolve |
`precursor_sequence_alert` columns: `device_id`, `device_agent_id`, `sequence_type`, `phases_detected[]`, `confidence` (precision 4 scale 3), `triggered_at`, `auto_response_taken`, `resolved_at`, `resolved_by`, `institution_key`, `sequence_id`. See [platform.data-architecture].
## API endpoints (`PrecursorSequenceAlertController`, all `hasRole('BCOS_ADMIN')`)
| Method | Path | Purpose |
|--------|------|---------|
| GET | `/api/aria/sequences` | paged list; params `resolved`, `sequenceType`, `deviceAgentId`, `page`, `size` (default size 25); sorted `triggeredAt` desc |
| GET | `/api/aria/sequences/{id}/events` | threat events sharing the alert's `sequence_id` |
| POST | `/api/aria/sequences/{id}/resolve` | idempotent resolve; stamps `resolved_at` + `resolved_by` (JWT email) |
Note the repository query precedence in `getSequences`: a non-blank `deviceAgentId` **overrides** the `resolved` and `sequenceType` filters (device query returns all statuses for that device). Only when no device filter is set do the resolved/type branches apply.
backend/src/main/resources/content/aria/precursor-alerts/claude.md
+53
View File
@@ -0,0 +1,53 @@
---
module: aria.precursor-alerts
title: Precursor Alerts
tab: Claude
order: 40
audience: internal
---
Terse agent reference. Owned by **hiveops-aria** (port 8095, DB `hiveiq_aria`). All facts below confirmed in code.
## Ownership
- Service: `hiveops-aria` — sole owner of detection + storage + API for this feature.
- Frontend: `hiveops-aria/frontend/src/components/PrecursorAlerts.svelte`, view id `sequences`.
## Endpoints (all `@PreAuthorize("hasRole('BCOS_ADMIN')")`)
| METHOD | Path | Notes |
|--------|------|-------|
| GET | `/api/aria/sequences` | params: `resolved` (bool), `sequenceType` (JACKPOTTING\|SKIMMING\|UNKNOWN), `deviceAgentId`, `page` (0), `size` (25); Page<> sorted `triggeredAt` desc |
| GET | `/api/aria/sequences/{id}/events` | `{id}` = alert Long id; returns `List<ThreatEvent>` by `sequence_id`; 404 if alert missing |
| POST | `/api/aria/sequences/{id}/resolve` | idempotent; sets `resolved_at`+`resolved_by`(JWT email); no un-resolve endpoint |
| POST | `/api/aria/ingest/events` | upstream source; `permitAll`, gated by header `X-Service-Secret` vs `ARIA_SERVICE_SECRET` (open if blank) |
## Tables (`hiveiq_aria`)
- `precursor_sequence_alert` — the alert record.
- `threat_event` — source signals; `sequence_id` links events to an alert.
## Kafka (produced only; NO consumers in aria)
- `hiveops.aria.sequences` — one per raised alert, key=`deviceAgentId`, payload `AriaSequenceEvent` (JSON).
- `hiveops.aria.threats` — per HIGH/CRITICAL ingested event, key=`deviceAgentId`.
## Roles
- BCOS_ADMIN **only** for all sequence endpoints. NOT `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. MSP_ADMIN → 403.
## Detection facts
- Window `ARIA_SEQUENCE_WINDOW_MINUTES` default 15.
- Needs ≥2 device events in window AND `PHYSICAL_ACCESS` phase, else no alert.
- JACKPOTTING = has `MALWARE_STAGING` or `MALWARE_EXECUTION`. SKIMMING = signal `CARD_READER_TAMPER` or `USB_INSERTION`.
- One active (unresolved) alert per device — dedup guard blocks new alerts until resolved.
## Gotchas
- `deviceAgentId` filter overrides `resolved`+`sequenceType` (returns all statuses for that device).
- `AriaSequenceEvent.alertId` = random UUID, NOT the DB alert id; use `sequenceId`/`deviceAgentId` to correlate.
- Alerts are auto-generated on ingestion; no manual create endpoint.
- `resolved_by` = `"unknown"` if JWT principal email absent.
- Auto-refresh is client-side 60s poll (`localStorage aria_alerts_autoRefresh`); does not affect backend.
## Do NOT
- Do NOT call these with an MSP_ADMIN token expecting data (403).
- Do NOT expect a re-open/un-resolve endpoint — none exists.
- Do NOT assume alerts appear without a `PHYSICAL_ACCESS` phase and ≥2 correlated events.
- Do NOT look for these tables/topics in incident/journal/devices — everything is in `hiveops-aria` / `hiveiq_aria`.
- Do NOT treat `AriaSequenceEvent.alertId` as a foreign key to `precursor_sequence_alert`.
- Do NOT invent an admin "force alert" or "auto-response" action from this screen (auto-response shutdown is off by default).
backend/src/main/resources/content/aria/precursor-alerts/internal.md
+51
View File
@@ -0,0 +1,51 @@
---
module: aria.precursor-alerts
title: Precursor Alerts
tab: Internal
order: 20
audience: internal
---
Ops/support view of the ARIA **Precursor Alerts** screen (multi-phase attack-sequence detection: jackpotting / skimming precursors). Served by **hiveops-aria** only. Nothing here touches incident/journal/devices data directly.
## Who can use it
- **All three endpoints require `hasRole('BCOS_ADMIN')`** — not the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. An MSP_ADMIN JWT will get **403** on this screen even though they can use other apps. This is the #1 "I can't see any alerts / it's empty for me" cause.
- Auth is a stateless JWT (ARIA's own `JwtAuthenticationFilter`). `resolvedBy` is stamped from the caller's JWT email; if the principal is missing it records `"unknown"`.
## What an operator does here
- **Review active alerts** — list defaults to Active (`resolved=false`), newest-triggered first.
- **Filter** by Device ID, Status (Active / Resolved / All), Type (Jackpotting / Skimming / Unknown).
- **Open a row** → side panel lists the individual threat events that make up the sequence (calls `/sequences/{id}/events`).
- **Resolve** a row or hit Mark Resolved in the panel. Resolve is idempotent — resolving an already-resolved alert just returns it unchanged.
- Auto-refresh polls the list every **60s** (client-side toggle, persisted in `localStorage` key `aria_alerts_autoRefresh`).
## Admin-only actions
- **Resolve alert:** `POST /api/aria/sequences/{id}/resolve` (BCOS_ADMIN). Sets `resolved_at`/`resolved_by`. There is **no un-resolve / re-open endpoint** — once resolved it stays resolved.
- Alerts themselves are **generated automatically** by the detection engine on event ingestion. There is no manual "create alert" action and no admin knob in the UI to force one.
## First things to check when it misbehaves
1. **"Empty list / 403"** → confirm the JWT role is **BCOS_ADMIN**. MSP_ADMIN is not enough here.
2. **"No alerts are ever raised despite threat events"** — walk the pipeline:
- Are events actually arriving? Check `POST /api/aria/ingest/events` (ThreatEvent ingestion). If ingestion is gated, the `X-Service-Secret` header must match `ARIA_SERVICE_SECRET`; **if that env var is blank, ingestion accepts anything** — so a silent-drop is more likely an empty request than auth.
- A sequence needs **≥2 threat events on the same device within the detection window** (`ARIA_SEQUENCE_WINDOW_MINUTES`, default **15 min**). Sparse/slow events won't correlate.
- Detection **requires a `PHYSICAL_ACCESS` attack phase** to be present. Without it, no jackpotting/skimming sequence is ever raised — this is by design, not a bug.
- Classification assigns the `attack_phase` and `severity`; if the classifier isn't tagging phases (bad/missing IOC catalog), sequences won't form. Check the IOC catalog (`GET /api/aria/ioc`).
3. **"Only one alert per device, second attack not showing"** — expected. There is a dedup guard: while a device has an **unresolved** alert, no new alert is created for it. Resolve the existing one to allow the next.
4. **"Resolved-by shows `unknown`"** → JWT reached the endpoint without a populated principal email; check the token/claims.
5. **"Alerts stopped updating on screen"** → auto-refresh toggle is off, or the 60s poll is failing silently; hit the manual ↻ button and watch the network call to `GET /api/aria/sequences`.
6. **Events panel empty ("No events tagged to this sequence yet")** → the alert's `sequence_id` has no matching rows in `threat_event`. Events are stamped with the sequence UUID at detection time; if they were purged/re-ingested the linkage is lost.
## Data locations (for support triage)
- DB: **`hiveiq_aria`** (Postgres on the database VM, ProxyJump). Tables: `precursor_sequence_alert`, `threat_event`.
- Logs: search for `ARIA SEQUENCE ALERT` (warn on every raised sequence) and `ARIA event ingested` in Loki. Sequence-detection failures log `Sequence detection failed for device ...`.
## Config that changes behavior
- `ARIA_SEQUENCE_WINDOW_MINUTES` (default 15) — correlation window.
- `ARIA_SERVICE_SECRET` — ingestion gate (blank = open).
- Auto-response (shutdown) is **off by default** (`aria.autoresponse.shutdown.enabled=false`); resolving here is manual triage, not an automated containment action.
backend/src/main/resources/content/aria/threat-events/architect.md
+88
View File
@@ -0,0 +1,88 @@
---
module: aria.threat-events
title: Threat Events
tab: Architect
order: 30
audience: internal
---
How the **Threat Events** feature is built. It is owned end-to-end by **`hiveops-aria`** (the threat-intelligence service, `spring.application.name=hiveops-aria`, DB `hiveiq_aria`). The screen is a thin read view; the interesting work happens in the ingest → classify → store → (optionally) publish pipeline. Cross-link [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Services involved
- **`hiveops-aria` (owner)** — serves the read API the screen calls, owns ingestion, classification, persistence, and Kafka production for this feature. No other service is required to render Threat Events.
- **Agents / internal callers (producers of data)** — POST batched signals to ARIA's ingest endpoint. They are the *source* of `threat_event` rows.
- **hiveops-incident (downstream consumer, optional)** — a potential consumer of the `hiveops.aria.threats` topic for auto-response; ARIA itself has **no Kafka consumers** for this feature.
## Data flow
```
Agent/service ──POST /api/aria/ingest/events──▶ ThreatEventIngestionController
(X-Service-Secret header, permitAll) │
▼
ThreatEventIngestionService.ingest()
per item:
ThreatEventClassifier.classify()
→ severity, matchedIoc, attackPhase
build ThreatEvent
eventRepository.save() ──▶ threat_event (hiveiq_aria)
if severity ∈ {CRITICAL, HIGH}:
AriaThreatProducer.publishThreatEvent()
──▶ Kafka topic hiveops.aria.threats
(key = deviceAgentId)
SequenceDetectionService.detectAndAlert(event)
→ may emit hiveops.aria.sequences
Browser (BCOS_ADMIN) ──GET /api/aria/events──▶ ThreatEventController.getEvents()
→ threat_event (paged, detectedAt DESC)
```
This is the standard **executor → record-store → Kafka** pattern: the ingestion service is the executor, `threat_event` is the record store, and `hiveops.aria.threats` is the fan-out for downstream reaction. Cross-link [platform.data-architecture].
## Classification
`ThreatEventClassifier.classify(item)` returns a `ClassificationResult(severity, matchedIoc, attackPhase)`:
- **IOC match** — signal is matched against active rows in `threat_ioc`; a hit sets `matched_ioc_id` (the **IOC Match** column).
- **severity** — one of `CRITICAL / HIGH / MEDIUM / LOW / INFO`.
- **attackPhase** — one of `PHYSICAL_ACCESS`, `MALWARE_STAGING`, `MALWARE_EXECUTION`, `PERSISTENCE`, `CLEANUP` (the UI relabels these to Physical Access / Staging / Execution / Persistence / Cleanup).
- `eventType` is parsed from the request; unknown values fall back to `OS_EVENT`.
- Classification happens **once, at ingest**. Rows are not re-classified later.
## DB tables (in `hiveiq_aria`, Flyway-managed, `ddl-auto=validate`)
| Table | Read/Written by this feature |
|-------|------------------------------|
| `threat_event` | **Written** by ingestion (`eventRepository.save`); **read** by the screen (`GET /api/aria/events`, `/stats`). Columns include `device_agent_id`, `device_id`, `signal_key`, `event_type`, `severity`, `matched_ioc_id`, `attack_phase`, `sequence_id`, `institution_key`, `raw_payload` (JSONB), `detected_at`. |
| `threat_ioc` | **Read** during classification to resolve `matched_ioc_id`; surfaced as the IOC Match badge (type + value). |
| `precursor_sequence_alert` | **Written** by `SequenceDetectionService` when an event completes a multi-phase sequence (adjacent feature, not this screen). |
Repository queries used by the read path (`ThreatEventRepository`): `findAllByOrderByDetectedAtDesc`, `findBySeverityOrderByDetectedAtDesc`, `findByDeviceAgentIdOrderByDetectedAtDesc` — all paged, sort `detectedAt DESC`.
## Kafka (produced only)
Cross-link [platform.kafka]. Topics defined in `AriaKafkaConfig` (JSON serializer):
| Topic | Producer | When | Key | Payload |
|-------|----------|------|-----|---------|
| `hiveops.aria.threats` | `AriaThreatProducer.publishThreatEvent` | ingest of a **CRITICAL or HIGH** event only | `deviceAgentId` (or `"unknown"`) | `AriaThreatEvent` (eventId, deviceId, deviceAgentId, institutionKey, signalKey, eventType, severity, description, matchedIocValue/Type, attackPhase, sequenceId, detectedAt, sourceService=`hiveops-aria`) |
| `hiveops.aria.sequences` | `AriaSequenceProducer` (via `SequenceDetectionService`) | a precursor sequence trips | — | sequence alert |
**MEDIUM / LOW / INFO events are stored but never published** (`shouldPublish` = CRITICAL||HIGH). ARIA has **no `@KafkaListener`** for this feature — it is a pure producer here.
## Key API endpoints (this feature)
| Method | Path | Auth | Role |
|--------|------|------|------|
| GET | `/api/aria/events` | JWT | `hasRole('BCOS_ADMIN')` |
| GET | `/api/aria/stats` | JWT | `hasRole('BCOS_ADMIN')` |
| POST | `/api/aria/ingest/events` | `X-Service-Secret` header | `permitAll` (secret checked only if `ARIA_SERVICE_SECRET` non-blank) |
`GET /api/aria/events` params: `severity?`, `deviceAgentId?`, `page` (0), `size` (50). Returns a Spring `Page<ThreatEvent>`.
## Frontend
`hiveops-aria/frontend/src/components/ThreatEvents.svelte` → `ariaApi.getEvents` in `src/lib/api.ts`. Base URL from `window.__APP_CONFIG__.apiUrl` (dev default `http://localhost:8017`, `/aria` suffix trimmed for the gateway var). Client-side 60s auto-refresh; filters map straight to the query params above.
## Design note (verified)
`getEvents` evaluates `severity` before `deviceAgentId` and returns early — so the two filters are **mutually exclusive** at the backend even though the UI can send both. If combined severity+device filtering is a requirement, the controller needs a combined repository query (`findBySeverityAndDeviceAgentId…`). See Internal/Claude tabs.
backend/src/main/resources/content/aria/threat-events/claude.md
+56
View File
@@ -0,0 +1,56 @@
---
module: aria.threat-events
title: Threat Events
tab: Claude
order: 40
audience: internal
---
Terse act-without-guessing reference. Owner service: **`hiveops-aria`** (`hiveiq_aria` DB, port 8095). All facts below confirmed in `hiveops-aria` source.
## Ownership
- **Read + ingest + classify + store + produce: `hiveops-aria` only.** No other service needed to render this screen.
- No Kafka **consumers** in aria for this feature — pure producer.
## Endpoints (METHOD path — role)
| Method | Path | Auth |
|--------|------|------|
| GET | `/api/aria/events` | `hasRole('BCOS_ADMIN')` |
| GET | `/api/aria/stats` | `hasRole('BCOS_ADMIN')` |
| POST | `/api/aria/ingest/events` | `permitAll` + `X-Service-Secret` header |
| GET | `/api/aria/ioc` | `hasRole('BCOS_ADMIN')` (IOC catalog behind the IOC Match column) |
- `GET /api/aria/events` params: `severity?` (CRITICAL/HIGH/MEDIUM/LOW/INFO), `deviceAgentId?`, `page` (def 0), `size` (def 50). Returns `Page<ThreatEvent>`, sort `detectedAt DESC`.
- Ingest body: `ThreatEventIngestionRequest` (deviceId, deviceAgentId, institutionKey, `events[]` of `ThreatEventItem{signalKey, eventType, rawPayload, detectedAt}`). Returns 202.
## Tables (DB `hiveiq_aria`)
- `threat_event` — the rows this screen lists. Cols: `device_agent_id`, `device_id`, `signal_key`, `event_type`, `severity`, `matched_ioc_id`→`threat_ioc`, `attack_phase`, `sequence_id`, `institution_key`, `raw_payload` (JSONB), `detected_at`.
- `threat_ioc` — IOC catalog; source of the IOC Match badge.
- `precursor_sequence_alert` — sequence alerts (sibling feature).
## Kafka topics
- Produce `hiveops.aria.threats` — key = `deviceAgentId` (else `"unknown"`). **Only CRITICAL/HIGH events published.** Payload `AriaThreatEvent`, `sourceService="hiveops-aria"`.
- Produce `hiveops.aria.sequences` — sequence alerts.
- Consume: **none.**
## Roles
- Read path is **`BCOS_ADMIN` only** — NOT `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. MSP_ADMIN → 403.
- Ingest gated by `X-Service-Secret` == `ARIA_SERVICE_SECRET`, **only when that env var is non-blank**; blank ⇒ open.
## Gotchas
- MSP_ADMIN gets 403 on this screen — by design.
- `getEvents` checks `severity` first and returns early → **`deviceAgentId` is ignored whenever `severity` is also set.** Filters are effectively mutually exclusive server-side.
- MEDIUM/LOW/INFO are stored but never hit Kafka — downstream services will not see them.
- Classification (severity/IOC/phase) is done once at ingest by `ThreatEventClassifier`; rows are never re-classified.
- Unknown `eventType` on ingest silently falls back to `OS_EVENT`.
- Attack-phase enums: `PHYSICAL_ACCESS`, `MALWARE_STAGING`, `MALWARE_EXECUTION`, `PERSISTENCE`, `CLEANUP` (UI relabels).
- Auto-refresh is client-side (60s), `localStorage` key `aria_threats_autoRefresh`. Staleness ⇒ ingest problem, not UI.
- Blank `ARIA_SERVICE_SECRET` ⇒ ingest accepts anything (no auth).
## Do NOT
- Do NOT tell an MSP_ADMIN they should see this screen — they can't (BCOS_ADMIN gate).
- Do NOT expect a MEDIUM/LOW/INFO event on `hiveops.aria.threats`.
- Do NOT assume filtering by device + severity together works — it doesn't.
- Do NOT look for the data in another service — `hiveops-aria` / `threat_event` owns it exclusively.
- Do NOT re-classify or mutate historical rows to "fix" a label; fix the classifier/IOC catalog and re-ingest.
- Do NOT confuse `/api/aria/ingest/events` (write, service-secret) with `/api/aria/events` (read, BCOS_ADMIN JWT).
backend/src/main/resources/content/aria/threat-events/internal.md
+56
View File
@@ -0,0 +1,56 @@
---
module: aria.threat-events
title: Threat Events
tab: Internal
order: 20
audience: internal
---
Ops/support/admin view of the ARIA **Threat Events** screen. This is a read-only list rendered from the `hiveops-aria` service (`ThreatEvents.svelte`); there are no per-row admin actions on this screen — the only controls are filters, pagination, and auto-refresh. Grounded in `hiveops-aria` source.
## Who can see it
- The screen calls **`GET /api/aria/events`**, which is gated **`@PreAuthorize("hasRole('BCOS_ADMIN')")`**.
- ⚠️ **BCOS_ADMIN only** — this is *not* the usual `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`. An MSP_ADMIN JWT will get **403** and the list will show the "Could not load threat events" toast. If a customer/MSP admin reports an empty or failing screen, this is expected — they are not authorized. Same gate applies to the stats endpoint (`GET /api/aria/stats`).
## What feeds the list
- Rows come from the `threat_event` table in the **`hiveiq_aria`** database, newest first (`ORDER BY detectedAt DESC`).
- Rows are written by the **ingestion pipeline**, not by anyone using this screen. Agents/services POST batches to **`POST /api/aria/ingest/events`**; `ThreatEventIngestionService` classifies each item (severity, matched IOC, attack phase) and saves a `threat_event` row.
- The **IOC Match** column is populated only when the classifier matched the signal against an active row in `threat_ioc`. No match → `—`.
## First things to check when it misbehaves
1. **403 / "Could not load threat events"** — confirm the caller's JWT role is `BCOS_ADMIN`. MSP_ADMIN is rejected.
2. **Screen loads but is empty** — check whether events are actually being ingested:
- Are agents/services POSTing to `/api/aria/ingest/events`? Look for `ARIA event ingested: device=… signal=… severity=… phase=…` INFO logs in `hiveops-aria`.
- `401` on ingest means the `X-Service-Secret` header did not match `aria.internal.service-secret` (`ARIA_SERVICE_SECRET`). Look for `Rejected ingestion request from … — invalid service secret` WARN logs.
- ⚠️ If `ARIA_SERVICE_SECRET` is **blank/unset**, the ingest endpoint accepts *any* request (no auth). A blank secret means the header is never checked.
3. **Device filter seems ignored** — see "Known quirk" below. Searching a Device (Agent) ID **while a severity is also selected** returns severity-filtered results ignoring the device. Clear the severity filter to search by device.
4. **IOC Match always empty** — the IOC catalog may be empty/inactive. IOCs live in `threat_ioc`; check via `GET /api/aria/ioc` (also BCOS_ADMIN). If the feed refresh hasn't run or seed migration (V2, FBI FLASH) is missing, nothing will match.
5. **List looks stale** — auto-refresh runs client-side every **60s** (toggle persisted in `localStorage` key `aria_threats_autoRefresh`). The `↻` button forces a reload. Staleness is a data/ingest problem, not a refresh problem — the UI just re-calls `GET /api/aria/events`.
6. **Wrong severity/phase on a row** — classification is done at ingest time by `ThreatEventClassifier`; rows are not re-classified. A bad label means the classifier or the IOC catalog was wrong when the event arrived.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| 403 loading events | Caller is MSP_ADMIN, not BCOS_ADMIN | Use a BCOS_ADMIN account/JWT |
| Empty list, no ingest logs | Agents not posting, or wrong service URL | Verify agent/service target + `ARIA_SERVICE_SECRET` match |
| Ingest returns 401 | `X-Service-Secret` mismatch | Align caller header with `ARIA_SERVICE_SECRET` |
| Ingest accepts anything | `ARIA_SERVICE_SECRET` unset/blank | Set the secret in the ARIA env |
| IOC Match column all `—` | No active IOCs matched | Check `threat_ioc` / feed refresh (`GET /api/aria/feeds/status`) |
| Device search returns severity results | Severity + device combined (see quirk) | Clear severity, then search device |
| Downstream (incident, etc.) missing an event | Only CRITICAL/HIGH are published to Kafka | Expected — MEDIUM/LOW/INFO are stored but not published |
## Known quirk (verified in code)
`ThreatEventController.getEvents` checks `severity` **before** `deviceAgentId` and returns early. So when the UI sends both (severity dropdown + device search active at once), the **device filter is silently ignored**. Workaround: filter by only one at a time. Track/report as a backend bug if it causes support confusion.
## Related admin surfaces (same service, BCOS_ADMIN)
- `GET /api/aria/stats` — dashboard counters (total events, critical/high 24h, active IOCs, active sequences, low-posture/EOL devices).
- `GET /api/aria/ioc`, `POST/PUT/DELETE /api/aria/ioc` — IOC catalog management.
- `GET /api/aria/sequences` — precursor-sequence alerts (multi-phase attack chains).
See the **Architect** and **Claude** tabs for data flow and exact endpoint/table/topic references.
backend/src/main/resources/content/claims/claim-detail/architect.md
+71
View File
@@ -0,0 +1,71 @@
---
module: claims.claim-detail
title: Claim Detail
tab: Architect
order: 30
audience: internal
---
How the **Claim Detail** panel is built end-to-end. See [platform.service-ownership] — this feature is owned entirely by **hiveops-claims**; no other HiveOps service is called for claim, message, note, attachment, or payment data. The only external dependency is the device mirror (below).
## Services involved
| Piece | What it is | Role for this feature |
|-------|-----------|-----------------------|
| `hiveiq-claims-frontend` | Svelte SPA (port 5179, `claims.bcos.cloud`) | Renders `ClaimPanel.svelte`; one component, five tabs |
| `hiveiq-claims` | Spring Boot backend (port 8092, DB `hiveiq_claims`) | Serves all panel data + mutations |
| SMTP (`JavaMailSender`) | outbound mail | Vendor notifications on message send / attachment upload |
| devices service (upstream) | publishes `hiveops.device.events` | Feeds `device_summary` read-model (terminal/location/ATM list) |
There is **no** executor→Kafka→record-store pattern here (that's the fleet-task shape). Claims is a straight synchronous REST + JPA CRUD service; the only Kafka involvement is the inbound device mirror. See [platform.kafka].
## Component → controller wiring
`ClaimPanel.svelte` takes a `claimUuid` prop and:
1. **On mount** — `Promise.all([ claimsAPI.get(uuid), vendorAPI.list() ])`
- `GET /api/claims/{uuid}` → `ClaimController.getClaim` → `ClaimService.getClaim` → returns `ClaimSummaryResponse` with `payments` **embedded** (so the Payment tab needs no extra call).
- `GET /api/claims/vendors` → `VendorController` (used for Edit dropdowns + Messages recipient list).
2. **Tabs lazy-load on first open** (via `ClaimActivityController`, base `/api/claims/{claimUuid}`):
- Messages → `GET /messages?vendorUuid=` (optional filter)
- Notes → `GET /notes`
- Files → `GET /attachments`
## Data flow — mutations
All mutations are synchronous REST → `ClaimService` / `ClaimActivityService` → JPA repositories.
- **Overview → Save** — `PUT /api/claims/{uuid}` body `{status, claimType, citVendorUuid|clearCitVendor, spVendorUuid|clearSpVendor}` → `ClaimService.updateClaim`. Sets status/type; assigning/clearing a vendor adds or deletes the matching `claim_payments` row (no re-split).
- **Messages → Send** — `POST /messages` `{messageText, vendorUuid?, additionalEmail?}` → `ClaimActivityService.sendMessage`: persists `claim_messages` row then calls `EmailService.sendMessageNotification` (recipient = vendor `contactEmail`, else `additionalEmail`; silently skipped if neither resolves or SMTP unconfigured). Sender party is derived server-side; UI renders `fromParty == INSTITUTION` on the right.
- **Notes → Add** — `POST /notes` `{noteText}` → `claim_notes`. Institution-only, no email, never shown to vendors.
- **Files → Upload** — `POST /attachments` multipart `file` → writes bytes to `UPLOADS_DIR` (default `/tmp/claims-uploads`), persists `claim_attachments` row, then `EmailService.sendAttachmentNotification` to assigned CIT/SP vendors. **Download** streams back via `GET /attachments/{attachmentUuid}/download`.
- **Payment → Save card** — `PUT /payments/{party}` `{amountOwed, amountPaid}` (`party` ∈ `INSTITUTION|CIT|SP`) → updates one `claim_payments` row.
## DB tables (read/written by this feature) — DB `hiveiq_claims`
| Table | Read | Written | By |
|-------|------|---------|-----|
| `claims` | ✔ | ✔ (PUT) | get/update claim |
| `claim_payments` | ✔ (embedded) | ✔ | payment edit, vendor assign/clear |
| `claim_messages` | ✔ | ✔ | Messages tab |
| `claim_notes` | ✔ | ✔ | Notes tab |
| `claim_attachments` | ✔ | ✔ | Files tab (+ bytes on local disk) |
| `vendors` | ✔ | — | Edit + Messages dropdowns |
| `email_templates` | ✔ | — | resolves vendor-notification subject/body |
| `device_summary` | ✔ | — (Kafka-fed) | ATM/terminal metadata |
See [platform.data-architecture]. `dev` profile uses H2 (`create-drop`, Flyway off); `prod` uses PostgreSQL with Flyway `validate`.
## Kafka topics + consumers
- **Consumed:** `hiveops.device.events` — `DeviceEventConsumer` (`@KafkaListener`, group `hiveops-claims-device-sync`, constants in `DeviceEventKafkaConfig`). Upserts/deletes `device_summary`; `DEVICE_DELETED` removes the row; missing `institutionKey` defaults to `"BCOS"`. This is what backs the terminal/location shown on the panel.
- **Produced:** **none.** Claim state changes do not emit events. Downstream services do not learn about claim/payment updates.
Claims is one of the `device_summary` mirror consumers (incident/fleet/reports/recon/claims); there is **no reconciler**, so a missed publish leaves the mirror stale silently. See [platform.kafka].
## Security model (this feature)
- JWT-authentication-only; `ClaimsPrincipal(userId, email, role, institutionKey)`. No role checks.
- **List** scopes by `institutionKey`; **detail + activity endpoints resolve by UUID only** (no institution scoping) — see the Internal tab and uncertainties for the resulting cross-institution exposure.
For the terse endpoint/table/topic map, see [claims.claim-detail] Claude tab.
backend/src/main/resources/content/claims/claim-detail/claude.md
+67
View File
@@ -0,0 +1,67 @@
---
module: claims.claim-detail
title: Claim Detail
tab: Claude
order: 40
audience: internal
---
Terse agent reference for the **Claim Detail** panel (`ClaimPanel.svelte`).
## Ownership
- **Service:** `hiveops-claims` OWNS all of this — claims, payments, messages, notes, attachments. No other service serves panel data.
- **Backend:** `hiveiq-claims`, port 8092, base path `/api/claims`.
- **External route:** `api.bcos.cloud/claims/` → `hiveiq-claims:8092/api/claims/`.
- **Frontend:** `hiveiq-claims-frontend`, `claims.bcos.cloud`, port 5179, component `ClaimPanel.svelte`.
- **DB:** `hiveiq_claims` (Postgres prod on `10.10.10.188`; H2 in `dev`).
## Endpoints (METHOD path) — all require JWT; `{uuid}`/`{claimUuid}` = claim UUID
| Method | Path | Use |
|--------|------|-----|
| GET | `/api/claims/{uuid}` | Load claim + embedded `payments` |
| PUT | `/api/claims/{uuid}` | Body `{status,claimType,citVendorUuid\|clearCitVendor,spVendorUuid\|clearSpVendor}` |
| GET | `/api/claims/vendors` | Edit + message dropdowns (`?type=CIT\|SP`) |
| GET | `/api/claims/{claimUuid}/messages` | `?vendorUuid=` optional filter |
| POST | `/api/claims/{claimUuid}/messages` | `{messageText,vendorUuid?,additionalEmail?}` → sends SMTP email |
| GET | `/api/claims/{claimUuid}/notes` | internal notes |
| POST | `/api/claims/{claimUuid}/notes` | `{noteText}` |
| GET | `/api/claims/{claimUuid}/attachments` | list |
| POST | `/api/claims/{claimUuid}/attachments` | multipart field `file` → sends SMTP email |
| GET | `/api/claims/{claimUuid}/attachments/{attachmentUuid}/download` | stream file |
| GET | `/api/claims/{claimUuid}/payments` | (panel uses embedded instead) |
| PUT | `/api/claims/{claimUuid}/payments/{party}` | `party`∈`INSTITUTION\|CIT\|SP`; body `{amountOwed,amountPaid}` |
## Tables (`hiveiq_claims`)
- `claims`, `claim_payments`, `claim_messages`, `claim_notes`, `claim_attachments`, `vendors`, `email_templates`, `device_summary`.
## Kafka
- **Consume:** `hiveops.device.events` (group `hiveops-claims-device-sync`) → `device_summary`.
- **Produce:** none. Claim/payment changes emit no events.
## Enums
- `ClaimStatus`: `OPEN`, `WAITING_CIT`, `WAITING_SP`, `WAITING_INSTITUTION`, `PENDING_PAYMENT`, `RESOLVED_PAID`, `RESOLVED_OFFAGE_FOUND`, `RESOLVED_DENIED`.
- `ClaimType`: `WITHDRAW`, `DEPOSIT`. Payment `Party`: `INSTITUTION`, `CIT`, `SP`.
## Roles
- **None.** `SecurityConfig` = `anyRequest().authenticated()`. No `hasAnyRole`. Isolation = `institutionKey` from JWT (`ClaimsPrincipal`), enforced in service layer for LIST only.
## Gotchas
- `GET/PUT /api/claims/{uuid}` and every `/{claimUuid}/…` activity endpoint resolve by UUID with **no `institutionKey` check** → cross-institution read/write (IDOR). Only `GET /api/claims` (list) is scoped.
- Panel `PUT` sends `clearCitVendor/clearSpVendor=true` when a vendor select is blank → saving Edit can delete a vendor + its payment row.
- Assigning a vendor later adds its `claim_payments` row at `amountOwed=0` — no re-split.
- Message/attachment emails go via **SMTP (`JavaMailSender`)**, not Kafka; silently skipped if no recipient resolves or SMTP unconfigured (record still saves).
- Message/notes/attachments tabs swallow load errors (empty `catch {}`) — no toast; check logs.
- Attachments stored on local disk `UPLOADS_DIR` (default `/tmp/claims-uploads`); lost if not a volume. Cap 50MB/file, 52MB/request.
- List filter params are `status`,`claimNumber`,`terminalId` (NOT `search`).
- `dev` = H2 create-drop, Flyway off; `prod` = Postgres validate.
## Do NOT
- Do NOT assume role gates exist — there are none; do not tell users an action is "admin-only."
- Do NOT expose claim UUIDs as safe identifiers — they bypass institution scoping.
- Do NOT expect a Kafka event after a claim/payment change — none is produced.
- Do NOT look in incident/mgmt/devices for claim data — it's all in `hiveops-claims`.
- Do NOT trust `device_summary` freshness — no reconciler; verify against the devices service.
- Do NOT call `/api/internal/**` for panel work — it's email-template only, `X-Internal-Secret`-gated.
See [claims.claim-detail] Internal + Architect tabs. [platform.service-ownership] · [platform.kafka] · [platform.data-architecture]
backend/src/main/resources/content/claims/claim-detail/internal.md
+53
View File
@@ -0,0 +1,53 @@
---
module: claims.claim-detail
title: Claim Detail
tab: Internal
order: 20
audience: internal
---
Ops/support reference for the **Claim Detail** panel (`ClaimPanel.svelte`, opens right-side over the claims list). Backend is **hiveops-claims** (`hiveiq-claims`, port 8092, DB `hiveiq_claims`). Frontend `claims.bcos.cloud`; API `api.bcos.cloud/claims/` → `hiveiq-claims:8092/api/claims/`.
## Who can do what
- **No role guards.** `SecurityConfig` is `anyRequest().authenticated()` — there is **no** `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` anywhere in this service. Any user with a valid HiveIQ JWT can open, edit, message, note, upload, and change payments on the panel.
- **Isolation is by `institutionKey`** carried in the JWT (`ClaimsPrincipal`). The *list* (`GET /api/claims`) filters to your institution — but the *detail* endpoints below do **not** (see failure modes). There is no admin-vs-user distinction inside claims.
- **Internal email-template endpoints** (`/api/internal/**`) are `permitAll()` at security level but gated by an `X-Internal-Secret` header (env `INTERNAL_SERVICE_SECRET`, default empty). Not used by this panel.
## What each tab calls (first things to check)
| Tab | Action | Call |
|-----|--------|------|
| (open) | Load claim + payments + vendor list | `GET /api/claims/{uuid}`, `GET /api/claims/vendors` |
| Overview | Save Edit | `PUT /api/claims/{uuid}` |
| Messages | Load / Send | `GET /api/claims/{uuid}/messages?vendorUuid=`, `POST …/messages` |
| Files | List / Upload / Download | `GET/POST …/attachments`, `GET …/attachments/{aUuid}/download` |
| 🔒 Notes | List / Add | `GET/POST …/notes` |
| Payment | (embedded) / Save card | payments come inside the claim; `PUT …/payments/{party}` |
`{party}` is `INSTITUTION` / `CIT` / `SP`.
## First things to check when it misbehaves
1. **Panel opens blank / "Could not load claim details"** — the open path fires `GET /api/claims/{uuid}` **and** `GET /api/claims/vendors` in parallel; if either 401/500s the whole panel shows the load-failed toast. Check the JWT (expired → 401) and that the vendors endpoint is healthy first.
2. **Messages/Files/Notes empty but you expect data** — these tabs lazy-load only on first open and only when the local array is empty. A failed load is swallowed silently (empty `catch {}`) — no toast. Check the network tab / `docker logs hiveiq-claims`, don't trust the empty state.
3. **Vendor email didn't arrive after sending a message or uploading a file** — messages and attachments trigger `EmailService` (SMTP `JavaMailSender`), **not** Kafka. If `mailSender` is null / SMTP unconfigured, or no recipient can be resolved, the email is **silently skipped** and the message/note/file still saves. Check `mail.from` / SMTP env and that the vendor has a `contactEmail` (or that an additional recipient email was supplied).
4. **CIT/SP vendor dropdown is empty in Edit** — depends on `GET /api/claims/vendors` returning vendors of that `type`; register vendors first (Vendors page). The Messages "to vendor" dropdown only lists vendors **assigned to this claim** (`citVendorUuid`/`spVendorUuid`).
5. **Download does nothing** — `GET …/attachments/{attachmentUuid}/download` streams from the local uploads dir (`UPLOADS_DIR`, default `/tmp/claims-uploads`). If the container was recreated and that dir isn't a persistent volume, the file rows exist but the bytes are gone.
6. **Terminal/location/ATM list looks stale or wrong** — device data (`device_summary`) is a Kafka mirror fed by `hiveops.device.events`. There is **no reconciler**; if the devices service didn't publish a change, claims shows stale data. Cross-check the devices service.
## Common failure modes + fixes
- **Cross-institution data visible / editable (IDOR).** `GET/PUT /api/claims/{uuid}` and all `/{claimUuid}/…` activity endpoints resolve by UUID with **no** `institutionKey` check. Anyone with a claim UUID from any institution can read/modify it. Treat claim UUIDs as sensitive; flagged as a real bug (see uncertainties). No config toggle — needs a code fix.
- **Save Edit clears a vendor unexpectedly.** The panel sends `clearCitVendor: true` / `clearSpVendor: true` whenever its dropdown is blank. Opening Edit and saving with a vendor field showing "— None —" **removes** that party (and deletes its payment row). Don't save the Overview edit unless the vendor selects are correct.
- **Payment balance looks wrong after assigning a vendor.** Assigning a CIT/SP vendor later adds that party's payment row at `amountOwed = 0` — it does **not** re-split the shortage. Adjust owed amounts manually on the Payment tab.
- **50MB upload rejected.** Multipart cap is 50MB/file, 52MB/request. Larger files 413.
- **Attachments lost after redeploy** — confirm `UPLOADS_DIR` is a mounted volume, not the container-local default.
## Where to look
- Logs: `docker logs hiveiq-claims` (or Loki, `localhost:3100` on services VM).
- Health: `GET /actuator/health` (permitAll).
- DB (separate VM `10.10.10.188` via ProxyJump): tables `claims`, `claim_payments`, `claim_messages`, `claim_notes`, `claim_attachments`, `vendors`, `device_summary`.
See [claims.claim-detail] Architect tab for how the pieces fit together.
backend/src/main/resources/content/claims/dashboard/architect.md
+68
View File
@@ -0,0 +1,68 @@
---
module: claims.dashboard
title: Claims Dashboard
tab: Architect
order: 30
audience: internal
---
How the **Claims Dashboard** is built. It is a thin read view over one service — **hiveops-claims** owns all the data it shows. See [platform.service-ownership].
## Components involved
| Component | Role for this view |
|-----------|-------------------|
| `hiveiq-claims-frontend` (Svelte SPA, port 5179, `claims.bcos.cloud`) | `Dashboard.svelte` renders stat cards + table; polls every 60s |
| `hiveiq-claims` backend (Spring Boot, port 8092) | `ClaimController` → `ClaimService` → JPA repositories |
| PostgreSQL `hiveiq_claims` | Source of truth for claims + payments |
| Kafka `hiveops.device.events` | Feeds the `device_summary` mirror used by the *File New Claim* ATM picker (not the dashboard table itself) |
The dashboard does **not** call any other microservice. Unlike the `device_summary` mirror consumers, its own table/stat data is fully local to hiveops-claims.
## Data flow
```
Dashboard.svelte ──GET /api/claims/stats──▶ ClaimController.getStats
└─GET /api/claims?…──────▶ ClaimController.listClaims
│
▼
ClaimService (@Transactional readOnly)
│
▼
ClaimRepository (JPA) ──▶ hiveiq_claims.claims (+ claim_payments)
```
- `JwtAuthenticationFilter` populates a `ClaimsPrincipal` record (`userId, email, role, institutionKey`) from the Bearer token. Both controller methods read `principal.institutionKey()` and pass it down as the scope — data isolation is enforced in the service/repo layer, not by role annotations.
- **List** (`GET /api/claims`): `ClaimService.listClaims` picks one of two `@Query` repository methods:
- no status filter → `findFilteredNoStatus(institutionKey, claimNumber, terminalId, pageable)`
- status(es) present → parses the CSV into `List<Claim.ClaimStatus>` and calls `findFilteredWithStatuses(...)`.
Both apply case-insensitive `LIKE %…%` on `claimNumber` and `terminalId`, return a `Page<Claim>`, and map to `ClaimSummaryResponse` (includes CIT/SP vendor uuid+name, `hasUpdate`, `submittedAt`, `updatedAt`).
- **Stats** (`GET /api/claims/stats`): `ClaimService.getStats` loads `findByInstitutionKey(institutionKey)` (all claims) and computes in Java: active count (any non-`RESOLVED_*`), deposit/withdraw exposure (sum of `shortageAmount` over active), awaiting (`OPEN`/`WAITING_*`), deposit/withdraw recovered (sum of `claim_payments.amount_paid`), and `recoveryPercent = totalRecovered / totalShortage * 100` (`HALF_UP`). Returned as `ClaimStatsResponse`.
## DB tables
| Table | Read | Written | Notes |
|-------|:----:|:-------:|-------|
| `claims` | ✓ | — | Dashboard is read-only; writes happen via *File New Claim* (`POST /api/claims`) |
| `claim_payments` | ✓ | — | `amount_paid` drives the Recovered cards; `amount_owed` set at claim creation |
| `vendors` | ✓ | — | Joined for CIT/SP vendor names in the table |
| `device_summary` | ✓ | — | Only for the ATM picker (`GET /api/claims/atms`), not the dashboard grid |
No table is written by the dashboard itself. See [platform.data-architecture].
## Kafka
The dashboard read path produces and consumes **nothing**. The service as a whole consumes `hiveops.device.events` (`DeviceEventConsumer`, group `hiveops-claims-device-sync`) to keep `device_summary` in sync — this is the standard mirror pattern shared by incident/fleet/reports/recon/claims and only surfaces in the claim-creation ATM dropdown, not this view. There is no executor→Kafka→record-store write pattern here; claim/stat state is committed synchronously to Postgres. See [platform.kafka].
## Key API endpoints (this view)
| METHOD | Path | Returns |
|--------|------|---------|
| GET | `/api/claims/stats` | `ClaimStatsResponse` |
| GET | `/api/claims?status=&claimNumber=&terminalId=&page=&size=&sort=` | `Page<ClaimSummaryResponse>` |
Externally routed as `api.bcos.cloud/claims/…` → `hiveiq-claims:8092/api/claims/…`.
## Refresh model
Client-side polling only: `Dashboard.svelte` runs a 1s ticker; every 60s (when Auto-refresh is on) it fires `loadStats()` + `loadClaims()` in parallel. Toggle state persists in `localStorage['claims_autoRefresh']`. No websockets, no SSE, no server push.
backend/src/main/resources/content/claims/dashboard/claude.md
+61
View File
@@ -0,0 +1,61 @@
---
module: claims.dashboard
title: Claims Dashboard
tab: Claude
order: 40
audience: internal
---
Terse agent reference for the Claims Dashboard view. Owner service: **hiveops-claims** (`hiveiq-claims`, port 8092, image `hiveiq-claims`). Frontend: `Dashboard.svelte` in `hiveops-claims/frontend`.
## Owns this view
- Backend: `hiveops-claims` only. No cross-service call in the dashboard read path.
- DB: `hiveiq_claims` (PostgreSQL prod; H2 in dev profile).
- External route: `api.bcos.cloud/claims/…` → `hiveiq-claims:8092/api/claims/…`.
## Endpoints (this view)
| METHOD | Path | Notes |
|--------|------|-------|
| GET | `/api/claims/stats` | → `ClaimStatsResponse`; scoped by JWT `institutionKey` |
| GET | `/api/claims` | `Page<ClaimSummaryResponse>`; params below |
Query params for `GET /api/claims`:
- `status` — CSV of enum values (`OPEN,WAITING_CIT,…`); unknown value = 500
- `claimNumber` — case-insensitive substring
- `terminalId` — case-insensitive substring
- `page`, `size` (default 25), `sort` (default `createdAt`; e.g. `shortageAmount,asc`)
Sortable fields: `claimNumber`, `terminalId`, `lossStart`, `shortageAmount`, `status`, `submittedAt`, `updatedAt`, `createdAt`.
## Enums
- `ClaimStatus`: `OPEN`, `WAITING_CIT`, `WAITING_SP`, `WAITING_INSTITUTION`, `PENDING_PAYMENT`, `RESOLVED_PAID`, `RESOLVED_OFFAGE_FOUND`, `RESOLVED_DENIED`
- `ClaimType`: `DEPOSIT`, `WITHDRAW`
## Tables
- `claims` (read) · `claim_payments` (read; `amount_paid`, `amount_owed`) · `vendors` (read, join for names) · `device_summary` (read, only for `GET /api/claims/atms`, not the grid)
## Topics
- Dashboard read path: **none produced, none consumed.**
- Service-wide: consumes `hiveops.device.events` (group `hiveops-claims-device-sync`) → `device_summary`. Affects ATM picker only.
## Auth / roles
- JWT Bearer required; `anyRequest().authenticated()`.
- **No `hasAnyRole`** anywhere. Scope = `institutionKey` from `ClaimsPrincipal`.
- `permitAll`: `/actuator/health`, `/actuator/info`, swagger, `/h2-console/**`, `/error`, `/api/internal/**`.
## Gotchas
- `depositRecoveredYtd` / `withdrawRecoveredYtd` are **all-time**, not year-scoped (name is misleading — no year filter in `getStats`).
- `awaitingResponse` counts `OPEN` + all `WAITING_*` active claims (includes fresh OPEN).
- Exposure cards sum full `shortageAmount` of active claims — not remaining owed.
- `getStats` loads all institution claims and computes in-memory (no SQL aggregate).
- Filtering is two params (`claimNumber`, `terminalId`), **not** a single `search` param (older docs are stale).
- Stats errors are swallowed client-side (`catch {}`) → cards silently vanish; curl to see real error.
- DB env is split form `DB_HOST`/`DB_PORT`/`DB_NAME`/`DB_USERNAME`/`DB_PASSWORD` — **not** `DB_URL`.
## Do NOT
- Do NOT assume role gates protect claims data — only `institutionKey` scoping does.
- Do NOT add a claims endpoint/table to another service; hiveops-claims owns claims. See [platform.service-ownership].
- Do NOT trust "YTD" stat labels as year-bounded.
- Do NOT use a `search=` param on `/api/claims` — it is ignored.
- Do NOT expect Kafka/websocket push for the dashboard; refresh is 60s client polling.
- Do NOT pass unlisted `sort` fields or unknown `status` values — the latter 500s.
backend/src/main/resources/content/claims/dashboard/internal.md
+48
View File
@@ -0,0 +1,48 @@
---
module: claims.dashboard
title: Claims Dashboard
tab: Internal
order: 20
audience: internal
---
Ops/support view for the HiveIQ **Claims Dashboard** — the landing view of the `claims.bcos.cloud` SPA. It renders six stat cards plus a filtered, paginated, sortable claims table. All data comes from **hiveops-claims** (`hiveiq-claims`, port 8092) via two calls: `GET /api/claims/stats` and `GET /api/claims`.
## Who can see what
- **No role guards.** `SecurityConfig` ends in `anyRequest().authenticated()` — there is **no** `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` check anywhere in this service. Any authenticated user with a valid HiveIQ JWT can load the dashboard.
- **Isolation is by `institutionKey`, not role.** Both endpoints read `principal.institutionKey()` from the JWT and scope every query to it. A user only ever sees their own institution's claims. There is no "see all institutions" admin path in the dashboard (a null `institutionKey` would return everything, but real tokens always carry one).
- No admin-only buttons on this view. The only mutating action reachable from the dashboard is **+ File New Claim** (opens a separate form → `POST /api/claims`).
## First things to check when it misbehaves
1. **Blank table / "Load failed" toast** — `GET /api/claims` failed. Check the JWT is present and valid (401 = auth filter rejected it), then check `hiveiq-claims` is up (`docker compose ps`, `curl localhost:8092/actuator/health`).
2. **Stat cards missing entirely** — `GET /api/claims/stats` failed silently. The frontend swallows stats errors (`catch {}`), so the cards just don't render while the table may still load. Curl `/api/claims/stats` directly with a real token to see the error.
3. **"Wrong institution's claims" / empty for a user who should have data** — check the `institutionKey` claim in their JWT. Wrong or missing key = wrong scope. This is a token problem, not a claims-service problem.
4. **ATM dropdown empty in File New Claim** (adjacent to dashboard) — that list comes from the `device_summary` mirror table, kept in sync by the `hiveops.device.events` Kafka consumer. If devices didn't publish, the mirror is stale. Not a dashboard-table issue.
5. **Numbers look wrong** — see "Known gotchas" below before filing a bug; several are by-design or naming quirks.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| Table empty, no error | Filters still applied (status pills / Claim # / Terminal ID) | Click **✕ Clear all** in the Filters sidebar |
| 401 on both calls | Expired/absent JWT | Re-login; confirm `VITE_AUTH_URL` reachable |
| 403 / CORS error in console | Origin not in `CORS_ALLOWED_ORIGINS` | Ensure `claims.bcos.cloud` (or `localhost:5179` in dev) is allowed |
| Cards render but table 500s | `hiveiq_claims` DB unreachable / schema drift | Check `DB_HOST/DB_PORT/DB_NAME` env; prod uses Flyway `validate` |
| Stale/short claim list for whole institution | DB connectivity, not mirror | Mirror only affects the ATM picker, not the claims table |
## Search & filter behavior (so you can reproduce a report)
- **Status pills** are multi-select; the frontend joins them CSV and sends `?status=OPEN,WAITING_CIT,…`. Values must match the enum exactly (`OPEN`, `WAITING_CIT`, `WAITING_SP`, `WAITING_INSTITUTION`, `PENDING_PAYMENT`, `RESOLVED_PAID`, `RESOLVED_OFFAGE_FOUND`, `RESOLVED_DENIED`) — an unknown value throws `IllegalArgumentException` (500).
- **Claim #** and **Terminal ID** are separate params (`?claimNumber=`, `?terminalId=`), each a case-insensitive substring `LIKE %…%` match. There is **no** single free-text `search` param despite older docs mentioning one.
- **Sorting** is server-side via Spring `sort=field,dir`. Default `createdAt,desc`. Sortable columns: `claimNumber`, `terminalId`, `lossStart`, `shortageAmount`, `status`, `submittedAt`, `updatedAt` (and default `createdAt`). Location, Type, and the two Vendor columns are **not** sortable.
- **Pagination** default page size is 25 (backend `@PageableDefault`); the frontend overrides with the user's saved page size.
- **Amber left stripe** on a row = `hasUpdate = true` on that claim (`Claim.getHasUpdate()`), surfacing a new vendor/party update.
## Known gotchas (verify before filing a "bug")
- **"Recovered YTD" is not year-scoped.** `depositRecoveredYtd` / `withdrawRecoveredYtd` sum `amountPaid` across **all** of the institution's claims with no year filter — the "YTD" label is a misnomer, the value is all-time recovered. (See uncertainties.)
- **"Awaiting Response" counts more than truly-waiting claims.** It counts active claims in `OPEN`, `WAITING_CIT`, `WAITING_SP`, or `WAITING_INSTITUTION` — including brand-new `OPEN` claims nobody has been asked to respond to yet.
- **Exposure = shortage, not remaining owed.** `depositExposure` / `withdrawExposure` sum the full `shortageAmount` of active (non-resolved) claims; they do not subtract amounts already paid.
- **Stats are computed in-memory.** `getStats` loads every claim for the institution and streams them in Java — fine at current volumes, but it is O(all claims) per dashboard load, not a SQL aggregate.
backend/src/main/resources/content/claims/settings/architect.md
+75
View File
@@ -0,0 +1,75 @@
---
module: claims.settings
title: Settings
tab: Architect
order: 30
audience: internal
---
How the Claims **Settings** (vendor email templates) feature is built. Single-service, synchronous, DB-backed — **no Kafka, no executor/record-store pattern** in this feature. See [platform.service-ownership].
## Services involved
- **`hiveops-claims`** owns this feature end to end (backend image `hiveiq-claims:8092`, frontend `hiveiq-claims-frontend:5179`). No other service participates.
- No cross-service calls, no Kafka producer/consumer for templates. (The wider claims service *does* consume `hiveops.device.events` into `device_summary`, but that is unrelated to Settings — see [platform.kafka].)
## Data flow — reading the effective template
```
SettingsPage.svelte
└─ EmailTemplatesSection.svelte (onMount → emailTemplateAPI.effective())
└─ GET api.bcos.cloud/claims/settings/email-templates/effective [JWT]
└─ EmailTemplateController.effective(ClaimsPrincipal)
└─ EmailTemplateRepository.findByInstitutionKeyAndIsDefaultTrue(institutionKey)
.or( findByInstitutionKeyAndIsDefaultTrue("__GLOBAL__") )
└─ 200 {name, subject, global} or 204 No Content → UI fallback
```
The `global` boolean in the response is computed as `"__GLOBAL__".equals(template.institutionKey)` so the UI can label an inherited default.
## Data flow — using the template (send path)
The template is consumed at message-send time, not just displayed:
```
POST /api/claims/{claimUuid}/messages
└─ ClaimActivityService.addMessage(...) (persists claim_messages row)
└─ EmailService.sendMessageNotification(claim, vendor, text, sender, additionalEmail)
└─ EmailTemplateRepository.findByInstitutionKeyAndIsDefaultTrue(claim.institutionKey)
├─ present → resolvePlaceholders(subject/bodyHtml)
└─ empty → __GLOBAL__ default → else hardcoded fallback body
└─ JavaMailSender.send(MimeMessage) (skipped if mail bean not configured)
```
Placeholders substituted in subject + body (HTML-escaped): `{{claimNumber}}`, `{{terminalId}}`, `{{location}}`, `{{shortageAmount}}`, `{{claimStatus}}`, `{{vendorName}}`, `{{messageText}}`, `{{senderEmail}}`. Attachment-notification emails (`sendAttachmentNotification`) use a **hardcoded** body, not the template.
## Write path & the "single default" invariant
- `EmailTemplateService.create/update`: when the request sets `isDefault=true`, it first runs the `@Modifying` query `clearDefaultForInstitution(institutionKey)` (UPDATE ... SET is_default=false WHERE institution_key=:key), then saves the new/updated row as default. This keeps at most one default per institution, which `findByInstitutionKeyAndIsDefaultTrue` (returns `Optional`) relies on.
- `update`/`delete` enforce ownership: `institutionKey.equals(template.institutionKey)` or throw 403; missing `uuid` → 404.
## Tables read/written
| Table | R/W | By |
|-------|-----|----|
| `email_templates` (DB `hiveiq_claims`) | R | `effective`, `list`, send path |
| `email_templates` | W | `create`, `update`, `delete`, `clearDefaultForInstitution` |
Schema from `V2__email_templates.sql`: `id BIGSERIAL`, `uuid UUID UNIQUE`, `institution_key VARCHAR(255)` (indexed), `name`, `subject VARCHAR(500)`, `body_html TEXT`, `is_default BOOLEAN`, `created_at`, `updated_at`. The special `institution_key = '__GLOBAL__'` row is the shared inherited default. See [platform.data-architecture].
## API endpoints (all served by hiveops-claims)
| METHOD | Path | Auth | Notes |
|--------|------|------|-------|
| GET | `/api/claims/settings/email-templates/effective` | JWT | 200 `{name,subject,global}` or 204 |
| GET | `/api/claims/settings/email-templates` | JWT | list for caller's institution |
| POST | `/api/claims/settings/email-templates` | JWT | 201; body `{name,subject,bodyHtml,isDefault}` |
| PUT | `/api/claims/settings/email-templates/{uuid}` | JWT | ownership-checked |
| DELETE | `/api/claims/settings/email-templates/{uuid}` | JWT | 204; ownership-checked |
| GET/POST/PUT/DELETE | `/api/internal/email-templates` | `X-Internal-Secret` + `?institutionKey=` | `permitAll` at security layer, secret-gated in controller; used for `__GLOBAL__` / cross-institution |
## Security model
`SecurityConfig` permits `/actuator/health|info`, swagger, `/h2-console/**`, `/error`, `/api/internal/**`, then `anyRequest().authenticated()`. **No role-based guards** — isolation is by `institutionKey` from the JWT (`JwtAuthenticationFilter` → `ClaimsPrincipal`). The `/api/internal/**` path is publicly routable but every method rejects unless `X-Internal-Secret` matches the non-blank `internal.service-secret` (`INTERNAL_SERVICE_SECRET`).
Cross-links: [platform.data-architecture] · [platform.kafka] · [platform.service-ownership]
backend/src/main/resources/content/claims/settings/claude.md
+61
View File
@@ -0,0 +1,61 @@
---
module: claims.settings
title: Settings
tab: Claude
order: 40
audience: internal
---
Terse reference. Feature = **vendor email templates** shown on the Claims Settings screen (read-only display of the *effective* template).
## Ownership
- **Owner service:** `hiveops-claims` (image `hiveiq-claims`, port 8092, DB `hiveiq_claims`). No other service. No Kafka in this feature.
- External route: `api.bcos.cloud/claims/...` → `hiveiq-claims:8092/api/claims/...`
- Frontend view: `hiveops-claims/frontend/src/components/Settings/EmailTemplatesSection.svelte` (calls `effective` only).
## Endpoints (METHOD path)
| METHOD | Path | Auth | Result |
|--------|------|------|--------|
| GET | `/api/claims/settings/email-templates/effective` | JWT | 200 `{name,subject,global}` / **204** if none |
| GET | `/api/claims/settings/email-templates` | JWT | list (own institution) |
| POST | `/api/claims/settings/email-templates` | JWT | 201; `{name,subject,bodyHtml,isDefault}` |
| PUT | `/api/claims/settings/email-templates/{uuid}` | JWT | 403 if not owner, 404 if uuid unknown |
| DELETE | `/api/claims/settings/email-templates/{uuid}` | JWT | 204 |
| GET/POST/PUT/DELETE | `/api/internal/email-templates?institutionKey=<key>` | header `X-Internal-Secret` | cross-institution / `__GLOBAL__` |
## Table
- `email_templates` (DB `hiveiq_claims`, migration `V2__email_templates.sql`)
- Cols: `id`, `uuid`, `institution_key`, `name`, `subject`, `body_html`, `is_default`, `created_at`, `updated_at`
- Global inherited default row: `institution_key = '__GLOBAL__'`, `is_default = true`
## Topics
- None for this feature. (Service consumes `hiveops.device.events` for `device_summary` only — unrelated.)
## Roles / auth
- **No `hasAnyRole` anywhere.** `SecurityConfig` → `anyRequest().authenticated()`. Any valid JWT works.
- Scope = `institutionKey` from JWT (`ClaimsPrincipal`). Write ops enforce `institutionKey == template.institution_key` else 403.
- `/api/internal/**` = `permitAll` in security, but each method requires `X-Internal-Secret` == `INTERNAL_SERVICE_SECRET` (blank default ⇒ all rejected).
## Resolution logic (effective + send path, identical)
1. own institution default (`is_default=true`) →
2. `__GLOBAL__` default →
3. none → 204 (UI) / hardcoded fallback body (email send)
## Template placeholders (subject + body_html, HTML-escaped)
`{{claimNumber}}` `{{terminalId}}` `{{location}}` `{{shortageAmount}}` `{{claimStatus}}` `{{vendorName}}` `{{messageText}}` `{{senderEmail}}`
## Gotchas
- `{uuid}` path var is the `uuid` column, NOT numeric `id`.
- Only ONE default per institution — setting `isDefault=true` auto-clears others (`clearDefaultForInstitution`).
- Settings SPA is **read-only**: `list/create/update/delete` exist in `api.ts` + backend but are NOT wired to any UI. Manage templates via API only.
- Template used by `EmailService.sendMessageNotification` (message emails). **Attachment** emails use a hardcoded body, not the template.
- Email send silently skips if `MAIL_USERNAME`/`MAIL_PASSWORD` unset (no `JavaMailSender` bean); look for DEBUG "Mail not configured".
- Dev profile = H2 create-drop, Flyway off → templates vanish on restart.
- `MAIL_FROM` default `noreply@bcos.cloud`; `EmailService` uses `noreply@hiveops.com` only when blank.
## Do NOT
- Do NOT add/expect a role check (`MSP_ADMIN`/`BCOS_ADMIN`) — none exists here; don't document one.
- Do NOT edit another institution's template via the JWT endpoints (403); use `/api/internal/**` with the secret for `__GLOBAL__`/cross-institution.
- Do NOT call `/api/internal/**` without a non-blank `INTERNAL_SERVICE_SECRET` set on the container — every call 401s.
- Do NOT assume a Kafka topic or executor→record-store flow for this feature — it's synchronous DB CRUD.
- Do NOT reference "HiveOps" in user-facing copy — brand is **HiveIQ**; `hiveops-ai` = **Adoons**.
backend/src/main/resources/content/claims/settings/internal.md
+60
View File
@@ -0,0 +1,60 @@
---
module: claims.settings
title: Settings
tab: Internal
order: 20
audience: internal
---
Ops/support view of the Claims **Settings** screen. The screen is a **read-only** display of the *effective* vendor-email template for the logged-in user's institution. Template management is **API-only** — there is no create/edit UI wired into this page (the SPA calls only the `effective` endpoint).
## What the screen actually does
- Frontend: `hiveops-claims/frontend/src/components/Settings/EmailTemplatesSection.svelte` calls `emailTemplateAPI.effective()` on mount and renders one of three states.
- Backend: `GET /api/claims/settings/email-templates/effective` (owner service `hiveops-claims`, image `hiveiq-claims`, port 8092).
- Resolution order (from `EmailTemplateController.effective`):
1. This institution's own default (`is_default = true`, `institution_key = <JWT institutionKey>`)
2. Global default (`institution_key = '__GLOBAL__'`, `is_default = true`) → shown as **Global Template (inherited)**
3. Neither → HTTP **204** → UI shows **No template configured** (built-in fallback body is used at send time)
## Roles required
- **None beyond authentication.** `SecurityConfig` ends in `anyRequest().authenticated()` — there is **no** `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` guard anywhere in this service. Any user with a valid HiveIQ JWT can read the effective template and (via API) manage their own institution's templates.
- Isolation is purely by `institutionKey` taken from the JWT principal (`ClaimsPrincipal`), not by role. A user only ever sees/edits templates where `institution_key` matches their own JWT.
## Admin-only actions (how templates get created)
There is no in-app editor, so templates are created one of two ways:
1. **Authenticated API (per-institution)** — acting as a user of that institution:
- `POST /api/claims/settings/email-templates` `{ name, subject, bodyHtml, isDefault }`
- `PUT /api/claims/settings/email-templates/{uuid}`
- `DELETE /api/claims/settings/email-templates/{uuid}`
2. **Internal service API (cross-institution / global)** — `/api/internal/email-templates`, gated by an `X-Internal-Secret` header (see failure modes). Requires `?institutionKey=` query param. Use `institutionKey=__GLOBAL__` to set the shared default that every institution inherits.
Setting `isDefault: true` on create/update runs `clearDefaultForInstitution` first, so **only one default per institution** is ever active — you don't have to clear the old one manually.
## First things to check when it misbehaves
- **User sees "No template configured" but a template exists** → the template's `is_default` is `false`. Only the default is resolved. Set one template's `isDefault=true`.
- **User sees the Global template instead of their own** → their institution has no row with `is_default=true`; only `__GLOBAL__` does. Create/mark an institution-specific default.
- **Wrong institution's template appears** → check the JWT's `institutionKey` claim; scoping is entirely JWT-driven. A mis-issued token = wrong template.
- **Vendor emails go out with the wrong wording** → the *send path* (`EmailService.sendMessageNotification`) re-resolves the default the same way (own → `__GLOBAL__` → hardcoded fallback). If the Settings screen shows template X, message emails use X. Attachment notification emails do **not** use templates (hardcoded body) — that's expected.
- **204/500 on the endpoint** → confirm the service is up: `curl http://localhost:8092/actuator/health` on the services VM; route externally as `api.bcos.cloud/claims/settings/email-templates/effective` with a real Bearer token.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Screen stuck on "Loading…" or shows fallback | `effective` request failed (network/401); the component catches errors and sets `template=null` | Check JWT validity + nginx `claims/` route; look at `hiveiq-claims` logs |
| `/api/internal/email-templates` → 401 "Invalid service secret" | `INTERNAL_SERVICE_SECRET` env unset (default blank) or header mismatch | Set `INTERNAL_SERVICE_SECRET` on the container **and** send matching `X-Internal-Secret`. If blank, ALL internal calls are rejected |
| PUT/DELETE → 403 | Template's `institution_key` ≠ caller's JWT institutionKey | Edit only your own institution's templates; use the internal API for cross-institution |
| PUT/DELETE → 404 "Template not found" | Wrong `uuid` (this uses the `uuid` column, not the numeric `id`) | Use the `uuid` from the list response |
| Vendor emails never arrive | `MAIL_USERNAME`/`MAIL_PASSWORD` unset → `JavaMailSender` bean absent → send silently skipped (logged at DEBUG) | Configure `MAIL_*` env; check logs for "Mail not configured — skipping" |
## Where the data lives
- Table `email_templates` in DB `hiveiq_claims` (PostgreSQL, prod). Columns: `id`, `uuid`, `institution_key`, `name`, `subject`, `body_html`, `is_default`, `created_at`, `updated_at` (migration `V2__email_templates.sql`).
- Dev profile is H2 in-memory (`create-drop`, Flyway disabled) — templates reset on restart.
See the customer-facing Overview tab for the end-user description, and the Architect tab for the build.
backend/src/main/resources/content/claims/vendors/architect.md
Vendored
+80
View File
@@ -0,0 +1,80 @@
---
module: claims.vendors
title: Vendors
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** How the Vendor Registry feature is built. All facts grounded in hiveops-claims source (2026-07-01). See [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Shape of the feature
Vendors is the simplest slice of hiveops-claims: a single-entity CRUD-lite (create + read only) registry, fully self-contained in the **hiveops-claims** service. There are **no other services, no Kafka, and no cross-service calls** in this feature — it is one Svelte page, one controller, one service, one repository, one table.
```
VendorsPage.svelte / AddVendorModal.svelte
│ vendorAPI.list() / vendorAPI.create() (frontend/src/lib/api.ts)
▼
GET|POST api.bcos.cloud/claims/api/claims/vendors (NGINX → hiveiq-claims:8092)
▼
VendorController ──► VendorService ──► VendorRepository (JPA) ──► Postgres: vendors
```
Unlike claims (which do the executor→Kafka→record-store and `device_summary` mirror dance), **vendors never touch Kafka**. hiveops-claims consumes `hiveops.device.events` only to keep its `device_summary` read-model current for claim creation — that path is unrelated to the vendor registry.
## Services involved
- **hiveops-claims** — owns everything for this feature. Backend `hiveiq-claims` (port 8092, `/api/claims/...`), frontend `hiveiq-claims-frontend` (`claims.bcos.cloud`, port 5179).
- No incident / journal / devices / mgmt involvement for vendor read or write.
## Data flow
**Read (`GET /api/claims/vendors`)**
1. `VendorController.listVendors` reads `principal.institutionKey()` (populated by `JwtAuthenticationFilter` from the JWT `institutionKey` claim; principal type `ClaimsPrincipal(userId, email, role, institutionKey)`).
2. Optional `?type=CIT|SP` binds to `Vendor.VendorType`. With a type → `listVendorsByType`; without → `listVendors`.
3. `VendorService` → `VendorRepository.findByInstitutionKeyOrderByNameAsc` (or `...AndTypeOrderByNameAsc`).
4. Rows map to `VendorResponse` (uuid, name, type, contactName/Email/Phone, createdAt).
5. Frontend `VendorsPage` receives the flat list and splits it **client-side** into CIT vs SP (`vendors.filter(v => v.type === 'CIT')`), rendering two tables. (Note: the page's `typeFilter` variable exists but is not bound to any UI control, so the server-side `?type=` filter is effectively unused today.)
**Write (`POST /api/claims/vendors`)**
1. `AddVendorModal` posts `{ name, type, contactName?, contactEmail?, contactPhone? }` (empty optionals omitted).
2. `CreateVendorRequest` validates `@NotBlank name`, `@NotNull type`. Email/phone are free text server-side.
3. `VendorService.createVendor` builds a `Vendor`, stamps `institutionKey` from the principal, and saves. `uuid`, `created_at`, `updated_at` are DB/entity-generated (`@GeneratedValue`, `@CreationTimestamp`, `@UpdateTimestamp`).
4. Returns `201 CREATED` with the `VendorResponse`. Frontend re-runs `load()` on `onCreated`.
## Persistence
Single table, `hiveiq_claims` DB (Flyway `V1__init_schema.sql`):
| Column | Type | Notes |
|--------|------|-------|
| `id` | BIGSERIAL PK | internal FK target |
| `uuid` | UUID UNIQUE | public identifier in API |
| `name` | VARCHAR(255) NOT NULL | required |
| `type` | VARCHAR(3) NOT NULL | CHECK IN ('CIT','SP') |
| `contact_name` / `contact_email` / `contact_phone` | VARCHAR | optional |
| `institution_key` | VARCHAR(255) | scoping key; **nullable, no index** |
| `created_at` / `updated_at` | TIMESTAMPTZ | defaults NOW() |
- **Referenced by** `claims.cit_vendor_id` and `claims.sp_vendor_id` (`BIGINT REFERENCES vendors(id)`). This FK is why vendors have no delete endpoint — a referenced vendor can't be removed. Also referenced conceptually by `claim_messages.vendor_id` / `claim_attachments.vendor_id`.
- No unique constraint on `(institution_key, name)` — duplicates are allowed.
- `dev` profile = H2 in-memory (`create-drop`, Flyway off); `prod` = Postgres + Flyway `validate`.
## API endpoints (this feature only)
| METHOD | Path | Handler |
|--------|------|---------|
| GET | `/api/claims/vendors` (`?type=CIT|SP`) | `VendorController.listVendors` |
| POST | `/api/claims/vendors` | `VendorController.createVendor` |
Externally routed as `api.bcos.cloud/claims/api/claims/vendors`. Both require a valid JWT; there is no role authority check (see [claims.vendors] Internal tab).
## Kafka
None for vendors. (The service-level `hiveops.device.events` consumer feeds `device_summary` for claim filing, not the registry.) See [platform.kafka].
## Design gaps worth knowing
- Create/read only — no update, no delete, no dedup (all confirmed absent in `VendorController`/`VendorRepository`).
- Scoping relies entirely on a correctly-populated JWT `institutionKey`; a blank claim silently pools vendors under a null key.
backend/src/main/resources/content/claims/vendors/claude.md
Vendored
+57
View File
@@ -0,0 +1,57 @@
---
module: claims.vendors
title: Vendors
tab: Claude
order: 40
audience: internal
---
> Act-without-guessing reference. All items code-confirmed in hiveops-claims (2026-07-01) unless marked (unverified).
## Ownership
- **Service:** hiveops-claims (`hiveiq-claims`, port 8092). Owns vendor read + write end to end.
- **Frontend:** `hiveiq-claims-frontend`, `claims.bcos.cloud`, port 5179. Page: `frontend/src/components/Vendors/VendorsPage.svelte`; create panel: `components/modals/AddVendorModal.svelte`; api: `lib/api.ts` → `vendorAPI`.
- No other service, no Kafka, no cross-service call in this feature.
## Endpoints (exact)
| METHOD | Path | Notes |
|--------|------|-------|
| GET | `/api/claims/vendors` | optional `?type=CIT` or `?type=SP` (binds `Vendor.VendorType`); no type → all |
| POST | `/api/claims/vendors` | body `{name*, type*, contactName?, contactEmail?, contactPhone?}`; returns `201` |
- External route: `https://api.bcos.cloud/claims/api/claims/vendors`.
- Auth: `Authorization: Bearer <JWT>` required. Scope key = JWT `institutionKey` claim.
- Required fields: `name` (`@NotBlank`), `type` (`@NotNull`, `CIT`|`SP`). Email validated **client-side only**.
- Response fields: `uuid, name, type, contactName, contactEmail, contactPhone, createdAt`.
## Storage
- **DB:** `hiveiq_claims` (Postgres, `10.10.10.188` via ProxyJump). **Table:** `vendors`.
- Columns: `id, uuid, name, type('CIT'|'SP'), contact_name, contact_email, contact_phone, institution_key(nullable, no index), created_at, updated_at`.
- FK in: `claims.cit_vendor_id`, `claims.sp_vendor_id` → `vendors(id)`.
- **Kafka topics:** none for this feature.
## Gotchas
- **No update / no delete endpoint** — `VendorController` has only GET + POST. Typos are fixable only via direct SQL `UPDATE vendors`.
- **No role gate** — `SecurityConfig` = `anyRequest().authenticated()`. Do NOT assume `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`; isolation is `institutionKey` only.
- **No dedup** — `(institution_key, name)` is not unique; POST twice = two rows.
- **Blank `institutionKey` JWT** pools vendors under a null key (nullable column) → apparent "missing" or cross-user leakage.
- Frontend splits CIT/SP **client-side**; `typeFilter` in `VendorsPage` is declared but not bound to any UI control, so `?type=` is currently unused.
- Deleting a vendor referenced by a claim fails the FK — unassign on the claim first.
- `dev` profile = H2 (`create-drop`, Flyway off); `prod` = Postgres + Flyway `validate`.
## Do NOT
- Do NOT tell users to edit or delete a vendor in the UI — no such action exists.
- Do NOT claim vendor changes emit Kafka events — they don't.
- Do NOT invent an MSP/all-institutions vendor view — every query is `institutionKey`-scoped.
- Do NOT expect a role/permission error path — auth failures are 401 (missing/invalid token), not 403 by role.
- Do NOT run `docker exec hiveiq-postgres` on the services VM — Postgres is on `10.10.10.188`.
- Do NOT reference this AI assistant as anything but "Adoons" externally.
## Cross-links
[claims.vendors] Overview · Internal · Architect. Service facts: [technical.claims]. Platform: [platform.service-ownership], [platform.kafka], [platform.data-architecture].
backend/src/main/resources/content/claims/vendors/internal.md
Vendored
+64
View File
@@ -0,0 +1,64 @@
---
module: claims.vendors
title: Vendors
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support/admin.** The Vendor Registry (`VendorsPage.svelte`) is a per-institution list of CIT (Cash-in-Transit) and SP (Service Provider) companies used when filing a claim. Backend owner: **hiveops-claims** (`hiveiq-claims`, port 8092). Grounded in code 2026-07-01.
## Who can do what
- **No role gate.** hiveops-claims security is **JWT-authentication-only** (`SecurityConfig` → `.anyRequest().authenticated()`). There is **no** `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` check on vendor endpoints — unlike most HiveOps services. Any authenticated user with a valid JWT can list and create vendors.
- **Isolation is by `institutionKey`, not role.** The controller passes `principal.institutionKey()` (read from the JWT `institutionKey` claim in `JwtAuthenticationFilter`) into every query. A user only ever sees/creates vendors for their own institution. Vendors are private to the institution — there is no cross-institution or MSP "all vendors" view.
## What actions exist (and what's missing)
Only **two** operations are wired end to end:
| Action | UI | API |
|--------|----|-----|
| List vendors (CIT + SP) | Vendor Registry tables, split by type client-side | `GET /api/claims/vendors` |
| Register a vendor | **+ Add Vendor** panel (`AddVendorModal.svelte`) | `POST /api/claims/vendors` |
- **There is NO edit endpoint and NO delete endpoint.** `VendorController` exposes only `@GetMapping` and `@PostMapping`. Once a vendor is created it is permanent — a typo in name/email/phone **cannot be corrected through the app**. Fixing it requires a direct SQL `UPDATE` on the `vendors` table (see below).
- **No de-duplication.** Nothing enforces unique `(institution_key, name)`. Registering "Acme CIT" twice creates two rows; both show in the list and both are assignable to claims.
## First things to check when it misbehaves
1. **"I don't see any vendors" / empty tables** — the list is scoped to the caller's `institutionKey`. If a user logged in with a JWT whose `institutionKey` claim is missing/blank, the query runs against a null/blank key and returns only vendors saved under that same blank key. Confirm the JWT actually carries `institutionKey` (auth service issue, not claims).
2. **"Vendor I added is gone / in the wrong institution"** — same root cause: the vendor was saved with whatever `institutionKey` the JWT had at create time. `institution_key` is nullable and never re-derived. Check the row directly.
3. **"Can't add vendor" toast** — the create panel surfaces the backend message. `name` and `type` are required (`@NotBlank name`, `@NotNull type`); email is only format-validated **client-side** (there is no server email validation). A 400 usually means missing name or an invalid `type` value.
4. **Wrong "Added" date format** — cosmetic only; `createdAt` comes straight from the DB `created_at` (`@CreationTimestamp`).
5. **Vendor exists but not selectable when filing a claim** — that's the claim side, not the registry. The vendor must exist first; assignment happens in the claim (`PUT /api/claims/{uuid}`).
## Direct DB access (for the edit/delete gap)
DB `hiveiq_claims` on the database VM (`10.10.10.188`, via ProxyJump — never the services VM). Table `vendors`.
```bash
# List an institution's vendors
ssh -i ~/.ssh/id_ed25519_hiveiq -J bcosadmin@173.231.195.250 bcosadmin@10.10.10.188 \
"docker exec hiveiq-postgres psql -U hiveiq -d hiveiq_claims \
-c \"SELECT uuid, name, type, contact_email, institution_key FROM vendors WHERE institution_key='PBNC' ORDER BY name;\""
```
- **Fix a typo (no app path exists):** `UPDATE vendors SET contact_email='new@vendor.com' WHERE uuid='...';`
- **Remove a bad/duplicate vendor:** deleting is only safe if it is not referenced. `claims.cit_vendor_id` / `claims.sp_vendor_id` are `BIGINT REFERENCES vendors(id)` — a `DELETE` on a referenced vendor will fail the FK constraint. Clear the reference on the claim first, or leave the vendor in place.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Vendor list empty for one user only | JWT missing/blank `institutionKey` | Fix at auth/mgmt; vendors are scoped to that claim |
| Duplicate vendor rows | No unique constraint; double-submit or re-registration | Delete the unreferenced duplicate via SQL |
| Can't edit vendor details | No update endpoint exists (by design/gap) | `UPDATE vendors ...` directly |
| Delete fails with FK error | Vendor assigned to a claim (`cit_vendor_id`/`sp_vendor_id`) | Unassign on the claim, then delete |
| 401 on `/api/claims/vendors` | Missing/expired Bearer token | Re-auth; endpoint requires authentication |
## Related
- User-facing behavior: [claims.vendors] Overview tab.
- Build-it view: [claims.vendors] Architect tab.
- Service-wide facts: [technical.claims] overview.
backend/src/main/resources/content/dashboard/cash-stats/architect.md
+87
View File
@@ -0,0 +1,87 @@
---
module: dashboard.cash-stats
title: Cash Stats
tab: Architect
order: 30
audience: internal
---
How the **Cash Stats** page is built end-to-end. It is a thin read view over data that another service derives — the interesting work happens in `hiveops-journal`, not in the dashboard. See [platform.service-ownership].
## Services involved
| Service | Role for this feature |
|---------|----------------------|
| `hiveops-dashboard` (SPA, 5181) | Renders `CashStatus.svelte`; no backend |
| `hiveops-incident` (8080) | Owns `journal_events` + `atms`; serves the read API |
| `hiveops-journal` (8087, **upload** role) | Parses ATM journals, extracts cassette counts, **writes into incident's DB** |
## Data flow
```
Agent → agent-proxy → journal (chunked upload) ─┐
│ parse (per-institution worker)
journal.parse-requests.{INSTITUTION} ────────────┘
│ worker parses file, persists JournalTransaction (STARTUP has rawLines)
▼
journal.file-parsed ──► CassetteInventoryService (upload role, groupId journal-cassette-inventory)
│ extracts [DENOMINATION] / RCY USD / MIX / RETRACT slots from the newest STARTUP
▼
incident DB: journal_events (event_type='CASSETTE_INVENTORY', event_source='JOURNAL')
▲
│ GET /api/journal-events/fleet/cassette-inventory (+ GET /api/atms)
CashStatus.svelte ── merges events with ATM list, colours pills, sorts empty→low→ok→config→nodata
```
Key point: this is the **executor→Kafka→record-store** pattern applied across a service boundary. `hiveops-journal` is the executor (parse), `journal.file-parsed` is the event, and the "record store" it writes to is **incident's `journal_events` table** via a second read/write JDBC datasource (`incidentJdbcTemplate`), not its own DB. See [platform.kafka], [platform.data-architecture].
## Kafka
- **Topic:** `journal.file-parsed` (constant `JournalKafkaConfig.TOPIC_FILE_PARSED`), produced by `FileParsedEventProducer`.
- **Consumer for this feature:** `CassetteInventoryService.onFileParsed` — `@KafkaListener(groupId="journal-cassette-inventory", auto.offset.reset=latest)`. Runs **only in the upload role** (`@ConditionalOnProperty journal.role=upload`).
- Offset reset = **latest**: on a fresh consumer it ignores backlog; use the admin backfill to seed history.
## Write path (journal → incident)
`CassetteInventoryService.processFile(fileId, atmId)`:
1. Load STARTUP transactions for the file; take the most recent; read `rawLines`.
2. `extractSlots()` — regex the STARTUP text:
- `[DENOMINATION]` table (preferred) → per-slot denom + count, type `RECYCLER`.
- else `RCY USD` lines (older firmware).
- `MIX CASSETTE COUNT` → slot `E` (type MIX), `RETRACT CASSETTE COUNT` → slot `R` (type RETRACT).
- Slots sorted by denomination, lettered `A, B, C...`.
3. `resolveNumericId(atmId)` → `SELECT id FROM atms WHERE atm_id=?` (incident). Null → skip + warn.
4. **Freshness guard:** if incident already has a newer `CASSETTE_INVENTORY.event_time` for that ATM, skip.
5. Otherwise `DELETE` existing `CASSETTE_INVENTORY` rows for the ATM, then `INSERT` fresh slots (`event_source='JOURNAL'`, `cassette_currency='USD'`).
`backfillAll()` does the same across every ATM with STARTUPs (admin sync endpoint).
## Read path (incident → dashboard)
`JournalEventController.getFleetCassetteInventory()` → `JournalEventService.getFleetCassetteInventory(scope)`:
- Scope from `InstitutionContext.resolveScope()` → resolved to institution *names* via `institutionKeyService`.
- Repo query returns the **latest event per (atm_id, cassettePosition)** using `MAX(id)` grouped by `(atm.id, COALESCE(cassettePosition,''))`, with `JOIN FETCH e.atm` (no N+1). Three variants: unrestricted / single-institution / `institution IN (...)`.
- For ATMs with **no** live inventory events, it synthesizes slots from `AtmProperties`:
- Source 1: agent-synced `cassetteConfigs` map (`position → "currency:denomination"`).
- Source 2: manually applied `CassetteConfiguration` template.
- These are flagged `event_source='CASSETTE_CONFIG'` → the UI renders them as "(no data)" config-only pills.
The frontend (`CashStatus.svelte`) does the rest client-side: groups events by ATM, computes empty/low/ok (`LOW_BILL_THRESHOLD = 200` bills), sorts empty→low→ok→config→no-data, and merges in ATMs from `/api/atms` that returned no events at all as "No inventory reported."
## DB tables
| Table | Service / DB | Access |
|-------|--------------|--------|
| `journal_events` | incident (`hiveiq_incident`) | read by API; **written by journal** via `incidentJdbcTemplate` |
| `atms` | incident | read (both by API and by journal's `resolveNumericId`) |
| `journal_transactions` | journal (`hiveiq_journal`) | read by `CassetteInventoryService` (STARTUP rawLines) |
## API endpoints (this feature)
| Method | Path | Service | Notes |
|--------|------|---------|-------|
| GET | `/api/journal-events/fleet/cassette-inventory` | incident | institution-scoped fleet inventory |
| GET | `/api/atms` | incident | ATM list for merge/labels |
| POST | `/api/journal/admin/sync-cassette-inventory` | journal | admin backfill (`MSP_ADMIN`/`BCOS_ADMIN`) |
Cross-links: [platform.data-architecture], [platform.kafka], [platform.service-ownership].
backend/src/main/resources/content/dashboard/cash-stats/claude.md
+56
View File
@@ -0,0 +1,56 @@
---
module: dashboard.cash-stats
title: Cash Stats
tab: Claude
order: 40
audience: internal
---
Terse agent reference for `dashboard.cash-stats`. Read view over cassette bill counts.
## Ownership
- **Frontend:** `hiveops-dashboard` → `frontend/src/components/Dashboard/CashStatus.svelte` (no backend).
- **Read API owner:** `hiveops-incident` — `JournalEventController` / `JournalEventService`.
- **Data producer:** `hiveops-journal` (upload role) — `CassetteInventoryService` writes into incident's DB.
- Dashboard SPA has **no server**; all data is incident's.
## Endpoints (METHOD path → service)
| Method | Path | Service | Auth |
|--------|------|---------|------|
| GET | `/api/journal-events/fleet/cassette-inventory` | incident | any app role, institution-scoped |
| GET | `/api/atms` | incident | any app role |
| POST | `/api/journal/admin/sync-cassette-inventory` | journal | `MSP_ADMIN`/`BCOS_ADMIN` |
| POST | `/api/journal/admin/reconcile` | journal | `MSP_ADMIN`/`BCOS_ADMIN` (run post-deploy) |
- Frontend calls these via configured API base (`window.__APP_CONFIG__.apiUrl`); paths above are the incident controller mappings.
## Tables
- `journal_events` — incident DB (`hiveiq_incident`). Filter `event_type='CASSETTE_INVENTORY'`. **Written by journal**, read by incident API.
- `atms` — incident DB. Journal resolves `atm_id` (string) → `id` (numeric) here.
- `journal_transactions` — journal DB (`hiveiq_journal`). Source STARTUP `rawLines`.
## Topics
- `journal.file-parsed` — consumed by `CassetteInventoryService` (groupId `journal-cassette-inventory`, `auto.offset.reset=latest`).
- No topic is consumed/produced by the dashboard itself.
## Roles
- Read endpoints: `USER, CUSTOMER, ADMIN, MSP_ADMIN, QDS_ADMIN, BCOS_ADMIN` (incident `/api/**`).
- Scope: `InstitutionContext.resolveScope()` → `CUSTOMER`/`MSP_ADMIN` scoped to JWT institution keys; others unrestricted.
- Journal admin: `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`.
## Gotchas
- `event_source='CASSETTE_CONFIG'` = **synthetic config-only** slot (no real count) — UI shows "(no data)". Real data = `event_source='JOURNAL'` (from journal) or agent ingest.
- Journal write **deletes then inserts** all `CASSETTE_INVENTORY` rows for an ATM per update; has a freshness guard (skips if existing `event_time` newer).
- Read query uses `MAX(id)` per `(atm_id, COALESCE(cassette_position,''))` as recency proxy — assumes monotonic ids.
- Currency hardcoded `USD`; type from STARTUP is `RECYCLER`/`MIX`/`RETRACT`. MIX/RETRACT have null denomination.
- Low threshold = **200 bills** (frontend `LOW_BILL_THRESHOLD`), empty = 0 bills, evaluated only on slots with denomination > 0.
- `CassetteInventoryService` runs **only in journal upload role** (`journal.role=upload`).
- ATMs whose `atm_id` isn't in incident `atms` get skipped (logged warn), so they show as "No Data."
## Do NOT
- Do NOT add a cassette/cash endpoint to `hiveops-dashboard` — it has no backend; use incident.
- Do NOT write `CASSETTE_INVENTORY` rows from anywhere but journal's `CassetteInventoryService` (single writer owns the delete+insert).
- Do NOT expect Kafka backlog replay — consumer is `auto.offset.reset=latest`; use `sync-cassette-inventory` to backfill.
- Do NOT assume "last reported" = now; it's the STARTUP transaction time.
- Do NOT bypass institution scoping by querying the table directly for a customer context.
- Do NOT invent per-position history — read path keeps only the latest per slot (getCassetteInventory keeps latest + previous for single-ATM only).
backend/src/main/resources/content/dashboard/cash-stats/internal.md
+54
View File
@@ -0,0 +1,54 @@
---
module: dashboard.cash-stats
title: Cash Stats
tab: Internal
order: 20
audience: internal
---
Ops / support view for the **Cash Stats** page on `dashboard.bcos.cloud`. It shows the latest cassette bill counts per ATM across the fleet. The page is **read-only** — there are no admin actions on the page itself; all levers are on the backend that feeds it.
## Who serves this
- **Frontend:** `hiveops-dashboard` (SPA, port 5181) — `CashStatus.svelte`. No backend of its own.
- **Data API:** `hiveops-incident` backend (port 8080) — `JournalEventController`.
- **Data producer:** `hiveops-journal` (upload role) parses ATM STARTUP journals and writes `CASSETTE_INVENTORY` rows into the **incident** DB.
The page makes exactly two calls on load and every 60 s:
- `GET /api/journal-events/fleet/cassette-inventory` (incident)
- `GET /api/atms` (incident)
## Roles required
- Both endpoints fall under incident's `/api/**` rule → **any authenticated app user**: `USER, CUSTOMER, ADMIN, MSP_ADMIN, QDS_ADMIN, BCOS_ADMIN`. (`/api/journal-events` POST also allows `AGENT`, but that's ingest, not this page.)
- **Institution scoping is automatic** (`InstitutionContext.resolveScope`): `CUSTOMER`/`MSP_ADMIN` see only their granted institution keys; `BCOS_ADMIN/ADMIN/QDS_ADMIN/USER` see the whole fleet. If a customer says "I only see some ATMs," that's correct scoping, not a bug.
- **Admin backfill/sync is `MSP_ADMIN` or `BCOS_ADMIN` only** (journal `AdminController`).
## First things to check when it misbehaves
1. **Whole page errors / red banner** → incident backend down or unreachable. `curl` the two endpoints with a real JWT; expect `200` + JSON array.
2. **Everything shows "No Data" / "No inventory reported"** → no `CASSETTE_INVENTORY` rows in incident `journal_events`. Usual cause: journal parsing stalled or the cassette-inventory Kafka consumer isn't running. Check `hiveops-journal` (upload role) is up and that files are parsing (`parse_status=PARSED`).
3. **One ATM stuck stale / never updates** → its STARTUP journals aren't parsing, or the ATM's `atm_id` string doesn't resolve to a numeric id in incident's `atms` table (`resolveNumericId` returns null → row skipped, logged as a warning).
4. **Counts look wrong for a recycler** → the journal parser reads the `[DENOMINATION]` table (or falls back to `RCY USD` lines) out of the STARTUP record; a firmware format it doesn't match yields missing/zero slots.
5. **Cassettes show but say "(no data)"** → those are **config-only** synthetic slots. The ATM has a cassette *configuration* (agent-synced or a manually applied template) but has reported no actual bill count yet. Not an error.
## Admin actions (backend, not on the page)
- **Force a fleet re-derive of cash inventory** from the latest STARTUP journal of every ATM:
`POST /api/journal/admin/sync-cassette-inventory` (journal service, `MSP_ADMIN`/`BCOS_ADMIN`). Runs `CassetteInventoryService.backfillAll()`.
- After any journal deploy, run the standard journal **reconcile** (`POST /api/journal/admin/reconcile`) so stranded files finish parsing — otherwise new inventory never lands.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| All "No Data" | journal upload role down / no parsed STARTUPs | Bring up `hiveiq-journal`; verify files reach `PARSED`; then `sync-cassette-inventory` |
| One ATM never updates | `atm_id` not in incident `atms`, or no STARTUP parsed | Confirm ATM exists in incident; check journal parse logs for that ATM |
| Newer count not showing | journal skips write because existing `event_time` is newer | Expected guard; a genuinely newer STARTUP will overwrite |
| Customer sees subset of ATMs | institution scoping from JWT | Working as intended |
| 401/403 on load | expired/invalid JWT, or user has none of the allowed roles | Re-auth; check role |
## Data freshness
- The "last reported" time per ATM is the `event_time` of its newest cassette event — for journal-derived data this is the **STARTUP transaction time**, not "now." A stale time means the ATM hasn't produced a fresh STARTUP journal, not that the page is broken.
- Page auto-refreshes every 60 s and on the **↻** button; there's no server-side push.
backend/src/main/resources/content/dashboard/overview/architect.md
+44
View File
@@ -0,0 +1,44 @@
---
module: dashboard.overview
title: Fleet Overview
tab: Architect
order: 30
audience: internal
---
## Shape of the feature
**Fleet Overview** is a **read-only aggregation view with no backend of its own.** `hiveops-dashboard` is a static Svelte/Vite SPA (port 5181, `dashboard.bcos.cloud`). All state lives in three owning services; the SPA is a thin client that composes their responses into KPI cards, health bars, task stats, and a recent-incidents table.
Component: `hiveops-dashboard/frontend/src/components/Dashboard/Dashboard.svelte`
Data layer: `frontend/src/lib/api.ts` (single axios instance) + `frontend/src/lib/stores.ts` (`atmSummary`, `dashboardStats`, `fleetStats` writables, `loadAll()`).
## Services involved (three-way fan-out)
The SPA has **one** base URL (`window.__APP_CONFIG__.apiUrl` from `VITE_API_URL`, injected by `frontend/entrypoint.sh` into `config.js` at container start). It calls four relative paths that are **owned by three different services**, so the deployment relies on NGINX path-routing to fan a single origin out to the right backend (the exact nginx map is in the ops repo, not here — verify there if a path 404s):
| Path | Service | Handler | Backing store |
|---|---|---|---|
| `GET /atms/summary` | hiveops-devices | `AtmController#getAtmSummary` (`@RequestMapping({"/api/atms","/api/devices"})`) | `hiveiq_devices.atms` (+ `atm_properties.last_heartbeat`) |
| `GET /atms/dashboard/stats` | hiveops-incident | `AtmDashboardController#getDashboardStats` (`/api/atms/dashboard/stats`) → `AtmService#getDashboardStats` | `hiveiq_incident.incidents`, `hiveiq_incident.atms` |
| `GET /fleet/tasks/stats` | hiveops-fleet | `FleetApiController#getStats` (`/api/fleet/{stats,tasks/stats}`) → `FleetTaskService#getStats` | `hiveiq_fleet.fleet_tasks`, `hiveiq_fleet.fleet_artifacts` |
| `GET /users/me` | hiveops-incident | `UserController#getCurrentUser` (`/api/users/me`) | JWT claims only (no DB) |
## Data flow
1. `onMount` in `Dashboard.svelte` calls `loadAll()` → `Promise.all([loadAtmSummary(), loadDashboardStats(), loadFleetStats()])`, then repeats on a 60s tick (auto-refresh) or manual button.
2. Each loader wraps its call in an isolated `try/catch`; a failure zeroes only that store, so cards degrade independently.
3. `App.svelte` separately calls `GET /users/me` to resolve `userRole` (CUSTOMER hides the "View all" incidents link; Electron `electronAPI.getUserProfile` fallback).
### Per-service internals
- **devices `/atms/summary`** — loads institution-scoped ATMs, computes `connected/disconnected/neverConnected` from `AtmDTO.agentConnectionStatus` (derived from `last_heartbeat`) and a `byModel` histogram. Pure read; no Kafka in the request path.
- **incident `/atms/dashboard/stats`** — `@Cacheable("dashboardStats")` keyed by institution scope. Reads `incidents` (open/critical/this-week counts + last 10 for the table) and `atms` (`totalAtms`, `operationalAtms` where `status == IN_SERVICE`). Institution names resolved via `institutionKeyService.resolveInstitutionNames`.
- **fleet `/fleet/tasks/stats`** — `taskRepository.count()` for total, `countByTaskKindAndStatus(AUTO_UPDATE, PENDING)` for pending, `findByStatus(RUNNING).size()` for running, `artifactRepository.count()` for artifacts. `completed`, `failed`, and `totalModulesReporting` are **not computed** (see uncertainties). No institution filter applied.
## Kafka / mirrors (context, not request-path)
This feature reads each service's **own** primary tables, so Kafka is not in the live read path. It is relevant to correctness because device identity is mirrored fleet-wide: hiveops-devices publishes `hiveops.device.events`, which hiveops-fleet (and incident/reports/recon) consume into a local `device_summary` mirror. The dashboard's ATM counts come from the devices `atms` table directly; fleet task stats come from `fleet_tasks`. If device naming/institution on the dashboard disagrees with Fleet, suspect a stale `device_summary` mirror rather than the dashboard. See [platform.kafka], [platform.data-architecture].
## Ownership boundaries
- Device inventory & connection status → **hiveops-devices** (source of truth).
- Incidents & the recent-incident list → **hiveops-incident**.
- Fleet task/artifact aggregates → **hiveops-fleet**.
- The dashboard SPA owns **nothing** — it must never grow its own endpoints or tables; new numbers belong in the owning service and are surfaced here. See [platform.service-ownership].
Cross-links: [platform.data-architecture] · [platform.kafka] · [platform.service-ownership] · [dashboard.overview]
backend/src/main/resources/content/dashboard/overview/claude.md
+53
View File
@@ -0,0 +1,53 @@
---
module: dashboard.overview
title: Fleet Overview
tab: Claude
order: 40
audience: internal
---
## Identity
- Frontend: `hiveops-dashboard` (Svelte/Vite SPA, port 5181, `dashboard.bcos.cloud`). **No backend, no DB, no Kafka of its own.**
- Component: `hiveops-dashboard/frontend/src/components/Dashboard/Dashboard.svelte`; data: `frontend/src/lib/{api.ts,stores.ts}`.
- Single axios base: `window.__APP_CONFIG__.apiUrl` (`VITE_API_URL`, default `http://localhost:8080`), set by `frontend/entrypoint.sh`.
## Endpoints (METHOD + path, relative to `apiUrl`)
| Method | Path | Owner service | Backend mapping | Auth |
|---|---|---|---|---|
| GET | `/atms/summary` | hiveops-devices | `AtmController` `/api/atms` + `/api/devices` → `/summary` | JWT, institution-scoped, no `@PreAuthorize` |
| GET | `/atms/dashboard/stats` | hiveops-incident | `AtmDashboardController` `/api/atms/dashboard/stats` | JWT, institution-scoped, `@Cacheable("dashboardStats")` |
| GET | `/fleet/tasks/stats` | hiveops-fleet | `FleetApiController` `/api/fleet/{stats,tasks/stats}` | `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
| GET | `/users/me` | hiveops-incident | `UserController` `/api/users/me` | JWT (claims only) |
## Owning service per data element
- ATM totals / connected / disconnected / neverConnected / byModel → **devices** (`atms` table, status from `agentConnectionStatus`/`last_heartbeat`).
- openIncidents / criticalIncidents / incidentsThisWeek / recentIncidents / totalAtms / operationalAtms → **incident** (`incidents`, `atms`).
- totalTasks / pendingTasks / runningTasks / completedTasks / failedTasks / totalArtifacts / totalModulesReporting → **fleet** (`fleet_tasks`, `fleet_artifacts`).
## Tables
- `hiveiq_devices.atms`, `hiveiq_devices.atm_properties`
- `hiveiq_incident.incidents`, `hiveiq_incident.atms`
- `hiveiq_fleet.fleet_tasks`, `hiveiq_fleet.fleet_artifacts` (also `fleet_task_history`, `device_summary` — NOT read by this feature's stats)
## Kafka
- `hiveops.device.events` — published by devices, consumed into `device_summary` mirrors in fleet/incident/reports/recon. **Not in this feature's request path** (dashboard reads primary tables directly).
## Gotchas
- `/atms/dashboard/stats` exists in **both** hiveops-incident AND hiveops-devices. Only the **incident** one populates incident fields. Routing determines which answers.
- `/atms/summary` (devices) and `/atms/dashboard/stats` (incident) share the `/atms/` prefix but land on **different services** — relies on nginx path routing (ops repo, not in tree).
- `FleetTaskService.getStats()` hard-codes `completedTasks=0`, `failedTasks=0`; never sets `totalModulesReporting` (→ always 0 on the page). Do NOT treat as an outage.
- `getStats()` `pendingTasks` = only `AUTO_UPDATE` PENDING (undercounts other kinds).
- `getStats()` ignores its `institutionKeys` arg → task/artifact totals are fleet-wide even for scoped users (inconsistent with scoped ATM/incident numbers).
- Each SPA loader `try/catch`es independently: one failing call zeroes only its card, silently (`console.error` only).
- CUSTOMER role only hides the "View all" incidents link; not an admin surface.
- Auto-refresh state in `localStorage['dashboard_autoRefresh']`; 60s tick.
## Do NOT
- Do NOT add endpoints or tables to `hiveops-dashboard` — it is a pure client; put new data in the owning service.
- Do NOT assume `/atms/dashboard/stats` hits incident — verify routing before diagnosing wrong incident counts.
- Do NOT report Completed/Failed/Modules-Reporting = 0 as a bug in the dashboard; it originates in `FleetTaskService.getStats`.
- Do NOT expect fleet task/artifact totals to respect institution scope.
- Do NOT write to `device_summary` to "fix" dashboard ATM counts — those come from the devices `atms` table.
- Do NOT add role guards to `/atms/summary` or `/atms/dashboard/stats` assuming MSP_ADMIN-only; current auth is any valid JWT, institution-scoped.
Refs: [platform.service-ownership] · [platform.kafka] · [platform.data-architecture] · [dashboard.overview]
backend/src/main/resources/content/dashboard/overview/internal.md
+42
View File
@@ -0,0 +1,42 @@
---
module: dashboard.overview
title: Fleet Overview
tab: Internal
order: 20
audience: internal
---
Ops/support view of the **Fleet Overview** landing page (`dashboard.bcos.cloud`, hiveops-dashboard, port 5181). This is a **frontend-only SPA** — it holds no data of its own. Every number on the page is fetched live from three other backends. If a card is wrong, the bug is almost always in the owning backend, not the dashboard.
## What the page actually calls
The SPA loads everything through one axios base URL (`window.__APP_CONFIG__.apiUrl` = `VITE_API_URL`, default `http://localhost:8080`) and fans out to four endpoints:
| Card / section | Endpoint (relative to `apiUrl`) | Owning service |
|---|---|---|
| Total ATMs, Connected, Fleet Health bars, By Model | `GET /atms/summary` | **hiveops-devices** |
| Open Incidents, Critical, Incidents this week, Recent Incidents | `GET /atms/dashboard/stats` | **hiveops-incident** |
| Active Tasks, Fleet Operations counts, Artifacts, Modules Reporting | `GET /fleet/tasks/stats` | **hiveops-fleet** |
| Sidebar user / role (hides "View all" for CUSTOMER) | `GET /users/me` | **hiveops-incident** |
`loadAll()` fires all three data calls in parallel every 60s (auto-refresh toggle, persisted in `localStorage['dashboard_autoRefresh']`) plus a manual refresh button.
## Roles required
- **`/atms/summary`** and **`/atms/dashboard/stats`** — no method-level `@PreAuthorize`; access is a valid JWT enforced by each service's SecurityFilterChain. Results are **institution-scoped** via `InstitutionContext.resolveScope()` (a scoped user sees only their institutions' ATMs/incidents).
- **`/fleet/tasks/stats`** — `@PreAuthorize("hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')")`.
- **CUSTOMER** role only changes one thing on this page: the **"View all →"** link on Recent Incidents is hidden.
- There are **no admin-only actions** on this page — it is read-only. Data mutations happen in the Incidents / Fleet / Devices apps.
## First things to check when it misbehaves
1. **A whole card reads 0 / blank** → open DevTools Network. Each of the 3 data calls fails independently (each has its own `try/catch` that only `console.error`s). A single 401/403/500 silently zeroes just that card; the rest of the page still renders.
2. **"Completed" and "Failed" task counts are always 0, "Modules Reporting" always 0** → this is **expected / a known backend limitation**, not a data outage. `FleetTaskService.getStats()` hard-codes `completed=0`, `failed=0` and never sets `totalModulesReporting`. Do not chase it as a dashboard bug.
3. **"Pending" tasks look too low** → `getStats()` counts only `AUTO_UPDATE` PENDING tasks, not all pending kinds. Also a backend limitation.
4. **Task/Artifact totals ignore institution** → `getStats()` does **not** apply `institutionKeys`; totals are fleet-wide even for a scoped user, while the ATM and incident numbers above ARE scoped. Expect the "Active Tasks" number to look inconsistent with the scoped ATM counts.
5. **Recent Incidents / incident counts wrong** → suspect nginx routing of `/atms/dashboard/stats`. This exact path exists in **both** hiveops-incident and hiveops-devices; only the incident version populates `recentIncidents`, `openIncidents`, `criticalIncidents`, `incidentsThisWeek`. If those fields are empty but ATM counts are fine, the call is landing on devices instead of incident.
6. **Numbers stale for ~a while** → `getDashboardStats` in incident is `@Cacheable("dashboardStats")` keyed by institution scope. A cache TTL/eviction lag, not a broken query.
7. **Auto-refresh "stuck"** → check the toggle; state persists in `localStorage`. A user who turned it off sees a frozen page until manual refresh.
8. **Fleet Health "Never Connected" spike** → device-side status; a machine added but never checked in (`agentConnectionStatus == NEVER_CONNECTED`). Not a dashboard fault.
## Connection-status definitions (as computed in devices)
`connected/disconnected/neverConnected` come from `AtmDTO.getAgentConnectionStatus()`, derived from each ATM's last heartbeat (`CONNECTED` < grace, `DISCONNECTED` beyond it, `NEVER_CONNECTED` = never checked in). If health bars disagree with the Devices app, both read the same `hiveiq_devices` source — reconcile there.
See also: [dashboard.overview] (customer tab), [platform.service-ownership].
backend/src/main/resources/content/devices/agent-profile/architect.md
+80
View File
@@ -0,0 +1,80 @@
---
module: devices.agent-profile
title: Agent Profile
tab: Architect
order: 30
audience: internal
---
How the **Agent Profile** feature is built end to end. Profiles are lightweight hardware-grouping records; the "which modules" logic is resolved at render time by joining profile records against a CDN-hosted module manifest. See [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Services involved
| Component | Role |
|-----------|------|
| `hiveops-devices` frontend (Svelte, port 5177) | `components/Settings/AgentProfile.svelte` — CRUD table + modal, module-count expansion |
| `hiveops-devices` backend (Spring Boot, port 8096) | `AgentProfileController` / `AgentProfileService` / `AgentProfileRepository` — owns profile records and proxies the CDN manifest |
| CDN (`cdn.bcos.cloud`) | Hosts `downloads/agent/modules/modules.json` — the module→profile mapping |
| Agent (on-ATM) | Reads its own `atm.hardware.profile` property and pulls matching modules |
There is **no cross-service call** for this feature: it is entirely within `hiveops-devices`, which is the source of truth for device/agent-config data. The only external dependency is the CDN fetch.
## Data flow
### Profile CRUD (admin write path)
```
AgentProfile.svelte ──HTTP──▶ AgentProfileController ──▶ AgentProfileService ──▶ agent_profiles (hiveiq_devices)
getAll/create/update/delete (@PreAuthorize on mutations) (@Transactional)
```
- `getAll()` → `repo.findAllByOrderBySortOrderAscProfileKeyAsc()` (sort_order asc, then key).
- `create()` → 409 if PK exists; key is trimmed + lowercased.
- `update()` → mutates label/description/enabled/sortOrder only; **key is immutable** (PK, disabled in the UI).
- `delete()` → guarded by `isProfileInUse(key)`: `SELECT COUNT(a) > 0 FROM Atm a WHERE a.hardwareProfile = :key`; returns 409 when any ATM still references it.
- `@PrePersist`/`@PreUpdate` set `created_at`/`updated_at`.
### Module resolution (read path — the interesting part)
```
AgentProfile.svelte ─GET /modules-manifest─▶ AgentProfileController
│ 60s in-memory cache (AtomicReference + AtomicLong expiry)
▼
HTTPS GET cdn.bcos.cloud/downloads/agent/modules/modules.json
│ adds X-CDN-Token header (agent.cdn.token) server-side
▼
{"modules":[{name,version,filename,profiles?[]}]}
```
The SPA joins the manifest against each profile **client-side**:
- module with empty/absent `profiles[]` → installs on all ATMs (universal).
- module whose `profiles[]` contains the profileKey → **exclusive** to that profile (★ / "+N exclusive" badge).
So a profile record itself stores no module list; membership is derived. This keeps module bundles versioned on the CDN independent of the DB.
### Agent-side effect (out of scope of this screen, for context)
An ATM's `atm.hardware.profile` agent property is matched against `agent_profiles.profile_key` and against `modules.json`. The `enabled` flag pauses provisioning for that profile.
## Kafka
**None for this feature.** Profile CRUD does not produce or consume Kafka events. (The broader `hiveops-devices` service publishes `hiveops.device.events` for device rows via `DeviceEventPublisher`, and consumes `hiveops.fleet.task.events` / `journal.replenishments` — but agent-profile writes touch neither.) See [platform.kafka].
## DB tables
| Table | Access | Notes |
|-------|--------|-------|
| `agent_profiles` (`hiveiq_devices`) | read/write | PK `profile_key` (varchar 40); `label`, `description` (TEXT), `enabled` (bool), `sort_order` (int), `created_at`, `updated_at`. Flyway `V14__agent_profiles_and_hardware_profile.sql`. |
| `atms` (`hiveiq_devices`) | read-only (from this feature) | `hardware_profile` (varchar 40) — the device→profile link, read by the delete-guard count query. |
## Key API endpoints
| Method + Path | Auth | Purpose |
|---------------|------|---------|
| GET `/api/agent-profiles` | authenticated | List profiles (sorted) |
| POST `/api/agent-profiles` | MSP_ADMIN/BCOS_ADMIN | Create |
| PUT `/api/agent-profiles/{key}` | MSP_ADMIN/BCOS_ADMIN | Update (no key change) |
| DELETE `/api/agent-profiles/{key}` | MSP_ADMIN/BCOS_ADMIN | Delete (guarded) |
| GET `/api/agent-profiles/modules-manifest` | authenticated | 60s-cached CDN manifest proxy, injects `X-CDN-Token` |
## Design notes
- **No executor→Kafka→record-store pattern here** — this is a plain CRUD + read-through-cache feature, not a long-running fleet action. (That pattern applies to fleet tasks, not profiles.)
- **CDN token stays server-side.** The token is only ever attached in `AgentProfileController.getModulesManifest`; the SPA never sees it.
- **Manifest cache is process-local** (`AtomicReference`), so each backend replica caches independently and CDN changes take up to 60s per replica to appear.
- See parent architect doc [technical.devices] for the full devices service map.
backend/src/main/resources/content/devices/agent-profile/claude.md
+56
View File
@@ -0,0 +1,56 @@
---
module: devices.agent-profile
title: Agent Profile
tab: Claude
order: 40
audience: internal
---
Terse act-without-guessing reference. Feature = hardware profiles controlling agent module provisioning.
## Ownership
- **Service:** `hiveops-devices` backend (Spring Boot, port 8096), DB `hiveiq_devices`. Frontend: `hiveops-devices` SPA port 5177 (`devices.bcos.cloud`).
- **Component:** `frontend/src/components/Settings/AgentProfile.svelte`
- **Backend:** `AgentProfileController` / `AgentProfileService` / `AgentProfileRepository` / entity `AgentProfile`.
## Endpoints (all under `/api/agent-profiles`)
| Method + Path | Auth | Notes |
|---|---|---|
| GET `/api/agent-profiles` | any JWT | list, sorted by sort_order asc, then key |
| POST `/api/agent-profiles` | `MSP_ADMIN`/`BCOS_ADMIN` | 400 if blank key/label; 409 if key exists; key lowercased+trimmed |
| PUT `/api/agent-profiles/{key}` | `MSP_ADMIN`/`BCOS_ADMIN` | 400 if blank label; key immutable |
| DELETE `/api/agent-profiles/{key}` | `MSP_ADMIN`/`BCOS_ADMIN` | 409 if any ATM uses it; 404 if missing |
| GET `/api/agent-profiles/modules-manifest` | any JWT | proxies CDN modules.json, 60s cache; 502 `{"modules":[]}` on failure |
## Tables (DB `hiveiq_devices`)
| Table | Use |
|---|---|
| `agent_profiles` | PK `profile_key`(varchar40); `label`,`description`(TEXT),`enabled`(bool),`sort_order`(int),`created_at`,`updated_at`. Flyway V14. |
| `atms` | `hardware_profile`(varchar40) = device→profile link; read by delete-guard `COUNT(*) WHERE hardware_profile=:key`. |
## Kafka topics
- **None** for profile CRUD. Do NOT expect `hiveops.device.events` on profile writes (that topic is device rows only).
## External deps
- CDN manifest: `https://cdn.bcos.cloud/downloads/agent/modules/modules.json` (shape: `{"modules":[{name,version,filename,profiles?[]}]}`).
- Token: `agent.cdn.token` / env `CDN_TOKEN` → sent as header `X-CDN-Token`, server-side only.
## Module→profile logic (client-side join)
- module `profiles` absent/empty ⇒ installs on ALL ATMs (universal).
- module `profiles` contains profileKey ⇒ exclusive (★) to that profile.
## Gotchas
- Read endpoints (`GET list`, `GET modules-manifest`) have **no `@PreAuthorize`** — open to any authenticated user; only mutations are admin-gated.
- `profileKey` is the PK and **immutable**; UI disables the key field on edit. "Rename" = create new + reassign devices + delete old.
- Manifest cache is **per-process** (`AtomicReference`); CDN changes take up to 60s per replica.
- Manifest failure ⇒ HTTP **502** body `{"modules":[]}`; UI shows "Could not load modules manifest from CDN" and blanks the Modules column.
- `enabled=false` pauses provisioning for that profile; ATM match is on `atm.hardware.profile` agent property (lowercase, exact).
- Module membership is NOT in the DB — it lives in CDN `modules.json`. Never look for a module list on the profile row.
## Do NOT
- Do NOT invent module-list columns on `agent_profiles` — modules come from CDN `modules.json`.
- Do NOT try to change a profile's key via PUT — it's ignored/immutable.
- Do NOT expect Kafka events from profile writes.
- Do NOT route these calls through incident/journal — `hiveops-devices` owns them.
- Do NOT expose `X-CDN-Token` to the SPA; only the backend attaches it.
- Do NOT delete a profile still referenced by any `atms.hardware_profile` — reassign first.
backend/src/main/resources/content/devices/agent-profile/internal.md
+44
View File
@@ -0,0 +1,44 @@
---
module: devices.agent-profile
title: Agent Profile
tab: Internal
order: 20
audience: internal
---
Ops/support view for the **Agent Profile** screen (Devices › Settings). Profiles are hardware groupings (`profileKey` + label) that decide which agent modules an ATM receives. This tab covers who can change them, what breaks, and how to fix it fast.
## Who can do what
| Action | Endpoint | Role required |
|--------|----------|---------------|
| List profiles | GET `/api/agent-profiles` | any authenticated user (no `@PreAuthorize`) |
| View modules manifest | GET `/api/agent-profiles/modules-manifest` | any authenticated user |
| Add profile | POST `/api/agent-profiles` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
| Edit profile | PUT `/api/agent-profiles/{key}` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
| Delete profile | DELETE `/api/agent-profiles/{key}` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
All served by `hiveops-devices` backend (port 8096). The read endpoints are open to any valid JWT; only mutations are admin-gated.
## First things to check when it misbehaves
1. **Modules column shows "—" for every profile.** The CDN modules manifest failed to load. The frontend shows the banner *"Could not load modules manifest from CDN."* The backend proxies `https://cdn.bcos.cloud/downloads/agent/modules/modules.json` and returns HTTP **502** with `{"modules":[]}` when the CDN is unreachable or non-200. Check: is the CDN up, and is `agent.cdn.token` / `CDN_TOKEN` set correctly on the devices backend? The token is sent as the `X-CDN-Token` header; if the CDN now requires it and it's missing/stale, the fetch returns non-200 and the column blanks out. Manifest is cached 60s, so allow up to a minute after a fix.
2. **Module counts look wrong / an "exclusive" module isn't showing.** Module-to-profile mapping lives entirely in the CDN `modules.json` (each module's optional `profiles: []` array), NOT in the devices DB. A module with no `profiles` (or an empty array) installs on **every** ATM; a module listing a profileKey installs **only** on ATMs with that profile (the ★ exclusive badge). If a module shows on the wrong profile, fix `modules.json` on the CDN, not the profile record.
3. **Profile save fails with 400.** Backend rejects blank `label` (create + update) and blank `profileKey` (create) with `400 Bad Request`. The form also enforces this client-side.
4. **"Profile key already exists" (409).** Create is refused when the key is already present — keys are the primary key of `agent_profiles`. Edit the existing one instead.
5. **Delete fails with 409 "assigned to one or more devices".** A profile can't be deleted while any ATM has it. See below.
## Common failure modes + fixes
- **Can't delete a profile.** `DELETE` returns **409 Conflict** if `SELECT COUNT(*) FROM atms WHERE hardware_profile = :key > 0`. Reassign or clear those devices' profile first (device detail page, or group bulk-assign), then retry. To find the blockers: query `atms` for `hardware_profile = '<key>'`.
- **ATM not getting its modules.** Two independent gates: (a) the profile row must be **enabled** (Active badge) — a disabled profile pauses provisioning; (b) the ATM's `atm.hardware.profile` agent property must exactly match a `profileKey` (lowercase, no spaces). A mismatch means the ATM falls back to only the universal (non-exclusive) modules and never gets the exclusive ones.
- **Key case/format issues.** Backend lowercases and trims `profileKey` on create (`applyFields`), but the ATM-side `atm.hardware.profile` value is matched as-is against `modules.json` and the `atms` table. Keep everything lowercase.
- **Edit doesn't change the key.** The key field is disabled on edit — `profileKey` is immutable (it's the PK). To "rename" a key you must create a new profile, reassign devices, and delete the old one.
## Data location
- **DB:** `hiveiq_devices` › table `agent_profiles` (columns: `profile_key` PK, `label`, `description`, `enabled`, `sort_order`, `created_at`, `updated_at`). Created by Flyway `V14__agent_profiles_and_hardware_profile.sql`.
- **Device assignment:** `atms.hardware_profile` (varchar 40).
- **Module manifest:** CDN file, not DB — `https://cdn.bcos.cloud/downloads/agent/modules/modules.json`.
No Kafka is involved in editing a profile. Profile writes do **not** publish `hiveops.device.events` (that topic tracks device rows, not profile rows).
backend/src/main/resources/content/devices/agent-scripts/architect.md
+89
View File
@@ -0,0 +1,89 @@
---
module: devices.agent-scripts
title: Agent Scripts
tab: Architect
order: 30
audience: internal
---
> **Architect · internal.** Written 2026-07-01 from source. Describes both what is **built** (a devices-owned CRUD store) and the **intended** signed-delivery pipeline that partly exists in hiveops-fleet + the agent but is **not yet connected** to this module's Push action.
## Ownership & shape
`devices.agent-scripts` is a thin CRUD feature fully owned by **hiveops-devices** (Spring Boot, port 8096, DB `hiveiq_devices`). No Kafka topics, no consumers, no cross-service calls, no agent contact originate from this module. See [platform.service-ownership].
```
Devices SPA (AgentScripts.svelte, Settings)
│ agentScriptAPI → devicesApi (baseURL = window.__APP_CONFIG__.apiUrl, e.g. api.bcos.cloud/devices)
▼
hiveops-devices : AgentScriptController (/api/fleet/scripts)
▼
AgentScriptService → AgentScriptRepository (JPA)
▼
Postgres hiveiq_devices : agent_scripts
```
## Components (hiveops-devices)
| Layer | Class |
|-------|-------|
| Controller | `com.hiveops.devices.controller.AgentScriptController` (`@RequestMapping("/api/fleet/scripts")`) |
| Service | `com.hiveops.devices.service.AgentScriptService` (`@Transactional`) |
| Repository | `com.hiveops.devices.repository.AgentScriptRepository extends JpaRepository<AgentScript, Long>` |
| Entity | `com.hiveops.devices.entity.AgentScript` (`@Table("agent_scripts")`) |
| DTO | `com.hiveops.devices.dto.AgentScriptRequest` (`name`, `description`, `content`, `institutionKey`) |
## Data — `agent_scripts` (Flyway `V15__create_agent_scripts.sql`)
| Column | Type | Notes |
|--------|------|-------|
| `id` | BIGSERIAL PK | `GenerationType.IDENTITY` |
| `name` | VARCHAR(200) NOT NULL | file name; extension implies type |
| `description` | TEXT | nullable |
| `content` | TEXT NOT NULL | raw script body |
| `institution_key` | VARCHAR(50) | nullable; indexed (`idx_agent_scripts_institution_key`) |
| `created_at` | TIMESTAMPTZ | `@CreationTimestamp` / `DEFAULT now()` |
| `updated_at` | TIMESTAMPTZ | `@UpdateTimestamp` / `DEFAULT now()` |
Repository queries: `findAllByOrderByNameAsc()` and `findByInstitutionKeyOrderByNameAsc(String)`. This is the **only** persistence — no mirror tables, no `device_summary` involvement. See [platform.data-architecture].
## API surface
| Method + Path | Behaviour |
|---------------|-----------|
| `GET /api/fleet/scripts?institutionKey=` | Lists all, or filtered by institution key. |
| `POST /api/fleet/scripts` | 400 if name/content blank; else 201 with saved entity. Sets `institutionKey` from body. |
| `PUT /api/fleet/scripts/{id}` | 404 if missing; updates name (if non-blank), description, content. |
| `DELETE /api/fleet/scripts/{id}` | 404 if missing; else 204. |
| `POST /api/fleet/scripts/{id}/push` | **Stub** — returns `200` with `List.of()`. Comment in code: "requires fleet service integration (not yet implemented)". |
All gated by `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")` at class level.
## Kafka
**None for this module.** It neither produces nor consumes. (The devices service overall publishes `hiveops.device.events` for device inventory, but agent-scripts writes never touch that path.) See [platform.kafka].
## Intended signed-delivery pipeline (NOT wired to this module)
The Overview/SPA copy promises "signed server-side with RSA-SHA256, agent verifies before writing" and "SCRIPT_UPDATE fleet tasks." That machinery exists — but in **hiveops-fleet** and the **agent**, not behind the devices Push endpoint:
```
(intended) devices push ──X──▶ hiveops-fleet FleetTaskService (kind = UPDATE_SCRIPT)
│ ScriptSigningService.sign(scriptBytes) [SHA256withRSA, private key]
▼ configPatch += signature
fleet_tasks ──▶ agent-proxy ──▶ ATM agent
│
hiveops-agent hiveops-file-collection ProcessScriptUpdate
verifySignature(...) [SHA256withRSA public key] ──▶ writes to scripts dir
```
- **hiveops-fleet:** `FleetTaskKind.UPDATE_SCRIPT` (and `RUN_SCRIPT`); `ScriptSigningService` loads a PKCS8 RSA private key from `script.signing.private-key` (env `SCRIPT_SIGNING_PRIVATE_KEY`) and signs `SHA256withRSA`. Signed tasks flow through the standard fleet-task → agent-proxy path (executor→task-store pattern).
- **agent:** `com.hiveops.filemgmt.ProcessScriptUpdate` requires `scriptName`, `scriptContent` (base64), `scriptSignature`; rejects unsigned scripts; verifies `SHA256withRSA` against an embedded public key; only writes files with `.cmd`/`.sh`/`.ps1` extensions.
The gap: the devices `push` endpoint is a no-op stub and never calls hiveops-fleet, so nothing in this module actually creates a task or signs anything today. There is also a **config-patch key mismatch** between the fleet signer and the agent consumer (see Claude tab / uncertainties) that would need reconciling if the two are ever wired together.
## Cross-links
- [platform.data-architecture] — where device-domain data lives.
- [platform.kafka] — device event bus (not used by this module).
- [platform.service-ownership] — devices owns `agent_scripts`; fleet owns task delivery + signing.
backend/src/main/resources/content/devices/agent-scripts/claude.md
+50
View File
@@ -0,0 +1,50 @@
---
module: devices.agent-scripts
title: Agent Scripts
tab: Claude
order: 40
audience: internal
---
> Act-without-guessing reference. Only confirmed facts (hiveops-devices source, 2026-07-01).
## Owner
- **Service:** `hiveops-devices` (Spring Boot, port `8096`, DB `hiveiq_devices`).
- Served under `/api/fleet/scripts` **but owned by devices, NOT hiveops-fleet.**
- SPA: `hiveops-devices/frontend` → `components/Settings/AgentScripts.svelte`, `lib/api.ts` → `agentScriptAPI` on `devicesApi` (base `window.__APP_CONFIG__.apiUrl` else `http://localhost:8096`).
## Endpoints (all `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`)
| Method | Path | Result |
|--------|------|--------|
| GET | `/api/fleet/scripts` | list; optional `?institutionKey=` |
| POST | `/api/fleet/scripts` | 400 if name/content blank, else 201 |
| PUT | `/api/fleet/scripts/{id}` | 404 if missing |
| DELETE | `/api/fleet/scripts/{id}` | 204 (hard delete) |
| POST | `/api/fleet/scripts/{id}/push` | **STUB → 200 + `[]`, does nothing** |
## Data
- **Table:** `agent_scripts` (Flyway `V15`). Cols: `id`, `name` VARCHAR(200), `description` TEXT, `content` TEXT, `institution_key` VARCHAR(50), `created_at`, `updated_at`. Index: `idx_agent_scripts_institution_key`.
- **Entity/Repo/Service/Controller:** `AgentScript` / `AgentScriptRepository` / `AgentScriptService` / `AgentScriptController` (pkg `com.hiveops.devices.*`).
- Repo methods: `findAllByOrderByNameAsc`, `findByInstitutionKeyOrderByNameAsc`.
## Kafka / topics
- **None.** This module produces/consumes nothing.
## Signed delivery (lives elsewhere, NOT wired here)
- hiveops-fleet: `FleetTaskKind.UPDATE_SCRIPT` + `ScriptSigningService` (`SHA256withRSA`, key `script.signing.private-key` / env `SCRIPT_SIGNING_PRIVATE_KEY`).
- agent: `com.hiveops.filemgmt.ProcessScriptUpdate` verifies `SHA256withRSA`, needs patch keys `scriptName` + `scriptContent` (base64) + `scriptSignature`.
## Gotchas
- Devices Push button is a **no-op**; SPA still shows a success toast. Never treat a push as delivered.
- Path says `fleet` but the backend is **devices** — route to `api.bcos.cloud/devices`, not `/fleet`.
- Name extension check (`.cmd/.sh/.ps1`) is **client-side only**; backend accepts any non-blank name.
- SPA `getAll()` never sends `institutionKey` → lists ALL institutions; `create()` never sends it → rows saved with `institution_key = NULL`.
- Edit locks `name` (SPA sends only `{description, content}` on PUT).
- Fleet signer writes patch key `signature` over raw `script`; agent reads `scriptSignature` + base64 `scriptContent` — key/format mismatch (see uncertainties).
## Do NOT
- Do NOT claim scripts reach ATMs via this module's Push endpoint — it's a stub.
- Do NOT look for a Kafka topic or `device_summary` mirror for scripts — none exist.
- Do NOT expect institution scoping from the UI — it lists/creates unscoped.
- Do NOT search hiveops-fleet for `/api/fleet/scripts` — that path is a devices controller.
- Do NOT rely on backend name-extension validation.
backend/src/main/resources/content/devices/agent-scripts/internal.md
+63
View File
@@ -0,0 +1,63 @@
---
module: devices.agent-scripts
title: Agent Scripts
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support.** Grounded in `hiveops-devices` source (2026-07-01). This is a CRUD store today; the "Push to fleet" delivery path is **not wired up** in the backend — read the failure notes below before telling a customer a script was delivered.
## What this module actually is
A per-fleet library of shell scripts (`.cmd` / `.ps1` / `.sh`) stored in the **hiveops-devices** backend. Screen lives under **Settings → Agent Scripts** in the Devices SPA (`AgentScripts.svelte`). Owned entirely by hiveops-devices — no other service, no Kafka, no agent contact from this module's own endpoints.
- **Service:** `hiveops-devices` (Spring Boot, port 8096, DB `hiveiq_devices`)
- **Table:** `agent_scripts`
- **API base:** `/api/fleet/scripts` (yes — `fleet` in the path, but it is served by devices, not hiveops-fleet)
## Roles required
Every endpoint on `AgentScriptController` is gated at the class level:
```java
@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")
```
So **all** actions (list, create, edit, delete, push) require `MSP_ADMIN` or `BCOS_ADMIN`. There is no read-only role. A user without one of these roles gets 403 on the initial `GET /api/fleet/scripts` and the list simply fails to load.
## Admin actions
| Action | Endpoint | Notes |
|--------|----------|-------|
| List scripts | `GET /api/fleet/scripts` | Optional `?institutionKey=`; the SPA never sends it, so the list shows **all** institutions' scripts. |
| Add | `POST /api/fleet/scripts` | Name + content required (blank → 400). Name must end `.cmd`/`.sh`/`.ps1` (validated **client-side only**). |
| Edit | `PUT /api/fleet/scripts/{id}` | SPA sends `{description, content}` only — name is locked after create. |
| Delete | `DELETE /api/fleet/scripts/{id}` | Hard delete, 204. No soft-delete, no audit row. |
| Push | `POST /api/fleet/scripts/{id}/push` | **No-op stub** — see below. |
## First things to check when it misbehaves
1. **"I pushed a script but nothing happened on the ATMs."** Expected. `POST /api/fleet/scripts/{id}/push` is a **stub** — it returns `200 OK` with an empty list and creates **no** fleet task. The SPA still shows a green "pushed to N ATMs" toast. Do not tell the customer a script was delivered based on that toast. (The real signed-delivery pipeline lives in **hiveops-fleet** as `UPDATE_SCRIPT` tasks, but this module's Push button is not connected to it.)
2. **List won't load / 403.** User lacks `MSP_ADMIN`/`BCOS_ADMIN`, or the JWT didn't reach the devices backend. Check the token role claim and that the request hit `api.bcos.cloud/devices` (devices backend), not the incident/fleet base.
3. **List empty but scripts exist.** Confirm you're pointed at the right DB — `agent_scripts` in `hiveiq_devices` (devices VM path via ProxyJump to `10.10.10.188`). Scripts created here have `institution_key = NULL` (SPA never sets it), so any institution-scoped query returns nothing.
4. **Can't rename a script.** By design — the name field is disabled on edit and the SPA doesn't send `name` on `PUT`. To rename, delete + recreate.
5. **Save rejected "Name must end with .cmd, .sh, or .ps1".** That check is **client-side only**. A direct `POST` with any name will be accepted by the backend as long as name+content are non-blank.
## Common failure modes + fixes
| Symptom | Cause | Fix |
|---------|-------|-----|
| Push toast succeeds, no ATM change | Backend push is a stub (returns `[]`) | Deliver via hiveops-fleet `UPDATE_SCRIPT` fleet task instead; treat this Push button as non-functional. |
| 403 on load | Missing admin role | Grant `MSP_ADMIN`/`BCOS_ADMIN`. |
| Scripts from other institutions visible | SPA lists unscoped (no `institutionKey`) | Known; filter is supported in the API but unused in the UI. |
| Edit blanks a description | `PUT` overwrites `description` with the request value each time | Always resend the full description on edit (SPA does this). |
## Quick DB check
```sql
-- on hiveiq_devices (database VM 10.10.10.188, via ProxyJump)
SELECT id, name, institution_key, updated_at FROM agent_scripts ORDER BY name;
```
See also the customer-facing **Overview** tab, the **Architect** tab for the intended signed-delivery design, and [platform.service-ownership].
backend/src/main/resources/content/devices/config-audit/architect.md
+84
View File
@@ -0,0 +1,84 @@
---
module: devices.config-audit
title: Config Audit
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture view.** How Config Audit is built inside `hiveops-devices`. Grounded in source 2026-07-01.
## Shape at a glance
Config Audit is a **single-service, synchronous, DB-backed** feature. There is **no Kafka, no async executor, and no record-store fan-out** for this feature — the audit row is written in the same transaction as the config save it records. (Contrast with the executor→Kafka→record-store pattern used elsewhere in the platform; it does **not** apply here.)
- **Service owner:** `hiveops-devices` (backend port **8096**, SPA `devices.bcos.cloud` port 5177). Devices owns all device/institution config data. See [platform.service-ownership].
- **Database:** `hiveiq_devices`. See [platform.data-architecture] / [platform.data-stores].
- **Messaging:** none for this feature. See [platform.kafka] for topics the rest of `hiveops-devices` produces/consumes (`hiveops.device.events`, etc.) — Config Audit touches none of them.
## Components
| Layer | Class | Role |
|-------|-------|------|
| Frontend | `Settings/ConfigAudit.svelte` | Table + institution filter + paging + per-row diff expander |
| Frontend API | `institutionKeyAPI.getConfigAudit()` (`lib/api.ts`) → `devicesApi` (base `VITE_API_URL`, else `http://localhost:8096`) | Calls `GET /api/institution-keys/config-audit` |
| Controller | `InstitutionKeyController` (`@RequestMapping("/api/institution-keys")`) | `getConfigAudit` (read), `saveAgentProperties` (write trigger) |
| Service | `InstitutionKeyService` | `getAuditLog()` (read + diff), `saveAgentProperties()` (write + audit insert) |
| Repository | `InstitutionConfigAuditRepository` (JPA) | `findAllByOrderByChangedAtDesc`, `findByInstitutionKeyOrderByChangedAtDesc` |
| Entity / table | `InstitutionConfigAudit` → `institution_config_audit` | Stored snapshots |
| Scoping | `InstitutionContext.resolveScope()` | Per-role institution key scope from JWT |
## Write path (how a row is created)
The audit row is a **side effect of saving agent properties** — there is no standalone "create audit" API.
```
Agent Properties save (PUT /api/institution-keys/{key}/agent-properties)
→ InstitutionKeyController.saveAgentProperties() [@PreAuthorize MSP_ADMIN|BCOS_ADMIN, verifyKeyAccess]
→ InstitutionKeyService.saveAgentProperties(key, dto)
1. load existing InstitutionAgentProperties (institution_agent_properties)
2. previousConfig = entity.hiveopsConfig (old JSON string)
3. newConfig = serialize(dto.hiveopsConfig)
4. save entity (updates institution_agent_properties) ← evicts agentProperties cache
5. changedBy = SecurityContext principal name, else "system"
6. auditRepository.save(InstitutionConfigAudit{key, changedBy, now, previousConfig, newConfig})
```
- Snapshots stored are the **serialized `hiveopsConfig` JSON only** (the `ejConfig` half of the DTO is persisted to `institution_agent_properties` but is **not** captured in the audit snapshot).
- Both writes happen in the same service call; a serialization failure throws before either save (`RuntimeException("Failed to save agent properties")`).
## Read path (how the diff is produced)
```
GET /api/institution-keys/config-audit?institutionKey&page&size (no @PreAuthorize)
→ InstitutionKeyController.getConfigAudit()
scopedKeys = InstitutionContext.resolveScope()
effectiveKey = (scoped) ? scopedKeys.get(0) : institutionKey // param used only for unrestricted roles
→ InstitutionKeyService.getAuditLog(effectiveKey, page, size)
- PageRequest.of(page, size)
- repo query: byInstitutionKey…Desc if key present, else findAll…Desc
- for each row: computeDiff(previous_config, new_config) → Map<field, {from,to}>
- map to InstitutionConfigAuditDTO {id, institutionKey, institutionName, changedBy, changedAt, changes}
```
- **Diff is computed on read**, not stored. `computeDiff` parses both JSON snapshots with Jackson and walks fields: added/changed (`!newVal.equals(oldVal)`) then removed (present in prev, absent in next → `to = null`). Parse errors are swallowed (warn-logged), yielding a partial/empty diff.
- `institutionName` is resolved by joining every audit row against an in-memory map built from `institution_keys.findAll()` on each request (cheap; small table).
- Paging: server returns Spring `Page` (`content/totalPages/number`); default size 50, matching the frontend request.
## DB tables
| Table | Access | Notes |
|-------|--------|-------|
| `institution_config_audit` | **written** on save, **read** for the log | `id` (identity), `institution_key` (≤10), `changed_by`, `changed_at` (default `now()`), `previous_config` TEXT, `new_config` TEXT. Indexes: `(institution_key)`, `(changed_at DESC)`. Created in `V7__create_institution_config_audit.sql`. |
| `institution_agent_properties` | written on save, read for previous snapshot | Source of the config being audited (`hiveops_config`, `ej_config`) |
| `institution_keys` | read | Key → institution name resolution + filter dropdown |
## Key endpoints
| Method | Path | Purpose | Auth |
|--------|------|---------|------|
| `GET` | `/api/institution-keys/config-audit` | Paged audit log (+ computed diffs) | authenticated; scoped by role |
| `PUT` | `/api/institution-keys/{key}/agent-properties` | Save agent props → creates audit row | `MSP_ADMIN`/`BCOS_ADMIN` |
| `GET` | `/api/institution-keys` | Filter dropdown (delegates to `/accessible`) | authenticated; scoped |
Cross-links: [platform.data-architecture], [platform.kafka], [platform.service-ownership], [technical.devices].
backend/src/main/resources/content/devices/config-audit/claude.md
+55
View File
@@ -0,0 +1,55 @@
---
module: devices.config-audit
title: Config Audit
tab: Claude
order: 40
audience: internal
---
> Terse agent reference. All facts below verified in `hiveops-devices` source (2026-07-01).
## Ownership
- **Service:** `hiveops-devices` (backend port **8096**; SPA `devices.bcos.cloud` port 5177).
- **DB:** `hiveiq_devices` (Postgres, separate DB VM — ProxyJump to reach).
- **Kafka:** none for this feature. Do not look for a topic.
- **Frontend:** `frontend/src/components/Settings/ConfigAudit.svelte`; API in `lib/api.ts` → `institutionKeyAPI.getConfigAudit`.
## Endpoints (backend-relative; base = `VITE_API_URL` / `http://localhost:8096`)
| Method | Path | Auth | Notes |
|--------|------|------|-------|
| `GET` | `/api/institution-keys/config-audit?institutionKey={k}&page={n}&size={s}` | authenticated, **no `@PreAuthorize`** | Paged log; diffs computed on read; default size 50 |
| `PUT` | `/api/institution-keys/{key}/agent-properties` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` | The ONLY thing that creates an audit row |
| `GET` | `/api/institution-keys` (→ `/accessible`) | authenticated, scoped | Dropdown source |
## Tables
- `institution_config_audit` — the log. Cols: `id, institution_key, changed_by, changed_at, previous_config(TEXT), new_config(TEXT)`. Indexes `(institution_key)`, `(changed_at DESC)`. Migration `V7`.
- `institution_agent_properties` — the audited config (`hiveops_config`, `ej_config`).
- `institution_keys` — key→name resolution.
## Roles / scoping (`InstitutionContext.resolveScope()`)
- `null` (unrestricted): `BCOS_ADMIN`, `ADMIN`, `QDS_ADMIN`, `USER` → `institutionKey` param honored (blank = all).
- scoped list: `CUSTOMER`, `MSP_ADMIN` → forced to `scopedKeys.get(0)`; **param ignored**.
## Gotchas
- Audit row is a **side effect of `PUT …/agent-properties`** — there is no create/delete audit endpoint.
- **Diffs are NOT stored.** Only full `previous_config`/`new_config` JSON snapshots are; `InstitutionKeyService.computeDiff` regenerates the field diff per read.
- Only `hiveopsConfig` is snapshotted in the audit; `ejConfig` changes are saved to props but **not** captured in the audit diff.
- First save → `previous_config = null` → UI badge "Initial save", all fields `from = null`.
- `changed_by` = SecurityContext principal name, else literal `"system"`.
- Scoped user (esp. multi-institution `MSP_ADMIN`) sees **only their first institution key**; dropdown filter is a no-op for them.
- `computeDiff` swallows JSON parse errors (warn log) → can return empty/partial diff silently.
- `institution_key` column is `VARCHAR(10)`.
## Do NOT
- Do **not** claim Config Audit publishes/consumes Kafka — it does not.
- Do **not** claim the diff is persisted — it is computed on read.
- Do **not** assume `MSP_ADMIN` can filter across institutions here — backend pins to the first scoped key.
- Do **not** look for this data in `hiveops-incident` — it lives in `hiveops-devices` / `hiveiq_devices`.
- Do **not** invent an "add/edit/delete audit" API — the log is append-only via config saves.
- Do **not** run `docker exec hiveiq-postgres` on the services VM — Postgres is on the separate DB VM.
backend/src/main/resources/content/devices/config-audit/internal.md
+58
View File
@@ -0,0 +1,58 @@
---
module: devices.config-audit
title: Config Audit
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support view.** Grounded in `hiveops-devices` source (2026-07-01). Config Audit is the read-only change log of every save to an institution's **agent properties** (the per-institution `hiveops_config` blob edited on the Agent Properties screen).
## What it actually is
- A history table in the `hiveops-devices` backend. **One row per save** to an institution's agent properties.
- The row stores a full `previous_config` / `new_config` JSON snapshot; the field-level before/after diff shown in the UI is computed **on read**, not stored.
- **Read-only.** Viewing the audit never changes any config. Nothing here pushes config to devices — that is a separate CONFIG_UPDATE fleet action.
## Who can do what (roles)
| Action | Endpoint | Auth |
|--------|----------|------|
| **View** the audit log | `GET /api/institution-keys/config-audit` | Any authenticated user (no `@PreAuthorize`) — results scoped by role, see below |
| **Create** an audit row (side effect of saving agent props) | `PUT /api/institution-keys/{key}/agent-properties` | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
- There is **no admin-only action inside Config Audit itself** — it is purely a viewer. The only way to add a row is for an `MSP_ADMIN`/`BCOS_ADMIN` to save agent properties.
- `changedBy` is captured from the Spring `SecurityContext` principal name at save time (falls back to literal `"system"` if no auth present).
## Scoping (who sees which rows)
Scope comes from `InstitutionContext.resolveScope()` (institution keys are carried in the JWT):
- **BCOS_ADMIN / ADMIN / QDS_ADMIN / USER** → unrestricted. The `institutionKey` query param is honored (blank = all institutions).
- **CUSTOMER / MSP_ADMIN** → scoped. The backend **ignores the query param** and forces the log to the caller's **first** scoped institution key only.
## First things to check when it misbehaves
1. **"Log is empty / No config changes recorded yet."** Expected until someone actually saves agent properties for that institution. Confirm a save has happened: check `institution_agent_properties.updated_at` for the key. No save → no audit row, by design.
2. **"An MSP_ADMIN can only see one institution's history."** This is current behavior, not a glitch — scoped callers are pinned to `scopedKeys.get(0)` and the institution dropdown filter is effectively a no-op for them. See uncertainties in the Architect tab. Reproduce/confirm by hitting the endpoint as an `MSP_ADMIN` with a multi-institution JWT.
3. **"Changed By shows `system`."** Save was made without an authenticated principal in context (e.g. an internal/service call path). Confirm the caller's JWT is present and being parsed by the auth filter.
4. **"Diff shows the whole config as changed on the first entry."** Correct — the first save has `previous_config = null`, so every field is rendered as an **Initial save** (badge `Initial save`, all fields `from = null`).
5. **Filter dropdown empty / institutions missing.** The dropdown is populated by `GET /api/institution-keys` (delegates to `/accessible`). If it's empty for a scoped user, their JWT institution-key claim is missing/empty — an auth/token issue, not a Config Audit issue.
6. **403 on save (no new audit rows appearing).** The saver lacks `MSP_ADMIN`/`BCOS_ADMIN`, or `verifyKeyAccess` rejected them (scoped user editing an institution outside their scope → `403 Access denied`).
7. **500 / nothing loads.** Backend down or DB unreachable. Service = `hiveops-devices` (port **8096**, DB `hiveiq_devices`). Check the container and Postgres connectivity, then Loki logs for `InstitutionKeyService`.
## Common failure modes → fixes
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| No rows for an institution | Nobody saved agent props yet | Save agent properties once (as admin); a row appears |
| Scoped user can't filter to another institution | By-design pinning to first scoped key | Use an unrestricted admin account to view other institutions |
| `Changed By = system` | Save with no auth principal | Ensure request carries a valid JWT |
| Save succeeds but no audit row | Serialization error thrown before audit save | Check `InstitutionKeyService` logs for `Failed to serialize agent properties` |
| Diff looks wrong / partial | `computeDiff` swallows parse errors (logs warn, returns partial) | Inspect `previous_config`/`new_config` TEXT columns directly in `institution_config_audit` |
## Direct DB inspection (when the UI won't tell you)
Table: `institution_config_audit` in **`hiveiq_devices`** (separate DB VM, reach via ProxyJump — see platform gotchas). Columns: `id, institution_key, changed_by, changed_at, previous_config, new_config`. Ordered newest-first by `changed_at` (indexed).
See also: [devices.config-audit] Overview (customer view), [technical.devices], [platform.service-ownership].
backend/src/main/resources/content/devices/device-detail/architect.md
+80
View File
@@ -0,0 +1,80 @@
---
module: devices.device-detail
title: Device Detail
tab: Architect
order: 30
audience: internal
---
How the single-device page is built. Component `AtmDetail.svelte` (~4.2k lines) is a tabbed shell; the "Details" surface is owned by **hiveops-devices**, but the page is a **multi-service aggregator** — most tabs fan out to other backends. See [platform.service-ownership] and [technical.devices].
## Services involved
| Service | Port | Role for this page |
|---------|------|--------------------|
| hiveops-devices (backend) | 8096 | **Owner** of the device record; serves Details/Agent/Hardware/Software/Cassettes/Config-Changes + logs; publishes device events |
| hiveops-incident | 8081 | Journal events, cassette inventory, incident history |
| hiveops-fleet | — | CONFIG_UPDATE fleet task, packet captures, fitted module status, task history |
| hiveops-journal | 8087 | Transactions / replenishments |
Frontend axios clients (`lib/api.ts`): `devicesApi` (base `apiUrl`, default `:8096`), `incidentApi` (`<gateway>/incident`), `gatewayApi` (gateway root, used for `/fleet/*` and `/journal/*`). `gatewayUrl` is derived by stripping a trailing `/devices` from the configured `apiUrl` (`window.__APP_CONFIG__.apiUrl`).
## Data flow
```
AtmDetail.svelte
├─ devicesApi GET /api/atms/{id} (+ /properties) ← Atm + AtmProperties (source of truth)
├─ devicesApi GET /api/atms/{id}/hardware|config/changes
├─ devicesApi GET /api/atms/{id}/latest-log | log-collects (FileSystemResource off disk)
├─ incidentApi GET /journal-events/atm/{id}/paginated | /cassette-inventory
├─ incidentApi GET /incidents/atm/{id}
├─ gatewayApi GET /fleet/captures?atmId | /fleet/modules/atm/{id} | /fleet/tasks/history?atmId
└─ gatewayApi GET /journal/transactions?atmId&type=REPLENISHMENT
Auto-refresh: setInterval 60_000ms → loadAtms() + loadAtmProperties(id); skipped while editing.
```
## Write paths (all admin-gated) and the executor → Kafka pattern
Every mutating device call runs through `AtmService`, persists to the `atms`/`atm_properties` tables, then publishes to Kafka so the four **device_summary mirrors** stay in sync. There is **no reconciler** — a write that skips `DeviceEventPublisher` silently desyncs mirrors.
| Endpoint | Service method | Kafka |
|----------|----------------|-------|
| `PUT /api/atms/{id}` | `updateAtm` | `publishUpdated` → `DEVICE_UPDATED` |
| `PATCH /api/atms/{id}/institution` | `moveInstitution` (upper-cases key) | `publishUpdated` |
| `PATCH /api/atms/{id}/status` | `updateAtmStatus` | `publishUpdated` |
| `POST /api/atms/{id}/reset-state` | `resetAtmState` (IN_SERVICE, clears supervisor) | *(status write; see code)* |
| `PUT /api/atms/{id}/hardware-profile` | `setHardwareProfile` | `publishUpdated` |
| `DELETE /api/atms/{id}` | `deleteAtm` (soft if active, hard if INACTIVE) | `publishDeleted` → `DEVICE_DELETED` (even on soft-delete) |
### Kafka (see [platform.kafka])
- **Produced:** `hiveops.device.events` (`KafkaProducerConfig.DEVICE_EVENTS_TOPIC`), **keyed by `atmId`**, payload `DeviceEventType ∈ {DEVICE_CREATED, DEVICE_UPDATED, DEVICE_DELETED}` via `DeviceEventPublisher`.
- **Consumers:** the `device_summary` mirrors in incident / fleet / reports / recon (and vault). This page's Details tab is the write origin; the other apps consume its events. See [platform.data-architecture].
## The hardware-profile CONFIG_UPDATE flow (frontend-orchestrated)
Distinct from a pure backend action — the UI does two calls in sequence:
```
1. devicesApi PUT /api/atms/{id}/hardware-profile { profileKey } → persists profile, publishUpdated
2. gatewayApi POST /fleet/tasks → queues the machine update
{ atmIds:[id], taskKind:'CONFIG_UPDATE',
configPatch:{ 'atm.hardware.profile': profileKey } }
```
Step 2 is what actually reaches the ATM; the fleet task executor delivers the config patch. The devices backend does **not** create the fleet task itself.
## DB tables (in `hiveiq_devices`, read/written for this page)
- `atms` (`Atm`) — core record; `atm_properties` (`AtmProperties`) — service state, supervisor flags, custom fields.
- `atm_hardware_changes` / `atm_config_changes` — change audit + review badges (acknowledged here).
- `atm_log_collects` (`AtmLogCollect`) — LOG_COLLECT bundle index; log/bundle bytes live on disk (`atmLogService`).
- `device_cassette_config` + `cassette_configurations`/`cassette_slots` — Cassettes tab.
- Cross-service reads hit incident/journal/fleet DBs, not `hiveiq_devices`.
## Cross-links
- [platform.service-ownership] — devices owns the device record; incident/journal/fleet own their tabs.
- [platform.kafka] — `hiveops.device.events` topic + mirror consumers.
- [platform.data-architecture] — device_summary mirror pattern (publish-on-every-change, no reconciler).
backend/src/main/resources/content/devices/device-detail/claude.md
+77
View File
@@ -0,0 +1,77 @@
---
module: devices.device-detail
title: Device Detail
tab: Claude
order: 40
audience: internal
---
Act-without-guessing reference for `devices.device-detail` (`AtmDetail.svelte`).
## Ownership
- **Owner:** `hiveops-devices` backend, port **8096**, DB **`hiveiq_devices`**, controller `AtmController` (`@RequestMapping({"/api/atms","/api/devices"})`).
- Page is a multi-service aggregator: tabs also call **incident**, **fleet**, **journal**.
## Endpoints — devices-backend (`AtmController` unless noted)
| Method + Path | Purpose | Auth |
|---------------|---------|------|
| `GET /api/atms/{id}` | Device detail | authed |
| `GET /api/atms/{id}/properties` | AtmProperties | authed |
| `PUT /api/atms/{id}` | Update / rename | `MSP_ADMIN`\|`BCOS_ADMIN` |
| `PUT /api/atms/{id}/properties` | Update properties | admin |
| `PATCH /api/atms/{id}/institution` | Move institution (upper-cases key) | admin |
| `PATCH /api/atms/{id}/status` | Set status | admin |
| `POST /api/atms/{id}/reset-state` | → IN_SERVICE, clear supervisor | admin |
| `PUT /api/atms/{id}/hardware-profile` | Set profile (does NOT queue task) | admin |
| `DELETE /api/atms/{id}` | Soft-delete if active, hard if INACTIVE | admin |
| `GET /api/atms/{id}/hardware/changes` | Hardware changes | authed |
| `POST /api/atms/{id}/hardware/changes/{changeId}/acknowledge` | Clear badge | admin |
| `GET /api/atms/{id}/config/changes` | Config changes | authed |
| `POST /api/atms/{id}/config/changes/{changeId}/acknowledge` | Clear badge | admin |
| `POST /api/atms/{id}/cassette-configuration/{configId}` | Apply template | admin |
| `DELETE /api/atms/{id}/cassette-configuration` | Clear template | admin |
| `GET /api/atms/{id}/latest-log` | Text log (from disk) | authed |
| `GET /api/atms/{id}/latest-log-collect` | Latest LOG_COLLECT zip | authed |
| `GET /api/atms/{id}/log-collects[/{collectId}/download]` | Bundle list / download | authed |
| `GET /api/atms/{id}/fleet-tasks` (`DeviceFleetTaskController`) | Task history | admin |
## Endpoints — other services (called from tabs)
| Method + Path | Service |
|---------------|---------|
| `GET /incident/journal-events/atm/{id}/paginated` | incident |
| `GET /incident/journal-events/atm/{id}/cassette-inventory` | incident |
| `GET /incident/incidents/atm/{id}` | incident |
| `POST /fleet/tasks` (CONFIG_UPDATE for hw profile) | fleet |
| `GET /fleet/captures?atmId=` · `/fleet/modules/atm/{id}` · `/fleet/tasks/history?atmId=` | fleet |
| `GET /journal/transactions?atmId=&type=REPLENISHMENT` | journal |
## Tables (`hiveiq_devices`)
- `atms`, `atm_properties`, `atm_hardware_changes`, `atm_config_changes`, `atm_log_collects`, `device_cassette_config`, `cassette_configurations`, `cassette_slots`.
## Kafka
- **Produces:** `hiveops.device.events` — key = `atmId`, `DeviceEventType ∈ {DEVICE_CREATED, DEVICE_UPDATED, DEVICE_DELETED}` (`DeviceEventPublisher`).
- Consumed by device_summary mirrors in incident/fleet/reports/recon (no reconciler).
## Gotchas
- Hardware profile = **two calls**: `PUT .../hardware-profile` THEN `POST /fleet/tasks` (`configPatch:{'atm.hardware.profile':<key>}`). Backend alone does not queue the task.
- `DELETE` on an active device is a **soft delete** (status→INACTIVE) yet still emits `DEVICE_DELETED`; call again to hard-delete.
- `moveInstitution` upper-cases + trims the key; blank is rejected.
- `latest-log*` are `FileSystemResource` reads → error/404 if no file uploaded yet (not a bug).
- Auto-refresh every **60s**, paused while `editing`.
- `server.error.include-message=never` → terse 5xx; get real errors from Loki (`localhost:3100`, services VM).
- Frontend base URL from `window.__APP_CONFIG__.apiUrl`; `gatewayUrl` = that minus trailing `/devices`.
## Do NOT
- Do NOT set a hardware profile without also queuing the CONFIG_UPDATE fleet task — the ATM won't get it.
- Do NOT add device-write endpoints that skip `DeviceEventPublisher` — mirrors silently desync.
- Do NOT expect journal/incident/transaction/capture tabs from devices — they belong to other services; debug the owner.
- Do NOT write incidents/fleet-tasks/journal data into `hiveiq_devices` — respect [platform.service-ownership].
- Do NOT assume reads are admin-gated (they aren't); do NOT assume writes are open (all are `MSP_ADMIN`/`BCOS_ADMIN`).
- Do NOT call it "AI" — the assistant is **Adoons**; brand is **HiveIQ**.
backend/src/main/resources/content/devices/device-detail/internal.md
+57
View File
@@ -0,0 +1,57 @@
---
module: devices.device-detail
title: Device Detail
tab: Internal
order: 20
audience: internal
---
Ops/support view of the single-device page (`AtmDetail.svelte`). The page is served by the **hiveops-devices** frontend (port 5177, `devices.bcos.cloud`) and reads/writes through the **hiveops-devices** backend (port 8096, DB `hiveiq_devices`). Several tabs pull from *other* services — see the failure table below.
## Roles required
- **Read** (device detail, tabs, logs): any authenticated HiveIQ user (institution scoping applies via `verifyAtmAccess`).
- **Every mutating action on this page** is gated by `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`. A non-admin sees the tabs but 403s on save/rename/move/reset/delete/acknowledge/apply-template.
- `GET /api/atms/{id}/fleet-tasks` (Task History tab) is also admin-gated — non-admins get 403 on that tab specifically.
## Admin-only actions (Details tab) → what fires
| UI action | Call | Backend effect |
|-----------|------|----------------|
| Edit → Save | `PUT /api/atms/{id}` | Updates the `atms` row, publishes `DEVICE_UPDATED` |
| Rename (ATM ID / display name) | `PUT /api/atms/{id}` | Same update path |
| Move Institution | `PATCH /api/atms/{id}/institution` | Key is **upper-cased + trimmed**; blank rejected; publishes `DEVICE_UPDATED` |
| Reset to In Service | `POST /api/atms/{id}/reset-state` | Sets status `IN_SERVICE`, `serviceState=IN_SERVICE`, `inSupervisor=false`, clears `supervisorEnteredAt` |
| Deactivate / Delete | `DELETE /api/atms/{id}` | **Soft-delete** if active (status → `INACTIVE`); **hard-delete** if already `INACTIVE` (bulk JPQL, DB `ON DELETE CASCADE`) |
| Set hardware profile | `PUT /api/atms/{id}/hardware-profile` **then** `POST /fleet/tasks` (CONFIG_UPDATE) | Two separate calls — see gotcha below |
| Lookup Coords | `fetch('/api/geocode?address=…')` | Geocode proxy fills lat/long |
| Acknowledge hardware change | `POST /api/atms/{id}/hardware/changes/{changeId}/acknowledge` | Clears the review badge |
| Acknowledge config change | `POST /api/atms/{id}/config/changes/{changeId}/acknowledge` | Clears the Config Changes badge |
| Apply / clear cassette template | `POST` / `DELETE /api/atms/{id}/cassette-configuration[/{configId}]` | Applies or clears the cassette template |
## First things to check when it misbehaves
| Symptom | Likely cause | Fix / where to look |
|---------|--------------|---------------------|
| Save/rename/move returns 403 | User lacks `MSP_ADMIN`/`BCOS_ADMIN`, or JWT expired | Confirm role in the JWT; re-login |
| "Delete" left the device visible but INACTIVE | Expected — first delete is a **soft delete** (status→INACTIVE). Delete again to hard-remove | Filter list by INACTIVE to confirm |
| Agent tab "latest log" 404s / errors | `GET /api/atms/{id}/latest-log` reads a file off disk (`atmLogService.getLatestLogPath`); no file yet if the agent never uploaded | Check the agent actually uploaded; nothing to serve = expected error |
| Log Collect download fails | `GET /api/atms/{id}/latest-log-collect` / `.../log-collects/{id}/download` are FileSystemResource reads | Verify the LOG_COLLECT bundle exists on the services VM |
| Journal / Incidents / Transactions / Captures / Modules tab empty or 5xx | **Those tabs are NOT devices** — they call incident, journal, or fleet backends | Check the owning service (table below), not devices |
| Status/institution shown in another app is wrong | Stale **device_summary mirror** — devices must publish on every change | Confirm `DEVICE_UPDATED` fired; no reconciler exists |
| Hardware/Config Changes red badge won't clear | Acknowledge call 403'd or hit wrong service | Re-run as admin; badge clears only after the acknowledge POST 200s |
| Page not refreshing | Auto-refresh (every 60s) is paused while `editing` is true | Cancel the edit form; polling resumes |
| Terse 500 with no message | `server.error.include-message=never` in devices | Pull the real stack from Loki (`localhost:3100` on services VM) |
## Which backend serves each tab
- **hiveops-devices** (`/api/atms/*`, `/api/institution-keys`, `/api/agent-profiles`, `/api/cassette-configurations`, `/api/atm-makes|models|field-definitions`): Details, System Info, Agent, Software, Hardware, Cassettes, Config Changes, Task History.
- **hiveops-incident** (`/incident/journal-events/*`, `/incident/incidents/*`): Journal events, cassette inventory, Incidents history.
- **hiveops-fleet** (`/fleet/*`): Captures, fitted Modules, fleet task creation (CONFIG_UPDATE), Task History artifacts.
- **hiveops-journal** (`/journal/transactions`): Transactions / replenishments.
## Gotchas
- **Hardware profile is a two-step, frontend-orchestrated flow.** The UI first calls `PUT /api/atms/{id}/hardware-profile`, then separately `POST /fleet/tasks` with `taskKind=CONFIG_UPDATE`, `configPatch={ "atm.hardware.profile": <profileKey> }`. Setting the profile via the backend endpoint alone (or any non-UI caller) persists the profile but does **not** queue the CONFIG_UPDATE — the machine won't be told.
- **Move Institution never blanks a key** and always upper-cases; a wrong-token/mis-imaged box keeps its sticky wrong institution until moved here.
- **Soft-delete still emits `DEVICE_DELETED`** — deactivating a device publishes a delete event, so it disappears from all mirror apps even though the `atms` row survives as INACTIVE.
backend/src/main/resources/content/devices/device-groups/architect.md
+79
View File
@@ -0,0 +1,79 @@
---
module: devices.device-groups
title: Device Groups
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Grounded in `hiveops-devices` backend + `AtmGroups.svelte` + `api.ts` (2026-07-01).
## Ownership
Device Groups are owned end-to-end by **hiveops-devices** (Spring Boot, port **8096**, DB **`hiveiq_devices`**). No other service owns group state. See [platform.service-ownership]. The one cross-service interaction is the hardware-profile push, which reaches into **fleet** (via NGINX `/fleet/*`) and emits a device event on **Kafka**.
## Components (this feature only)
- **Frontend:** `AtmManagement/AtmGroups.svelte` — table + slide-in panels for create/edit and Assign Devices, plus an Assign-Profile modal. Two-panel "shuttle" pickers manage membership client-side before saving. Calls `atmGroupAPI` (and `fleetAPI`, `agentProfileAPI`) in `lib/api.ts`.
- **Controller:** `AtmGroupController` — `@RequestMapping({"/api/atm-groups","/api/device-groups"})` (Phase-2 rename alias).
- **Service:** `AtmGroupService` (`@Transactional(readOnly=true)`, write methods `@Transactional`).
- **Repository:** `AtmGroupRepository` — uses `LEFT JOIN FETCH g.atms` queries (`findByIdWithAtms`, `findAllWithAtms`, `findAllWithAtmsByInstitutionKeyIn`) to avoid lazy-load blowups on the `@ManyToMany`.
- **Entity:** `AtmGroup` → table `atm_groups`; membership is a `@ManyToMany` to `Atm` via join table `atm_group_members` (`group_id`, `atm_id`).
- **DTO:** `AtmGroupDTO` (adds computed `atmIds` sorted + `atmCount`); request DTO `CreateAtmGroupRequest`.
## Data stores (read/written)
| Table | Access | Notes |
|-------|--------|-------|
| `atm_groups` | R/W | group name (`UNIQUE`, global), description, `color`, `institution_key` (len 10), timestamps |
| `atm_group_members` | R/W | join table; membership replace/add/remove all mutate here |
| `atms` | R (membership resolve) / W (hardware-profile only) | `setGroupHardwareProfile` writes `atm.hardware_profile` per member |
See [platform.data-architecture]. Schema is Flyway-managed; `ddl-auto=validate`.
## Data flow — plain group CRUD
1. Frontend `atmGroupAPI.*` → devices backend controller → service → repository → `atm_groups` / `atm_group_members`.
2. Reads are institution-filtered by `InstitutionContext.resolveScope()`; single-institution callers have their `institution_key` forced on create.
3. **No Kafka is emitted for group create/update/delete/membership changes.** Groups are a devices-local concept and are *not* mirrored into the `device_summary` tables of incident/fleet/reports/recon.
## Data flow — Assign Profile (the cross-service path)
This is the executor → Kafka → downstream pattern for this feature:
```
User picks profile in Assign-Profile modal
→ POST /api/atm-groups/{id}/hardware-profile { profileKey } (devices backend)
AtmGroupService.setGroupHardwareProfile():
for each member Atm:
atm.setHardwareProfile(profileKey.trim().toLowerCase()) // normalised
atmRepository.save(atm)
DeviceEventPublisher.publishUpdated(atm) ──► Kafka hiveops.device.events (DEVICE_UPDATED, keyed by atmId)
returns { atmIds: [...] } // sorted member ids
→ Frontend, if atmIds non-empty:
POST /fleet/tasks { atmIds, taskKind: "CONFIG_UPDATE",
configPatch: { "atm.hardware.profile": profileKey } } (fleet backend, via /fleet/*)
```
So a single profile assign produces **two** effects: (a) N `DEVICE_UPDATED` Kafka events consumed by the device-summary mirrors ([platform.kafka]), and (b) one batch of `CONFIG_UPDATE` fleet tasks that the agents pull to actually apply `atm.hardware.profile`. The Kafka publish and the fleet task are independent — the fleet task is orchestrated by the **frontend**, not the backend, so a backend-only caller of `/hardware-profile` updates DB + emits events but does **not** queue the config push.
## Key API endpoints
| Method + Path | Auth | Purpose |
|---------------|------|---------|
| `GET /api/atm-groups` | authed | List (institution-scoped) |
| `GET /api/atm-groups/{id}` | authed | One group |
| `POST /api/atm-groups` | admin | Create |
| `PUT /api/atm-groups/{id}` | admin | Update (partial; non-null fields only) |
| `DELETE /api/atm-groups/{id}` | admin | Delete (clears members first) |
| `PUT /api/atm-groups/{id}/atms` | admin | **Replace** all members (`List<Long>` body) |
| `POST /api/atm-groups/{id}/atms/{atmId}` | admin | Add one member |
| `DELETE /api/atm-groups/{id}/atms/{atmId}` | admin | Remove one member |
| `POST /api/atm-groups/{id}/hardware-profile` | admin | Bulk-set `atm.hardware_profile`, returns `{atmIds}` |
| `POST /fleet/tasks` | authed | (fleet) `CONFIG_UPDATE` push queued by frontend after profile assign |
admin = `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`.
## Kafka
- **Produced (only via profile assign):** `hiveops.device.events`, type `DEVICE_UPDATED`, keyed by `atmId`, from `DeviceEventPublisher.publishUpdated`. Consumed by the device-summary mirrors in incident/fleet/reports/recon. See [platform.kafka].
- **No group-specific topic exists.** Group identity/membership is never published.
## Design notes / trade-offs
- **Default group is not persisted server-side** — it lives in browser `localStorage` (`hiveops_defaultGroupId`, `stores.ts`) and only drives the `groupId` filter on `/api/atms/paginated`.
- **Group name uniqueness is global**, not per-institution — a multi-tenant naming collision surfaces as a DB constraint error.
- **Membership save is a full-set replace** (`assignAtms` / `updateGroup` with `atmIds`), so the client must send the complete intended set.
- Cross-link: [platform.data-architecture], [platform.kafka], [platform.service-ownership], and the service-level [technical.devices].
backend/src/main/resources/content/devices/device-groups/claude.md
+56
View File
@@ -0,0 +1,56 @@
---
module: devices.device-groups
title: Device Groups
tab: Claude
order: 40
audience: internal
---
## Owner
- Service: **hiveops-devices** (port 8096, DB `hiveiq_devices`). Groups live entirely here.
- Controller `AtmGroupController`, base `@RequestMapping({"/api/atm-groups","/api/device-groups"})` (Phase-2 alias — both route).
- External via NGINX: `https://api.bcos.cloud/devices/api/atm-groups`.
## Endpoints (METHOD path — auth)
| Method | Path | Auth | Body / returns |
|--------|------|------|----------------|
| GET | `/api/atm-groups` | any authed | list, institution-scoped |
| GET | `/api/atm-groups/{id}` | any authed | one group |
| POST | `/api/atm-groups` | admin | `{name, description?, color?, institutionKey?, atmIds?[]}` |
| PUT | `/api/atm-groups/{id}` | admin | partial update; only non-null fields applied |
| DELETE | `/api/atm-groups/{id}` | admin | 204 |
| PUT | `/api/atm-groups/{id}/atms` | admin | body = `[atmId,...]` → **replaces** all members |
| POST | `/api/atm-groups/{id}/atms/{atmId}` | admin | add one |
| DELETE | `/api/atm-groups/{id}/atms/{atmId}` | admin | remove one |
| POST | `/api/atm-groups/{id}/hardware-profile` | admin | `{"profileKey":"..."}` → `{"atmIds":[...]}` |
| POST | `/fleet/tasks` | authed | fleet svc; frontend queues `CONFIG_UPDATE` after profile assign |
- admin = `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`.
- GET endpoints have **no** `@PreAuthorize` (only institution scoping).
## Tables
- `atm_groups` — id, name (**UNIQUE, global**), description, `color`, `institution_key` (len 10), createdAt, updatedAt.
- `atm_group_members` — join (`group_id`, `atm_id`); `@ManyToMany AtmGroup↔Atm`.
- `atms` — written only by `/hardware-profile` (sets `atm.hardware_profile`, lowercased+trimmed).
## Kafka
- Topic: `hiveops.device.events`, type `DEVICE_UPDATED`, key = atmId.
- Emitted **only** by `/hardware-profile` (one event per member, via `DeviceEventPublisher.publishUpdated`).
- Group create/update/delete/membership changes emit **nothing**.
## Gotchas
- **Default group is client-side only** — browser `localStorage` key `hiveops_defaultGroupId`. Not in DB, not per-account. Drives `groupId` filter on `/api/atms/paginated`.
- **Group name UNIQUE is global**, not per-institution → duplicate name = DB constraint error; `server.error.include-message=never` hides detail.
- **`PUT .../atms` replaces the whole set** — not additive. Must send full membership.
- Single-institution caller: create **forces** `institution_key = scope` regardless of request `institutionKey`.
- `verifyGroupAccess` only blocks when `institution_key != null`; **null ("All") groups are editable/deletable by any in-scope admin**.
- Profile assign = 2 effects: DB+Kafka from backend, `CONFIG_UPDATE` fleet task from **frontend** (skipped if `atmIds` empty). Calling `/hardware-profile` directly does NOT queue the fleet task.
- `configPatch` key used by the push = `atm.hardware.profile`.
## Do NOT
- Do NOT expect `Set Default` to persist server-side or across browsers.
- Do NOT assume group changes propagate via Kafka / device_summary mirrors — they don't; only per-ATM hardware-profile updates do.
- Do NOT treat `PUT .../atms` as add — it overwrites membership.
- Do NOT invent a group Kafka topic or a `hiveiq_*` table other than `atm_groups` / `atm_group_members`.
- Do NOT queue a fleet task by calling only the backend `/hardware-profile`; the fleet push is a separate frontend call.
- Do NOT rely on error bodies — messages are suppressed in prod.
backend/src/main/resources/content/devices/device-groups/internal.md
+63
View File
@@ -0,0 +1,63 @@
---
module: devices.device-groups
title: Device Groups
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support/admin.** Grounded in `hiveops-devices` backend + `AtmGroups.svelte` (2026-07-01). Owner service: **hiveops-devices** (port 8096, DB `hiveiq_devices`).
## What this module actually is
A CRUD screen over the `atm_groups` table plus its `atm_group_members` join table. Groups are named, colored collections of devices used for **filtered device lists** and **bulk hardware-profile pushes**. Deleting a group or dropping a device only touches the join table — no device is ever deleted.
## Roles required
- **Read** (list groups, view a group): any authenticated JWT. `GET /api/atm-groups` and `GET /api/atm-groups/{id}` have **no `@PreAuthorize`** — they are readable by any logged-in user, scoped by institution (see below).
- **All mutations** (create, update, delete, assign ATMs, add/remove single ATM, assign hardware profile): `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`. A non-admin PUT/POST/DELETE returns **403**.
## Institution scoping (why a user "can't see a group")
`AtmGroupService` calls `InstitutionContext.resolveScope()` on every read/write:
- scope `null` → full access (MSP/BCOS admin) — sees all groups.
- scope empty list → sees **zero** groups.
- scope non-empty → only groups whose `institution_key` is in scope (`findAllWithAtmsByInstitutionKeyIn`).
- On **create**, if the caller is scoped to exactly one institution, the group's `institution_key` is **forced** to that key — the Institution dropdown selection in the form is ignored for single-institution users.
First thing to check when "my group vanished / I can't edit it": the caller's JWT institution scope vs the group's `institution_key`.
## Admin-only actions on the screen
- **+ New Group** → `POST /api/atm-groups`.
- **Edit** → `PUT /api/atm-groups/{id}` (name/description/color/institution/membership; only non-null fields are applied).
- **Assign Devices** → `PUT /api/atm-groups/{id}/atms` with a full `[atmId, ...]` list — this **replaces** the whole membership set, it is not additive.
- **Assign Profile** → `POST /api/atm-groups/{id}/hardware-profile` `{ "profileKey": "..." }`, then the **frontend** fires a `POST /fleet/tasks` (`CONFIG_UPDATE`, patch `atm.hardware.profile`).
- **Delete** → `DELETE /api/atm-groups/{id}`.
## Set Default / Clear Default — NOT server state
The **Set Default** / **✕ Clear** buttons are **purely client-side**. The chosen group id is stored in browser `localStorage` under key **`hiveops_defaultGroupId`** (`stores.ts`). It is:
- per-browser, per-machine — never synced to the account or DB;
- used only to pre-filter the device list (passes `groupId` to `/api/atms/paginated`).
"My default group didn't follow me to another machine" is expected behaviour, not a bug. To reset it: clear site data / `localStorage`.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|--------|--------------|-----|
| **409 / 500 on create or edit** with terse message | `atm_groups.name` is `UNIQUE` **globally** (not per-institution). A duplicate name — even across institutions — violates the constraint. `server.error.include-message=never` hides the detail. | Pick a unique name; check DB for the colliding row. |
| **403 on any save** | Caller lacks `MSP_ADMIN`/`BCOS_ADMIN`. | Use an admin JWT. |
| **Group list empty for a real user** | Institution scope is empty, or all groups are tied to institutions outside scope. | Check JWT `institutionKey` claim / scope. |
| **"Assign Profile" says N devices but no config lands on ATMs** | Two steps: profile save succeeded but the follow-up fleet task failed, or ATMs are offline/haven't polled. | Check `/fleet/tasks` for the queued `CONFIG_UPDATE`; verify agents are polling. |
| **Assigning devices wiped others out** | `Assign Devices` is a full replace, not add. | Re-open, select the complete intended set. |
| **Removed a device but it's "back"** | Edit form / Assign panel was opened from stale data. | Reload groups (`GET /api/atm-groups`) and retry. |
| **Group shows "All" institution but a scoped admin can still edit it** | `verifyGroupAccess` only blocks when `institution_key` is non-null. Null (All) groups are editable by any in-scope admin. | Intended today; note it before blaming data. |
## Quick checks
```bash
# List groups (any authed JWT)
curl -s https://api.bcos.cloud/devices/api/atm-groups -H "Authorization: Bearer $JWT"
# DB: group + member counts
SELECT g.id, g.name, g.institution_key, count(m.atm_id) AS atms
FROM atm_groups g LEFT JOIN atm_group_members m ON m.group_id = g.id
GROUP BY g.id ORDER BY g.name;
```
See also [devices.device-groups] Overview (customer) and the Architect tab for data flow.
backend/src/main/resources/content/devices/device-list/architect.md
+53
View File
@@ -0,0 +1,53 @@
---
module: devices.device-list
title: Device List
tab: Architect
order: 30
audience: internal
---
How the Device List is built. Frontend: `hiveops-devices/frontend/src/components/AtmManagement/AtmList.svelte`. Backend of record: **hiveops-devices** (Spring Boot, port **8096**, DB `hiveiq_devices`). See [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Services involved (for this feature)
- **hiveops-devices** — owns the device inventory and serves every grid query. `AtmController` (`@RequestMapping({"/api/atms","/api/devices"})`) → `AtmService` → `AtmRepository` / `AtmPropertiesRepository` on `hiveiq_devices`.
- **hiveops-incident** — only for **📍 Lookup Coords**: `GeocodingController` (`@RequestMapping("/api/geocode")`) proxies a US Census geocoder. The Device List calls it as a relative `fetch('/api/geocode?address=...')`; nginx must route `/api/geocode` to incident.
- **auth/mgmt (JWT)** — the bearer token carries role + institution keys, consumed by `InstitutionContext` for row-level scoping.
## Data flow — reading the grid
1. `AtmList.fetchAtms()` builds params (page, size, sort, search, status[], model[], agentStatus[], groupId, deviceType, agentVersion[]) and calls `loadAtmsPaginated()` in `stores.ts`.
2. `GET /api/atms/paginated` → `AtmController.getAllAtmsPaginated` → `InstitutionContext.resolveScope()` derives the institution scope from the JWT (scoped for `ROLE_CUSTOMER`/`ROLE_MSP_ADMIN`; null = whole fleet for BCOS_ADMIN) → `atmService.getAllAtmsPaginated(...)`.
3. `AtmService` reads `atms` + `atm_properties`, and per row derives `agentConnectionStatus` via `calculateConnectionStatus(lastHeartbeat)`:
- `null` → `NEVER_CONNECTED`
- ≤ 15 min → `CONNECTED`
- > 15 min → `DISCONNECTED`
4. Returns a Spring `Page<AtmDTO>`; the store maps `page.totalElements / totalPages / number / size`.
5. **Status badge** is computed client-side in the grid: `AGENT_OFFLINE` if disconnected/never-connected, else `IN_SUPERVISOR` if `inSupervisor`, else `serviceState || status`. The `Status` column sorts on backend field `effectiveStatus`.
6. Filter dropdowns are populated by `GET /api/atms/distinct-models` and `GET /api/atms/distinct-agent-versions`; groups by `GET /api/atm-groups`.
## Data flow — Add Device (write) + Kafka mirrors
1. `POST /api/atms` (`@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`) → `AtmService` persists to `atms` (+ `atm_properties`).
2. On create, `deviceEventPublisher.publishCreated(savedAtm)` sends a `DEVICE_CREATED` event to Kafka topic **`hiveops.device.events`** (`KafkaProducerConfig.DEVICE_EVENTS_TOPIC`), keyed by `atmId`. Updates/institution-moves/deletes publish `DEVICE_UPDATED` / `DEVICE_DELETED` the same way.
3. Mirror consumers in **incident / fleet / reports / recon** (and vault) keep their `device_summary` copies in sync from this topic. There is **no reconciler** — every write path must publish or the mirrors go stale.
4. `POST /api/atms/republish-all` (admin) re-emits events to rebuild mirrors.
## DB tables (this feature)
| Table | Read/Write | Role |
|-------|-----------|------|
| `atms` | R (grid) / W (add) | Core device inventory, source of truth |
| `atm_properties` | R (heartbeat, version) / W | Per-device props incl. `lastHeartbeat`, agent version |
| `atm_groups` | R | Group filter |
## Kafka
| Topic | Direction | Producer/Notes |
|-------|-----------|----------------|
| `hiveops.device.events` | produced | `DeviceEventPublisher`; `DEVICE_CREATED/UPDATED/DELETED`, keyed by atmId; consumed by incident/fleet/reports/recon mirrors |
## Key API endpoints
- `GET /api/atms/paginated` — the grid (institution-scoped, server-side filter/sort/page)
- `GET /api/atms/distinct-models`, `GET /api/atms/distinct-agent-versions` — filter options
- `POST /api/atms` — Add Device (admin) → publishes `DEVICE_CREATED`
- `GET /api/geocode?address=` — coord lookup (**incident**, not devices)
## Notes
- `/api/atms*` also answers on the `/api/devices*` Phase-2 alias (ATM→device rename); both must stay routable in nginx.
- Devices backend uses `DB_URL` (full JDBC URL), not split host/port/name vars.
backend/src/main/resources/content/devices/device-list/claude.md
+50
View File
@@ -0,0 +1,50 @@
---
module: devices.device-list
title: Device List
tab: Claude
order: 40
audience: internal
---
Terse agent reference. Frontend: `hiveops-devices/frontend/src/components/AtmManagement/AtmList.svelte`.
## Owner
- **hiveops-devices** backend (port 8096, DB `hiveiq_devices`) owns the grid data + all mutations.
- **hiveops-incident** owns ONLY `/api/geocode` (Lookup Coords).
## Endpoints (METHOD path → purpose · role)
| Method + Path | Purpose | Role |
|---|---|---|
| GET `/api/atms/paginated` | Grid rows (params: page,size,sort,search,status[],model[],agentStatus[],groupId,deviceType,agentVersion[]) | any auth, institution-scoped |
| GET `/api/atms/distinct-models` | Model filter options | any auth |
| GET `/api/atms/distinct-agent-versions` | Agent version filter options | any auth |
| GET `/api/atm-groups` | Group filter | any auth |
| GET `/api/institution-keys/accessible` | Add-Device institution dropdown | any auth |
| POST `/api/atms` | Add Device | `MSP_ADMIN`/`BCOS_ADMIN` |
| PATCH `/api/atms/{id}/institution` | Reassign institution | `MSP_ADMIN`/`BCOS_ADMIN` |
| GET `/api/geocode?address=` | Lookup Coords (**incident**) | any auth |
- Controller: `AtmController` `@RequestMapping({"/api/atms","/api/devices"})` → all paths also work under `/api/devices*`.
## Tables
- `atms` (source of truth), `atm_properties` (`lastHeartbeat`, agent version), `atm_groups`.
## Kafka
- Produces `hiveops.device.events` (`DeviceEventPublisher`; `DEVICE_CREATED/UPDATED/DELETED`, key=atmId) on every write. Consumed by incident/fleet/reports/recon mirrors. **No reconciler.**
## Gotchas
- Connection status = heartbeat age: `null`→`NEVER_CONNECTED`, ≤15min→`CONNECTED`, >15min→`DISCONNECTED` (`calculateConnectionStatus`). NOTE: customer overview.md still says "5 min" — code uses **15**.
- Status badge is derived **client-side**: `AGENT_OFFLINE` if disconnected/never-connected overrides real `serviceState`; then `IN_SUPERVISOR`; else `serviceState||status`. Status column sorts on backend `effectiveStatus`.
- Grid filtering/sorting/paging is **server-side** (re-queries on every change). `AGENT OFFLINE` status-filter option is translated to `agentStatus=disconnected,never_connected` client-side.
- Institution scoping via `InstitutionContext.resolveScope()`: `ROLE_CUSTOMER`/`ROLE_MSP_ADMIN` scoped (keys from JWT credentials); other roles (BCOS_ADMIN) unscoped. Empty list for one user = scoping, not a bug.
- `/api/geocode` is a relative `fetch` → depends on nginx routing to incident, not devices.
- Backend: `server.error.include-message=never` → terse prod errors; read logs (Loki).
- Devices backend config uses `DB_URL` (full JDBC URL).
## Do NOT
- Do NOT add device read/write endpoints to incident/fleet/etc — device domain is owned by **hiveops-devices** only.
- Do NOT assume `AGENT OFFLINE` means a service fault — it means no heartbeat in 15 min.
- Do NOT expect other apps to reflect a device edit unless `hiveops.device.events` was published (no reconciler); use `POST /api/atms/republish-all` (admin) to rebuild mirrors.
- Do NOT look for geocode in hiveops-devices — it lives in hiveops-incident.
- Do NOT hit `POST /api/atms` as a non-admin (403).
- Do NOT invent split DB host/port env vars — it's `DB_URL`.
backend/src/main/resources/content/devices/device-list/internal.md
+41
View File
@@ -0,0 +1,41 @@
---
module: devices.device-list
title: Device List
tab: Internal
order: 20
audience: internal
---
Ops/support view of the Device List (frontend `AtmManagement/AtmList.svelte`, served by the **hiveops-devices** backend, port **8096**). This is the fleet inventory grid at `devices.bcos.cloud`.
## What powers each part
- **Grid rows / Total count** → `GET /api/atms/paginated` (`AtmController.getAllAtmsPaginated`). Read-only, no admin role required — but results are **institution-scoped** (see below).
- **Model filter dropdown** → `GET /api/atms/distinct-models`.
- **Agent Version filter dropdown** → `GET /api/atms/distinct-agent-versions`.
- **Group filter** → groups loaded via `GET /api/atm-groups`.
- **+ Add Device** → `POST /api/atms` — **admin only** (`hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`).
- **📍 Lookup Coords** → `GET /api/geocode?address=...` — this is served by **hiveops-incident** (`GeocodingController`), NOT devices. Called as a relative `fetch('/api/geocode')`, so it depends on nginx routing.
## Roles required
- **Viewing the list, filters, distinct-models/versions:** any authenticated user; data is scoped by institution.
- **Add Device / any mutation** (`POST/PUT/PATCH/DELETE /api/atms*`): `MSP_ADMIN` or `BCOS_ADMIN`. A `CUSTOMER` gets 403 on Add Device.
- **Institution scoping** (`InstitutionContext.resolveScope`): callers with `ROLE_CUSTOMER` or `ROLE_MSP_ADMIN` are scoped to their institution key(s) (from JWT credentials); `BCOS_ADMIN` (and other unscoped roles) see the whole fleet.
## First things to check when it misbehaves
1. **List is empty / "No ATMs found" for one user only** → almost always institution scoping. Confirm the user's JWT institution keys; an MSP/customer only sees their own devices. Compare against a `BCOS_ADMIN` login (unscoped).
2. **Device shows "AGENT OFFLINE" unexpectedly** → the **Status** badge is overridden to `AGENT OFFLINE` whenever `agentConnectionStatus` is `DISCONNECTED` or `NEVER_CONNECTED`, regardless of the real service state. Connection status is derived from **last heartbeat**: `CONNECTED` ≤15 min, `DISCONNECTED` >15 min, `NEVER_CONNECTED` if never seen. So an offline badge = no heartbeat in 15 min, not a service fault. Check the device's actual `lastHeartbeat` / agent connectivity.
3. **Wrong institution stuck on a device** → institution is auto-derived from the agent token; a mis-imaged box keeps the wrong institution. Fix via `PATCH /api/atms/{id}/institution` (admin). Cross-check LOCATION to find the true owner.
4. **A device edit doesn't show up in Incident/Fleet/Reports/Recon** → those are **Kafka mirrors** of `hiveops.device.events`. If the devices backend can't reach Kafka, the mirrors go stale (there is no reconciler). Check devices-backend Kafka connectivity; a stale mirror is the usual cause of "wrong data in app X but right in Devices."
5. **Add Device fails silently / terse error** → devices backend runs `server.error.include-message=never`, so prod errors are terse. Check the container logs (Loki) rather than the UI message.
6. **"Lookup Coords" fails but everything else works** → geocode is a different service (incident) on a different nginx path. Verify `/api/geocode` routes to incident and that the upstream (US Census / geocoder) is reachable.
7. **Filters don't narrow the list** → the grid re-queries the backend on every filter change (server-side filtering). `AGENT OFFLINE` in the Status filter is translated client-side into an `agentStatus` filter (disconnected + never_connected), not a DB status value.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| One user sees far fewer devices | Institution scoping via JWT | Expected; verify with BCOS_ADMIN |
| Status = AGENT OFFLINE | No heartbeat >15 min | Check agent/network, not service state |
| Add Device → 403 | Non-admin role | Needs MSP_ADMIN / BCOS_ADMIN |
| Edit not reflected in other apps | Kafka mirror stale | Check devices→Kafka; republish (`POST /api/atms/republish-all`, admin) |
| Lookup Coords broken | incident/geocode down or nginx | Check incident + `/api/geocode` route |
| Terse/empty error text | `include-message=never` | Read backend logs, not UI |
backend/src/main/resources/content/devices/map/architect.md
+68
View File
@@ -0,0 +1,68 @@
---
module: devices.map
title: Map
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Grounded in `AtmMap.svelte`, `frontend/src/lib/{api,stores}.ts`, and the devices/incident controllers, 2026-07-01.
## Shape of the feature
The Map is a **pure read/aggregate view** with **zero server-side state of its own**. There is no `devices.map` controller, table, executor, or Kafka topic. The Svelte component orchestrates two backends and does all clustering/colour/filter logic in the browser. See [platform.service-ownership].
```
AtmMap.svelte (devices SPA, port 5177)
├─ loadAtms() → devicesApi GET /api/atms → hiveops-devices (8096) → atms table
├─ loadIncidents() → incidentApi GET /incident/incidents → hiveops-incident (8080) → incidents table
└─ loadAtmGroups() → devicesApi GET /api/atm-groups → hiveops-devices (8096) → atm_groups table
(Leaflet circleMarkers, client-side grouping + colour + filter)
```
## Services involved & how they interact
- **hiveops-devices** — system of record for the pins. `AtmController` (`@RequestMapping({"/api/atms","/api/devices"})`) serves the ATM inventory incl. `latitude`/`longitude`/`status`/`inSupervisor`/`agentConnectionStatus`. `AtmGroupController` (`/api/atm-groups`) serves group definitions for the Group filter. See [technical.devices].
- **hiveops-incident** — supplies open incidents. `IncidentController.getAllIncidents()` (`GET /api/incidents`, institution-scoped via `InstitutionContext.resolveScope()`). The map keeps only `status === 'OPEN'`, indexes them by `atmId`, and uses that to (a) colour a pin orange, (b) build the Incident Type filter, (c) render incident tags in the pin popup.
The two calls are **independent** — an incident-backend outage degrades colour/type-filter but still renders every pin.
### Frontend API wiring (`lib/api.ts`)
- `devicesApi` baseURL = `window.__APP_CONFIG__.apiUrl` (prod `https://api.bcos.cloud/devices`, dev `http://localhost:8096`).
- `incidentApi` baseURL = `${gatewayUrl}/incident`, where `gatewayUrl` = devices URL with the trailing `/devices` stripped. So `incidentAPI.getAll()` hits `…/incident/incidents`; nginx rewrites that to the controller's `/api/incidents`.
## Data flow (no Kafka in the read path)
The map does **not** consume Kafka. It reads live from both services on each refresh. Note the platform-level relationship for context: `hiveops-devices` publishes `hiveops.device.events` (`DEVICE_CREATED/UPDATED/DELETED`) which mirrors device data into incident/fleet/reports/recon `device_summary` tables — but **this map bypasses those mirrors** and queries devices + incident directly, so it is never stale-mirror-affected. See [platform.kafka], [platform.data-architecture].
Auto-refresh is a browser `setInterval`; each tick calls `loadAtms()` + `loadIncidents()` (interval = `settings.dashboardRefreshIntervalSeconds`).
## DB tables (read-only)
| Table | DB | Service | Used for |
|-------|----|---------|----------|
| `atms` (cols incl. `latitude`, `longitude`, `status`, `in_supervisor`, `agent_connection_status`) | `hiveiq_devices` | devices | Pin position, colour, popup |
| `atm_groups` | `hiveiq_devices` | devices | Group filter options |
| `incidents` | `hiveiq_incident` | incident | Open-incident colour + Incident Type filter + popup tags |
No table is written by this view.
## Key API endpoints
| Method + Path | Service | Role | Purpose |
|---------------|---------|------|---------|
| `GET /api/atms` | devices | authenticated (no `@PreAuthorize`) | Pins. Falls back to `GET /api/atms/paginated?size=9999&groupId=<gid>` when a default group is set |
| `GET /api/atm-groups` | devices | authenticated | Group filter |
| `GET /api/incidents` | incident | authenticated, institution-scoped | Open incidents |
| `PUT /api/atms/{id}` | devices | `MSP_ADMIN`/`BCOS_ADMIN` | Set lat/long to un-hide an ATM (done from ATM Detail, not the map) |
## Client-side logic worth knowing
- **Clustering:** ATMs are grouped by exact `"${latitude},${longitude}"` string key; a group of >1 renders one `circleMarker` + a count badge `divIcon`, coloured by the worst member (`STATUS_PRIORITY`: DOWN < MAINTENANCE < INACTIVE < OUT_OF_CASH < IN_SERVICE).
- **Pin colour** (`getPinColor`): not IN_SERVICE/OUT_OF_CASH → red; open incident → amber; OUT_OF_CASH → amber; `inSupervisor` → blue; else green.
- **Persistence** is entirely `localStorage`: `hiveops_map_filters_v2` (current filters + presets), `devices_map_autoRefresh`, `hiveops_defaultGroupId`.
## Cross-links
- [platform.data-architecture] · [platform.kafka] · [platform.service-ownership] · [technical.devices]
backend/src/main/resources/content/devices/map/claude.md
+68
View File
@@ -0,0 +1,68 @@
---
module: devices.map
title: Map
tab: Claude
order: 40
audience: internal
---
> Terse agent reference. Grounded in `AtmMap.svelte`, `lib/api.ts`, `lib/stores.ts`, `AtmController`, `AtmGroupController`, `IncidentController` (2026-07-01). Read-only view — no writes, no dedicated backend.
## Ownership
- **View:** `hiveops-devices` frontend, `frontend/src/components/AtmMap/AtmMap.svelte` (Leaflet, port 5177).
- **Pin/group data owner:** `hiveops-devices` backend (8096).
- **Incident data owner:** `hiveops-incident` backend (8080).
- No `devices.map` controller / table / Kafka topic exists.
## Endpoints (all GET, all read-only)
| Method + Path | Service | Auth | Notes |
|---|---|---|---|
| `GET /api/atms` | devices | any JWT (no `@PreAuthorize`) | pins; frontend `atmAPI.getAll()` |
| `GET /api/atms/paginated?size=9999&groupId=<gid>&sort=atmId,asc` | devices | any JWT | used instead when `localStorage.hiveops_defaultGroupId` set |
| `GET /api/atm-groups` | devices | any JWT | Group filter (`atmGroupAPI.getAll()`) |
| `GET /api/incidents` | incident | any JWT, institution-scoped | open-incident colour + type filter; frontend calls gateway path `…/incident/incidents` |
| `PUT /api/atms/{id}` | devices | `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` | ONLY way to set lat/long (unmapped fix); done in ATM Detail, not the map |
- Frontend base URLs: `devicesApi` = `window.__APP_CONFIG__.apiUrl` (`https://api.bcos.cloud/devices`); `incidentApi` = that with `/devices` stripped + `/incident`. nginx injects `/api` for the incident call.
- Devices controller also answers Phase-2 aliases `/api/devices` and `/api/device-groups`.
## Tables (read-only)
- `atms` (`hiveiq_devices`) — cols used: `latitude`, `longitude`, `status`, `in_supervisor`, `agent_connection_status`.
- `atm_groups` (`hiveiq_devices`).
- `incidents` (`hiveiq_incident`).
## Topics
- None consumed/produced by this view. (Platform context only: devices publishes `hiveops.device.events`; the map does NOT read those mirrors — it queries devices + incident live.)
## Client-only state (localStorage keys)
- `hiveops_map_filters_v2` — current filters + saved presets.
- `devices_map_autoRefresh` — auto-refresh toggle.
- `hiveops_defaultGroupId` — forces group-scoped `loadAtms()`.
## Enums
- `status`: `IN_SERVICE`, `DOWN`, `MAINTENANCE`, `INACTIVE`, `OUT_OF_CASH`.
- `agentConnectionStatus`: `CONNECTED`, `DISCONNECTED`, `NEVER_CONNECTED`.
- incident: only `status === 'OPEN'` colours a pin; `severity`: `CRITICAL`/`HIGH`/other.
## Gotchas
- Pins need **non-null lat AND long**; others dropped + counted in amber banner.
- Clustering keys on **exact** `"lat,lon"` string — near-but-not-equal coords do not merge.
- Two-service view: incident outage ⇒ pins still render, but no orange/incident-type filter.
- Default group active ⇒ map capped at 9999 rows and scoped to that group.
- Presets/filters are per-browser only; not persisted server-side.
- Reads have no role gate; only lat/long edits (`PUT /api/atms/{id}`) need `MSP_ADMIN`/`BCOS_ADMIN`.
## Do NOT
- Do NOT invent a `devices.map` endpoint/table/topic — none exists; it aggregates existing device + incident reads.
- Do NOT expect an incident endpoint at `/incident/api/incidents`; frontend uses `/incident/incidents` (nginx adds `/api`).
- Do NOT tell users presets are recoverable after clearing browser data — they are `localStorage`-only.
- Do NOT attribute stale pins to Kafka `device_summary` mirrors — this view bypasses them.
- Do NOT add write actions to the map; corrections belong in ATM Detail (devices) / Incident app.
backend/src/main/resources/content/devices/map/internal.md
+54
View File
@@ -0,0 +1,54 @@
---
module: devices.map
title: Map
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support.** Grounded in `hiveops-devices/frontend/src/components/AtmMap/AtmMap.svelte` + the devices/incident backends, 2026-07-01.
## What this view actually does
The Fleet Map is a **read-only** Leaflet view. It makes no writes. On mount and on each auto-refresh it pulls three lists and does all clustering, colouring, and filtering **client-side**:
| Data | Source (frontend fn) | Backend |
|------|----------------------|---------|
| ATMs (pins) | `loadAtms()` → `atmAPI.getAll()` | `hiveops-devices` `GET /api/atms` |
| Open incidents (orange colour + Incident Type filter) | `loadIncidents()` → `incidentAPI.getAll()` | `hiveops-incident` `GET /api/incidents` |
| Groups (Group filter) | `loadAtmGroups()` → `atmGroupAPI.getAll()` | `hiveops-devices` `GET /api/atm-groups` |
**Key consequence:** the map straddles **two services**. Pins come from devices; the orange "Open Incident" colour and the Incident Type filter come from incident. They fail independently.
## Roles required
- **To view the map:** any authenticated JWT. The three GET endpoints above have **no `@PreAuthorize`** — reads are open to any logged-in role (incidents are still institution-scoped server-side via `InstitutionContext.resolveScope()`).
- **To fix an unmapped ATM (set lat/long):** `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` — coordinates live on the `atms` row and are edited through the admin-gated `PUT /api/atms/{id}` (ATM detail → properties).
- **To manage the groups** that populate the Group filter: `MSP_ADMIN`/`BCOS_ADMIN` (`/api/atm-groups` mutations).
## First things to check when it misbehaves
**"Pins are missing / fleet looks half-empty"**
1. Almost always **missing coordinates**. Only ATMs with non-null `latitude` AND `longitude` are plotted; the rest are silently dropped and counted in the amber "N ATMs have no coordinates" banner. Fix: open each via **View in ATM List →** and set lat/long (admin).
2. A **default group** is active. If the user has a default group set (browser `localStorage: hiveops_defaultGroupId`), `loadAtms()` switches to `GET /api/atms/paginated?size=9999&groupId=<gid>` — so the map only ever shows that group, capped at 9999 rows.
3. An **active status/type/group filter** is narrowing the set — check the "Filters:" tag bar and the legend count (`X of Y filtered`). Hit **Clear all**.
**"No ATMs show as Open Incident (nothing goes orange) / Incident Type filter is empty"**
- The incident backend call failed or returned nothing. Pins still render (devices is independent), but incident colouring/filtering is gone. Check `hiveops-incident` health and that `GET /api/incidents` returns 200 for this user's institution.
- Only incidents with `status === 'OPEN'` count — resolved/closed incidents never colour a pin.
**"Two ATMs at the same site aren't clustered into one pin"**
- Clustering keys on the **exact** `"lat,lon"` string. Coordinates that differ even slightly get separate pins. To force a shared count-badge pin, give the co-located ATMs identical lat/long.
**"My saved presets / filters vanished"**
- Presets and the last filter state are **browser-local only** (`localStorage: hiveops_map_filters_v2`), never stored server-side. Clearing site data, a different browser, or a different machine = presets gone. This is expected, not a bug.
**"Map won't stop reloading / reloads too often (or never)"**
- Auto-refresh is a client timer; interval = `settings.dashboardRefreshIntervalSeconds`. The toggle state persists in `localStorage: devices_map_autoRefresh`. Each tick re-runs `loadAtms()` + `loadIncidents()`.
**"401/403 loading the map"**
- Expired/again JWT, or the incident call is being blocked by nginx. The incident list is fetched at gateway path `/incident/incidents` (nginx injects the `/api` segment to reach the controller's `/api/incidents`); a stale nginx block for `incident.*`/`devices.*` is the usual culprit for a one-service failure.
## What you can NOT fix from this screen
- You can't edit coordinates, status, or incidents here — the map only reads. All corrections happen in **ATM Detail** (devices) or the **Incident** app.
backend/src/main/resources/content/devices/sw-deploy/architect.md
+77
View File
@@ -0,0 +1,77 @@
---
module: devices.sw-deploy
title: Software Deployment
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Written 2026-07-01 from `hiveops-devices/frontend` + `hiveops-fleet/backend`. Verify against code before relying on specifics.
## Ownership — this feature lives in Fleet, rendered by Devices
The SW Deploy tab is a **Devices SPA view** (`DeviceSoftwareTab.svelte`) but the domain — software download/install tasks — is owned entirely by **hiveops-fleet**. Devices contributes only the device identity and the UI shell. See [platform.service-ownership].
| Concern | Owner |
|---------|-------|
| Tab UI / `atm` object | hiveops-devices (frontend, port 5177) |
| `SOFTWARE` / `SOFTWARE_INSTALL` task lifecycle, storage, dispatch | **hiveops-fleet** (backend, port **8097**, DB `hiveiq_fleet`) |
| Agent auth / device identity source of truth | hiveops-devices backend (`atms`) → mirrored to fleet `device_summary` via Kafka |
## Data flow
### 1. Read path (rendering the tab)
`DeviceSoftwareTab` → `fleetAPI.getTasksPaginated({ atmId, taskKind:['SOFTWARE','SOFTWARE_INSTALL'] })` → gateway `/fleet/tasks/paginated` → `FleetApiController.getFiltered` → `FleetTaskService.getFilteredPaginated` → `FleetTaskSpecification.withFilters` over the **live `fleet_tasks`** table. Frontend derives the two step cards (`latestDownload`, `latestInstall`) and the history table client-side.
### 2. Create path (Schedule Install)
`fleetAPI.createTask({ atmIds:[id], taskKind:'SOFTWARE_INSTALL', scheduledAt? })` → `POST /fleet/tasks` → `FleetTaskService.createTask`:
- Resolves each `atmId` against `device_summary` (`deviceSummaryRepository.findById`); skips if missing, **offline (>15 min since `lastHeartbeat`)**, or out of institution scope.
- `SOFTWARE`/`SOFTWARE_INSTALL` (and `HOTFIX`/`HOTFIX_INSTALL`) are created with initial status **`PENDING_APPROVAL`**.
- If no `scheduledAt` and kind is `SOFTWARE_INSTALL`, defers to the device's assigned **patch window** (`PatchWindowService.nextOccurrence`).
- Persists to `fleet_tasks`, then `FleetTaskEventPublisher.publishCreated` → Kafka.
### 3. Approval → dispatch
`POST /fleet/tasks/{id}/approve` (`FLEET_APPROVER` / `BCOS_ADMIN`) moves the task `PENDING_APPROVAL → PENDING`. The agent (via agent-proxy, role `INTERNAL`) polls `GET /api/internal/fleet/tasks/next?atm={atmId}&kind=...`; `getNextPendingTaskOfKind` returns the oldest `PENDING` task whose `scheduledAt` is null/past, marks it `QUEUED`, and returns artifact metadata (name, version, sha256, CDN url + token) for `SOFTWARE`. `SOFTWARE_INSTALL` carries no artifact — the agent runs `softwareinstall.cmd` against its staged `software/` dir.
### 4. Status reporting → archival (executor → record-store)
Agent reports progress via `POST /api/internal/fleet/tasks/{taskId}/status` (`downloadOffset`, `totalSize`, `status`) → `FleetTaskService.updateTaskStatus`:
- `RUNNING` sets `startedAt`; `COMPLETED`/`FAILED` set `completedAt`; download progress percent is derived from offset/total.
- Every update publishes `FleetTaskEventPublisher.publishStatusUpdate`.
- On **`COMPLETED`**, the row is **archived to `fleet_task_history` and deleted from `fleet_tasks`** (`archiveAndDelete`, also emits `publishArchived`). A `@Scheduled` sweep every 5 min archives stray `COMPLETED`/`CANCELLED`/`REJECTED`; history is purged after 30 days. This is the classic **live-table + archived-record-store** split. See [platform.data-architecture].
## Kafka
| Topic | Direction | Class | Notes |
|-------|-----------|-------|-------|
| `hiveops.fleet.task.events` | **produced** by fleet | `FleetTaskEventPublisher` / `FleetTaskKafkaConfig.FLEET_TASK_EVENTS_TOPIC` | eventTypes `TASK_CREATED`, `TASK_STATUS`, `TASK_ARCHIVED`; keyed per task. Consumed downstream by devices (`FleetTaskEventConsumer` → `device_fleet_tasks` mirror). |
| `hiveops.device.events` | **consumed** by fleet | `DeviceEventConsumer` (group `hiveops-fleet-device-sync`) | Upserts/deletes `device_summary` from devices' `DEVICE_CREATED/UPDATED/DELETED`. This is how fleet knows the device exists, its `institutionKey`, and `lastHeartbeat` seed. |
See [platform.kafka].
## Tables (DB `hiveiq_fleet`)
| Table | Entity | Role in this feature |
|-------|--------|----------------------|
| `fleet_tasks` | `FleetTask` | Live SOFTWARE/SOFTWARE_INSTALL rows — the tab's read source |
| `fleet_task_history` | `FleetTaskHistory` | Archived terminal tasks (completed installs/downloads land here) |
| `device_summary` | `DeviceSummary` | Fleet's mirror of devices; target resolution, online check, institution scope |
| `fleet_artifacts` | `FleetArtifact` | Software package metadata for `SOFTWARE` downloads (enabled flag, CDN url, sha256) |
| patch window tables | `PatchWindow` / `PatchWindowDevice` | Auto-schedule for `SOFTWARE_INSTALL` |
## Key endpoints
| Method + Path | Purpose | Auth |
|---------------|---------|------|
| `GET /api/fleet/tasks/paginated` | Tab read (filter `taskKind`, `atmId`) | USER…BCOS_ADMIN |
| `POST /api/fleet/tasks` | Create install/download task | ADMIN/MSP_ADMIN/BCOS_ADMIN |
| `POST /api/fleet/tasks/{id}/approve` \| `/reject` | Approve gate | FLEET_APPROVER / BCOS_ADMIN |
| `POST /api/fleet/tasks/{id}/cancel` \| `/retry` | Lifecycle | ADMIN/MSP_ADMIN/BCOS_ADMIN |
| `GET /api/fleet/tasks/history` | Archived (completed) tasks | USER…BCOS_ADMIN |
| `GET /api/internal/fleet/tasks/next` | Agent poll/dispatch | INTERNAL |
| `POST /api/internal/fleet/tasks/{id}/status` | Agent status/progress | INTERNAL |
## Architectural note (see uncertainties)
The tab reads the **live** `fleet_tasks` table, but `COMPLETED` tasks are archived+deleted immediately. Completed software history therefore lives only in `fleet_task_history` / the `/tasks/history` endpoint, which this tab does not query. Treat the tab as "in-flight + failed" state; use history for the completed record.
Cross-links: [platform.data-architecture] · [platform.kafka] · [platform.service-ownership]
backend/src/main/resources/content/devices/sw-deploy/claude.md
+59
View File
@@ -0,0 +1,59 @@
---
module: devices.sw-deploy
title: Software Deployment
tab: Claude
order: 40
audience: internal
---
**Owner service:** `hiveops-fleet` (port **8097**, DB **`hiveiq_fleet`**). Frontend `DeviceSoftwareTab.svelte` lives in `hiveops-devices` but calls fleet only. NOT owned by devices/incident.
## Task kinds (this tab)
- `SOFTWARE` = download (has `FleetArtifact`).
- `SOFTWARE_INSTALL` = install; **no artifact**; agent runs `softwareinstall.cmd` on staged `software/` dir.
## Endpoints (METHOD path — auth)
| Method | Path | Auth |
|--------|------|------|
| GET | `/api/fleet/tasks/paginated?atmId={id}&taskKind=SOFTWARE&taskKind=SOFTWARE_INSTALL` | USER…BCOS_ADMIN |
| POST | `/api/fleet/tasks` body `{atmIds:[id],taskKind,artifactId?,scheduledAt?}` | ADMIN/MSP_ADMIN/BCOS_ADMIN |
| POST | `/api/fleet/tasks/{id}/approve` \| `/reject` | `FLEET_APPROVER` or `BCOS_ADMIN` |
| POST | `/api/fleet/tasks/{id}/cancel` \| `/retry` | ADMIN/MSP_ADMIN/BCOS_ADMIN |
| GET | `/api/fleet/tasks/history?atmId={id}` | USER…BCOS_ADMIN |
| GET | `/api/fleet/tasks/pending-approval` | FLEET_APPROVER or MSP_ADMIN/BCOS_ADMIN |
| GET | `/api/internal/fleet/tasks/next?atm={atmId}&kind=` | `INTERNAL` (agent) |
| POST | `/api/internal/fleet/tasks/{id}/status` body `{status,downloadOffset,totalSize}` | `INTERNAL` |
Frontend routes hit gateway prefix `/fleet/...` (nginx). Controller base = `/api/fleet`, internal base = `/api/internal/fleet`.
## Tables (`hiveiq_fleet`)
- `fleet_tasks` — live tasks (tab read source).
- `fleet_task_history` — archived terminal tasks (completed downloads/installs land here).
- `device_summary` — device mirror; target lookup + online check + institution scope.
- `fleet_artifacts` — software package (enabled flag, cdn_url, sha256).
- patch window: `PatchWindow` / `PatchWindowDevice`.
## Kafka topics
- Produce: `hiveops.fleet.task.events` (`FleetTaskEventPublisher`; TASK_CREATED/TASK_STATUS/TASK_ARCHIVED).
- Consume: `hiveops.device.events` (group `hiveops-fleet-device-sync` → upsert `device_summary`).
## Status lifecycle
`PENDING_APPROVAL → PENDING → QUEUED → RUNNING → COMPLETED|FAILED` (also `CANCELLED`,`REJECTED`).
- `SOFTWARE`/`SOFTWARE_INSTALL` START in `PENDING_APPROVAL` (need approve).
- On `COMPLETED`: row archived to `fleet_task_history` and **deleted** from `fleet_tasks` (`archiveAndDelete`). 5-min sweep archives COMPLETED/CANCELLED/REJECTED; FAILED stays live.
## Gotchas
- Completed SOFTWARE tasks are gone from `fleet_tasks` → the tab (live paginated) won't show them; use `/tasks/history`.
- Create silently skips a device if: not in `device_summary`, `lastHeartbeat` >15 min old (offline), or out of institution scope. No error surfaced — check fleet logs.
- Disabled artifact → `400 "Artifact '…' is disabled"`.
- `SOFTWARE_INSTALL` with no `scheduledAt` auto-defers to device patch window if assigned.
- Only `PENDING` + (scheduledAt null/past) tasks dispatch; `PENDING_APPROVAL` never reaches agent.
- Create is admin-role; approve is a SEPARATE authority (`FLEET_APPROVER`/`BCOS_ADMIN`) — MSP_ADMIN alone can create but not approve.
- Frontend `getTasksPaginated` sends repeated `taskKind` params (`paramsSerializer indexes:null`).
## Do NOT
- Do NOT look in `hiveops-devices` / `hiveops-incident` DB for these tasks — they are in `hiveiq_fleet`.
- Do NOT expect completed installs in `/tasks/paginated` — query `/tasks/history`.
- Do NOT assume creating the task dispatches it — it sits in `PENDING_APPROVAL` until approved.
- Do NOT invent an artifact for `SOFTWARE_INSTALL` — it has none.
- Do NOT create tasks for offline/out-of-scope devices expecting success — they are skipped silently.
backend/src/main/resources/content/devices/sw-deploy/internal.md
+56
View File
@@ -0,0 +1,56 @@
---
module: devices.sw-deploy
title: Software Deployment
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support.** Written 2026-07-01 from source (`hiveops-devices/frontend` + `hiveops-fleet/backend`). The SW Deploy tab is a **thin view over Fleet tasks** — the data and all actions live in the **hiveops-fleet** service, not devices.
## What you're actually looking at
The **SW Deploy** tab (`DeviceSoftwareTab.svelte`, under a device's detail view in `devices.bcos.cloud`) renders two fleet task kinds for one device:
- **Step 1 — Software Download** = fleet task kind `SOFTWARE` (delivers the package to the ATM).
- **Step 2 — Software Install** = fleet task kind `SOFTWARE_INSTALL` (runs `softwareinstall.cmd` against the staged `software/` dir on the device).
It calls **only** the fleet backend: `GET /fleet/tasks/paginated?atmId={id}&taskKind=SOFTWARE&taskKind=SOFTWARE_INSTALL` to read, and `POST /fleet/tasks` to create an install. No devices-backend or incident call is involved for this tab's data.
## Roles required
| Action | Endpoint | Required authority |
|--------|----------|--------------------|
| View history / step cards | `GET /api/fleet/tasks/paginated` | `USER`,`CUSTOMER`,`ADMIN`,`MSP_ADMIN`,`BCOS_ADMIN` |
| Create the install task ("Schedule Install") | `POST /api/fleet/tasks` | `hasAnyRole('ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
| **Approve/reject** the SOFTWARE / SOFTWARE_INSTALL task | `POST /api/fleet/tasks/{id}/approve` \| `/reject` | `hasAuthority('FLEET_APPROVER') or hasRole('BCOS_ADMIN')` |
| Cancel / retry | `POST /api/fleet/tasks/{id}/cancel` \| `/retry` | `hasAnyRole('ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
**Gotcha:** creating the install is admin-gated, but **approving it is a separate gate** — `SOFTWARE` and `SOFTWARE_INSTALL` are born in status `PENDING_APPROVAL` (see below). A plain `MSP_ADMIN` who is not a `FLEET_APPROVER` can create the task but **cannot approve it**; a `BCOS_ADMIN` (or `FLEET_APPROVER`) must approve before the agent will ever see it.
## The dispatch lifecycle (why a task "isn't doing anything")
`SOFTWARE` / `SOFTWARE_INSTALL` follow the approval flow:
`PENDING_APPROVAL` → (approve) → `PENDING` → (agent polls) → `QUEUED` → `RUNNING` → `COMPLETED` / `FAILED`
- The agent only receives tasks in `PENDING` whose `scheduledAt` is null or in the past (`getNextPendingTaskOfKind`). So **PENDING_APPROVAL never dispatches** — it waits for an approver.
- On create, the target device must be **online** (heartbeat within **15 minutes**) and in the caller's **institution scope**, or the task is silently skipped (logged as a warning, no error to the user).
- `SOFTWARE_INSTALL` with no explicit schedule auto-defers to the device's **patch window** if one is assigned (`PatchWindowService.nextOccurrence`).
## First things to check when it misbehaves
1. **Nothing in the tab / expected history missing.** The tab reads the **live `fleet_tasks` table only**. Completed tasks are archived to `fleet_task_history` and deleted from `fleet_tasks` the instant the agent reports `COMPLETED` (`archiveAndDelete`), plus a 5-minute sweep (`archiveStaleTerminalTasks`) for `COMPLETED`/`CANCELLED`/`REJECTED`. So **completed downloads/installs disappear from this tab** — look in fleet task **history** (`GET /api/fleet/tasks/history?atmId={id}`) or `fleet_task_history` in DB. `FAILED` stays in the live table and is visible.
2. **"Schedule Install" button never appears.** Frontend `canInstall` requires the latest `SOFTWARE` download to be `status === 'COMPLETED'` *and still present in the live list*. Because completion archives+deletes the row, the button window is effectively zero — see uncertainties/architect notes. Workaround: create the `SOFTWARE_INSTALL` directly via `POST /fleet/tasks` with `taskKind:"SOFTWARE_INSTALL"`.
3. **Task created but stuck in PENDING_APPROVAL.** Needs an approver — `FLEET_APPROVER` authority or `BCOS_ADMIN`. Check `GET /api/fleet/tasks/pending-approval`.
4. **Task silently not created for a device.** Almost always **offline device** (no heartbeat in 15 min) or **out of institution scope**. Confirm via the device's `lastHeartbeat` in `device_summary` and the caller's institution scope. Check fleet backend logs for `offline — skipping` / `out of scope`.
5. **Download package rejected on create.** Artifact must be **enabled** — a disabled artifact returns `400 "Artifact '…' is disabled"`. `SOFTWARE_INSTALL` itself carries no artifact (it acts on the already-staged dir).
6. **Stuck in QUEUED/RUNNING.** A task idle in `QUEUED`/`RUNNING` past the stale threshold is auto-reset to `PENDING` (offset zeroed) on the next agent poll for re-dispatch.
## Where to look
- Frontend: `hiveops-devices/frontend/src/components/AtmDetail/DeviceSoftwareTab.svelte`
- Backend service: **hiveops-fleet** (port **8097**, DB **`hiveiq_fleet`**), `FleetTaskService` + `FleetApiController` + `InternalFleetController`.
- Logs: fleet backend (Loki `hiveops-fleet`), not devices.
See also [devices.sw-deploy] Overview (customer) and the Architect tab for the full data flow.
backend/src/main/resources/content/fleet/dashboard/architect.md
+90
View File
@@ -0,0 +1,90 @@
---
module: fleet.dashboard
title: Fleet Dashboard
tab: Architect
order: 30
audience: internal
---
How the **Fleet Stats** dashboard is built. It is a thin read-only view over hiveops-fleet's own database — no cross-service HTTP at request time. Cross-link [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Services involved
| Concern | Owner | Role for this screen |
|---------|-------|----------------------|
| Dashboard UI | `hiveops-fleet/frontend` (Svelte, port 5176) | `FleetDashboard.svelte`, aggregates ATMs client-side |
| Stats + device list API | `hiveops-fleet/backend` (Spring Boot 3.4, port 8097) | `FleetApiController` → `FleetTaskService.getStats(...)` + `DeviceSummaryRepository` |
| Device state feed | `hiveops-devices` | Publishes `hiveops.device.events`; fleet mirrors it |
Only hiveops-fleet is on the request path. hiveops-devices contributes **asynchronously** via Kafka, never a live call. This is the deliberate no-cross-service-HTTP pattern for fleet.
## Data flow
**Task/artifact counters (Task Statistics + Fleet Overview bars):**
```
FleetDashboard onMount / 60s timer
→ GET /api/fleet/tasks/stats
→ FleetTaskService.getStats(InstitutionContext.resolveScope())
→ fleet_tasks (count / countByTaskKindAndStatus / findByStatus)
→ fleet_artifacts (count)
→ FleetDashboardStats DTO
```
**ATMs by Model / by Institution tables:**
```
FleetDashboard onMount (once, not on the 60s timer)
→ GET /api/fleet/devices
→ InstitutionContext.resolveScope() (null = unrestricted; else List<institutionKey>)
→ deviceSummaryRepository.findAll() OR findByInstitutionKeyIn(scope)
→ List<Map> {atmId, institution(Key), status, model, ...}
→ frontend groups by model & institution, counts operational/offline
```
The device rows are populated out-of-band by the Kafka consumer, so the dashboard reads a **local mirror**, decoupled from device-service availability.
## Kafka (async, off the request path) — [platform.kafka]
**Consumed by fleet (feeds this screen's device tables):**
| Topic | Consumer | Effect |
|-------|----------|--------|
| `hiveops.device.events` | `DeviceEventConsumer` (`@KafkaListener`, groupId `hiveops-fleet-device-sync`, `auto-offset-reset=earliest`) | Upserts `device_summary` rows read by `GET /api/fleet/devices` |
**Produced by fleet (NOT read by this screen — listed for completeness):**
| Topic | Producer | Notes |
|-------|----------|-------|
| `hiveops.fleet.task.events` | `FleetTaskEventPublisher` via `FleetTaskService` | created/status/archived events for downstream consumers; the dashboard reads `fleet_tasks` directly, not this topic |
There is **no reconciler** — if devices stops publishing, `device_summary` (and therefore the ATM tables) goes stale silently. Cross-link [platform.data-architecture].
## DB tables (all in `hiveiq_fleet`)
| Table | Access for this screen | Notes |
|-------|------------------------|-------|
| `fleet_tasks` | READ (counts) | active tasks (PENDING/QUEUED/RUNNING) |
| `fleet_artifacts` | READ (count) | uploadable/CDN-imported artifacts |
| `device_summary` | READ | Kafka-synced mirror of device state; the only source for the ATM tables |
`fleet_task_history`, `patch_windows`, `atm_captures` exist in fleet but are **not** touched by the dashboard.
## Key API endpoints (this feature)
| Method Path | Handler | Auth |
|-------------|---------|------|
| `GET /api/fleet/tasks/stats` (alias `/api/fleet/stats`) | `FleetApiController.getStats` → `FleetTaskService.getStats` | `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
| `GET /api/fleet/devices` | `FleetApiController.listDevices` | same |
Base default in frontend: `http://localhost:8097/api/fleet`; prod `https://api.bcos.cloud/fleet`.
## Executor → Kafka → record-store pattern (context)
The broader fleet task lifecycle follows executor → `hiveops.fleet.task.events` → devices record-store, and terminal tasks archive to `fleet_task_history`. **This dashboard sits outside that loop** — it only aggregates current-state reads (task/artifact counts + the device mirror). Worth knowing so you don't wire the dashboard into the task-event pipeline.
## Notable implementation facts (verify before relying on displayed numbers)
- `getStats` currently hard-codes `completedTasks`/`failedTasks` to `0`, scopes `pendingTasks` to `AUTO_UPDATE`+`PENDING` only, and **ignores** its `institutionKeys` argument for the global `fleet_tasks`/`fleet_artifacts` counts.
- The `FleetDashboardStats` DTO has no `totalModulesReporting` field, so the frontend's "Modules Reporting" always renders its 0 default.
- Device status mapping is client-side: `IN_SERVICE`→operational; `OUT_OF_SERVICE`/`DOWN`/`INACTIVE`→offline.
backend/src/main/resources/content/fleet/dashboard/claude.md
+53
View File
@@ -0,0 +1,53 @@
---
module: fleet.dashboard
title: Fleet Dashboard
tab: Claude
order: 40
audience: internal
---
Terse agent reference for the **Fleet Stats** dashboard (`FleetDashboard.svelte`).
## Ownership
- Service: **hiveops-fleet** (backend port 8097, DB `hiveiq_fleet`, package `com.hiveops.fleet`).
- Frontend: hiveops-fleet/frontend, port 5176, `components/FleetManagement/FleetDashboard.svelte`.
- No other service on the request path. hiveops-devices contributes only via Kafka.
## Endpoints (the only two this screen calls)
| Method | Path | Returns | Auth |
|--------|------|---------|------|
| GET | `/api/fleet/tasks/stats` (alias `/api/fleet/stats`) | `FleetDashboardStats` | `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
| GET | `/api/fleet/devices` | `List<Map>` device rows | same |
- Frontend base: `http://localhost:8097/api/fleet` (dev) / `https://api.bcos.cloud/fleet` (prod).
## Tables (read-only, `hiveiq_fleet`)
- `fleet_tasks` — task counts.
- `fleet_artifacts` — artifact count.
- `device_summary` — ATM rows for the model/institution tables (Kafka mirror).
## Kafka
- Consumes `hiveops.device.events` → `DeviceEventConsumer`, groupId `hiveops-fleet-device-sync` → upserts `device_summary`.
- Produces `hiveops.fleet.task.events` (NOT read by this screen).
## Scoping
- `InstitutionContext.resolveScope()`: `ROLE_CUSTOMER`/`ROLE_MSP_ADMIN` → scoped to JWT-credential institution-key list; else `null` = unrestricted.
- `/devices` IS scoped. `/tasks/stats` is NOT (see gotchas).
## Gotchas (grounded in current code — do not "fix" the numbers blindly)
- `getStats`: `completedTasks` and `failedTasks` are hard-coded `0`. They never move.
- `getStats`: `pendingTasks` = only `AUTO_UPDATE` + `PENDING` (not all pending kinds).
- `getStats`: ignores its `institutionKeys` arg → `totalTasks`/`totalArtifacts` are GLOBAL counts even for scoped users.
- `FleetDashboardStats` DTO has NO `totalModulesReporting` field → frontend "Modules Reporting" always 0. Real module data is `GET /api/fleet/modules/summary`.
- 60s auto-refresh reloads **stats only**; ATM tables load once on mount (stale until page reload).
- Stats fetch failures are swallowed (`console.warn`), no on-screen error → zeros ≠ "no data".
- Device status → operational/offline map: `IN_SERVICE`=operational; `OUT_OF_SERVICE`/`DOWN`/`INACTIVE`=offline; other = counted in Total only.
- `device_summary` has no reconciler; stale if devices stops publishing.
## Do NOT
- Do NOT add an endpoint/table for this screen in incident/journal/devices — fleet owns it.
- Do NOT trust Completed/Failed/Modules-Reporting counters as live data (see gotchas).
- Do NOT expect `/tasks/stats` numbers to respect institution scope.
- Do NOT read `hiveops.fleet.task.events` to build this view — read `fleet_tasks` directly.
- Do NOT call hiveops-devices over HTTP to refresh ATM tables — data arrives via Kafka only.
- Do NOT restrict this screen to admin roles; `USER`/`CUSTOMER` are allowed to read.
backend/src/main/resources/content/fleet/dashboard/internal.md
+53
View File
@@ -0,0 +1,53 @@
---
module: fleet.dashboard
title: Fleet Dashboard
tab: Internal
order: 20
audience: internal
---
Ops/support view for the **Fleet Stats** screen (`FleetDashboard.svelte`, route inside `fleet.bcos.cloud`). It is a read-only dashboard: two stat bars (Task Statistics, Fleet Overview) plus two expandable tables (ATMs by Model, ATMs by Institution). It calls exactly **two** endpoints — everything on the screen comes from these:
- `GET /api/fleet/tasks/stats` (alias `/api/fleet/stats`) → the five task counters + Artifacts + Modules Reporting.
- `GET /api/fleet/devices` → the ATM rows, aggregated client-side by model/institution.
Owned end-to-end by **hiveops-fleet** (backend port 8097, DB `hiveiq_fleet`). No other service is called for this screen.
## Roles required
- Both endpoints: `@PreAuthorize("hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')")`. This is **not** admin-only — any authenticated fleet user can view it.
- There are **no admin-only actions on this screen.** It is display-only (no create/cancel/approve here — those live on the Fleet Tasks screen).
- Data is **institution-scoped** for `CUSTOMER` and `MSP_ADMIN` tokens: `InstitutionContext.resolveScope()` reads the institution-key list from the JWT credentials and filters `device_summary`. `ADMIN`/`BCOS_ADMIN`/`USER` see everything (unrestricted scope).
## Refresh behaviour (know this before chasing "stale" reports)
- On mount the component loads **both** stats and ATMs once.
- The 60-second auto-refresh timer reloads **stats only** (`loadFleetStats`). The **ATMs by Model / by Institution tables do NOT auto-refresh** — they only update on a full page reload. A user reporting "device counts didn't update for an hour" is expected behaviour, not a bug.
- The accordion "N models · M ATMs" header count is derived from the same one-time ATM load.
## First things to check when it misbehaves
1. **Blank / all-zero task counters** → hit `GET /api/fleet/tasks/stats` directly with a real JWT. On failure the frontend silently swallows the error (`console.warn` only) and keeps the last values/zeros — there is no on-screen error banner for stats.
2. **"No ATM data available" in both tables** → `GET /api/fleet/devices` returned empty. Root cause is almost always an empty/stale `device_summary` table, not the dashboard. `device_summary` is a Kafka mirror — see failure modes below.
3. **Wrong institution split** → stale `device_summary` mirror. Fleet does not reconcile; it only trusts `hiveops.device.events`. If hiveops-devices stopped publishing, the split is stale. Cross-check against the devices service.
4. **403 on load** → token missing one of the five allowed roles, or an expired session (frontend maps 401→"session expired", 403→"no permission").
5. **Scoped user sees more/less than expected** → confirm the JWT's institution-key credentials list; `resolveScope()` returns `null` (unrestricted) for anyone lacking `ROLE_CUSTOMER`/`ROLE_MSP_ADMIN`.
## Known-quirks that look like bugs but are current behaviour
> These are grounded in `FleetTaskService.getStats(...)` as written today. Treat "the numbers look wrong" reports as product limitations, not incidents, until the service is changed.
- **Completed and Failed always show 0.** The service hard-codes `completed = 0` and `failed = 0`. The counters render, but never move.
- **Modules Reporting always shows 0.** The backend `FleetDashboardStats` DTO has no `totalModulesReporting` field, so the frontend default (0) is never overwritten. (The real module numbers live behind `GET /api/fleet/modules/summary`, a different screen.)
- **Pending counts only auto-update tasks.** `pendingTasks` = count of `AUTO_UPDATE` tasks in `PENDING` only — not all pending tasks of every kind.
- **Total tasks / Artifacts are NOT institution-scoped.** `getStats` ignores the scope argument and uses global `taskRepository.count()` / `artifactRepository.count()`. So a scoped MSP_ADMIN sees fleet-wide task and artifact totals but only their own ATMs in the tables — an intentional-looking mismatch that is actually the scope bug above.
## Failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Both ATM tables empty | `device_summary` empty/stale | Check the `hiveops.device.events` consumer (`groupId hiveops-fleet-device-sync`) is running and lag is 0; confirm hiveops-devices is publishing |
| Stats stuck at last values | `/tasks/stats` failing silently | Curl the endpoint; check fleet backend logs (Loki), not the browser |
| 401/403 | expired/under-privileged JWT | Re-login; verify role + `fleetTaskApprover` not required here |
| Institution split wrong for one inst | stale mirror for that inst | Reconcile via devices service; fleet has no reconciler |
| "Operational/Offline" miscounts | status mapping | `IN_SERVICE`→operational; `OUT_OF_SERVICE`/`DOWN`/`INACTIVE`→offline; anything else counts toward Total only |
backend/src/main/resources/content/fleet/modules/architect.md
+53
View File
@@ -0,0 +1,53 @@
---
module: fleet.modules
title: Modules
tab: Architect
order: 30
audience: internal
---
> **Internal · architecture.** Written 2026-07-01 from `hiveops-fleet` source. Verify against code before relying on specifics.
## Shape of the feature
The **Agent Modules** screen is intentionally thin. It is a **client-owned catalog** plus a **single fleet-wide summary endpoint**. There is no dedicated module-status table, no Kafka topic, and no executor→record-store pipeline behind this specific screen today. Cross-link [platform.service-ownership].
### Two data sources, one of them static
1. **Module catalog (static, frontend):** `MODULE_CATALOG` in `hiveops-fleet/frontend/src/components/FleetManagement/FleetModules.svelte` — a hardcoded array of 16 `{ key, displayName, description, category }` entries across four categories: `Data Collection`, `Monitoring`, `Fleet Management`, `Updates`. The `key`s (e.g. `journal-events`, `command-center`, `auto-update`) mirror the agent's module identifiers. No backend call produces this list.
2. **Status summary (dynamic, backend):** `GET /api/fleet/modules/summary` on `hiveops-fleet`, deserialized to `ModuleSummary[]` in the frontend store `moduleSummary` (`loadModuleSummary()` in `stores.ts`). The component joins the two by matching `ModuleSummary.moduleName === catalog.key`.
## Service ownership
- **Owner:** `hiveops-fleet` (port 8097, DB `hiveiq_fleet`). It serves both the SPA (`fleet.bcos.cloud`) and the `/api/fleet` REST surface.
- No other service participates in **this** feature. (Broadly, fleet syncs a `device_summary` mirror from `hiveops.device.events` and publishes task events on `hiveops.fleet.task.events` — see [technical.fleet] — but neither pipeline feeds the Modules screen.)
## Data flow (current)
```
FleetModules.svelte (onMount)
→ loadModuleSummary() [stores.ts]
→ fleetAPI.getModuleSummary() [api.ts] GET {apiUrl}/modules/summary
→ FleetApiController.getModuleSummary() [hiveops-fleet]
→ return List.of(); ← STUB: always empty
→ moduleSummary store = []
→ component renders 16 static cards; no badges (summaryFor() always null)
```
### Endpoint detail (confirmed)
| Method + Path | Controller | Auth | Returns |
|---------------|-----------|------|---------|
| `GET /api/fleet/modules/summary` | `FleetApiController.getModuleSummary()` | `@PreAuthorize hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` | `List<ModuleSummaryDTO>` — currently `List.of()` (empty) |
`ModuleSummaryDTO` fields: `moduleName` (String), `runningCount`, `disabledCount`, `failedCount`, `totalCount` (all `long`). Frontend `ModuleSummary` interface mirrors these.
## DB tables read/written
- **None for this feature.** The summary endpoint returns a static empty list; it does not query any table. The `hiveiq_fleet` schema (V1__init_fleet.sql: `fleet_tasks`, `fleet_task_history`, `fleet_artifacts`, `patch_windows`, `atm_captures`, `device_summary`) has **no module-status table**. Cross-link [platform.data-architecture].
## Kafka
- **None for this feature.** No producer or consumer touches module status. (Fleet's Kafka wiring — `FleetTaskEventPublisher` → `hiveops.fleet.task.events`, and `DeviceEventConsumer` ← `hiveops.device.events` — is unrelated to the Modules screen.) Cross-link [platform.kafka].
## If/when this gets wired for real
The DTO already models per-module aggregate counts (running/disabled/failed/total), which implies the intended source is a per-ATM module-status feed aggregated by `moduleName`. That feed does not exist in fleet today. A future implementation would need agents to report module state (via agent-proxy internal endpoints, the pattern used elsewhere: agent → `X-Internal-Secret` `/api/internal/fleet/**`), land it in a new fleet table, and aggregate in `getModuleSummary()`. Until then, treat the screen as documentation-only.
## Cross-links
- [platform.data-architecture]
- [platform.kafka]
- [platform.service-ownership]
- [technical.fleet] — fleet task orchestration, artifact delivery, the real Kafka/table surface of `hiveops-fleet`
backend/src/main/resources/content/fleet/modules/claude.md
+42
View File
@@ -0,0 +1,42 @@
---
module: fleet.modules
title: Modules
tab: Claude
order: 40
audience: internal
---
> **Internal · agent reference.** Terse. Confirmed against `hiveops-fleet` source 2026-07-01.
## Owner
- Service: **`hiveops-fleet`** (port **8097**, DB **`hiveiq_fleet`**). Serves SPA `fleet.bcos.cloud` + `/api/fleet`.
- Frontend view: `hiveops-fleet/frontend/src/components/FleetManagement/FleetModules.svelte`.
## Endpoints (confirmed)
| Method | Path | Auth (`@PreAuthorize`) | Behavior |
|--------|------|------------------------|----------|
| GET | `/api/fleet/modules/summary` | `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` | Returns `List<ModuleSummaryDTO>` — **hardcoded `List.of()` (always empty)** |
Base path `@RequestMapping("/api/fleet")` in `FleetApiController`. Frontend base = `window.__APP_CONFIG__.apiUrl` (default `http://localhost:8097/api/fleet`).
## DTO
- `ModuleSummaryDTO { String moduleName; long runningCount; long disabledCount; long failedCount; long totalCount; }`
- Joined in UI by `moduleName === catalog.key`.
## Data facts
- Module catalog = **16 entries hardcoded in `FleetModules.svelte`** (`MODULE_CATALOG`), NOT from DB/API. Categories: `Data Collection`, `Monitoring`, `Fleet Management`, `Updates`.
- **No DB table** backs module status. **No Kafka topic** backs module status.
- Status badges never render today (summary is empty → `summaryFor()` returns null → `{#if summary}` skipped).
## Gotchas
- `/modules/summary` returning `[]` is **by design (stub)**, not a bug/outage. "No badges" = expected.
- Client methods `getModules()`→`GET /modules`, `getModulesByAtm(id)`→`GET /modules/atm/{id}` exist in `api.ts` but **have NO backend endpoint** in `FleetApiController` (would 404). Not called by this view. (unverified whether any other service implements them — none found in `hiveops-fleet`.)
- Module enable/disable is agent-config driven (`CONFIG_UPDATE` fleet task / config server), NOT this screen. This screen is read-only.
- `/api/fleet/**` = JWT plane (`JWT_SECRET`). Internal agent endpoints are `/api/internal/fleet/**` (`hasRole('INTERNAL')`, `X-Internal-Secret`) — unrelated to this feature.
## Do NOT
- Do NOT claim a `module_status` / `agent_module` table or a `hiveops.*.modules` Kafka topic exists — none do.
- Do NOT tell anyone module counts are "live" or reflect real ATM state — the endpoint returns empty.
- Do NOT route users to `/modules` or `/modules/atm/{id}` — no backend, will 404.
- Do NOT look in `hiveops-incident`/`devices`/`journal` for this screen's data — owner is `hiveops-fleet`, and the summary is a fleet stub.
- Do NOT edit the module list in the backend — the catalog lives in `FleetModules.svelte` (frontend), rebuild the fleet frontend to change it.
backend/src/main/resources/content/fleet/modules/internal.md
+43
View File
@@ -0,0 +1,43 @@
---
module: fleet.modules
title: Modules
tab: Internal
order: 20
audience: internal
---
> **Internal · ops/support.** Written 2026-07-01 from `hiveops-fleet` source. Verify against code before relying on specifics.
## What this screen actually is
The **Agent Modules** screen (`FleetModules.svelte`) is a mostly **static reference catalog**. The list of 16 modules, their display names, descriptions, and the four categories (**Data Collection / Monitoring / Fleet Management / Updates**) are **hardcoded in the frontend** (`MODULE_CATALOG` const in `FleetModules.svelte`) — they are not fetched from the backend.
The only live data is the per-module status badge counts (running / disabled / failed / not reporting), fetched from `GET /api/fleet/modules/summary` on `hiveops-fleet` (port 8097).
## ⚠️ Known state: status badges are not wired up
`FleetApiController.getModuleSummary()` currently returns a **hardcoded empty list** (`return List.of();`). Because of this:
- The screen renders all 16 module cards with names + descriptions, but **no status badges appear** (the `{#if summary}` block in the component never matches, since `$moduleSummary` is always empty).
- This is expected current behaviour, **not** an outage. If someone reports "no running/failed counts on the Modules screen," that is the stub endpoint, not a broken agent or a data-sync problem.
## Roles required
- **Read (this screen):** `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` on `GET /api/fleet/modules/summary`. Any logged-in fleet user can view it.
- There are **no admin-only mutating actions** on this screen — it is strictly read-only. There is nothing to enable, disable, or approve here. Module enablement is driven by agent config (`CONFIG_UPDATE` fleet tasks / config server), not from this view.
## First things to check when it misbehaves
1. **Page loads but no badges** → expected: `/modules/summary` is a stub returning `[]`. Not a bug to escalate for a customer.
2. **Page blank / cards missing entirely** → frontend issue. The catalog is client-side, so a blank page means the SPA failed to load, not a backend outage. Check the fleet frontend bundle / nginx for `fleet.bcos.cloud`.
3. **500 / 401 / 403 on `/modules/summary`** → auth plane. `/api/fleet/**` (non-internal) uses the JWT `JWT_SECRET`; a 401/403 means a bad/expired JWT or a role the token lacks. Confirm the caller's JWT role is one of the allowed roles above.
4. **404 on `/modules/summary`** → wrong base path or service down. Confirm requests hit `hiveops-fleet` at `/api/fleet` (nginx location block), not a stale blue/green backend. `curl http://localhost:8097/actuator/health` on the services VM.
5. **Descriptions look wrong / a module is missing** → it's a frontend catalog edit, not data. Update `MODULE_CATALOG` in `FleetModules.svelte`, rebuild the fleet frontend.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| No status badges on any card | `/modules/summary` returns `List.of()` (stub) | Expected today; wiring the real summary is a backend change, not an ops fix |
| Whole page blank | SPA/bundle didn't load | Check nginx for `fleet.bcos.cloud`, rebuild/redeploy fleet frontend |
| 401/403 on the API call | Bad/expired JWT or missing role | Re-auth; confirm role is USER/CUSTOMER/ADMIN/MSP_ADMIN/BCOS_ADMIN |
| 404 on the API call | Stale blue/green backend or wrong path | Recheck nginx `/api/fleet/` upstream; hit active fleet color |
| Module counts look stale (if/when wired) | Would depend on the future data source (agent status reporting) | N/A today — no live source exists yet |
## Notes for support
- This screen does **not** tell you whether a specific ATM has a module running — it is a fleet-wide catalog/summary, and the summary is currently empty. To check a real ATM's module state, look at agent config / fleet tasks / the device record, not here.
- Related client methods `getModules()` → `/modules` and `getModulesByAtm()` → `/modules/atm/{id}` exist in the fleet API client but have **no matching backend endpoint** in `FleetApiController` and are **not** used by this screen — do not point people at them.
backend/src/main/resources/content/incident/dashboard/architect.md
+75
View File
@@ -0,0 +1,75 @@
---
module: incident.dashboard
title: Dashboard
tab: Architect
order: 30
audience: internal
---
> **How it's built.** The Dashboard is a thin read-only aggregation view: one Svelte component, three GET endpoints, all served by **hiveops-incident**. There is no dedicated dashboard aggregation service — most of the "analytics" (trend chart, incidents-by-type, device/agent status breakdowns) is computed **client-side** in the component from two data stores. See [platform.service-ownership].
## Owning service
`hiveops-incident` owns every endpoint the Dashboard calls, plus both source tables (`incidents`, `device_summary`). The device data in `device_summary` is a **Kafka-fed mirror** — the source of truth is `hiveops-devices`. See [platform.data-architecture].
## Data flow
```
hiveops-devices ──hiveops.device.events──▶ DeviceEventConsumer ──▶ device_summary (hiveiq_incident)
(group hiveops-incident-device-sync) │
▼
Dashboard.svelte ── GET /api/device-summary ─────────────────────────────────────▶ $atms store
│
agent-proxy / ARIA / journal ──▶ IncidentAutoCreationService ──▶ incidents (hiveiq_incident)
▼
Dashboard.svelte ── GET /api/incidents/paginated (90d, ≤2000) ───────────────────▶ $incidents store
Dashboard.svelte ── GET /api/incident-management/stats/summary ───────────────────▶ incidentStats
```
- **Device Status / Agent Connection panels** and the top "Device Status" cards are derived in-browser from `$atms` (the `device_summary` snapshot): fields `status` / `serviceState`, `inSupervisor`, `deviceType` (ATM vs TCR), `agentConnectionStatus`.
- **Incident Trends** and **Incidents by Type** are derived in-browser from `$incidents` (last 90 days, ≤2000 rows), bucketed by `createdAt` / `type` / `severity` / `actualResolutionTime`.
- **Ticket cards** (Open/Assigned/In Progress/Critical) come from the server-computed `IncidentStatisticsDTO` (all-time, institution-scoped), so they don't inherit the 90-day/2000 cap.
## Kafka
See [platform.kafka].
**Consumed by incident (relevant to this view):**
| Topic | Listener (group) | Effect on Dashboard |
|---|---|---|
| `hiveops.device.events` | `DeviceEventConsumer` (`hiveops-incident-device-sync`) | Upserts/deletes `device_summary` rows → device counts + agent-connection panel; also drives `IncidentAutoCreationService` |
| `hiveops.aria.threats` | `AriaThreatEventConsumer` | Auto-creates incidents → shows up in ticket/trend/type panels |
No topics are produced *for* the Dashboard; incident's lifecycle producers (`hiveops.incidents.created/updated/resolved`) fan out to other services (analytics/reports/recon mirrors) but the Dashboard reads Postgres directly, not those topics.
There is **no reconciler** for `device_summary` — if `hiveops-devices` stops publishing, the mirror (and therefore the device counts) silently goes stale. This is the single most common architectural failure point for this view.
## Tables read
| Table (DB `hiveiq_incident`) | Read by | For |
|---|---|---|
| `device_summary` | `DeviceSummaryController` → `DeviceSummaryRepository` | device + agent-connection stats |
| `incidents` (joins `atm.institution`) | `IncidentController.getIncidentsPaginated`, `IncidentService.getStatistics` | trend/type panels, ticket cards |
| `atm_groups` (via `AtmGroupRepository.findByIdWithAtms`) | `DeviceSummaryController` when `groupId` set | group-scoped device filtering |
The Dashboard **writes nothing** — it is pure read.
## Key API endpoints
| Method + path | Controller | Auth | Scope |
|---|---|---|---|
| `GET /api/device-summary` | `DeviceSummaryController` | authenticated | `InstitutionContext.resolveScope()` |
| `GET /api/device-summary/paginated?groupId&page&size` | `DeviceSummaryController` | authenticated | scope + group ATM-id set |
| `GET /api/incidents/paginated` | `IncidentController` | authenticated (no `@PreAuthorize`) | scope + `dateFrom`/`groupId` |
| `GET /api/incident-management/stats/summary` | `IncidentManagementController` | authenticated (no `@PreAuthorize`) | `resolveScope()` → institution names |
`stats/summary` counts by loading incidents (`findAll` or an `atm.institution` specification) and reducing in memory — it is not a SQL aggregate, so cost grows with incident volume.
## Notable design points
- **Read-open by design:** the Dashboard's four endpoints carry no role gate; only incident *mutations* require `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`.
- **Two different truth windows on one screen:** ticket cards = all-time server stats; trend/type/"This Week" = client-computed over a capped 90-day window. Expect small mismatches on busy fleets — this is structural, not a bug in the endpoint.
- Institution isolation is enforced at the query layer (`InstitutionContext` + `institutionKeyService.resolveInstitutionNames`), not in the frontend.
Cross-links: [platform.data-architecture] · [platform.kafka] · [platform.service-ownership]
backend/src/main/resources/content/incident/dashboard/claude.md
+53
View File
@@ -0,0 +1,53 @@
---
module: incident.dashboard
title: Dashboard
tab: Claude
order: 40
audience: internal
---
## Owner
- Service: **hiveops-incident** (port 8080, NGINX `/api/incident/`). Owns all Dashboard endpoints + both source tables.
- Frontend: `hiveops-incident/frontend/src/components/Dashboard/Dashboard.svelte` (Vite+Svelte SPA; h1 renders literal text "Overview").
- Read-only view. No writes, no mutations.
## Endpoints (all authenticated; NONE role-gated)
| Method | Path | Returns | Feeds |
|---|---|---|---|
| GET | `/api/device-summary` | `DeviceSummary[]` | device + agent-connection panels |
| GET | `/api/device-summary/paginated?groupId&page&size` | `Page<DeviceSummary>` | group-scoped device counts |
| GET | `/api/incidents/paginated?page=0&size=2000&dateFrom=<90d>&groupId` | `Page<Incident>` | trends, by-type, "This Week" |
| GET | `/api/incident-management/stats/summary` | `IncidentStatisticsDTO` | ticket cards (all-time) |
- `IncidentStatisticsDTO` fields: `totalIncidents, openIncidents, assignedIncidents, inProgressIncidents, resolvedIncidents, closedIncidents, criticalCount, highCount, avgResolutionTime`.
- Controllers: `DeviceSummaryController`, `IncidentController.getIncidentsPaginated`, `IncidentManagementController.getIncidentStats`.
## Tables (DB `hiveiq_incident`)
- `device_summary` — device/agent state (read via `DeviceSummaryRepository`). **Kafka-fed mirror, not authored here.**
- `incidents` — read via `IncidentController` + `IncidentService.getStatistics` (joins `atm.institution`).
- `atm_groups` — group→ATM-id set when `groupId` present.
## Kafka
- Consume `hiveops.device.events` → `DeviceEventConsumer` (group **`hiveops-incident-device-sync`**) → upserts `device_summary`. Constant: `DeviceEventKafkaConfig.TOPIC`.
- Consume `hiveops.aria.threats` → auto-creates incidents.
- Dashboard produces **no** topics.
## Scoping / auth
- Every endpoint filters by `InstitutionContext.resolveScope()` (from caller JWT). `null` scope = all institutions.
- View requires only a valid JWT. `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` gates only incident *mutations* elsewhere, NOT this view.
## Gotchas
- Trend/by-type/"This Week" derive from a **90-day, ≤2000-row** client slice (`loadIncidentsForDashboard`) → silently truncates on busy fleets. Ticket cards (server stats) are all-time and unaffected.
- `fetchIncidentStats()` failure is **swallowed** (`catch {}`) → UI silently falls back to capped-store counts; ticket cards under-report with no error shown.
- Device counts stale = `device_summary` mirror stale. **No reconciler.** Root cause is upstream (hiveops-devices not publishing, or consumer group lag) — not incident.
- Agent Connection "Disconnected" bucket = everything not `CONNECTED`/`NEVER_CONNECTED`, including null → can inflate.
- Active `defaultGroupId` scopes both ATMs and incidents; stale group id → 404 "Group not found" on `device-summary/paginated`.
- Auto-refresh toggle persists in `localStorage['incident_dashboard_autoRefresh']`; interval = `settings.dashboardRefreshIntervalSeconds`.
- Frontend `API_BASE` = `/api` (dev) → prod resolves via `window.__APP_CONFIG__.apiUrl`; through NGINX = `/api/incident/...`.
## Do NOT
- Do NOT assume the Dashboard writes/owns device data — it mirrors `hiveops-devices` via Kafka; fix stale counts upstream.
- Do NOT treat trend/type/"This Week" as authoritative counts — they are capped (90d/2000).
- Do NOT add a role gate expecting to "lock down" the Dashboard endpoints; they are read-open by design and shared with other views.
- Do NOT invent a `stats/summary`-style aggregate SQL — current impl loads rows and reduces in memory.
- Do NOT look for Dashboard data in journal/mgmt/fleet — all four endpoints are hiveops-incident.
backend/src/main/resources/content/incident/dashboard/internal.md
+51
View File
@@ -0,0 +1,51 @@
---
module: incident.dashboard
title: Dashboard
tab: Internal
order: 20
audience: internal
---
> **Internal ops/support view.** The Dashboard is **read-only** — there are no admin-only actions *on* this screen. It renders live counts from three GET endpoints and derives the charts client-side. Most "the dashboard is wrong" tickets are upstream data problems, not dashboard bugs.
## What it actually loads
Component: `hiveops-incident/frontend/src/components/Dashboard/Dashboard.svelte`. On mount and on each refresh it fires **three** calls (all against `hiveops-incident`, NGINX path `/api/incident/`):
| Frontend call | Endpoint | Feeds |
|---|---|---|
| `loadAtms()` | `GET /api/device-summary` (or `/api/device-summary/paginated?groupId=` when a group is active) | Device Status cards, Device Status panel, Agent Connection panel |
| `loadIncidentsForDashboard()` | `GET /api/incidents/paginated?page=0&size=2000&dateFrom=<90d ago>` | Incident Trends, Incidents by Type, "This Week" |
| `fetchIncidentStats()` | `GET /api/incident-management/stats/summary` | Ticket cards: Open / Assigned / In Progress / Critical |
Key split to remember when triaging: **ticket cards come from the server** (`stats/summary`, accurate across all time); **trend + type panels + "This Week" are computed in the browser** from the last-90-day / max-2000-record incident slice.
## Roles / access
- All three endpoints are **read-open to any authenticated user** — no `@PreAuthorize` on `stats/summary`, `incidents/paginated`, or `device-summary`. There is no MSP_ADMIN gate to *view* the Dashboard.
- The admin-only actions (`hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`) live on the incident *management* mutations (assign/transition/bulk/notes) — not on anything the Dashboard renders.
- Numbers are **institution-scoped** via `InstitutionContext.resolveScope()` (derived from the caller's JWT). A user only ever sees their own institution's devices/incidents. "User X sees the wrong institution's numbers" = a JWT/scope or stale-mirror issue, not a dashboard bug.
## First things to check when it misbehaves
1. **All device counts are 0 / stale (but devices exist in the Devices app)** → the `device_summary` mirror in `hiveiq_incident` is stale. This table is fed by Kafka (`hiveops.device.events`), **not** written by incident itself. Check: is `hiveops-devices` publishing, and is the incident consumer group `hiveops-incident-device-sync` alive / not lagging? See [platform.kafka].
2. **Ticket cards (Open/Assigned/In Progress/Critical) show 0 but incidents exist** → `stats/summary` call is failing. The frontend **swallows that error silently** (`catch {}`) and falls back to counting the capped 90-day store, so the UI looks "fine" while under-reporting. Curl `GET /api/incident-management/stats/summary` with a real JWT.
3. **Trends / "Incidents by Type" / "This Week" look low on a busy fleet** → expected limitation, not a bug: `loadIncidentsForDashboard()` pulls only the **last 90 days, capped at 2000 records**. A fleet with >2000 incidents in 90 days will truncate these panels. Ticket cards (server stats) stay correct.
4. **Header shows a group name and numbers look small** → a device group is selected (`defaultGroupId`). Both ATMs and incidents are then filtered to that group. Clear the group to see the whole fleet.
5. **"Group not found" / 404 on load** → `device-summary/paginated` 404s when the active `groupId` no longer exists. Reset the saved group.
6. **Nothing loads at all** → check NGINX routing to `/api/incident/` and JWT validity; the error banner at the top of the card shows the last store error.
## Common failure modes → fix
| Symptom | Likely cause | Fix |
|---|---|---|
| Device totals frozen / wrong | `device_summary` mirror stale (no reconciler exists) | Confirm devices publishing `hiveops.device.events`; restart/inspect `hiveops-incident-device-sync` consumer |
| Agent Connection "Disconnected" inflated | Any status that isn't `CONNECTED`/`NEVER_CONNECTED` (incl. null) buckets as Disconnected | Verify agents actually reporting; null status ≠ truly disconnected |
| Wrong institution's counts | JWT scope / stale mirror institutionKey | Check caller JWT scope + device_summary institution_key |
| Ticket cards zero, panels populated | `stats/summary` erroring (silent fallback) | Curl the endpoint; check incident service logs |
| Auto-refresh stopped | Toggle off (persisted in `localStorage` key `incident_dashboard_autoRefresh`) | Flip Auto-refresh back on; interval = `settings.dashboardRefreshIntervalSeconds` |
## Auto-refresh mechanics
- Toggle state persists per-browser in `localStorage['incident_dashboard_autoRefresh']`.
- Countdown resets to `settings.dashboardRefreshIntervalSeconds` each cycle; the **↻** button forces an immediate `doRefresh()` (all three calls in parallel).
backend/src/main/resources/content/incident/incident-management/architect.md
+72
View File
@@ -0,0 +1,72 @@
---
module: incident.incident-management
title: Incident Management
tab: Architect
order: 30
audience: internal
---
> How the Incident Management screen is built. Owner service: **hiveops-incident**. See [platform.service-ownership], [platform.data-architecture], [platform.kafka].
## Shape
- **Frontend:** `hiveops-incident/frontend` — `components/Incident/IncidentList/IncidentList.svelte` (the Active/Parked grid, filters, bulk actions, link/group, fraud-close modal). API client `lib/api.ts` (`incidentAPI`, `journalEventAPI`, `helpdeskPersonAPI`), base `/api`.
- **Backend:** `hiveops-incident` (Spring Boot, port **8080**, NGINX `/api/incident/`). Two controllers back this screen:
- `IncidentController` — `@RequestMapping("/api/incidents")` — reads/CRUD (`/paginated`, `/status/open`, `/severity/critical`, `/atm|device/{id}`).
- `IncidentManagementController` — `@RequestMapping("/api/incident-management")` — workflow: assign, transition, notes, bulk, link/group, stats.
- **Database:** `hiveiq_incident` (Postgres) via **`DB_URL`** full JDBC URL; Flyway `validate`.
## Data it reads/writes
| Table / entity | Role in this feature |
|----------------|----------------------|
| `incidents` (`Incident`) | primary record — type, severity, status, assignment, timestamps |
| `incident_events` | audit stream (`IncidentEventService.record`, e.g. `STATUS_CHANGED`) |
| `incident_notes` / notes on the entity | free-text notes added via `/notes` and workflow transitions |
| `workflow_transitions` | valid status-edge definitions consulted by `workflowStateService` |
| `technicians`, `helpdesk_persons` | assignment targets |
| `atms` (`Atm`) | device the incident belongs to |
| `device_summary` | Kafka-fed mirror of device metadata (name/institution shown on rows) |
| `incident_embeddings` | Adoons/AI indexing of incidents |
## Workflow transition flow
`POST /api/incident-management/{id}/transition` (admin-gated):
1. Load incident, resolve `WorkflowTransition` for `(currentStatus → newStatus)`; reject 400 if no edge.
2. If `transition.requiresAssignment` and no assignee → 400.
3. `assertNoUnfixedChildren()` blocks closing a primary with unfixed linked children.
4. Mutate status; set `resolvedAt`/`actualResolutionTime` on RESOLVED, `closedAt` on CLOSED.
5. Append a note, record an `incident_events` `STATUS_CHANGED` row, re-index for Adoons.
6. Broadcast STOMP `/topic/incidents/status-changed` for live UI update.
`PARKED` is a first-class status: while any incident for a device is `PARKED`, `IncidentAutoCreationService` suppresses new auto-incidents for that device (#219).
## Kafka — incident lifecycle (producer side)
`IncidentLifecycleProducer` publishes on create and update (config `IncidentLifecycleKafkaConfig`, 3 partitions each):
| Topic | When |
|-------|------|
| `hiveops.incidents.created` | `publishCreated` — on incident create (`IncidentService:73`) |
| `hiveops.incidents.updated` | `publishUpdated` — non-resolved update (`IncidentService:189,433`) |
| `hiveops.incidents.resolved` | `publishUpdated` when status RESOLVED/CLOSED |
Downstream services (analytics, reports, messaging, etc.) consume these — this screen is the source of those events.
## Kafka — auto-creation (consumer side)
Incidents shown here are largely machine-generated. Consumers:
| Topic | Listener (group) | Effect |
|-------|------------------|--------|
| `hiveops.device.events` | `DeviceEventConsumer` (`hiveops-incident-device-sync`) | sync `device_summary`; `IncidentAutoCreationService` raises incidents from device events |
| `hiveops.aria.threats` | `AriaThreatEventConsumer` (`hiveops-incident-aria`) | raise incidents from ARIA threat events |
| `adoons.rule-suggestion.ready` | `RuleSuggestionResultConsumer` (`hiveops-incident-adoons`) | ingest completed Adoons rule suggestions |
Pattern: **executor/consumer → mutate incident → lifecycle producer → Kafka → downstream record stores**. The UI only ever sees the resulting `incidents` rows.
## Boundaries
- Agents do **not** hit this service directly. hiveops-agent-proxy calls incident `/api/internal/**` (secured by `X-Internal-Secret` → `ROLE_INTERNAL`) to ingest journal events / heartbeats that feed auto-creation. See [platform.service-ownership].
- Device master data is owned by **hiveops-devices**; incident holds a read-only `device_summary` mirror fed over Kafka — no reconciler, so publish-on-every-change is required upstream. See [platform.data-architecture].
- `JWT_SECRET` is shared with hiveops-auth / hiveops-mgmt; `mgmt.service.url` used for agent-token validation. See [platform.kafka] for bootstrap/topic conventions.
backend/src/main/resources/content/incident/incident-management/claude.md
+63
View File
@@ -0,0 +1,63 @@
---
module: incident.incident-management
title: Incident Management
tab: Claude
order: 40
audience: internal
---
## Ownership
- **Service:** `hiveops-incident` (Spring Boot, port **8080**, NGINX `/api/incident/`).
- **DB:** `hiveiq_incident` (Postgres). Env var **`DB_URL`** = single full JDBC URL (NOT split host/port/name).
- **Frontend view:** `hiveops-incident/frontend/src/components/Incident/IncidentList/IncidentList.svelte`; API client `lib/api.ts` (`incidentAPI`).
## Endpoints (METHOD path) — all under `/api`
| Method | Path | Role |
|--------|------|------|
| GET | `/api/incidents/paginated` | any auth |
| GET | `/api/incidents` · `/api/incidents/{id}` | any auth |
| GET | `/api/incidents/status/open` · `/api/incidents/severity/critical` | any auth |
| GET | `/api/incidents/atm/{atmId}` · `/api/incidents/device/{atmId}` (aliases) | any auth |
| PUT | `/api/incidents/{id}` | any auth |
| POST | `/api/incident-management` (create) | MSP_ADMIN/BCOS_ADMIN |
| POST | `/api/incident-management/{id}/assign` · `/reassign` · `/unassign` | MSP_ADMIN/BCOS_ADMIN |
| POST | `/api/incident-management/{id}/transition` | MSP_ADMIN/BCOS_ADMIN |
| POST | `/api/incident-management/{id}/notes` | MSP_ADMIN/BCOS_ADMIN |
| POST | `/api/incident-management/bulk/update` | MSP_ADMIN/BCOS_ADMIN |
| POST | `/api/incident-management/{id}/link` · `/group` | MSP_ADMIN/BCOS_ADMIN |
| DELETE | `/api/incident-management/{id}/link/{linkedId}` | MSP_ADMIN/BCOS_ADMIN |
| GET | `/api/incident-management/{id}/links` · `/audit-trail` · `/stats/summary` | any auth |
| GET | `/api/incident-management/technicians/available` | any auth |
Bulk close body: `{ "incidentIds": ["1","2"], "newStatus": "CLOSED" }` (ids are strings).
## Roles
- Every mutation: `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`. Reads: authenticated only.
## Tables (`hiveiq_incident`)
- `incidents`, `incident_events`, `incident_notes`, `workflow_transitions`, `atms`, `technicians`, `helpdesk_persons`, `device_summary` (Kafka mirror), `incident_embeddings`.
## Kafka topics
- **Produced:** `hiveops.incidents.created`, `hiveops.incidents.updated`, `hiveops.incidents.resolved` (`IncidentLifecycleProducer` / `IncidentLifecycleKafkaConfig`).
- **Consumed (auto-create):** `hiveops.device.events` (group `hiveops-incident-device-sync`), `hiveops.aria.threats` (group `hiveops-incident-aria`), `adoons.rule-suggestion.ready` (group `hiveops-incident-adoons`).
- WebSocket (STOMP): `/topic/incidents/status-changed`.
## Statuses
- `OPEN, ASSIGNED, IN_PROGRESS, RESOLVED, CLOSED, PARKED`.
- A device with any `PARKED` incident suppresses ALL auto-creation for that device (#219).
## Gotchas
- Transition requires a valid `workflow_transitions` edge; if `requiresAssignment`, assignee must be set first (else 400).
- Closing blocked by `assertNoUnfixedChildren()` if linked children unfixed.
- **Possible Fraud note-before-close is FRONTEND-only** (`IncidentList.svelte` fraud-close modal) — NOT enforced by the API.
- Agents never call this service; they go via **hiveops-agent-proxy** → `/api/internal/**` (`X-Internal-Secret`).
- `device_summary` is a read-only Kafka mirror from hiveops-devices; no reconciler → stale = fix upstream.
- `DB_URL` default points at a stale `hiveops_incident` DB — prod overrides to `hiveiq_incident`.
## Do NOT
- Do NOT add device/institution writes here — device data is owned by hiveops-devices.
- Do NOT assume the fraud-note rule protects direct API closes — it doesn't.
- Do NOT expect UPDATE/transition without an admin JWT — reads succeed, writes 403.
- Do NOT split `DB_URL` into host/port/name.
- Do NOT create incidents for a parked device expecting auto-flow — parking silences it.
- Do NOT invent transition endpoints per-status (start/resolve/close); all go through one `POST /{id}/transition` with `newStatus`.
backend/src/main/resources/content/incident/incident-management/internal.md
+52
View File
@@ -0,0 +1,52 @@
---
module: incident.incident-management
title: Incident Management
tab: Internal
order: 20
audience: internal
---
> Ops/support/admin view for the Incident Management screen (`IncidentList.svelte`). Served by **hiveops-incident** (port 8080, NGINX `/api/incident/`). All facts below grounded in source as of 2026-07-01.
## Who can do what
Read/list is open to any authenticated user. **Every mutating action requires `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`** (`@PreAuthorize` on `IncidentManagementController`):
| Action | Endpoint | Role gate |
|--------|----------|-----------|
| Assign / reassign / unassign | `POST /api/incident-management/{id}/assign` · `/reassign` · `/unassign` | admin |
| Workflow transition (Start/Resolve/Close/Reopen/Park) | `POST /api/incident-management/{id}/transition` | admin |
| Add note | `POST /api/incident-management/{id}/notes` | admin |
| Bulk close | `POST /api/incident-management/bulk/update` | admin |
| Link / unlink / group | `POST /api/incident-management/{id}/link` · `DELETE .../link/{linkedId}` · `POST .../group` | admin |
| Create incident (manual) | `POST /api/incident-management` or `POST /api/incidents` | admin |
If an admin reports "buttons do nothing / 403 on save," first confirm their JWT actually carries `MSP_ADMIN` or `BCOS_ADMIN` — a plain customer role can load the list but cannot transition or assign.
## First things to check when it misbehaves
1. **List empty / won't load** — the grid calls `GET /api/incidents/paginated`. Check the incident service is up: `curl http://localhost:8080/actuator/health` on the services VM. A 401/403 through NGINX usually means a missing/expired JWT, not a data problem.
2. **"No new incidents appearing for a device"** — the device may be **parked**. Any incident in `PARKED` status silences that device: `IncidentAutoCreationService.isDeviceParked()` suppresses all auto-creation (device events, log-errors, AGENT_OFFLINE) until the device is un-parked (issue #219). This is intended, not a bug — un-park the incident to resume auto-creation.
3. **Auto-created incidents not arriving at all** — auto-creation is driven by Kafka, not by the UI. Incidents are created by consumers of `hiveops.device.events` (`DeviceEventConsumer`) and `hiveops.aria.threats` (`AriaThreatEventConsumer`). If none are appearing fleet-wide, check Kafka connectivity (`KAFKA_BOOTSTRAP_SERVERS`) and consumer group lag for `hiveops-incident-device-sync` / `hiveops-incident-aria`.
4. **Wrong institution / stale device name on a row** — the row's device metadata comes from the `device_summary` mirror, fed by `hiveops.device.events` from **hiveops-devices**. There is no reconciler; a stale mirror means devices didn't publish. Fix at the source (devices), not here.
5. **Can't close an incident** — two guards:
- The incident has **unfixed linked children** — `assertNoUnfixedChildren()` blocks closing a primary while linked child incidents remain unfixed.
- **Possible Fraud** incidents require a note before closing — this is enforced **in the frontend** (`fraud-close-modal` in `IncidentList.svelte`), so it only applies via the UI, not direct API calls.
6. **Transition rejected (400)** — the target status must be a valid edge in the workflow (`workflowStateService.getTransition(from, to)`) and, if the transition `requiresAssignment`, the incident must already have an assignee. Assign first, then Start/Resolve.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| 403 on assign/transition/note | JWT lacks `MSP_ADMIN`/`BCOS_ADMIN` | issue a token with the right role |
| New tickets not raised for one device | device is parked (#219) | un-park its `PARKED` incident |
| No incidents fleet-wide | Kafka down / consumer lag | check `KAFKA_BOOTSTRAP_SERVERS`, consumer groups |
| Stale device name/institution on row | `device_summary` mirror stale | fix in hiveops-devices (source of truth) |
| Close button blocked | unfixed linked children or fraud-note missing | fix children first / add note in UI |
| Service can't reach DB | `DB_URL` misconfigured | it's a **single full JDBC URL** (`DB_URL`), not split host/port/name; default points at a stale `hiveops_incident` DB and MUST be overridden to `hiveiq_incident` |
## Notes for support
- Real-time list updates use a STOMP WebSocket topic `/topic/incidents/status-changed`; the "Auto-refresh" toggle also re-polls `/api/incidents/paginated`. If a user sees stale data, have them refresh — polling is the reliable path.
- Agents never write here directly. All agent journal/heartbeat traffic goes through **hiveops-agent-proxy** → incident `/api/internal/**` (secured by `X-Internal-Secret`). A gap in incoming events is an agent-proxy / agent problem, not an incident-UI problem.
- Adoons (AI) rule suggestions surface via `adoons.rule-suggestion.*` topics and separate rule screens — not part of this list view.
backend/src/main/resources/content/incident/incident-tracker/architect.md
+79
View File
@@ -0,0 +1,79 @@
---
module: incident.incident-tracker
title: Incident Tracker
tab: Architect
order: 30
audience: internal
---
> How the Incident Tracker is built. Grounded in `hiveops-incident` frontend + backend as of 2026-07-01. See [platform.data-architecture], [platform.kafka], [platform.service-ownership].
## Shape of the feature
The tracker is a **pure read view with client-side aggregation**. There is no server-side "tracker" projection, endpoint, or table — the Svelte component fans out to two existing controllers and assembles the board in the browser. All state (pipeline stages, severity rollup, sort, "recently closed" toggle) is computed in `IncidentTracker.svelte`.
Everything is owned by a **single service, `hiveops-incident`** ([platform.service-ownership]). The only cross-service dependency is the `device_summary` mirror, which is fed from **hiveops-devices** over Kafka.
## Services involved
| Service | Role for this feature |
|---------|-----------------------|
| `hiveops-incident` (backend, port 8080) | Serves `/api/device-summary` and `/api/incidents/paginated`; owns `incidents` + `device_summary` tables |
| `hiveops-incident` (frontend, Vite+Svelte SPA) | `IncidentTracker.svelte` — does all joining/aggregation |
| `hiveops-devices` | Source of truth for device data; publishes `hiveops.device.events` |
| NGINX | Routes `/api/incident/` → incident backend `:8080` |
## Data flow (per 60s refresh)
```
IncidentTracker.svelte ──(3 parallel GETs)──► hiveops-incident @ 8080
1. deviceSummaryAPI.getAll() → GET /api/device-summary → DeviceSummaryController.getAll() → device_summary
2. incidentAPI.getPaginated(open) → GET /api/incidents/paginated → IncidentController.getIncidentsPaginated() → incidents
3. incidentAPI.getPaginated(closed)→ GET /api/incidents/paginated (…) → same handler → incidents
```
Client-side assembly:
1. Build `dsMap` keyed by `device_summary.atm_id` (string natural key).
2. Concatenate active + closed incidents; group by `incident.atmName ?? String(incident.atmId)`.
3. Join each incident group to `dsMap` for location; if no match → stub row (`location: ''`).
4. Per ATM: `hasAction` if any active; `currentStage = min(stage index)` over active; `highestSeverity` = first of `[CRITICAL,HIGH,MEDIUM,LOW]` present; `closedStage = max(stage index)` over closed.
5. Filter to rows that `hasAction || recentlyClosed`; sort action-first, then by severity, then closed-most-recent, then atmId.
`incident.atmId` (integer FK) is retained only for the `viewAtmIncidents` dispatch that drills into the filtered `IncidentList`.
## How `device_summary` stays current (Kafka) — [platform.kafka]
The left column is **not** incident's own device data — it's a Kafka-synced mirror:
| Topic | Consumer (class / group) | Effect |
|-------|--------------------------|--------|
| `hiveops.device.events` | `DeviceEventConsumer` / `hiveops-incident-device-sync` | Upsert (`repo.save`) or delete rows in `device_summary`; also updates `atms` and can trigger `IncidentAutoCreationService` |
Topic + group are constants in `config/DeviceEventKafkaConfig` (`TOPIC = "hiveops.device.events"`, `CONSUMER_GROUP = "hiveops-incident-device-sync"`). No reconciler — if devices stops publishing, the mirror silently ages. This is the same `device_summary` mirror pattern used across incident/fleet/reports/recon ([platform.data-architecture]).
Incident lifecycle itself produces `hiveops.incidents.created/updated/resolved` (via `IncidentLifecycleProducer`), but **the tracker does not consume these** — it re-polls REST every 60s instead.
## DB tables
| Table | Access | Notes |
|-------|--------|-------|
| `incidents` (`@Table(name="incidents")`) | read | status, severity, atmName, atmId, resolvedAt, updatedAt |
| `device_summary` (`@Table(name="device_summary")`) | read | Kafka-fed mirror; keyed by `atm_id` string |
Both live in DB `hiveiq_incident`. Institution scoping is applied server-side in each controller via `InstitutionContext.resolveScope()` (repository `findByInstitutionKeyIn(scope)` when scoped, `findAll` when unrestricted).
## Key API endpoints (this feature)
| Method + path | Controller | Auth |
|---------------|-----------|------|
| `GET /api/device-summary` | `DeviceSummaryController.getAll` | any JWT; institution-scoped |
| `GET /api/incidents/paginated` | `IncidentController.getIncidentsPaginated` | any JWT; institution-scoped |
| `GET /api/incidents` | `IncidentController.getAllIncidents` | any JWT; institution-scoped (not used by tracker) |
Note this feature has **no executor→Kafka→record-store write path** — it is entirely read-side. The write/lifecycle path (create/assign/transition → `hiveops.incidents.*`) belongs to the broader incident domain, not to the tracker view.
## Cross-links
- [platform.data-architecture] — device_summary mirror pattern
- [platform.kafka] — topic/consumer conventions
- [platform.service-ownership] — incident owns incidents + journal-event store
backend/src/main/resources/content/incident/incident-tracker/claude.md
+60
View File
@@ -0,0 +1,60 @@
---
module: incident.incident-tracker
title: Incident Tracker
tab: Claude
order: 40
audience: internal
---
> Terse act-without-guessing reference. Confirmed against `hiveops-incident` code 2026-07-01.
## Owner
- Service: **hiveops-incident** (backend port 8080, NGINX `/api/incident/`). DB: **hiveiq_incident** (env `DB_URL` = full JDBC URL).
- Frontend: `hiveops-incident/frontend/src/components/IncidentTracker/IncidentTracker.svelte` (Vite+Svelte SPA, state-routed via `currentView`).
- The tracker is **read-only + client-side aggregated**. No tracker endpoint/table/topic exists.
## Endpoints the tracker calls (all GET, all via NGINX `/api/incident/` → base `/api`)
| Method + path | Params used | Handler |
|---------------|-------------|---------|
| GET `/api/device-summary` | — | `DeviceSummaryController.getAll` |
| GET `/api/incidents/paginated` | `status=OPEN,ASSIGNED,IN_PROGRESS&size=1000&groupId` | `IncidentController.getIncidentsPaginated` |
| GET `/api/incidents/paginated` | `status=RESOLVED,CLOSED&size=500&sort=updatedAt,desc&groupId` | same |
## Drill-down (View Incidents button)
- Dispatches `viewAtmIncidents { atmId: <integer FK>, statuses }` into `IncidentList` — filter is client-side, not a new endpoint.
## Tables
| Table | R/W here | Key |
|-------|----------|-----|
| `incidents` | read | join key = `atmName` (string), `atmId` = integer FK |
| `device_summary` | read | key = `atm_id` (string, natural key) |
## Kafka (feeds `device_summary`, not the tracker directly)
| Topic | Consumer / group |
|-------|------------------|
| `hiveops.device.events` | `DeviceEventConsumer` / `hiveops-incident-device-sync` |
Constants in `config/DeviceEventKafkaConfig`: `TOPIC`, `CONSUMER_GROUP`.
## Roles
- Viewing tracker: **any valid JWT**. Read endpoints have **no `@PreAuthorize`**; scoped by `InstitutionContext.resolveScope()`:
- CUSTOMER / MSP_ADMIN → scoped to JWT-granted institution keys.
- BCOS_ADMIN / ADMIN / QDS_ADMIN / USER → `null` = all institutions.
- Mutations (create/assign/transition/bulk): `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`.
## Gotchas
- Row join is **client-side** on `incident.atmName == device_summary.atm_id`. No match → stub row, blank location. Blank location = **stale `device_summary` mirror** (devices didn't publish; no reconciler).
- `groupId` filter applies to **incidents only**, not `device-summary` (per code comment).
- Board shows only ATMs with active incident, or resolved/closed when the toggle is on.
- Pipeline stage & severity are computed in JS (`Math.min` stage over active; severity precedence CRITICAL>HIGH>MEDIUM>LOW). To change a row, fix the `incidents` record.
- Silent caps: active `size=1000`, closed `size=500`, no UI pagination → truncates on very large fleets.
- Auto-refresh 60s (`setInterval`), no websocket. Lifecycle topics `hiveops.incidents.created/updated/resolved` are produced but **not consumed** by the tracker.
- `DB_URL` hardcoded default is a stale `hiveops_incident` on 192.168.200.103 — prod overrides to `hiveiq_incident`.
## Do NOT
- Do NOT look for a `/api/incident-tracker` endpoint or a `tracker` table — none exist.
- Do NOT expect the tracker to update `device_summary` — it's read-only; devices owns device data.
- Do NOT assume it consumes `hiveops.incidents.*` — it re-polls REST.
- Do NOT add a write to `incidents`/`device_summary` from this view — mutations belong to incident-management endpoints ([platform.service-ownership]).
- Do NOT call these agent-facing; agents never hit incident directly (agent-proxy → `/api/internal/**`).
- Do NOT report the tracker as "AI" — hiveops-ai is branded **Adoons**; the tracker has no AI.
backend/src/main/resources/content/incident/incident-tracker/internal.md
+53
View File
@@ -0,0 +1,53 @@
---
module: incident.incident-tracker
title: Incident Tracker
tab: Internal
order: 20
audience: internal
---
> Ops / support / admin view. What to check when the Incident Tracker board is wrong or empty. Grounded in `hiveops-incident` frontend + backend as of 2026-07-01.
## What the board actually is
The Incident Tracker is a **read-only, client-side aggregation**. The Svelte component (`frontend/src/components/IncidentTracker/IncidentTracker.svelte`) makes three GET calls in parallel and builds every row, pipeline stage, severity, and sort order **in the browser**. There is no dedicated "tracker" endpoint or table — nothing is written back.
Three calls per refresh (auto every **60s**, or manual ↻ button):
1. `GET /api/device-summary` — ATM inventory (id + location), authoritative source for the left column.
2. `GET /api/incidents/paginated?status=OPEN,ASSIGNED,IN_PROGRESS&size=1000&groupId=…` — active incidents.
3. `GET /api/incidents/paginated?status=RESOLVED,CLOSED&size=500&sort=updatedAt,desc&groupId=…` — recently closed.
Rows are joined **in JS** on the ATM business key: `incident.atmName` ↔ `device_summary.atm_id`.
## Roles required
- **Viewing the tracker** needs only a **valid JWT** — the two read endpoints (`/api/device-summary`, `/api/incidents/paginated`) carry **no `@PreAuthorize`**. Access is scoped by `InstitutionContext.resolveScope()`:
- `ROLE_CUSTOMER` / `ROLE_MSP_ADMIN` → **scoped** to their granted institution keys (list comes from the JWT credentials).
- `BCOS_ADMIN` / `ADMIN` / `QDS_ADMIN` / `USER` → `null` scope = **sees all institutions**.
- **Acting on an incident** (the "View Incidents" drill-down leads to create/assign/transition) requires `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` — enforced on `POST /api/incidents`, `POST /api/incident-management/{id}/assign`, `/transition`, `/bulk/update`.
## First things to check when it misbehaves
**Board is empty / "All ATMs clear" but you expect incidents**
- The board only shows ATMs with an **open** incident (`OPEN`/`ASSIGNED`/`IN_PROGRESS`) or, with the toggle on, a `RESOLVED`/`CLOSED` one. If every incident is closed and the toggle is off, you correctly see "All clear."
- Confirm the user's **institution scope** — an MSP_ADMIN only sees incidents for their granted institution keys. A wrong/empty scope in the JWT hides everything.
- Check the group filter: `defaultGroupId` store applies `groupId` to the **incident** queries. A stale/wrong default group can filter out all incidents.
**Rows show a blank location or the ATM ID looks raw**
- The left column comes from `device_summary`. If the join key (`incident.atmName`) has no matching `device_summary.atm_id`, the row falls back to a **stub** (`location: ''`). Root cause is almost always a **stale `device_summary` mirror**.
- `device_summary` is Kafka-fed from **hiveops-devices** via topic `hiveops.device.events` (consumer `DeviceEventConsumer`, group `hiveops-incident-device-sync`). If devices didn't publish, or the consumer lagged/failed, incident's mirror goes stale. Check `DeviceEventConsumer` logs (`Upserted device_summary for …` / `Removed device_summary for deleted device …`).
**Severity or pipeline stage looks wrong**
- Both are computed client-side. Pipeline stage = **lowest** stage index among that ATM's active incidents (`Math.min`); severity = the most urgent among active incidents (`CRITICAL > HIGH > MEDIUM > LOW`). This reflects the underlying `incidents.status` / `incidents.severity` values — fix the incident record, not the board.
**Board silently missing some incidents on a huge fleet**
- Active list is capped at `size=1000`, closed at `size=500`, both **unpaginated in the UI**. An institution with more open incidents than the cap will silently truncate. See uncertainties in the architect notes.
**Nothing loads / "Failed to load data"**
- One of the three GETs failed. Check browser network tab for which. Common causes: NGINX `/api/incident/` route down, expired JWT (401), or CORS. Backend port is **8080** behind NGINX path `/api/incident/`.
## Where to look
- Backend service: `hiveops-incident`, DB `hiveiq_incident` (env `DB_URL`, full JDBC URL — not split host/port/name).
- Tables read: `incidents`, `device_summary`.
- Logs: use **Loki** (services VM) for `hiveops-incident`, don't SSH+cat.
backend/src/main/resources/content/incident/journal-events/architect.md
+68
View File
@@ -0,0 +1,68 @@
---
module: incident.journal-events
title: Journal Events
tab: Architect
order: 30
audience: internal
---
How the **Event Journal** feature is built. This module is a thin read view over the journal-event store that **hiveops-incident** owns; the write side is agent ingestion routed through **hiveops-agent-proxy**. See [platform.service-ownership].
## Services involved
| Service | Role for this feature |
|---|---|
| incident-frontend (`JournalEvents.svelte`) | Vite+Svelte SPA view; per-ATM paginated table + filters |
| **hiveops-incident** | Owns reads and the `journal_events` store; `JournalEventController` (browser) + `InternalJournalController` (ingest) |
| **hiveops-agent-proxy** | Sole agent entry point; native `JournalEventController` forwards ingest to incident's internal API |
| hiveops-agent | Runs on the ATM; POSTs journal events |
Agents never call incident directly — see [platform.service-ownership].
## Data flow — read path
1. SPA selects an ATM (numeric `atms.id`) and calls `GET /api/journal-events/atm/{atmId}/paginated` with `page/size/sort/hoursBack/eventTypes[]` (axios base `/api`; `eventTypes` serialized as repeated keys, not indexed).
2. `JournalEventController.getEventsPaginated` → `JournalEventService.getEventsPaginated` builds a JPA `Specification` (`JournalEventSpecification.withFilters`) filtering on `atm.id`, `eventType IN (...)`, and `eventTime >= now-hoursBack`.
3. `journalEventRepository.findAll(spec, pageable)` → mapped to `JournalEventDTO`, returned as a Spring `Page` (`{content, page:{totalElements,totalPages,number}}`).
Aggregate reads (`/activity`, `/fleet/cassette-inventory`) additionally resolve institution scope via `InstitutionContext.resolveScope()` (null = unrestricted for `USER`/`ADMIN`/`QDS_ADMIN`/`BCOS_ADMIN`; scoped list for `CUSTOMER`/`MSP_ADMIN`). The per-ATM read endpoints do **not** apply that scope (they filter by `atmId` only).
## Data flow — write/ingest path
```
agent --POST /api/journal-events--> agent-proxy (ROLE_AGENT)
JournalEventController.createEvent
-> IncidentInternalService.createJournalEvent(body, contentType)
--POST /api/internal/journal-events (X-Internal-Secret)-->
incident InternalJournalController (ROLE_INTERNAL)
-> JournalEventService.createEvent(request)
```
`JournalEventService.createEvent`:
- Resolves/auto-registers the ATM via `atmService.findOrCreateAtm(agentAtmId, deviceType)` and updates last heartbeat.
- If `eventType` is not in the `event_type` lookup (`eventTypeService.isKnown`) → recorded via `eventExceptionService.record(...)`, returns `null` → HTTP `202`. Known types → saved, HTTP `201`.
- Side effects: flips ATM service state for `ATM_IN_SERVICE`/`ATM_OUT_OF_SERVICE`/`SUPERVISOR_ENTER`/`SUPERVISOR_EXIT`; runs `incidentAutoCreationService.processRecovery` + `processEvent` (may attach a newly created incident to the event).
- Wrapped in `@Retryable(PessimisticLockingFailureException, 3x, backoff)` — deadlock guard for the morning-reboot burst (incident#212).
**No Kafka in the ingest path** for this feature — journal ingestion is synchronous HTTP. This is *not* the executor→Kafka→record-store pattern; there is no `hiveops.journal.*` topic. (Downstream, incident *does* emit `hiveops.incidents.created/updated/resolved` when auto-incident creation fires, but that is the incidents domain, not the journal event itself.) See [platform.kafka].
## DB tables
- **`journal_events`** (DB `hiveiq_incident`, entity `JournalEvent`) — written on ingest, read by this view. Columns: `atm_id` (FK→`atms`), `incident_id` (nullable FK→`incidents`), `eventType`, `eventDetails` (TEXT), card-reader (`cardReaderSlot`,`cardReaderStatus`), cassette (`cassetteType`,`cassetteFillLevel`,`cassetteBillCount`,`cassetteCurrency`,`cassettePosition`,`cassetteDenomination`), `eventTime`, `eventSource`. Indexes: `idx_journal_events_atm_time (atm_id, eventTime DESC)`, `idx_journal_events_event_type`.
- **`atms`** — read for ATM name/location in the DTO; written (auto-register + heartbeat) on ingest.
- **`event_type`** — lookup that decides known vs. exception.
- **`incidents`** — linked when auto-incident creation fires.
See [platform.data-architecture].
## Key API endpoints
| Method + path | Service | Auth |
|---|---|---|
| `GET /api/journal-events/atm/{id}/paginated` | incident | JWT/browser role |
| `GET /api/journal-events/atm/{id}` `.../recent` `.../card-reader` `.../cassette` `.../cassette-inventory` | incident | JWT/browser role |
| `GET /api/journal-events/activity` · `/fleet/cassette-inventory` | incident | JWT/browser role (institution-scoped) |
| `POST /api/journal-events` | incident (`JournalEventController`) / agent-proxy native | AGENT or JWT/browser role |
| `POST /api/internal/journal-events` | incident (`InternalJournalController`) | `ROLE_INTERNAL` via `X-Internal-Secret` |
All `/atm/{id}/...` paths have a Phase-2 `/device/{id}/...` alias mapping to the same handler.
backend/src/main/resources/content/incident/journal-events/claude.md
+63
View File
@@ -0,0 +1,63 @@
---
module: incident.journal-events
title: Journal Events
tab: Claude
order: 40
audience: internal
---
Act-without-guessing reference. Everything below is confirmed in code.
## Ownership
- **Reads + store owner:** `hiveops-incident` (port 8080, NGINX `/api/incident/`).
- **Ingest gateway:** `hiveops-agent-proxy` (port 8093/host 8015). Agents NEVER hit incident directly.
- Frontend: `hiveops-incident/frontend/src/components/JournalEvents/JournalEvents.svelte` (view label "Event Journal").
## Endpoints (METHOD path)
| Method + path | Service | Auth |
|---|---|---|
| GET `/api/journal-events/atm/{atmId}/paginated` | incident | JWT/browser role |
| GET `/api/journal-events/atm/{atmId}` | incident | JWT/browser role |
| GET `/api/journal-events/atm/{atmId}/recent?hoursBack=24` | incident | JWT/browser role |
| GET `/api/journal-events/atm/{atmId}/card-reader` | incident | JWT/browser role |
| GET `/api/journal-events/atm/{atmId}/cassette` | incident | JWT/browser role |
| GET `/api/journal-events/atm/{atmId}/cassette-inventory` | incident | JWT/browser role |
| GET `/api/journal-events/activity?hoursBack=24` | incident | JWT/browser role (institution-scoped) |
| GET `/api/journal-events/fleet/cassette-inventory` | incident | JWT/browser role (institution-scoped) |
| POST `/api/journal-events` | incident controller / agent-proxy native | AGENT or JWT/browser role |
| POST `/api/internal/journal-events` | incident `InternalJournalController` | ROLE_INTERNAL (`X-Internal-Secret`) |
- `{atmId}` = numeric `atms.id`, NOT the `AT000xxx` agent id.
- Every `/atm/{id}/...` has an identical `/device/{id}/...` alias.
- Paginated query params: `page`, `size`, `sort=eventTime,desc`, `hoursBack`, `eventTypes` (repeated key, e.g. `eventTypes=CARD_READER_FAIL&eventTypes=...`).
## Roles
- Read `/api/journal-events/**`: `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','QDS_ADMIN','BCOS_ADMIN')`. NO `@PreAuthorize`; NOT admin-only.
- Ingest internal: `hasRole('INTERNAL')` (header `X-Internal-Secret` = env `INTERNAL_SERVICE_SECRET`).
- Agent path `POST /api/journal-events` also allows `ROLE_AGENT`.
## Tables / DB
- Table `journal_events` in DB **`hiveiq_incident`**. Entity `JournalEvent`.
- Reads also touch `atms`; ingest touches `atms`, `event_type` (known-type lookup), `incidents` (auto-link).
## Kafka
- NONE for journal events. Ingest is synchronous HTTP. No `hiveops.journal.*` topic exists.
## Ingest chain (exact)
`agent → agent-proxy JournalEventController POST /api/journal-events → IncidentInternalService.createJournalEvent → incident POST /api/internal/journal-events → JournalEventService.createEvent`
## Gotchas
- Unknown `eventType` (not in `event_type` lookup) → NOT stored; recorded as event exception; returns HTTP 202. Known → 201.
- `createEvent` is `@Retryable` on `PessimisticLockingFailureException` (incident#212 deadlock guard).
- `eventTime` parsed as `LocalDateTime` (no timezone); unparseable/blank → `now()`.
- Auto-registers ATM on `agentAtmId` via `findOrCreateAtm`; also updates last heartbeat + may flip service state (`ATM_IN_SERVICE`/`ATM_OUT_OF_SERVICE`/`SUPERVISOR_ENTER`/`SUPERVISOR_EXIT`) + trigger auto-incident.
- Category filter in UI is an allowlist of exact type strings (8 groups); types outside the groups are hidden when any category is ticked.
- Per-ATM read endpoints filter by `atmId` only — they do NOT apply `InstitutionContext` scope (only `/activity` and `/fleet/cassette-inventory` do).
## Do NOT
- Do NOT POST agent journal events straight to incident — go through agent-proxy.
- Do NOT call `/api/internal/journal-events` without `X-Internal-Secret` — 401/403.
- Do NOT pass the `AT000xxx` agent id where `{atmId}` (numeric) is expected.
- Do NOT expect a Kafka topic for journal ingest — there is none.
- Do NOT assume this view is admin-gated — any authenticated user role can read it.
- Do NOT treat a 202 on ingest as "stored" — it means the type was unknown.
backend/src/main/resources/content/incident/journal-events/internal.md
+45
View File
@@ -0,0 +1,45 @@
---
module: incident.journal-events
title: Journal Events
tab: Internal
order: 20
audience: internal
---
Ops/support view of the **Event Journal** screen (frontend component `JournalEvents.svelte`). It is a read-only, per-ATM event viewer served by **hiveops-incident**. There are no admin write actions in this view — the only mutation path is agent ingestion through hiveops-agent-proxy.
## Who can see it
- Read endpoints under `/api/journal-events/**` fall through to the `/api/**` rule in incident's `SecurityConfig` → `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','QDS_ADMIN','BCOS_ADMIN')`.
- **Not admin-gated.** There is no `@PreAuthorize` on `JournalEventController` — any authenticated browser (`USER`) or JWT user can read journal events. This is intentional (it is a diagnostic view), unlike `/api/incident-management/**` which is `MSP_ADMIN`/`BCOS_ADMIN` only.
- `USER` = HiveIQ Browser session; `CUSTOMER`/`MSP_ADMIN` = customer JWT.
## What the screen actually calls
- Selecting an ATM → `GET /api/journal-events/atm/{atmId}/paginated?page=&size=&sort=eventTime,desc&hoursBack=24&eventTypes=...`
- `{atmId}` is the **numeric DB id** of the `atms` row, not the human `AT000123` agent id.
- Category checkboxes expand client-side into concrete `eventTypes` (e.g. `cardReader` → `CARD_READER_DETECTED`,`CARD_READER_FAIL`,`CARD_READER_RECOVERED`). The backend filters on the exact `eventType` strings passed.
- Default window is `hoursBack=24`; options are 1/6/24/72/168.
## First things to check when it misbehaves
1. **"No events" for an ATM you know is live** — widen Period to 168h and clear category filters first. Category filters are an allowlist of exact event-type strings; a type the agent sends that isn't in one of the 8 category groups (e.g. `ATM_IN_SERVICE`, `CASSETTE_INVENTORY`) will be hidden whenever any category is ticked.
2. **Events stop arriving entirely** — the problem is almost always upstream of this view. Ingestion is `agent → hiveops-agent-proxy (POST /api/journal-events) → incident POST /api/internal/journal-events`. Check agent-proxy logs and confirm `INTERNAL_SERVICE_SECRET` matches between agent-proxy and incident (mismatch → 401/403 on the internal call, agent-proxy returns the error upstream).
3. **Unknown event type "vanishes"** — if `eventType` isn't in incident's `event_type` lookup, the event is **not** stored in `journal_events`; it is recorded as an *event exception* and the ingest returns `202 Accepted`. So a missing event may be a not-yet-registered type, not a lost message. Check the event-exceptions/settings area.
4. **Empty screen with a red error banner** — a 401/403 from the read call means the browser session/JWT lacks one of the six roles above; a 5xx means incident-backend is down. Confirm `GET /actuator/health` on incident (port 8080).
5. **Wrong ATM's data / cross-institution** — see the scoping caveat below.
## Common failure modes + fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| No new events fleet-wide | agent-proxy → incident internal call failing | Check `X-Internal-Secret` / `INTERNAL_SERVICE_SECRET` parity; check agent-proxy logs |
| Sporadic ingest 500s during morning reboot burst | Pessimistic lock contention (incident#212) between journal ingest and heartbeat | Already mitigated: `createEvent` is `@Retryable` (3 attempts, backoff). Persistent failures = investigate DB locks |
| Event type never appears in the table | Type not in `event_type` lookup → stored as exception, 202 | Register the event type, or confirm agent is sending the expected string |
| Times look off | `eventTime` parsed as `LocalDateTime` (no zone); unparseable → `now()` | Confirm agent sends ISO local date-time |
| "Details"/hardware fields empty on expand | Agent didn't populate cardReader*/cassette* fields for that type | Expected for non-hardware events |
## Notes
- Table/DB: `journal_events` in Postgres DB **`hiveiq_incident`** (incident owns it). No separate journal-events database.
- Ingestion also drives ATM service state: `ATM_IN_SERVICE`/`ATM_OUT_OF_SERVICE`/`SUPERVISOR_ENTER`/`SUPERVISOR_EXIT` flip device state as a side effect, and every known event runs auto-incident creation/recovery. So journal events are not purely cosmetic — a stuck ingest also freezes auto-incidents.
backend/src/main/resources/content/incident/processing-rules/architect.md
+102
View File
@@ -0,0 +1,102 @@
---
module: incident.processing-rules
title: Processing Rules
tab: Architect
order: 30
audience: internal
---
> **Architecture view.** How Processing Rules are built and how events flow through them. All rule storage + evaluation is owned by **hiveops-incident**; Adoons suggestions are delegated to **hiveops-ai** over Kafka + a sync proxy. Grounded in code 2026-07-01. See [platform.service-ownership].
## Services involved
| Service | Role in this feature |
|---------|----------------------|
| **hiveops-incident** | Owns rule tables, CRUD controllers, and the evaluation engine (`IncidentAutoCreationService`). Serves the frontend directly. |
| **hiveops-devices** | Produces `hiveops.device.events`; incident's `DeviceEventConsumer` drives auto-creation, which is where opening rules are evaluated. |
| **hiveops-ai (Adoons)** | Generates rule suggestions. Reached two ways: sync proxy (`AdoonsController`) and async via Kafka. |
| **hiveops-agent-proxy** | Agents post journal events here; proxy forwards to incident `/api/internal/journal-events`, feeding `JournalEventService` (which records Event Exceptions + triggers auto-creation/recovery). |
## Data model (all in `hiveiq_incident`, Flyway-managed)
| Table | Entity | Purpose |
|-------|--------|---------|
| `ticket_processing_rules` | `TicketProcessingRule` | Opening rules: `event_type`, `action` (CREATE_INCIDENT/SUPPRESS/OPEN_AND_CLOSE), `override_severity`, `override_incident_type`, `text_pattern`, `institution`, `enabled`, `draft`, `version` |
| `ticket_processing_rule_audit` | `TicketProcessingRuleAudit` | Full change log per rule (History view) |
| `incident_closing_rules` | `IncidentClosingRule` | Closing rules: `trigger_event_type`, `close_incident_types` (CSV), `text_pattern`, `target_status` (RESOLVED/CLOSED), `institution`, `enabled`, `version` |
| `event_exceptions` | `EventException` | Unrecognised event types, aggregated per ATM with occurrence count + first/last seen + sample |
| `rule_suggestions` | `RuleSuggestion` | Async Adoons suggestion rows: `request_id` (UUID), `kind` (PROCESSING/CLOSING), `status` (PENDING/READY/FAILED), suggested_* fields, `institution`, `requested_by` |
See [platform.data-architecture].
## Data flow — opening rules (the core path)
```
device event ──► hiveops.device.events ──► DeviceEventConsumer
(also: agent journal event ──► agent-proxy ──► /api/internal/journal-events ──► JournalEventService)
│
▼
IncidentAutoCreationService
│
findApplicableRule(eventType, institution, details)
│
┌──────────────┬───────────────────────────────┬───────────────────────┐
SUPPRESS OPEN_AND_CLOSE CREATE_INCIDENT no rule → default
(skip create) (create + immediate close, (create incident, (built-in behaviour)
transition OPEN→CLOSED) apply override sev/type)
```
Rule selection precedence inside `findApplicableRule`:
1. Filter enabled rules for the event type (`findEnabledByEventType`).
2. Drop rules whose `text_pattern` (if set) is not `contains`-ed by the lowercased event details.
3. Prefer **institution-scoped** rules (rule institution key → institution name, compared to ATM institution) over **Global** (`institution` null/blank).
4. Within a scope, **text-pattern-specific** rules win over catch-all (`Comparator` on pattern presence).
5. No match → built-in default incident behaviour.
## Data flow — closing rules
`IncidentAutoCreationService.processRecovery(event)` runs on recovery events: applies the built-in `RECOVERY_MAP` first, then `closingRuleService.findEnabledByTriggerEventType(...)`, resolving/closing incidents whose type is in the rule's `close_incident_types` CSV, to the rule's `target_status`.
## Data flow — Event Exceptions
When `JournalEventService` ingests an event whose type isn't recognised, it upserts an `event_exceptions` row (increment `occurrence_count`, update `last_seen`) rather than creating an incident. The screen reads these via `EventExceptionService.findAll()` and admins Dismiss them (single row or by type).
## Adoons rule suggestions — two paths
**Sync proxy** (`AdoonsController`, `/api/adoons/**`): validates user JWT, then calls hiveops-ai with `X-Internal-Secret`.
- `GET /status` → hiveops-ai `/actuator/health`
- `POST /suggest-rule` → hiveops-ai `POST /api/adoons/suggest-rule`
- `POST /suggest-closing-rule` → hiveops-ai `POST /api/adoons/suggest-closing-rule`
**Async / Kafka** (`AdoonsSuggestionController` + `RuleSuggestionAsyncService`, `/api/adoons/rule-suggestions`):
```
POST /api/adoons/rule-suggestions (kind=PROCESSING|CLOSING)
│ persist RuleSuggestion(status=PENDING), return 202 + requestId
▼
RuleSuggestionProducer ──► adoons.rule-suggestion.requested ──► hiveops-ai
│ (generates)
RuleSuggestionResultConsumer ◄── adoons.rule-suggestion.ready ◄─┘
(group hiveops-incident-adoons) → update row status READY/FAILED + suggested_*
▼
Frontend polls GET /api/adoons/rule-suggestions[/{requestId}] (institution-scoped)
```
See [platform.kafka].
## Kafka topics (this feature)
| Topic | Direction | Class | Group |
|-------|-----------|-------|-------|
| `hiveops.device.events` | consume | `DeviceEventConsumer` | `hiveops-incident-device-sync` |
| `adoons.rule-suggestion.requested` | produce | `RuleSuggestionProducer` | — |
| `adoons.rule-suggestion.ready` | consume | `RuleSuggestionResultConsumer` | `hiveops-incident-adoons` |
## Key API endpoints
- Opening rules: `GET/POST /api/ticket-processing-rules`, `/drafts`, `/{id}` (PUT), `/{id}/enabled` (PATCH), `/{id}/approve` (PATCH), `/{id}/audit`, `/audit`, DELETE `/{id}`
- Closing rules: `GET/POST /api/incident-closing-rules`, `/{id}` (PUT), `/{id}/enabled` (PATCH), DELETE `/{id}`
- Event exceptions: `GET /api/event-exceptions`, DELETE `/{id}`, DELETE `/by-type/{eventTypeRaw}`
- Adoons: `/api/adoons/status|suggest-rule|suggest-closing-rule|rule-suggestions`
Related: [platform.data-architecture] · [platform.kafka] · [platform.service-ownership] · [technical.incident]

Some files were not shown because too many files have changed in this diff Show More