content(guide): platform infra knowledge base — topology, nginx-routing, data-stores, kafka, gotchas (DRAFT, Refs #3)

Cross-cutting internal guides so admins/Claude can query HiveIQ topology,
routing, data ownership and Kafka flows without re-reading code. All
audience: internal, marked DRAFT pending review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

backend/src/main/resources/content/platform/data-stores/overview.md

@@ -0,0 +1,69 @@
+---
+module: platform.data-stores
+title: Data Stores — Postgres DBs & Table Ownership
+tab: Overview
+order: 10
+audience: internal
+---
+
+> **DRAFT · internal.** Written 2026-07-01. Pairs with **platform.data-architecture** (the
+> executor→Kafka→record-store pattern) and **platform.service-ownership**. Verify a table's owner
+> in the owning service's code before writing to it.
+
+## One database per service
+
+All databases run on the **hiveiq-database VM (10.10.10.188)**, user `hiveiq`, reached via
+ProxyJump (see **platform.topology**). Each service owns exactly one database — do not cross-write.
+
+| Database | Owning service |
+|----------|----------------|
+| `hiveiq_config` | hiveops-config |
+| `hiveiq_auth` | hiveops-auth |
+| `hiveiq_mgmt` | hiveops-mgmt (users, institutions, licenses, app_modules, allocations) |
+| `hiveiq_incident` | hiveops-incident |
+| `hiveiq_journal` | hiveops-journal |
+| `hiveiq_remote` | hiveops-remote |
+| `hiveiq_messaging` | hiveops-messaging |
+| `hiveiq_reports` | hiveops-reports |
+| `hiveiq_analytics` | hiveops-analytics |
+| `hiveiq_agent` | hiveops-agent-proxy (agent_connections, machine fingerprints) |
+
+> Newer services (devices, fleet, recon, vault, aria, claims) each own their own DB too — confirm
+> the exact name in that service's `application.yml` (the per-service guides will pin these).
+
+## Ownership rule (non-negotiable)
+
+Every endpoint/table belongs to **exactly one** service:
+- **incident** = incidents · **fleet** = fleet tasks · **devices** = everything device-related ·
+  **mgmt** = users/institutions/licenses.
+
+Never add an endpoint or write to another service's domain. See **platform.service-ownership**.
+
+## device_summary is MIRRORED into 4 services
+
+Device data is mirrored into a `device_summary` table inside **incident, fleet, reports, recon**
+(and now vault) via the Kafka topic **`hiveops.device.events`** (see **platform.kafka**).
+
+- **devices is the source of truth**; it must publish on **every** device change.
+- There is **no reconciler** — if devices fails to publish, all mirrors go stale independently.
+- Symptom of a stale mirror: *"wrong institution / stale device in app X but correct in devices."*
+  That is a missed/failed publish, not a bug in app X.
+
+## Journal specifics
+
+- `hiveops-journal` reads env var **`DB_URL`** (a full JDBC URL) — NOT the split
+  `DB_HOST`/`DB_PORT`/`DB_NAME` other services use.
+- Journal processing runs **per-institution workers** (1 upload + N institution + unassigned
+  containers) with a dedup guard. Always reconcile counts after a journal deploy.
+
+## Institution scoping
+
+Device institution is **auto-derived from the agent token's `X-Institution-Key`**, only when
+blank, and **never overwritten**. A mis-imaged box (wrong token) yields a sticky wrong
+institution — diagnose via `hiveiq_agent.agent_connections` and use device LOCATION to find the
+true owner.
+
+## Do NOT
+- Do not write to a table owned by another service — publish an event or call its API.
+- Do not treat a stale mirror as the owner being wrong — fix the publish path in devices.
+- Do not assume a service's DB name — confirm in its `application.yml`.

backend/src/main/resources/content/platform/gotchas/overview.md

@@ -0,0 +1,64 @@
+---
+module: platform.gotchas
+title: Common Mistakes & Gotchas
+tab: Overview
+order: 10
+audience: internal
+---
+
+> **DRAFT · internal.** Written 2026-07-01. The running list of things that have bitten us, in one
+> queryable place. Each links to the guide with the full story. If a fact here contradicts the
+> code, the code wins — update this.
+
+## Infrastructure
+- **Postgres is on a separate VM** (10.10.10.188), not the services VM. `docker exec
+  hiveiq-postgres` on `.250` always fails. Use ProxyJump. → **platform.topology**
+- **Bare metal is off-limits** — never SSH/deploy to `.229/.230/.231`. → **platform.topology**
+- **Production credentials:** Vaultwarden (pw.bcos.cloud) is the source of truth. When a
+  password/token fails, fetch the current value — never retry stale plaintext from memory.
+- **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod).
+- **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat.
+
+## Auth
+- **Login field is `email`, not `username`** — `POST /auth/api/login` with
+  `{"email": "...", "password": "..."}`. `username` returns 403.
+- **Every `@PreAuthorize` uses `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`** for admin endpoints.
+- **JWT_SECRET is shared** between hiveops-auth and hiveops-mgmt and must be identical.
+- **Hand-rolling a JWT:** use the raw JWT_SECRET string — do **not** base64-decode it.
+- **Route API calls through NGINX** with the service prefix: `https://api.bcos.cloud/incident/api/...`,
+  not bare `/api/...`.
+
+## Routing / NGINX
+- **NGINX owns CORS** — a per-location `add_header` resets inherited CORS headers. → **platform.nginx-routing**
+- **502 on relative `/api/`** after a deploy = stale blue/green hostname in a per-domain conf;
+  should use the `00-backends.conf` map var. → **platform.nginx-routing**
+- **Double-prefix 404s** — frontend + conf both prepend the service segment.
+
+## Data / ownership
+- **Empty executor endpoint ≠ missing data.** Fleet task endpoints are empty by design once a job
+  finishes; history lives in the devices record store. → **platform.data-architecture**
+- **Stale mirror ≠ owner wrong.** `device_summary` is mirrored into 4+ services with no
+  reconciler; fix the publish in devices. → **platform.data-stores**
+- **Service ownership is exclusive** — never add an endpoint/table to the wrong service's domain.
+  → **platform.service-ownership**
+- **Journal uses `DB_URL`** (full JDBC URL), not `DB_HOST/DB_PORT/DB_NAME`.
+- **Fleet tasks** use an `atmIds` array and forward-slash Windows paths.
+- **SW Update tab empty** for QDS/self-update installs is a **product limitation** (no device
+  record is created), not a data bug to investigate.
+
+## Process
+- **Always a Gitea issue before code** — any work, however phrased.
+- **Describe changes before coding**; for a new view/API shape, describe and wait.
+- **Read the Spring controller before writing any frontend `api.*` path.**
+- **Copy `hiveops-template`** (`TablePage.svelte`/`TemplatePage.svelte`) before any new `.svelte`;
+  header = text only, action buttons in `.toolbar-right`.
+- **"Deployed" requires a passing smoke test** — curl the live endpoint with a real JWT for 200 +
+  parseable body. A green container is not a smoke test.
+- **Production = blue/green on the standby slot only**, hotfix→PR main→approval→explicit go.
+- **Never bypass branch protection on `main`.**
+- **CI is disabled** — builds are manual.
+- **Work on `develop`**, never commit directly to `main`.
+
+## Branding
+- External brand is **HiveIQ** (repos stay `hiveops-*`). hiveops-ai is **"Adoons"** externally —
+  never surface "AI".

backend/src/main/resources/content/platform/kafka/overview.md

@@ -0,0 +1,58 @@
+---
+module: platform.kafka
+title: Kafka Topics & Event Flows
+tab: Overview
+order: 10
+audience: internal
+---
+
+> **DRAFT · internal.** Written 2026-07-01. Topic **names** below are grounded in the Java source;
+> exact per-service producer/consumer **edges** are being pinned in the per-service guides — where
+> this says "consumers include," treat it as non-exhaustive until the service guide confirms.
+
+## Why Kafka here
+
+HiveIQ services are single-responsibility and decoupled: an **executor** does a job and produces
+an event; the **record-keeping / mirror** service consumes and stores it. Kafka is for **durable
+history, cross-service mirrors, and eventual propagation** — not for live state or synchronous
+commands (those go over direct API). See **platform.data-architecture**.
+
+## Topic catalog (names verified in code)
+
+| Topic | Producer | Consumers (non-exhaustive) | Payload |
+|-------|----------|----------------------------|---------|
+| `hiveops.device.events` | devices | incident, fleet, reports, recon, vault | device create/update → `device_summary` mirrors |
+| `hiveops.messages` | messaging (`MessageEvent`) | messaging (`MessageConsumer`) | user/broadcast/action messages |
+| `hiveops.reports.completed` | reports | messaging (`ReportCompletedConsumer`) | report finished → notify |
+| `hiveops.aria.threats` | aria | aria | threat/IOC events |
+| `hiveops.aria.sequences` | aria | aria | attack sequence events |
+
+Confirmed consumers found in code: `hiveops-vault` (DeviceEventConsumer),
+`hiveops-messaging` (MessageConsumer, ReportCompletedConsumer),
+`hiveops-aria` (AriaKafkaConfig). The device_summary mirror set (incident/fleet/reports/recon) is
+operationally established — the per-service guides will pin each consumer class.
+
+## Key flows
+
+- **Device mirror fan-out** — devices publishes `hiveops.device.events` on every change; four+
+  services keep a local `device_summary`. **No reconciler** — a missed publish silently staleness
+  one or more mirrors. See **platform.data-stores**.
+- **Fleet task history** — the fleet **executor** publishes task completion; **devices** consumes
+  and stores the device-scoped `DeviceFleetTask` record (source of truth for a device's task
+  history, served at `GET /api/atms/{id}/fleet-tasks`). The fleet executor's own task endpoints
+  are empty by design once a job finishes.
+- **Adoons (AI) async pipeline** — incident → ai via Kafka (journey analysis); a legacy
+  synchronous path still exists. "Adoons" is the external brand for hiveops-ai — never surface
+  "AI".
+- **Report completion** — reports publishes `hiveops.reports.completed`; messaging notifies.
+
+## Serialization
+
+Messaging uses JSON with `spring.json.trusted.packages: com.hiveops.messaging.kafka` and a
+default value type of `com.hiveops.messaging.kafka.MessageEvent`. Cross-service event DTOs must be
+compatible on both ends — a package/class mismatch throws on deserialize.
+
+## Do NOT
+- Do not use Kafka for live/in-flight state or synchronous commands — use a direct API.
+- Do not assume a mirror self-heals — there is no reconciler; fix the producer.
+- Do not invent a topic name — the catalog above is grounded in code; add new ones here.

backend/src/main/resources/content/platform/nginx-routing/overview.md

@@ -0,0 +1,62 @@
+---
+module: platform.nginx-routing
+title: NGINX Routing, CORS & Blue/Green
+tab: Overview
+order: 10
+audience: internal
+---
+
+> **DRAFT · internal.** Written 2026-07-01. NGINX on the services VM (`173.231.195.250`) is the
+> single entry point for every `*.bcos.cloud` API. Most "CORS error" / "502 on /api" reports are
+> NGINX config, not the backend. Verify against the live `api.conf` before asserting a fix.
+
+## NGINX is the CORS authority — not the services
+
+CORS headers are set **once, in NGINX**, via the `cors_map.conf` + `cors_include.conf` pattern.
+The allowed origin is chosen by a `map` on the request `Origin`.
+
+- **Every new API `location` block must `include` the CORS snippet** — a location without it
+  emits no CORS headers and the browser call fails.
+- **A per-location `add_header` RESETS all inherited headers.** If you add any `add_header` inside
+  a location, you must re-add the CORS headers there too, or CORS silently breaks for that path
+  only. This is the #1 cause of "works on endpoint A, CORS-fails on endpoint B, same domain."
+- Backends should **not** also emit `Access-Control-*` — double headers = browser rejects.
+
+## Path prefixing — the double-prefix trap
+
+External API paths are namespaced by service: `https://api.bcos.cloud/<service>/api/...`
+(e.g. `/incident/api/...`, `/journal/api/...`). NGINX strips/rewrites the `<service>` segment
+when proxying to the container.
+
+- Some per-domain confs **prepend** a service prefix. If the frontend also sends the prefix you
+  get a **double prefix** (`/devices/api/devices/api/...`) → 404. Location blocks like
+  `/devices/api/`, `/fleet/api/`, `/transactions/api/` exist in `api.conf` to normalise this.
+- **Journal agent block MUST come before the generic UI block.** Generic path-prepend rules
+  otherwise double-prefix agent paths. Admin journal API external paths **drop** the
+  `/api/journal/` segment — do not add it back.
+- Order matters: **specific API `location`s first, SPA catch-all `location /` LAST.**
+
+## Blue/Green backends
+
+Seven+ services run blue/green. The active upstream is selected by a **`map` variable** in
+`00-backends.conf` (e.g. `set $backend hiveiq-<svc>-blue:PORT`), not a hardcoded hostname.
+
+- To switch slots: deploy to the **standby** color, verify, then flip the map var **and any
+  internal callers** in sync — keep the old slot until verified.
+- **Stale hardcoded hostnames = 502 on relative `/api/`.** Per-domain confs
+  (devices/incident/dashboard/aria) once pinned pre-blue/green hostnames and 502'd; resolved
+  2026-06-30 by moving them to the `map` vars. **If a domain 502s on `/api/` again, recheck this
+  first.**
+
+## Health-check noise
+
+Several backends returned 401/403 to Uptime Kuma because the NGINX catch-all prefix injected a
+path they auth-gate. Fix = an **exact-match `location =` passthrough** in `api.conf` for the
+health path. A 401 on a health probe often means "NGINX reached the backend and it challenged,"
+not "down."
+
+## Do NOT
+- Do not add CORS handling in a backend — NGINX owns it.
+- Do not add an `add_header` in a location without re-adding CORS.
+- Do not hardcode a blue/green hostname in a per-domain conf — use the `00-backends.conf` map var.
+- Do not place an API `location` after `location /`.

backend/src/main/resources/content/platform/topology/overview.md

@@ -0,0 +1,74 @@
+---
+module: platform.topology
+title: Infrastructure Topology — VMs, Access, Deploy Paths
+tab: Overview
+order: 10
+audience: internal
+---
+
+> **DRAFT · internal.** Written 2026-07-01 to stop the most expensive recurring mistake:
+> guessing where a service runs or how to reach the database. Verify a specific IP/path against
+> the live host before destructive actions.
+
+## The two clouds
+
+HiveIQ runs on **OpenMetal (OpenStack)**. Workloads run inside **Nova VMs** — never SSH or
+deploy to the `.229 / .230 / .231` bare-metal hypervisors.
+
+- **hiveiq-*** VMs = the HiveOps platform (services, db, cdn, ollama, dev).
+- **bcos-*** VMs = supporting services (mail, office/Nextcloud, share/Pingvin, proxy/NPM,
+  **pw/Vaultwarden**).
+
+## VM map (production)
+
+| VM | Floating IP | Internal IP | Purpose |
+|----|-------------|-------------|---------|
+| hiveiq-services | **173.231.195.250** | 10.10.10.164 | All microservices + NGINX |
+| hiveiq-database | *(internal only)* | **10.10.10.188** | PostgreSQL + pgAdmin |
+| hiveiq-cdn | **173.231.195.252** | 10.10.10.103 | Browser/agent CDN + Docker registry |
+| hiveiq-ollama | *(internal only)* | 10.10.10.7 | Ollama LLM inference |
+| hiveiq-dev | **173.231.195.251** | 10.10.10.235 | Dev environment (bcos.dev) — all-in-one, isolated |
+| bcos-share | **173.231.195.254** | 10.10.10.170 | Pingvin Share **and Vaultwarden (pw.bcos.cloud, :8200)** |
+| bcos-proxy | **173.231.195.253** | 10.10.10.12 | Nginx Proxy Manager (proxy.bcos.cloud) |
+
+## Access
+
+- **SSH user** `bcosadmin`, key `~/.ssh/id_ed25519_hiveiq`. Always
+  `ssh -i ~/.ssh/id_ed25519_hiveiq bcosadmin@<IP>`.
+- At the office use **floating IPs**; over WireGuard VPN use **internal IPs**.
+
+### The database is NOT on the services VM
+
+`hiveiq-postgres` runs on **10.10.10.188** (database VM), reachable only via ProxyJump through
+the services VM. Running `docker exec hiveiq-postgres …` on `.250` **always fails** —
+"No such container".
+
+```bash
+ssh -i ~/.ssh/id_ed25519_hiveiq -J bcosadmin@173.231.195.250 bcosadmin@10.10.10.188 \
+  "docker exec hiveiq-postgres psql -U hiveiq -d hiveiq_journal -c 'SELECT 1'"
+```
+
+## Deploy paths on the services VM (`173.231.195.250`)
+
+```
+~/hiveiq/hiveops-openmetal/hiveops/instances/services/   # microservices + NGINX compose
+```
+Standard microservice deploy: build/push locally, then on the services VM
+`docker compose pull hiveiq-<svc> && docker compose up -d hiveiq-<svc>`.
+
+- **dev = bcos.dev = 173.231.195.251** (all-in-one). CDN files there live under
+  `downloads/downloads/`.
+- **CDN VM = 173.231.195.252** — browser + agent installers served as static files; also hosts
+  the Docker registry `registry.bcos.cloud`.
+
+## Registries
+
+- **Dev:** `registry.bcos.dev` (tag `:dev`) — dev/bcos.dev only.
+- **Production:** `registry.bcos.cloud` (tag `:latest`/`:<sha>`).
+- **Never mix them.** `192.168.200.200:5000` is removed — never use it.
+
+## Do NOT
+- Do not SSH/deploy to `.229/.230/.231` (bare metal).
+- Do not assume Postgres is on the services VM — it is on 10.10.10.188.
+- Do not trust a plaintext credential from memory for production — **Vaultwarden (pw.bcos.cloud)
+  is the source of truth**; fetch the current value when one fails.