From b102bbabd01e5850d8999d7bf03a2c317c0dc761 Mon Sep 17 00:00:00 2001 From: Johannes Date: Wed, 1 Jul 2026 08:52:53 -0400 Subject: [PATCH] =?UTF-8?q?content(guide):=20platform=20infra=20knowledge?= =?UTF-8?q?=20base=20=E2=80=94=20topology,=20nginx-routing,=20data-stores,?= =?UTF-8?q?=20kafka,=20gotchas=20(DRAFT,=20Refs=20#3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-cutting internal guides so admins/Claude can query HiveIQ topology, routing, data ownership and Kafka flows without re-reading code. All audience: internal, marked DRAFT pending review. Co-Authored-By: Claude Opus 4.8 --- .../content/platform/data-stores/overview.md | 69 +++++++++++++++++ .../content/platform/gotchas/overview.md | 64 ++++++++++++++++ .../content/platform/kafka/overview.md | 58 +++++++++++++++ .../platform/nginx-routing/overview.md | 62 ++++++++++++++++ .../content/platform/topology/overview.md | 74 +++++++++++++++++++ 5 files changed, 327 insertions(+) create mode 100644 backend/src/main/resources/content/platform/data-stores/overview.md create mode 100644 backend/src/main/resources/content/platform/gotchas/overview.md create mode 100644 backend/src/main/resources/content/platform/kafka/overview.md create mode 100644 backend/src/main/resources/content/platform/nginx-routing/overview.md create mode 100644 backend/src/main/resources/content/platform/topology/overview.md diff --git a/backend/src/main/resources/content/platform/data-stores/overview.md b/backend/src/main/resources/content/platform/data-stores/overview.md new file mode 100644 index 0000000..7af7b6a --- /dev/null +++ b/backend/src/main/resources/content/platform/data-stores/overview.md @@ -0,0 +1,69 @@ +--- +module: platform.data-stores +title: Data Stores — Postgres DBs & Table Ownership +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01. Pairs with **platform.data-architecture** (the +> executor→Kafka→record-store pattern) and **platform.service-ownership**. Verify a table's owner +> in the owning service's code before writing to it. + +## One database per service + +All databases run on the **hiveiq-database VM (10.10.10.188)**, user `hiveiq`, reached via +ProxyJump (see **platform.topology**). Each service owns exactly one database — do not cross-write. + +| Database | Owning service | +|----------|----------------| +| `hiveiq_config` | hiveops-config | +| `hiveiq_auth` | hiveops-auth | +| `hiveiq_mgmt` | hiveops-mgmt (users, institutions, licenses, app_modules, allocations) | +| `hiveiq_incident` | hiveops-incident | +| `hiveiq_journal` | hiveops-journal | +| `hiveiq_remote` | hiveops-remote | +| `hiveiq_messaging` | hiveops-messaging | +| `hiveiq_reports` | hiveops-reports | +| `hiveiq_analytics` | hiveops-analytics | +| `hiveiq_agent` | hiveops-agent-proxy (agent_connections, machine fingerprints) | + +> Newer services (devices, fleet, recon, vault, aria, claims) each own their own DB too — confirm +> the exact name in that service's `application.yml` (the per-service guides will pin these). + +## Ownership rule (non-negotiable) + +Every endpoint/table belongs to **exactly one** service: +- **incident** = incidents · **fleet** = fleet tasks · **devices** = everything device-related · + **mgmt** = users/institutions/licenses. + +Never add an endpoint or write to another service's domain. See **platform.service-ownership**. + +## device_summary is MIRRORED into 4 services + +Device data is mirrored into a `device_summary` table inside **incident, fleet, reports, recon** +(and now vault) via the Kafka topic **`hiveops.device.events`** (see **platform.kafka**). + +- **devices is the source of truth**; it must publish on **every** device change. +- There is **no reconciler** — if devices fails to publish, all mirrors go stale independently. +- Symptom of a stale mirror: *"wrong institution / stale device in app X but correct in devices."* + That is a missed/failed publish, not a bug in app X. + +## Journal specifics + +- `hiveops-journal` reads env var **`DB_URL`** (a full JDBC URL) — NOT the split + `DB_HOST`/`DB_PORT`/`DB_NAME` other services use. +- Journal processing runs **per-institution workers** (1 upload + N institution + unassigned + containers) with a dedup guard. Always reconcile counts after a journal deploy. + +## Institution scoping + +Device institution is **auto-derived from the agent token's `X-Institution-Key`**, only when +blank, and **never overwritten**. A mis-imaged box (wrong token) yields a sticky wrong +institution — diagnose via `hiveiq_agent.agent_connections` and use device LOCATION to find the +true owner. + +## Do NOT +- Do not write to a table owned by another service — publish an event or call its API. +- Do not treat a stale mirror as the owner being wrong — fix the publish path in devices. +- Do not assume a service's DB name — confirm in its `application.yml`. diff --git a/backend/src/main/resources/content/platform/gotchas/overview.md b/backend/src/main/resources/content/platform/gotchas/overview.md new file mode 100644 index 0000000..8daef16 --- /dev/null +++ b/backend/src/main/resources/content/platform/gotchas/overview.md @@ -0,0 +1,64 @@ +--- +module: platform.gotchas +title: Common Mistakes & Gotchas +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01. The running list of things that have bitten us, in one +> queryable place. Each links to the guide with the full story. If a fact here contradicts the +> code, the code wins — update this. + +## Infrastructure +- **Postgres is on a separate VM** (10.10.10.188), not the services VM. `docker exec + hiveiq-postgres` on `.250` always fails. Use ProxyJump. → **platform.topology** +- **Bare metal is off-limits** — never SSH/deploy to `.229/.230/.231`. → **platform.topology** +- **Production credentials:** Vaultwarden (pw.bcos.cloud) is the source of truth. When a + password/token fails, fetch the current value — never retry stale plaintext from memory. +- **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod). +- **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat. + +## Auth +- **Login field is `email`, not `username`** — `POST /auth/api/login` with + `{"email": "...", "password": "..."}`. `username` returns 403. +- **Every `@PreAuthorize` uses `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`** for admin endpoints. +- **JWT_SECRET is shared** between hiveops-auth and hiveops-mgmt and must be identical. +- **Hand-rolling a JWT:** use the raw JWT_SECRET string — do **not** base64-decode it. +- **Route API calls through NGINX** with the service prefix: `https://api.bcos.cloud/incident/api/...`, + not bare `/api/...`. + +## Routing / NGINX +- **NGINX owns CORS** — a per-location `add_header` resets inherited CORS headers. → **platform.nginx-routing** +- **502 on relative `/api/`** after a deploy = stale blue/green hostname in a per-domain conf; + should use the `00-backends.conf` map var. → **platform.nginx-routing** +- **Double-prefix 404s** — frontend + conf both prepend the service segment. + +## Data / ownership +- **Empty executor endpoint ≠ missing data.** Fleet task endpoints are empty by design once a job + finishes; history lives in the devices record store. → **platform.data-architecture** +- **Stale mirror ≠ owner wrong.** `device_summary` is mirrored into 4+ services with no + reconciler; fix the publish in devices. → **platform.data-stores** +- **Service ownership is exclusive** — never add an endpoint/table to the wrong service's domain. + → **platform.service-ownership** +- **Journal uses `DB_URL`** (full JDBC URL), not `DB_HOST/DB_PORT/DB_NAME`. +- **Fleet tasks** use an `atmIds` array and forward-slash Windows paths. +- **SW Update tab empty** for QDS/self-update installs is a **product limitation** (no device + record is created), not a data bug to investigate. + +## Process +- **Always a Gitea issue before code** — any work, however phrased. +- **Describe changes before coding**; for a new view/API shape, describe and wait. +- **Read the Spring controller before writing any frontend `api.*` path.** +- **Copy `hiveops-template`** (`TablePage.svelte`/`TemplatePage.svelte`) before any new `.svelte`; + header = text only, action buttons in `.toolbar-right`. +- **"Deployed" requires a passing smoke test** — curl the live endpoint with a real JWT for 200 + + parseable body. A green container is not a smoke test. +- **Production = blue/green on the standby slot only**, hotfix→PR main→approval→explicit go. +- **Never bypass branch protection on `main`.** +- **CI is disabled** — builds are manual. +- **Work on `develop`**, never commit directly to `main`. + +## Branding +- External brand is **HiveIQ** (repos stay `hiveops-*`). hiveops-ai is **"Adoons"** externally — + never surface "AI". diff --git a/backend/src/main/resources/content/platform/kafka/overview.md b/backend/src/main/resources/content/platform/kafka/overview.md new file mode 100644 index 0000000..e2e9795 --- /dev/null +++ b/backend/src/main/resources/content/platform/kafka/overview.md @@ -0,0 +1,58 @@ +--- +module: platform.kafka +title: Kafka Topics & Event Flows +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01. Topic **names** below are grounded in the Java source; +> exact per-service producer/consumer **edges** are being pinned in the per-service guides — where +> this says "consumers include," treat it as non-exhaustive until the service guide confirms. + +## Why Kafka here + +HiveIQ services are single-responsibility and decoupled: an **executor** does a job and produces +an event; the **record-keeping / mirror** service consumes and stores it. Kafka is for **durable +history, cross-service mirrors, and eventual propagation** — not for live state or synchronous +commands (those go over direct API). See **platform.data-architecture**. + +## Topic catalog (names verified in code) + +| Topic | Producer | Consumers (non-exhaustive) | Payload | +|-------|----------|----------------------------|---------| +| `hiveops.device.events` | devices | incident, fleet, reports, recon, vault | device create/update → `device_summary` mirrors | +| `hiveops.messages` | messaging (`MessageEvent`) | messaging (`MessageConsumer`) | user/broadcast/action messages | +| `hiveops.reports.completed` | reports | messaging (`ReportCompletedConsumer`) | report finished → notify | +| `hiveops.aria.threats` | aria | aria | threat/IOC events | +| `hiveops.aria.sequences` | aria | aria | attack sequence events | + +Confirmed consumers found in code: `hiveops-vault` (DeviceEventConsumer), +`hiveops-messaging` (MessageConsumer, ReportCompletedConsumer), +`hiveops-aria` (AriaKafkaConfig). The device_summary mirror set (incident/fleet/reports/recon) is +operationally established — the per-service guides will pin each consumer class. + +## Key flows + +- **Device mirror fan-out** — devices publishes `hiveops.device.events` on every change; four+ + services keep a local `device_summary`. **No reconciler** — a missed publish silently staleness + one or more mirrors. See **platform.data-stores**. +- **Fleet task history** — the fleet **executor** publishes task completion; **devices** consumes + and stores the device-scoped `DeviceFleetTask` record (source of truth for a device's task + history, served at `GET /api/atms/{id}/fleet-tasks`). The fleet executor's own task endpoints + are empty by design once a job finishes. +- **Adoons (AI) async pipeline** — incident → ai via Kafka (journey analysis); a legacy + synchronous path still exists. "Adoons" is the external brand for hiveops-ai — never surface + "AI". +- **Report completion** — reports publishes `hiveops.reports.completed`; messaging notifies. + +## Serialization + +Messaging uses JSON with `spring.json.trusted.packages: com.hiveops.messaging.kafka` and a +default value type of `com.hiveops.messaging.kafka.MessageEvent`. Cross-service event DTOs must be +compatible on both ends — a package/class mismatch throws on deserialize. + +## Do NOT +- Do not use Kafka for live/in-flight state or synchronous commands — use a direct API. +- Do not assume a mirror self-heals — there is no reconciler; fix the producer. +- Do not invent a topic name — the catalog above is grounded in code; add new ones here. diff --git a/backend/src/main/resources/content/platform/nginx-routing/overview.md b/backend/src/main/resources/content/platform/nginx-routing/overview.md new file mode 100644 index 0000000..d317202 --- /dev/null +++ b/backend/src/main/resources/content/platform/nginx-routing/overview.md @@ -0,0 +1,62 @@ +--- +module: platform.nginx-routing +title: NGINX Routing, CORS & Blue/Green +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01. NGINX on the services VM (`173.231.195.250`) is the +> single entry point for every `*.bcos.cloud` API. Most "CORS error" / "502 on /api" reports are +> NGINX config, not the backend. Verify against the live `api.conf` before asserting a fix. + +## NGINX is the CORS authority — not the services + +CORS headers are set **once, in NGINX**, via the `cors_map.conf` + `cors_include.conf` pattern. +The allowed origin is chosen by a `map` on the request `Origin`. + +- **Every new API `location` block must `include` the CORS snippet** — a location without it + emits no CORS headers and the browser call fails. +- **A per-location `add_header` RESETS all inherited headers.** If you add any `add_header` inside + a location, you must re-add the CORS headers there too, or CORS silently breaks for that path + only. This is the #1 cause of "works on endpoint A, CORS-fails on endpoint B, same domain." +- Backends should **not** also emit `Access-Control-*` — double headers = browser rejects. + +## Path prefixing — the double-prefix trap + +External API paths are namespaced by service: `https://api.bcos.cloud//api/...` +(e.g. `/incident/api/...`, `/journal/api/...`). NGINX strips/rewrites the `` segment +when proxying to the container. + +- Some per-domain confs **prepend** a service prefix. If the frontend also sends the prefix you + get a **double prefix** (`/devices/api/devices/api/...`) → 404. Location blocks like + `/devices/api/`, `/fleet/api/`, `/transactions/api/` exist in `api.conf` to normalise this. +- **Journal agent block MUST come before the generic UI block.** Generic path-prepend rules + otherwise double-prefix agent paths. Admin journal API external paths **drop** the + `/api/journal/` segment — do not add it back. +- Order matters: **specific API `location`s first, SPA catch-all `location /` LAST.** + +## Blue/Green backends + +Seven+ services run blue/green. The active upstream is selected by a **`map` variable** in +`00-backends.conf` (e.g. `set $backend hiveiq--blue:PORT`), not a hardcoded hostname. + +- To switch slots: deploy to the **standby** color, verify, then flip the map var **and any + internal callers** in sync — keep the old slot until verified. +- **Stale hardcoded hostnames = 502 on relative `/api/`.** Per-domain confs + (devices/incident/dashboard/aria) once pinned pre-blue/green hostnames and 502'd; resolved + 2026-06-30 by moving them to the `map` vars. **If a domain 502s on `/api/` again, recheck this + first.** + +## Health-check noise + +Several backends returned 401/403 to Uptime Kuma because the NGINX catch-all prefix injected a +path they auth-gate. Fix = an **exact-match `location =` passthrough** in `api.conf` for the +health path. A 401 on a health probe often means "NGINX reached the backend and it challenged," +not "down." + +## Do NOT +- Do not add CORS handling in a backend — NGINX owns it. +- Do not add an `add_header` in a location without re-adding CORS. +- Do not hardcode a blue/green hostname in a per-domain conf — use the `00-backends.conf` map var. +- Do not place an API `location` after `location /`. diff --git a/backend/src/main/resources/content/platform/topology/overview.md b/backend/src/main/resources/content/platform/topology/overview.md new file mode 100644 index 0000000..23b9d92 --- /dev/null +++ b/backend/src/main/resources/content/platform/topology/overview.md @@ -0,0 +1,74 @@ +--- +module: platform.topology +title: Infrastructure Topology — VMs, Access, Deploy Paths +tab: Overview +order: 10 +audience: internal +--- + +> **DRAFT · internal.** Written 2026-07-01 to stop the most expensive recurring mistake: +> guessing where a service runs or how to reach the database. Verify a specific IP/path against +> the live host before destructive actions. + +## The two clouds + +HiveIQ runs on **OpenMetal (OpenStack)**. Workloads run inside **Nova VMs** — never SSH or +deploy to the `.229 / .230 / .231` bare-metal hypervisors. + +- **hiveiq-*** VMs = the HiveOps platform (services, db, cdn, ollama, dev). +- **bcos-*** VMs = supporting services (mail, office/Nextcloud, share/Pingvin, proxy/NPM, + **pw/Vaultwarden**). + +## VM map (production) + +| VM | Floating IP | Internal IP | Purpose | +|----|-------------|-------------|---------| +| hiveiq-services | **173.231.195.250** | 10.10.10.164 | All microservices + NGINX | +| hiveiq-database | *(internal only)* | **10.10.10.188** | PostgreSQL + pgAdmin | +| hiveiq-cdn | **173.231.195.252** | 10.10.10.103 | Browser/agent CDN + Docker registry | +| hiveiq-ollama | *(internal only)* | 10.10.10.7 | Ollama LLM inference | +| hiveiq-dev | **173.231.195.251** | 10.10.10.235 | Dev environment (bcos.dev) — all-in-one, isolated | +| bcos-share | **173.231.195.254** | 10.10.10.170 | Pingvin Share **and Vaultwarden (pw.bcos.cloud, :8200)** | +| bcos-proxy | **173.231.195.253** | 10.10.10.12 | Nginx Proxy Manager (proxy.bcos.cloud) | + +## Access + +- **SSH user** `bcosadmin`, key `~/.ssh/id_ed25519_hiveiq`. Always + `ssh -i ~/.ssh/id_ed25519_hiveiq bcosadmin@`. +- At the office use **floating IPs**; over WireGuard VPN use **internal IPs**. + +### The database is NOT on the services VM + +`hiveiq-postgres` runs on **10.10.10.188** (database VM), reachable only via ProxyJump through +the services VM. Running `docker exec hiveiq-postgres …` on `.250` **always fails** — +"No such container". + +```bash +ssh -i ~/.ssh/id_ed25519_hiveiq -J bcosadmin@173.231.195.250 bcosadmin@10.10.10.188 \ + "docker exec hiveiq-postgres psql -U hiveiq -d hiveiq_journal -c 'SELECT 1'" +``` + +## Deploy paths on the services VM (`173.231.195.250`) + +``` +~/hiveiq/hiveops-openmetal/hiveops/instances/services/ # microservices + NGINX compose +``` +Standard microservice deploy: build/push locally, then on the services VM +`docker compose pull hiveiq- && docker compose up -d hiveiq-`. + +- **dev = bcos.dev = 173.231.195.251** (all-in-one). CDN files there live under + `downloads/downloads/`. +- **CDN VM = 173.231.195.252** — browser + agent installers served as static files; also hosts + the Docker registry `registry.bcos.cloud`. + +## Registries + +- **Dev:** `registry.bcos.dev` (tag `:dev`) — dev/bcos.dev only. +- **Production:** `registry.bcos.cloud` (tag `:latest`/`:`). +- **Never mix them.** `192.168.200.200:5000` is removed — never use it. + +## Do NOT +- Do not SSH/deploy to `.229/.230/.231` (bare metal). +- Do not assume Postgres is on the services VM — it is on 10.10.10.188. +- Do not trust a plaintext credential from memory for production — **Vaultwarden (pw.bcos.cloud) + is the source of truth**; fetch the current value when one fails.