johannes
09fb6fc580
Internal (audience:internal) guides capturing the HiveIQ architecture that caused repeated mistakes: data-architecture (executor/Kafka/record-store), service-ownership, deployment topology, auth/JWT, git workflow, module map. For admins + Claude's reference.
2.3 KiB
2.3 KiB
module, title, tab, order, audience
| module | title | tab | order | audience |
|---|---|---|---|---|
| platform.data-architecture | Data Architecture — Executors, Kafka, Record Stores | Overview | 10 | internal |
DRAFT · internal. Written 2026-07-01 to stop the most expensive recurring mistake: assuming "where data lives." Verify against code before relying on any specific path.
The core pattern
HiveIQ services are single-responsibility and decoupled via Kafka producer/consumer. The rule that matters:
- An executor service does one job and stays clean — it does not keep long-term history.
- On completion or failure, the executor produces a Kafka event with the result.
- The owning / record-keeping service consumes the event and stores the record.
- ⇒ Query history from the record store; send commands to the executor.
Worked example — fleet tasks (the 2026-06-30 bug)
- Fleet = executor. It runs download/install/etc. It keeps no queryable history — the
standalone
/fleet/tasksand/fleet/tasks/paginatedendpoints are empty by design once a job finishes. An empty executor endpoint is NOT a bug or a stale instance. - On done/fail, fleet publishes to Kafka.
- Devices consumes and stores the device-scoped task record, served at
GET /api/atms/{id}/fleet-tasks(DeviceFleetTask). This is the source of truth for a device's task history. (Task History and the SW Deploy tab must read this, not the fleet executor.) - The install action is the exception — a command, sent directly to the fleet executor
(
fleetAPI.createTask).
Kafka vs direct API — judgment, not dogma
- Direct API when you need live/real-time state (in-flight progress) or a synchronous command (create a task), or immediate consistency.
- Kafka for durable history, cross-service mirrors, eventual propagation.
- One feature can legitimately read history from the record store and hit a direct API for live progress. Match the mechanism to the need.
Do NOT
- Do not say a service's data "lives in" whichever service happens to serve it. Ownership (execution) and record-keeping (storage) can be different services.
- Do not treat an empty executor endpoint as missing data.
- When unsure where data lives: check the code / ask, don't assert.