---
module: platform.data-architecture
title: Data Architecture — Executors, Kafka, Record Stores
tab: Overview
order: 10
audience: internal
---

> **DRAFT · internal.** Written 2026-07-01 to stop the most expensive recurring mistake:
> assuming "where data lives." Verify against code before relying on any specific path.

## The core pattern

HiveIQ services are **single-responsibility and decoupled via Kafka producer/consumer.** The rule
that matters:

1. An **executor service does one job and stays clean** — it does *not* keep long-term history.
2. On **completion or failure**, the executor **produces a Kafka event** with the result.
3. The **owning / record-keeping service consumes** the event and **stores the record**.
4. ⇒ **Query history from the record store; send commands to the executor.**

## Worked example — fleet tasks (the 2026-06-30 bug)

- **Fleet = executor.** It runs download/install/etc. It keeps **no queryable history** — the
  standalone `/fleet/tasks` and `/fleet/tasks/paginated` endpoints are **empty by design** once a
  job finishes. An empty executor endpoint is **NOT** a bug or a stale instance.
- On done/fail, fleet **publishes to Kafka**.
- **Devices consumes and stores** the device-scoped task record, served at
  `GET /api/atms/{id}/fleet-tasks` (`DeviceFleetTask`). **This is the source of truth for a
  device's task history.** (Task History and the SW Deploy tab must read *this*, not the fleet
  executor.)
- The **install action is the exception** — a *command*, sent directly to the fleet executor
  (`fleetAPI.createTask`).

## Kafka vs direct API — judgment, not dogma

- **Direct API** when you need **live/real-time state** (in-flight progress) or a **synchronous
  command** (create a task), or immediate consistency.
- **Kafka** for **durable history, cross-service mirrors, eventual propagation.**
- One feature can legitimately read history from the record store *and* hit a direct API for live
  progress. Match the mechanism to the need.

## Do NOT
- Do not say a service's data "lives in" whichever service happens to *serve* it. Ownership
  (execution) and record-keeping (storage) can be different services.
- Do not treat an empty executor endpoint as missing data.
- When unsure where data lives: check the code / ask, don't assert.