[session 1] docs(guide): Kafka recovery runbook + platform.branding module (#32)

2026-07-17 17:21:50 -04:00
parent 6c728bd96e
commit 6361d01e1b
2 changed files with 106 additions and 0 deletions
@@ -0,0 +1,49 @@
+---
+module: platform.branding
+title: Branding, Naming & Palette
+tab: Overview
+order: 10
+audience: dev
+---
+
+# Branding, naming & palette
+
+Rules for any user-facing or customer-facing output — UI copy, release notes, emails, docs,
+one-pagers. Internal code/repos are exempt (they stay `hiveops-*`).
+
+## Naming
+
+| Context | Correct | Never |
+|---------|---------|-------|
+| The product / platform (anything a customer reads) | **HiveIQ** | "HiveOps" (that's the internal repo/codebase name) |
+| The company name | **BCOS** (all caps) | "Bcos", "bcos" |
+| Domains / hostnames | `bcos.cloud`, `api.bcos.cloud` (lowercase) | — |
+| The AI/insights service (`hiveops-ai`) | **Adoons** | "AI", "artificial intelligence", "AI-powered" |
+
+- **HiveIQ** is the external brand; **HiveOps** is the internal repo name — never mix them in prose a human reads.
+- **BCOS** is always uppercase in copy; only the domain strings stay lowercase.
+- **Never use "AI" in user-facing content** (banking/financial-services compliance — customers have
+  strict AI-disclosure policies). Use **"Adoons"** (the product noun), "fleet intelligence" (the
+  capability), "insights" (the output), or a direct behavioural description ("surfaces patterns in
+  journal data", "flags anomalies across ATMs"). Applies to release notes, commit messages, UI copy,
+  docs, and What's New entries.
+
+## Palette
+
+**Source of truth:** `hiveops-website/frontend/src/app.css` `:root` (the www.bcos.cloud marketing
+site). Use these for any brand/customer-facing artifact — **not** the app-template blue (`#2563eb`),
+and never invent an accent.
+
+| Token | Hex |
+|-------|-----|
+| `--bcos-blue` (primary indigo) | **#212c95** · dark #171f6e · light #2d3bb5 · xlight #e8eaf8 |
+| Background ground (deep navy) | **#081a3d** |
+| `--bcos-white` (ivory) | #fefef4 |
+| `--bcos-black` | #090901 |
+| On-dark accent (periwinkle) | #6b84f5 / #8ea2ff / #a5b4fc |
+| Status | green #4ade80 · amber #fbbf24 · red #f87171 |
+
+- Hero gradient: `linear-gradient(150deg, #090901 0%, #181f6b 50%, #212c95 100%)`.
+- **Rule of thumb:** indigo **#212c95** as the accent on **light** surfaces; periwinkle **#8ea2ff**
+  on **dark** surfaces (indigo is too low-contrast on navy).
+- Font: system sans (`-apple-system, "Segoe UI", Roboto, …`).
@@ -0,0 +1,57 @@
+---
+module: platform.kafka
+title: Kafka Topics & Event Flows
+tab: Recovery
+order: 20
+audience: dev
+---
+
+# Single-node KRaft crash-loop recovery
+
+`hiveiq-kafka` on the services VM (`.250`) is a **single broker in KRaft mode**
+(`confluentinc/cp-kafka:7.8.0`, `restart: unless-stopped`). It is a **platform-wide SPOF** — the
+journal parse pipeline, devices, reports, analytics, messaging, and ai all produce/consume through
+it. Tracked in `hiveiq-devops#36`.
+
+## Failure mode: `DUPLICATE_BROKER_REGISTRATION`
+
+Observed 2026-07-13. The broker exited cleanly (ExitCode 0 — **not** OOM, disk was fine) and
+`restart: unless-stopped` auto-restarted it **faster than its broker session expired in the
+controller quorum**. Each restart then hit **`DUPLICATE_BROKER_REGISTRATION`**, timed out, and
+restarted again — a loop (RestartCount reached 44; `Kafka Server started` only completed a handful
+of times).
+
+Downstream symptom: journal per-institution workers log `TimeoutException: Expiring N record(s)`,
+`DisconnectException`, and `JournalTransactionEventProducer: Failed to publish` → the
+transaction/parse pipeline stalls or degrades. Some institutions keep up via producer retries
+during brief up-windows, so dashboards can look simply "frozen" rather than erroring.
+
+## Fix — break the loop
+
+```bash
+docker stop hiveiq-kafka      # unless-stopped → an explicit stop keeps it down
+sleep 45                       # let the stale broker session/registration expire in the quorum
+docker start hiveiq-kafka      # single clean registration
+```
+
+A rapid `docker restart` does **not** work — it re-triggers the duplicate registration. **The 45s
+gap is the key.**
+
+## Verify recovery (don't trust "healthy" alone)
+
+- **CLI tools have no `.sh` suffix** in this image: `/usr/bin/kafka-topics`, `kafka-consumer-groups`, …
+- **Advertised listener is `kafka:29092`** (`KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092`).
+  `--bootstrap-server localhost:9092` fails with "node -1 not available" — a false alarm, not an outage.
+- Good signals:
+  - `kafka-topics --bootstrap-server kafka:29092 --list` returns ~31 topics incl. `journal.parse-requests.*`
+  - `kafka-consumer-groups --bootstrap-server kafka:29092 --describe --group journal-parse-group.<INST>` shows **LAG 0**
+  - journal producer errors in Loki drop to 0 (`{service_name=~"hiveiq-journal.*"} |~ "TimeoutException|DisconnectException|Failed to publish"`)
+- Journal workers: `hiveiq-journal` + 10× `hiveiq-journal-worker-<inst>` (aicu/aocu/atec/bob/cbpa/fscu/gucu/laca/pbnc/secu/unassigned).
+- Quiet institutions legitimately lag (low traffic); **LAG 0 proves not-stuck**, not liveness.
+
+## Open follow-ups (`hiveiq-devops#36`)
+
+- SPOF — no replication factor > 1, no failover.
+- No Kafka-down alerting (this outage was only caught via missing transactions).
+- The `unless-stopped` restart policy perpetuates the loop — needs backoff / an unregistered-state healthcheck.
+- Root trigger of the initial clean exit is still unknown.