From 6361d01e1b44ea7f93ec1d05892566ffa1724cc5 Mon Sep 17 00:00:00 2001 From: hiveiq Date: Fri, 17 Jul 2026 17:21:50 -0400 Subject: [PATCH] [session 1] docs(guide): Kafka recovery runbook + platform.branding module (#32) --- .../content/platform/branding/overview.md | 49 ++++++++++++++++ .../content/platform/kafka/recovery.md | 57 +++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 backend/src/main/resources/content/platform/branding/overview.md create mode 100644 backend/src/main/resources/content/platform/kafka/recovery.md diff --git a/backend/src/main/resources/content/platform/branding/overview.md b/backend/src/main/resources/content/platform/branding/overview.md new file mode 100644 index 0000000..2af4ed3 --- /dev/null +++ b/backend/src/main/resources/content/platform/branding/overview.md @@ -0,0 +1,49 @@ +--- +module: platform.branding +title: Branding, Naming & Palette +tab: Overview +order: 10 +audience: dev +--- + +# Branding, naming & palette + +Rules for any user-facing or customer-facing output — UI copy, release notes, emails, docs, +one-pagers. Internal code/repos are exempt (they stay `hiveops-*`). + +## Naming + +| Context | Correct | Never | +|---------|---------|-------| +| The product / platform (anything a customer reads) | **HiveIQ** | "HiveOps" (that's the internal repo/codebase name) | +| The company name | **BCOS** (all caps) | "Bcos", "bcos" | +| Domains / hostnames | `bcos.cloud`, `api.bcos.cloud` (lowercase) | — | +| The AI/insights service (`hiveops-ai`) | **Adoons** | "AI", "artificial intelligence", "AI-powered" | + +- **HiveIQ** is the external brand; **HiveOps** is the internal repo name — never mix them in prose a human reads. +- **BCOS** is always uppercase in copy; only the domain strings stay lowercase. +- **Never use "AI" in user-facing content** (banking/financial-services compliance — customers have + strict AI-disclosure policies). Use **"Adoons"** (the product noun), "fleet intelligence" (the + capability), "insights" (the output), or a direct behavioural description ("surfaces patterns in + journal data", "flags anomalies across ATMs"). Applies to release notes, commit messages, UI copy, + docs, and What's New entries. + +## Palette + +**Source of truth:** `hiveops-website/frontend/src/app.css` `:root` (the www.bcos.cloud marketing +site). Use these for any brand/customer-facing artifact — **not** the app-template blue (`#2563eb`), +and never invent an accent. + +| Token | Hex | +|-------|-----| +| `--bcos-blue` (primary indigo) | **#212c95** · dark #171f6e · light #2d3bb5 · xlight #e8eaf8 | +| Background ground (deep navy) | **#081a3d** | +| `--bcos-white` (ivory) | #fefef4 | +| `--bcos-black` | #090901 | +| On-dark accent (periwinkle) | #6b84f5 / #8ea2ff / #a5b4fc | +| Status | green #4ade80 · amber #fbbf24 · red #f87171 | + +- Hero gradient: `linear-gradient(150deg, #090901 0%, #181f6b 50%, #212c95 100%)`. +- **Rule of thumb:** indigo **#212c95** as the accent on **light** surfaces; periwinkle **#8ea2ff** + on **dark** surfaces (indigo is too low-contrast on navy). +- Font: system sans (`-apple-system, "Segoe UI", Roboto, …`). diff --git a/backend/src/main/resources/content/platform/kafka/recovery.md b/backend/src/main/resources/content/platform/kafka/recovery.md new file mode 100644 index 0000000..6ecfa8c --- /dev/null +++ b/backend/src/main/resources/content/platform/kafka/recovery.md @@ -0,0 +1,57 @@ +--- +module: platform.kafka +title: Kafka Topics & Event Flows +tab: Recovery +order: 20 +audience: dev +--- + +# Single-node KRaft crash-loop recovery + +`hiveiq-kafka` on the services VM (`.250`) is a **single broker in KRaft mode** +(`confluentinc/cp-kafka:7.8.0`, `restart: unless-stopped`). It is a **platform-wide SPOF** — the +journal parse pipeline, devices, reports, analytics, messaging, and ai all produce/consume through +it. Tracked in `hiveiq-devops#36`. + +## Failure mode: `DUPLICATE_BROKER_REGISTRATION` + +Observed 2026-07-13. The broker exited cleanly (ExitCode 0 — **not** OOM, disk was fine) and +`restart: unless-stopped` auto-restarted it **faster than its broker session expired in the +controller quorum**. Each restart then hit **`DUPLICATE_BROKER_REGISTRATION`**, timed out, and +restarted again — a loop (RestartCount reached 44; `Kafka Server started` only completed a handful +of times). + +Downstream symptom: journal per-institution workers log `TimeoutException: Expiring N record(s)`, +`DisconnectException`, and `JournalTransactionEventProducer: Failed to publish` → the +transaction/parse pipeline stalls or degrades. Some institutions keep up via producer retries +during brief up-windows, so dashboards can look simply "frozen" rather than erroring. + +## Fix — break the loop + +```bash +docker stop hiveiq-kafka # unless-stopped → an explicit stop keeps it down +sleep 45 # let the stale broker session/registration expire in the quorum +docker start hiveiq-kafka # single clean registration +``` + +A rapid `docker restart` does **not** work — it re-triggers the duplicate registration. **The 45s +gap is the key.** + +## Verify recovery (don't trust "healthy" alone) + +- **CLI tools have no `.sh` suffix** in this image: `/usr/bin/kafka-topics`, `kafka-consumer-groups`, … +- **Advertised listener is `kafka:29092`** (`KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092`). + `--bootstrap-server localhost:9092` fails with "node -1 not available" — a false alarm, not an outage. +- Good signals: + - `kafka-topics --bootstrap-server kafka:29092 --list` returns ~31 topics incl. `journal.parse-requests.*` + - `kafka-consumer-groups --bootstrap-server kafka:29092 --describe --group journal-parse-group.` shows **LAG 0** + - journal producer errors in Loki drop to 0 (`{service_name=~"hiveiq-journal.*"} |~ "TimeoutException|DisconnectException|Failed to publish"`) +- Journal workers: `hiveiq-journal` + 10× `hiveiq-journal-worker-` (aicu/aocu/atec/bob/cbpa/fscu/gucu/laca/pbnc/secu/unassigned). +- Quiet institutions legitimately lag (low traffic); **LAG 0 proves not-stuck**, not liveness. + +## Open follow-ups (`hiveiq-devops#36`) + +- SPOF — no replication factor > 1, no failover. +- No Kafka-down alerting (this outage was only caught via missing transactions). +- The `unless-stopped` restart policy perpetuates the loop — needs backoff / an unregistered-state healthcheck. +- Root trigger of the initial clean exit is still unknown.