---
module: platform.production-checks
title: Production Health Checks
tab: Overview
order: 40
audience: dev
---

> **DRAFT · internal.** Written 2026-07-02. Defines the daily production health sweep that emails a
> report. Verify commands against the live hosts before relying on them.

## What this is

A **daily automated sweep** (cron on the services VM, ~06:30) that runs five checks and emails a
single report to `johannes@bcos.cloud` + `tristan@bcos.cloud` from `hiveiq@bcos.cloud`
(`mail.bcos.cloud:587`). **Sent every day** (silence must never hide a dead cron); the **verdict is
in the subject** — e.g. `HiveIQ Prod Health — 🔴 2 issues (2026-07-02)`.

Script: `/opt/hiveiq/prod-health-report.sh` on `173.231.195.250` (next to `backup-db-swift.sh`).
Section detail for backups + anti-virus lives under **devops.backups** and **devops.security**.

## Report shape

- **Subject** carries the overall verdict (🟢 all clear / 🔴 N issues).
- **Summary block** — one line per section with 🟢 / 🟡 / 🔴.
- **Details** — only non-green items expanded (which service, top errors, backup age, AV table).
- HTML with a plaintext fallback.

## 1. HiveIQ Platform health  (critical set = auth, mgmt, incident, agent-proxy → RED)

**Deep actuator** — parse the `components` block, not just top-level `status`, so a degraded
dependency (DB/Kafka/disk) shows even when the app reports UP. **Active blue/green slot only.**

```bash
# per service (host ports on 173.231.195.250): auth 8001, mgmt 8003, incident 8005, journal 8006,
# remote 8008, ai 8009, messaging 8010, reports 8011, analytics 8012, recon 8013, claims 8014, agent-proxy 8015
curl -fsS localhost:<port>/actuator/health | jq -e '.status=="UP" and (all(.components[]?; .status=="UP"))'
docker compose ps --format '{{.Name}} {{.State}} {{.Health}}'   # flag not-running / unhealthy / restarting
```

- **RED** if any **critical-set** service (auth/mgmt/incident/agent-proxy) is not UP or has a
  non-UP component; **YELLOW** for the rest.
- Confirm the **active** slot is the one nginx serves (`00-backends.conf` map) and it's healthy.
- Edge: NGINX 200 through `api.bcos.cloud` + TLS cert days-to-expiry.

## 2. Loki error check  (last 24h)

Query **Loki** (`localhost:3100` on the services VM — never SSH+cat logs). Count **`level=ERROR`
+ HTTP 5xx**, **highlight stacktraces**, and surface the **top-5 error signatures per service**
(normalise out timestamps/ids so the same error groups). Apply a **suppression list** for
known-benign noise (e.g. transient agent reconnects, AGENT_OFFLINE churn).

```bash
logcli --addr=http://localhost:3100 query --since=24h --quiet \
  'sum by (service_name) (count_over_time({service_name=~".+"} | level="ERROR" [24h]))'
# 5xx from the nginx access stream if present; group + rank signatures; drop suppressed patterns
```

- **YELLOW/RED** when a service crosses the flag threshold (tuned once real volume is known).
- Note: confirm Loki's actual label scheme (`service_name` vs `container`/`job`) on the box.

## 3. Backup check  → **devops.backups**

The nightly `pg_dumpall` → Swift job. Verify:
- `/var/log/hiveiq-backup.log` ends in `=== DB backup done ===` **within the last 24h**
- last dump size sane (**~1.3–1.4 GB**, not near-zero → catches truncated dumps)
- Swift object count ≈ 30 (`swift --insecure list hiveiq-backups --prefix db/` — newest object today)
- **RED** if last run failed or is >24h old. (Restore-test verification tracked in hiveiq-devops#14.)

## 4. Security check  → **devops.security**

Lightweight "are the controls up" across **all prod VMs** (services, database, cdn, proxy, share,
ollama). Per VM:
- **ClamAV** installed, `clamav-freshclam` **active**, signature DB **< 2 days old**
  (`/var/lib/clamav/daily.c[vl]d`), `clamav-daemon` running.
- Report `ufw` / `fail2ban` state (network firewall may be OpenStack security groups — see devops.security).
- **RED** if AV is missing/stale on any in-scope VM — loudest on the **database VM (PCI CDE)**.
  *(As of 2026-07-02 this is RED — remediation in hiveiq-devops#16/#17.)*

## 5. Email delivery

`hiveiq@bcos.cloud` via `mail.bcos.cloud:587` (STARTTLS). Recipients: johannes + tristan (so
Tristan knows the state when Johannes is out). Always send; verdict in subject.

## Related
Gaps found while building this: backups hiveiq-devops#13/#14/#15; security hiveiq-devops#16/#17.
See **devops.backups**, **devops.security**, **platform.gotchas**, **platform.ci-cd**.
