---
module: platform.gotchas
title: Common Mistakes & Gotchas
tab: Overview
order: 10
audience: dev
---

> **DRAFT · internal.** Written 2026-07-01. The running list of things that have bitten us, in one
> queryable place. Each links to the guide with the full story. If a fact here contradicts the
> code, the code wins — update this.

## Infrastructure
- **Postgres is on a separate VM** (10.10.10.188), not the services VM. `docker exec
  hiveiq-postgres` on `.250` always fails. Use ProxyJump. → **platform.topology**
- **Bare metal is off-limits** — never SSH/deploy to `.229/.230/.231`. → **platform.topology**
- **Production credentials:** Vaultwarden (pw.bcos.cloud) is the source of truth. When a
  password/token fails, fetch the current value — never retry stale plaintext from memory.
- **Registries never mix** — `registry.bcos.dev` (dev) vs `registry.bcos.cloud` (prod).
- **Logs:** use **Loki** (`localhost:3100` on the services VM), don't SSH+cat.
- **Check VM RAM before installing memory-heavy tools.** VMs are sized very differently
  (`bcos-proxy` = **957 MB**; CDE = 32 GB). Installing ClamAV (freshclam + a ~2 GB clamscan) on the
  tiny proxy VM **OOM'd the guest** — it hung mid-`apt` and needed a **hard** reboot
  (`openstack server reboot --hard`; soft doesn't work on a hung guest). Always `free -m` first;
  tiny VMs → agentless/remote, not on-host agents. → **devops.security**
- **Root-owned cron logs silently kill `bcosadmin` cron jobs.** Prod cron runs as **`bcosadmin`**,
  but several jobs redirect output to a `/var/log/*.log` that's `root:root` (or doesn't exist — and
  `bcosadmin` can't create files in root-owned `/var/log`). The shell redirect (`>> file`) then
  **fails before the command runs**, so the job dies producing *zero* output — looks like it's not
  scheduled when it actually is. Bit us hard: backups (#18/#19) dead for months and the **IDS
  auto-block dead ~2.5 months** (#22), all from this. Running the script with `sudo` "works" and
  masks it (root can write). **Diagnose:** `crontab -l` shows the job, syslog shows CRON executing
  it, but the target log is stale/empty → check the redirect target's owner (`stat -c %U`).
  **Fix:** `chown bcosadmin:bcosadmin` the log (create it first if missing); add a logrotate/tmpfiles
  rule with `create 0644 bcosadmin bcosadmin` so it survives rotation. → **devops.backups**, **hiveiq-devops#22**
- **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard
  resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is
  down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once:
  Claude Code sessions get `ConnectionRefused` (can't resolve `api.anthropic.com`), app boxes lose
  public DNS, and CI goes red (`failed to fetch task … server misbehaving`). Diagnose:
  `getent hosts example.com` fails = DNS, not the app. AdGuard up but public names return empty =
  its **upstream** is down (don't restart AdGuard). Stopgap: point the box at `.254`/`1.1.1.1`.
  HA fix pending. → **hiveiq-devops#11**, **platform.ci-cd**

## Auth
- **Login field is `email`, not `username`** — `POST /auth/api/login` with
  `{"email": "...", "password": "..."}`. `username` returns 403.
- **Every `@PreAuthorize` uses `hasAnyRole('MSP_ADMIN','BCOS_ADMIN')`** for admin endpoints.
- **JWT_SECRET is shared** between hiveops-auth and hiveops-mgmt and must be identical.
- **Hand-rolling a JWT:** use the raw JWT_SECRET string — do **not** base64-decode it.
- **Route API calls through NGINX** with the service prefix: `https://api.bcos.cloud/incident/api/...`,
  not bare `/api/...`.

## Routing / NGINX
- **NGINX owns CORS** — a per-location `add_header` resets inherited CORS headers. → **platform.nginx-routing**
- **502 on relative `/api/`** after a deploy = stale blue/green hostname in a per-domain conf;
  should use the `00-backends.conf` map var. → **platform.nginx-routing**
- **Double-prefix 404s** — frontend + conf both prepend the service segment.

## Data / ownership
- **Empty executor endpoint ≠ missing data.** Fleet task endpoints are empty by design once a job
  finishes; history lives in the devices record store. → **platform.data-architecture**
- **Stale mirror ≠ owner wrong.** `device_summary` is mirrored into 4+ services with no
  reconciler; fix the publish in devices. → **platform.data-stores**
- **Service ownership is exclusive** — never add an endpoint/table to the wrong service's domain.
  → **platform.service-ownership**
- **Journal uses `DB_URL`** (full JDBC URL), not `DB_HOST/DB_PORT/DB_NAME`.
- **Fleet tasks** use an `atmIds` array and forward-slash Windows paths.
- **SW Update tab empty** for QDS/self-update installs is a **product limitation** (no device
  record is created), not a data bug to investigate.

## Process
- **Always a Gitea issue before code** — any work, however phrased.
- **Describe changes before coding**; for a new view/API shape, describe and wait.
- **Read the Spring controller before writing any frontend `api.*` path.**
- **Copy `hiveops-template`** (`TablePage.svelte`/`TemplatePage.svelte`) before any new `.svelte`;
  header = text only, action buttons in `.toolbar-right`.
- **"Deployed" requires a passing smoke test** — curl the live endpoint with a real JWT for 200 +
  parseable body. A green container is not a smoke test.
- **Production = blue/green on the standby slot only**, hotfix→PR main→approval→explicit go.
- **Never bypass branch protection on `main`.**
- **CI is enabled and runs on every push** (build → Trivy → OWASP → Docker → deploy to bcos.dev on
  `develop`). A red pipeline is usually **infra** (runner DNS / Sonar host), not your build —
  confirm with a local `mvn clean package`. → **platform.ci-cd**
- **Work on `develop`**, never commit directly to `main`.

## Branding
- External brand is **HiveIQ** (repos stay `hiveops-*`). hiveops-ai is **"Adoons"** externally —
  never surface "AI".
