---
module: devops.dev
title: Dev Environment (bcos.dev)
tab: Overview
order: 40
audience: bcos
---

> **BCOS only.** Written 2026-07-13, when bcos.dev had **no health monitoring at all** — empty
> crontab, no `/opt/hiveiq`. It is the environment that is supposed to catch prod problems first.

## The rule

**Everything bcos.dev lives in `hiveiq-ops/hiveiq-openmetal-dev`. Everything production lives in
`hiveiq-ops/hiveiq-openmetal-prod`** (see **devops.prod**). Tooling lives in the same repo as the
config it watches, so dev tooling can never be pointed at prod boxes.

This rule was broken in *both* directions — prod ops scripts sat in the shared `hiveiq-devops` repo,
and `provision-dev-vm.sh` / `setup-dev-environment.sh` sat in the **prod** repo. Both corrected in
**hiveiq-devops#39**.

## The host

`bcos.dev` is a **single all-in-one VM** — fully isolated from production, with its **own** PostgreSQL
and its **own** registry.

| | |
|---|---|
| VM | `hiveiq-dev` — FIP `173.231.195.251`, internal `10.10.10.235` |
| Registry | `registry.bcos.dev` — **never** mix with `registry.bcos.cloud` |
| API | `api.bcos.dev` |
| Deployed config | `~/hiveiq/hiveops-openmetal/hiveops/instances/dev/` |
| Repo | `hiveiq-ops/hiveiq-openmetal-dev` → `hiveops/instances/dev/` |

**Postgres is local to `.251`.** A ProxyJump to `10.10.10.188` reaches **production** — do not reach
for the prod DB pattern out of habit when you are working on dev.

## ⚠ Memory is the real constraint

The whole platform runs on **one 16 GB VM** and sits chronically at **~93–97% RAM**.

**On deploys, recreate JVM services ONE AT A TIME.** Recreating several at once causes an OOM cascade.
This is the single most common way to break bcos.dev.

## Health report

`/opt/hiveiq/dev-health-report.sh --send`, cron **06:00** (30 min ahead of the prod report), log
`/var/log/hiveiq-dev-health-report.log`. Run without `--send` for a dry-run to stdout.
Source: `hiveiq-openmetal-dev/scripts/dev-health/`.

**Emails `johannes@bcos.cloud` only** — dev is engineering-only; the prod report goes wider.

Deliberately **not** a copy of the prod report. Dev has no backups, no ClamAV, no Loki and no certs of
its own; checking for them would be noise.

| check | notes |
|-------|-------|
| Platform | actuator per Spring backend, discovered from the Docker port map |
| Resources | RAM 🔴≥95% / 🟡≥90% · disk 🔴≥90% / 🟡≥85% |
| Container state | restarting/unhealthy = 🔴 |
| Config drift | deployed `compose/*.yml` vs `hiveiq-openmetal-dev@main` + stray `.bak` count |

Needs a **read-only** `GITEA_TOKEN` (`read:repository`) in `/opt/hiveiq/.health-report-env`; absent =
yellow "skipped", never a failed run.

## Gotchas in the health check — do not "fix" these back

- **The SPA frontends return HTTP 200 with `index.html` for *any* path**, including
  `/actuator/health`. A status-code check reports every frontend as healthy **forever**. The probe
  must require `"status":"UP"` **in the body**.
- **The probe retries 3×.** At ~93% RAM a single read blips — `feedback` once showed red while
  answering `{"status":"UP"}` in 14 ms. A report that cries wolf is a report nobody reads.
- **Exited `*-green` / `*-blue` containers are blue/green standby slots, not failures.** Alerting on
  them would fire every single day.
- **Cron runs as `bcosadmin`, not root.** Never test with `sudo` — `$HOME` becomes `/root`, every path
  resolves wrong, and everything reads as missing.

## Known drift (first run, 2026-07-13)

The deployed tree is **not a git checkout** here either. Six compose files differed from `main` —
**exactly the six with `.bak` siblings** (`agent-proxy`, `core`, `guide`, `incident`, `nginx`,
`simulators`), alongside 15 stray `.bak` files. Edited on the box, backed up, never committed.

The first run also caught **`hiveiq-fleet-blue` crash-looping** — nothing had been watching for it.

Tracked in **hiveiq-devops#39**.

## See also

**devops.prod** · **platform.ci-cd** · **platform.deployment**
