---
module: devops.prod
title: Production Environment (bcos.cloud)
tab: Overview
order: 30
audience: bcos
---

> **BCOS only.** Written 2026-07-13 after an audit that found the repo was **not** describing
> production. Read the "Known drift" section before you deploy anything from `main`.

## The rule

**Everything production lives in `hiveiq-ops/hiveiq-openmetal-prod`. Everything bcos.dev lives in
`hiveiq-ops/hiveiq-openmetal-dev`** (see **devops.dev**). Tooling lives in the same repo as the config
it watches, so dev tooling can never be pointed at prod boxes and a script cannot quietly fork into
two copies that drift apart.

`hiveiq-devops` is now **shared, environment-agnostic utilities only**. If what you are adding names a
specific VM, registry, domain, or cron on a specific box, it does not belong there.

This rule exists because it was broken: ops scripts lived in **both** repos and had already drifted —
`openmetal-prod`'s `aide-check.sh` was missing a fix that was already live in production, so deploying
from that repo would have silently reintroduced a bug we had already fixed (**hiveiq-devops#39**).

## Hosts

All workloads run in **Nova VMs** on the OpenStack cluster. The `.229`/`.230`/`.231` addresses are
bare-metal hypervisors — **never SSH or deploy to them.**

| VM | Floating IP | Internal | Purpose |
|----|-------------|----------|---------|
| hiveiq-services | `173.231.195.250` | `10.10.10.164` | all microservices + NGINX |
| hiveiq-database | *(internal only)* | `10.10.10.188` | PostgreSQL — reach via ProxyJump through `.250` |
| hiveiq-cdn | `173.231.195.252` | `10.10.10.103` | browser/agent CDN + Docker registry |
| hiveiq-ollama | *(internal only)* | `10.10.10.7` | LLM inference |
| bcos-share | `173.231.195.254` | `10.10.10.170` | Pingvin Share + Vaultwarden |
| bcos-proxy | `173.231.195.253` | `10.10.10.12` | Nginx Proxy Manager |
| bcos-office | *(internal only)* | `10.10.10.222` | Nextcloud (`office.bcos.cloud`) |
| bcos-mail | `173.231.195.249` | `10.10.10.41` | docker-mailserver |

Registry `registry.bcos.cloud` · domain `*.bcos.cloud` · SSH key `~/.ssh/id_ed25519_hiveiq`, user
`bcosadmin`.

## What lives in the repo

```
hiveiq-openmetal-prod/
├── hiveops/instances/          # compose + nginx per instance (services, cdn, database, bcos-office…)
└── scripts/
    ├── prod-health/            # daily health report -> .250 /opt/hiveiq/
    ├── ops/backup/             # Swift dumps, borgmatic, Cinder snapshots
    ├── ops/security/           # security-monitor, aide-check, clamav-remediate
    ├── ops/office-monitor.sh   # office.bcos.cloud uptime probe
    └── software-registry.py    # weekly PCI/SOC2 third-party inventory (.gitea workflow)
```

Deployed config on `.250` lives at `~/hiveiq/hiveops-openmetal/hiveops/instances/services/`.

## Health & drift

The daily report (`/opt/hiveiq/prod-health-report.sh --send`, cron **06:30**, emails johannes +
tristan) is documented in **platform.production-checks**. Since 2026-07-13 it has a **section 8 —
config drift**, which compares the deployed files against `hiveiq-openmetal-prod@main` and checksums
the patched Nextcloud core file on the office VM.

It needs a **read-only** `GITEA_TOKEN` (`read:repository`) in `/opt/hiveiq/.health-report-env`.
Without it the section reports a yellow "skipped" rather than failing the run. Do **not** put an admin
token there — `.250` is internet-facing.

## ⚠ Known drift — read before deploying from `main`

**The deployed tree on `.250` is not a git checkout.** It is loose files, hand-edited in place, so
`main` being clean proves nothing about what is running. On the drift check's first run (2026-07-13):

**33 of 46 services config files differed from `main`.**

The dangerous one — `main`'s `nginx/conf.d/00-backends.conf` names **different active blue/green slots
than production is actually serving**:

| backend | `main` says | prod actually runs |
|---------|-------------|--------------------|
| `devices_backend` | green | **blue** |
| `transactions_backend` | blue | **green** |
| `agent_proxy_backend` | blue | **green** |

Deploying that file believing the repo is authoritative flips three services to the wrong slot.
**Do not bulk-copy `main` over prod.** The reconcile has to go file by file and decide which side is
correct — and the answer is not always "the repo". Tracked in **hiveiq-devops#39**.

## Why this matters

Two failures in one day came through the same gap, and PRs caught neither, because a PR gates what
enters `main` — not what is running on a box:

- A **core Nextcloud file was patched inside a running container**, existed in no repo, and broke every
  calendar invite for two weeks before anyone diagnosed it (**hiveiq-devops#38**).
- An **abandoned blue/green cutover** sat as uncommitted working-tree edits for ten days — never a
  commit, so no PR ever saw it.

The drift check does not fix this; it makes it **impossible to hide**. The real fix is making the
deployed tree an actual checkout, at which point drift detection collapses into `git status`.

## See also

**devops.dev** · **platform.production-checks** · **platform.deployment** · **platform.nginx-routing**
· **devops.backups** · **devops.security**
