From 3c0058d892cb9d56caf0a29d9560140044479db6 Mon Sep 17 00:00:00 2001 From: Johannes Date: Thu, 2 Jul 2026 11:50:30 -0400 Subject: [PATCH] =?UTF-8?q?docs(guide):=20gotcha=20=E2=80=94=20root-owned?= =?UTF-8?q?=20cron=20logs=20silently=20kill=20bcosadmin=20cron=20jobs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The redirect (>> /var/log/x.log) fails before the command runs when the target is root-owned/absent; job dies with zero output. Caused backups #18/#19 + IDS #22 outages. Diagnose via redirect-target owner; fix with chown + logrotate create. Co-Authored-By: Claude Opus 4.8 --- .../resources/content/platform/gotchas/overview.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/backend/src/main/resources/content/platform/gotchas/overview.md b/backend/src/main/resources/content/platform/gotchas/overview.md index 5789fbe..e9dbda4 100644 --- a/backend/src/main/resources/content/platform/gotchas/overview.md +++ b/backend/src/main/resources/content/platform/gotchas/overview.md @@ -23,6 +23,16 @@ audience: dev tiny proxy VM **OOM'd the guest** — it hung mid-`apt` and needed a **hard** reboot (`openstack server reboot --hard`; soft doesn't work on a hung guest). Always `free -m` first; tiny VMs → agentless/remote, not on-host agents. → **devops.security** +- **Root-owned cron logs silently kill `bcosadmin` cron jobs.** Prod cron runs as **`bcosadmin`**, + but several jobs redirect output to a `/var/log/*.log` that's `root:root` (or doesn't exist — and + `bcosadmin` can't create files in root-owned `/var/log`). The shell redirect (`>> file`) then + **fails before the command runs**, so the job dies producing *zero* output — looks like it's not + scheduled when it actually is. Bit us hard: backups (#18/#19) dead for months and the **IDS + auto-block dead ~2.5 months** (#22), all from this. Running the script with `sudo` "works" and + masks it (root can write). **Diagnose:** `crontab -l` shows the job, syslog shows CRON executing + it, but the target log is stale/empty → check the redirect target's owner (`stat -c %U`). + **Fix:** `chown bcosadmin:bcosadmin` the log (create it first if missing); add a logrotate/tmpfiles + rule with `create 0644 bcosadmin bcosadmin` so it survives rotation. → **devops.backups**, **hiveiq-devops#22** - **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once: