hiveiq-src/hiveops-guide
4
0
Fork 0
Code Issues 6 Pull Requests Actions 1 Packages Projects Releases Wiki Activity

docs(guide): gotcha — root-owned cron logs silently kill bcosadmin cron jobs

Browse Source 
The redirect (>> /var/log/x.log) fails before the command runs when the target
is root-owned/absent; job dies with zero output. Caused backups #18/#19 + IDS
#22 outages. Diagnose via redirect-target owner; fix with chown + logrotate create.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Johannes
2026-07-02 11:50:30 -04:00
parent 80e4bbe415
commit 3c0058d892
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/main/resources/content/platform/gotchas/overview.md
+10
View File
@@ -23,6 +23,16 @@ audience: dev
tiny proxy VM **OOM'd the guest** — it hung mid-`apt` and needed a **hard** reboot tiny proxy VM **OOM'd the guest** — it hung mid-`apt` and needed a **hard** reboot
(`openstack server reboot --hard`; soft doesn't work on a hung guest). Always `free -m` first; (`openstack server reboot --hard`; soft doesn't work on a hung guest). Always `free -m` first;
tiny VMs → agentless/remote, not on-host agents. → **devops.security** tiny VMs → agentless/remote, not on-host agents. → **devops.security**
- **Root-owned cron logs silently kill `bcosadmin` cron jobs.** Prod cron runs as **`bcosadmin`**,
but several jobs redirect output to a `/var/log/*.log` that's `root:root` (or doesn't exist — and
`bcosadmin` can't create files in root-owned `/var/log`). The shell redirect (`>> file`) then
**fails before the command runs**, so the job dies producing *zero* output — looks like it's not
scheduled when it actually is. Bit us hard: backups (#18/#19) dead for months and the **IDS
auto-block dead ~2.5 months** (#22), all from this. Running the script with `sudo` "works" and
masks it (root can write). **Diagnose:** `crontab -l` shows the job, syslog shows CRON executing
it, but the target log is stale/empty → check the redirect target's owner (`stat -c %U`).
**Fix:** `chown bcosadmin:bcosadmin` the log (create it first if missing); add a logrotate/tmpfiles
rule with `create 0644 bcosadmin bcosadmin` so it survives rotation. → **devops.backups**, **hiveiq-devops#22**
- **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard - **DNS is a single-point-of-failure.** Both the app boxes and the CI runner use a single AdGuard
resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is resolver (`192.168.200.100`) via the `127.0.0.53` stub, with **no fallback**. When AdGuard is
down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once: down (e.g. DLX Proxmox maintenance), *everything* that resolves a hostname fails at once: