platform.deployment and processes.release both walked the reader through ./build.sh, never
mentioning that cd-develop/cd-main already build and scan. Following either one produces an
UNSCANNED image that races CI for the mutable :dev tag. That is what happened on 2026-07-13:
a full day of hand-built images reached bcos.dev without passing Trivy.
Verified against the workflow YAML and both runners on 2026-07-14:
- cd-develop does NOT deploy for 6 of 8 repos. Only incident and devices deploy to bcos.dev;
everywhere else CI stops at push. The old 'a green run redeploys the service' claim was wrong.
- hiveops-fleet CI builds the FRONTEND ONLY — the fleet backend is built by neither cd-develop
nor cd-main. hiveops-devices cd-main builds only the frontend too. Flagged as defects.
- runner-02 IS active (incident built on it today) — the 'not the active worker' note was wrong.
- CI now pushes an immutable :<sha8> alongside :dev/:latest, so deploys can pin a scanned image.
Also records the runner failure signatures ('lease does not exist', 'network ... not found'),
the daemon-restart + prune runbook, the standing warning never to garbage-collect or delete
/opt/docker-registry, and the fact that merging a batch of PRs at once overwhelms a runner's
4 shared job slots and breaks its Docker daemon.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirrors devops.dev 'Dev Environment (bcos.dev)'. The domain is what people
actually navigate by; OpenMetal is the hosting provider, which is an
implementation detail and is already covered in the body.
Two new BCOS-only modules documenting the two environments and the rule that
separates them:
devops.prod Production Environment (OpenMetal) — hosts, repo layout, health
report + config drift, and the KNOWN DRIFT warning: main's
00-backends.conf names different active blue/green slots than prod
actually serves (devices/transactions/agent-proxy), so deploying it
as-is flips three services to the wrong slot. Do NOT bulk-copy main
over prod.
devops.dev Dev Environment (bcos.dev) — single 16G all-in-one VM at ~93-97%
RAM (recreate JVM services ONE at a time), its own postgres and
registry, the new 06:00 health report (johannes only), and the
health-check gotchas that must not be "fixed" back: SPA frontends
return 200 for /actuator/health so the body must be checked, the
probe must retry, and exited *-green/*-blue are standby slots.
Both audience: bcos.
The rule, now stated in both:
everything bcos.dev -> hiveiq-openmetal-dev
everything prod -> hiveiq-openmetal-prod
shared, env-agnostic -> hiveiq-devops
Also corrects devops.backups and devops.security, which still told readers the
backup/security scripts live in `hiveops-devops/scripts/ops/`. They were moved to
`hiveiq-openmetal-prod/scripts/ops/` today — the Guide is the source of truth, so
a stale path in it is worse than none.
Refs: hiveiq-ops/hiveiq-devops#39, #38
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OWASP now fails builds on three HIGH CVEs in transitive deps. bom 1.0.7 takes the upstream patches
(no suppressions):
postgresql 42.7.11 -> 42.7.13 CVE-2026-54291 (CVSS4 8.2) — channelBinding downgrade / MITM
httpcore5 5.3.6 -> 5.4.3 CVE-2026-54399 (7.5), CVE-2026-54428 (7.5) — DoS
httpcore5 is transitive via httpclient5; Apache ships them as a matched pair, so bom 1.0.7 bumps
httpclient5 5.5.2 -> 5.6.2 which pulls the fixed core.
Verified locally: builds and tests pass on 1.0.7.
Refs bom#6
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace Open/All columns with: open issues with no milestone (unscheduled
backlog) and open issues in 2026-07; linked number is the count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 2026-07 page: authoritative counts from org-wide milestone-name search
(38 open across 12 repos; add devices=4, fleet #50, dashboard #51 that the
stale per-repo milestone counters had hidden).
- External http(s) links in rendered guide content now open in a new tab.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New top-level MILESTONE section (bcos-only), peer to PROCESSES, renders
uppercase via .sec-head; first module is milestone.2026-07 (per-repo
release-scope view with Gitea filter links).
- Nav UX: all sidebar sections start collapsed; no section/app auto-expands
on entry; landing pane shows the HiveIQ logo instead of auto-selecting Agent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document how to view a release milestone's issues in Gitea:
- per-repo web UI filter URL pattern (milestone=<id>)
- cross-repo rollup via issues/search by name
- a copy-paste one-liner that regenerates the current-month
per-repo links for any YYYY-MM (no hardcoded ids)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
incident.journal-events (Claude tab): the state flip is a dual-write — incident
writes its own atm_properties, then mirrors to devices, and it is devices the UI
reads. Adds the best-effort/afterCommit semantics and the nginx-listener URL rule.
technical.devices: document POST /api/internal/atms/{agentAtmId}/service-state,
that devices does not ingest journal events, the now()-stamped supervisor_entered_at,
and that the status badge is derived client-side (agent-offline masks IN_SUPERVISOR).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the mgmt portal 5-step New Customer + Institution wizard shipped to
production in hiveops-mgmt#43: the five steps, atomic-submit semantics, the
MSP_ADMIN agent-onboarding handoff message, and the gotchas (uppercased
institution key, unrecallable handoff, skipping step 4 skips the handoff,
MESSAGING_SERVICE_URL required).
Guide had no coverage for hiveops-template, so the copy-don't-rewrite rule
and the TemplatePage-vs-TablePage decision were only in a stale CLAUDE.md.
- decision table for which page to copy; ask the user when not obvious
- assigned dev ports read from the actual vite configs (next free = 5191;
CLAUDE.md still claims 5180, which is adoons)
- canonical .page-right spacing is 0.65rem/0.5rem, not the documented 1rem/0.75rem
- dark-mode filter-header values + collapsed-by-default sidebar
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The smoke step polled /guide/actuator/health for only 60s, but the Spring
Boot backend restart after `compose up -d` needs longer to warm up — so a
successful deploy was marked FAILED (run #6, PR #11). Poll 30x10s (~5 min)
and echo attempts. No change to the deploy itself.
Closes#12
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to hiveops-incident#253 (QDS_ADMIN removed from all 5 backends'
SecurityConfig/InstitutionContext, and from the hiveops-auth Role enum so no
JWT can carry it). The guide KB still quoted QDS_ADMIN in role/allow-list
descriptions, which no longer matched the code.
Removed QDS_ADMIN from 26 references across 23 content files; surviving roles
(USER, CUSTOMER, ADMIN, MSP_ADMIN, BCOS_ADMIN, AGENT, SYSTEM) left unchanged.
No frontmatter touched -> zero module-coverage impact. Doc-accuracy only.
Closes#10
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Notify Slack step could fail the whole job (turning a successful deploy
red, as just happened). A notification must never fail a deploy:
- continue-on-error: true on the step
- curl captures HTTP code and never propagates a non-zero exit (|| echo)
- logs "slack notify HTTP: <code>" so we can see if the runner reaches Slack
- first commit line via ${VAR%%$'\n'*} param-expansion (no head pipe)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reword journal Overview to drop the 'legacy SmartJournal dependency' mention
and remove smartjournal from the coverage-reconcile deprecated skip-list.
qds-monitor stays in the skip-list (deprecated, intentionally undocumented).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Appends a Notify Slack step to cd-develop (if: always()) that posts a
colored Block Kit card on every run — ✅ deployed / 🔴 FAILED — with commit,
branch, actor, run number, and a View-run button. Uses the org secret
SLACK_WEBHOOK_URL; no-ops if the secret is unset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
QDS_ADMIN was removed from the hiveops-auth Role enum (972a28a); no JWT can carry
ROLE_QDS_ADMIN. Strip the stale SecurityConfig hasAnyRole entry. Zero runtime change.
Refs hiveiq-src/hiveops-incident#253
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The guide bundles content into the backend jar at build time, so instances must be rebuilt +
redeployed to pick up doc changes (git-pull is insufficient). Automates it: on push to develop
(backend/** paths), mvn package -> docker build -> push :dev -> deploy to bcos.dev -> smoke.
mvn package runs FIRST (the Dockerfile COPYs a pre-built jar — skipping it ships stale content).
Auto-deploy scoped to dev only (docs, low risk). Audience filtering is runtime via
GUIDE_SERVED_AUDIENCES, so no build-time filtering — prod (future cd-main, gated) serves
customer,internal only.
Needs (coordinate w/ CI/runner session): BCOS_DEV_DEPLOY_KEY secret + runner reachability to .251.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
platform.nginx-routing: replace the stale 'flip map + internal callers in sync' guidance with the
new pattern — internal callers point at hiveiq-nginx:<port> and follow the 00-backends.conf map
(10-internal-backends.conf). Document the targeted-flip gotcha (global sed corrupts the derived
active_color table; bluegreen-deploy.sh has this bug) and 'never recreate the active color'.
platform.deployment: add a prod blue/green cutover section + the 2026-07-03 incident lesson.
Refs hiveiq-ops/hiveiq-devops#27, hiveiq-ops/hiveiq-openmetal-prod#20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the MSP_MANAGED/CUSTOMER_MANAGED incident ownership mode across the
Claude/Internal/Architect/Testing/Customer tabs: the write matrix, the
/institution-keys/{key}/incident-mode endpoint + MSP toggle, controller-layer
assertWritable enforcement, IncidentDTO.institution, V90 migration, and the
known UI follow-up (customer nav still hidden).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Milestones section to processes.gitea-issues: releases are tracked by per-repo
YYYY-MM milestones (not labels), Next = milestone with no due date, Backlog = no
milestone. Note the retired Release/* labels, cross-repo rollup via issues/search, and
assignment API. Cross-link from processes.release for release scoping.
SGs are the firewall of record — public surface 80/443, SSH/Loki restricted to
internal+admin. All VMs key-only (passwordauthentication no) -> brute-force
mitigated, fail2ban optional. Dead SSH-ingress SG (22->world, unattached) to remove.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The redirect (>> /var/log/x.log) fails before the command runs when the target
is root-owned/absent; job dies with zero output. Caused backups #18/#19 + IDS
#22 outages. Diagnose via redirect-target owner; fix with chown + logrotate create.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
devops.security: #16 remediated (freshclam + clamscan, no resident clamd); 5/6
VMs done incl CDE; proxy EXCLUDED (957MB, install OOM d it -> hard reboot).
platform.gotchas: check VM RAM before installing memory-heavy tools.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First pass had a partial view. Backups are 3 layers (Swift dumps + encrypted
off-cluster borg to DLX + Cinder snapshots), not just pg_dumpall — #15 (DR)
resolved, #13/#14 re-scoped. Security has iptables auto-block (security-monitor)
+ AIDE FIM — #17 re-scoped; ClamAV (#16) remains the one real gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The activeteller agent module now emits two connectivity-recovery events
(hiveops-agent#68) that drive incident auto-close. Adds them to the event lists
(internal + claude), bumps the count 9→11, and corrects the SymXchange behaviour
in architect.md: the connection-error streak resets (and IAT_CORE_RECOVERED fires)
on the next message header, not on a fault line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New platform.ci-cd page: runner topology (dlx-hiveiq-runner-01), no-internet
design, workflow set, access, and failure signatures. gotchas: correct the
stale \"CI is disabled\" line and add the AdGuard DNS SPOF (hits app boxes + CI
+ Claude sessions). Grounded in runner logs from the 2026-07-01 outage.
Refs hiveiq-ops/hiveiq-devops#11
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the agent-module coverage gap the reconciler flagged (activeteller + 14).
Grounded per module in hiveops-agent/hiveops-module-*; audience dev (dev-only).
Reconciler now green. Grounding surfaced agent-side bugs (log-error route
mismatch, atec poll-interval hardcoded) — to verify + file next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
scripts/coverage-reconcile.py diffs the code surface (services, agent modules,
SPAs) against documented module keys and reports gaps — catches undocumented
things (found 15 agent modules + qds-monitor). Also relabeled 63 customer tabs
Overview→Customer for role-tab consistency with the fleet pilot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4-tier ladder customer<internal<dev<bcos. Re-tagged Architect/Testing/Claude +
platform/technical from internal->dev; Processes stays bcos. Backend visible()
enforces a per-instance served-audiences cap at the API. bcos.dev/DLX serve all;
production sets GUIDE_SERVED_AUDIENCES=customer,internal so dev/bcos never leave
dev. Verified: capped instance returns only Customer+Internal, 404s dev/bcos.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend FeedbackController + GiteaFeedbackService (POST /api/v1/guide/feedback):
resolves/creates the Bug/Enhancement label, appends reporter (from JWT) + module
context, POSTs the issue via a server-side GITEA_FEEDBACK_TOKEN (reaches Gitea via
public route). Area→repo map mirrors the Browser. Reader gets a 💬 Send feedback
button + dark slide-in form (Bug/Enhancement, area prefilled from current module,
title, description) with success link. Verified e2e (created hiveops-guide #4).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ties in dlx-claude's gitea-notify webhook: closing a Gitea issue DMs the reporter
in their HiveIQ inbox using the Resolution field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New 3-tier gating: customer < internal (MSP/BCOS) < bcos (BCOS_ADMIN only).
GuideController.isBcos + service visible() honour audience: bcos. New 'processes'
app with Develop / Main / Release / Gitea-Issue process docs (audience bcos),
rendered as a Processes nav section only BCOS_ADMIN can see. Verified: BCOS 200,
MSP/customer 404.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every customer module now has the full 5-lens set (Customer/Internal/Architect/
Testing/Claude). Testing = frontend-only what-to-test -> pass/fail checklist vs
bcos.dev sim data (C1/C2/C3 + test logins), audience internal. Authored via
per-module fan-out (2 retried on Sonnet 5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
johannes
hiveiq