Incident binds bar-filters-active but its CSS is a no-op (#eef2f7 = the default
bar), so it never recolored. Devices/fleet never bound it. The blue #dbeafe was
the template's alone. No SPA sweep needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the template change: the Filters sidebar header no longer tints blue
when filters are active — the count pill + active-filter chips signal it. Aligns
with devices/fleet (which never recolored); the old behaviour was the outlier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gitea scans commits merged to main for closing keywords, so a commit SUBJECT
like `fix(x): ... close ... (#N)` closes #N on release — even when the intent
was only to reference it. This bypasses the Verify Fix -> Tristan gate exactly
like a PR-body `Closes #N`.
Happened for real: hiveops-incident#329 (carried Verify Fix, assigned for QA)
auto-closed when the fix commit landed on main via the 1.1.7 release, because
the subject read `... + close IN_PROGRESS faults back in service (#328)`.
Extends step 10 and the Do NOT list to say the Refs #N rule covers commit
messages, and to keep close/fix/resolve away from any #ref. Cross-refs the
real incidents and hiveiq-ops/hiveiq-devops#58.
Refs hiveiq-ops/hiveiq-devops#58
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hotfixes merged straight to `main` are not in `develop`. Nothing warns you
they have diverged; the bill arrives at the next release, as a conflict on
the prod branch.
hiveops-remote hit this on 2026-07-20: main had 5 commits develop had never
seen (OWASP suppressions, jackson-bom fix, jackson bump, cd-main rework,
sonar-pr removal) against develop's 9 including the hiveops-bom 1.0.2 -> 1.0.8
bumps. Both sides had fixed the SAME jackson/spring/tomcat CVEs by different
routes — main suppressed, develop bumped — so develop -> main conflicted in
pom.xml with main's hotfixes at risk of being clobbered.
Documents the two read-only probes (rev-list count, merge-tree exit code),
the back-merge-first ordering, and why resolving a divergence inside the
release PR is the wrong place for it.
Also records the second-order effect: hotfixes that bypass the develop PR
flow lose their issue paperwork. agent#62, bom#6 and devices#55/#60 all sat
open with a shipped fix, no Verify Fix label and no milestone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Profile had no technical.* module. Documents: frontend-only SPA (hiveiq-profile-frontend
:5184, no backend/DB — composes auth/mgmt/messaging), PROFILE_VERSION sha-pinning +
single-instance recreate, the profile.bcos.cloud->www.bcos.cloud root-redirect red herring,
and that Profile is the sole is_default module (browser landing page). Inbox unread badge
cross-links [profile.inbox] rather than duplicating. Captured from the 2026-07-17 prod deploy.
Records what actually governs atms.institution, after feedback #792 (NC000937
kept reverting from Peoples Bank to Five Star).
- Three writers, one lock: updateLastHeartbeat() (guarded, never overwrites),
syncAtmConfig() (agent-reported key; now skips a locked device and WARNs),
moveInstitution() (operator intent — sets atms.institution_locked).
- The fingerprint for debugging "it changed back": `Assigned institution ...` is
syncAtmConfig(); `Auto-assigned institution ...` is the guarded heartbeat path
and is not the culprit. devices#93 blamed the wrong one for weeks.
- The key a device reports is NOT its authenticated identity: the agent reads it
from `incident.institution.key` in its own hiveops.properties, and agent-proxy's
putIfAbsent keeps it over the token-derived key. Fixing agent.auth.token alone
does not stop a mis-imaged box re-asserting the wrong institution.
- Pre-V19 rows ship unlocked; re-run the move to lock one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hiveops-feedback has been live in production since 2026-07-09 and had no Guide
module; coverage-reconcile.py has been reporting it as MISSING. Coverage is now
green: 127 modules across 19 apps, no missing services/agent modules/SPAs.
Written from source (develop @ ea81eb4), matching the technical.* shape:
single overview.md, tab=Overview, audience=dev, order=10.
Records the things that actually cost time rather than restating the obvious:
- Split DB vars (DB_HOST/DB_PORT/DB_NAME), NOT DB_URL like journal/devices.
- Two auth realms: X-Submit-Key -> ROLE_SUBMITTER for intake; JWT MSP/BCOS for
triage. RELOCATED is terminal and immutable (409 on accept/review).
- Gitea relay: label lookups swallow their exceptions, so a token without
read:organization silently drops org labels (feedback#14).
- Rate limiting trusts X-Forwarded-For blindly and is per-instance Caffeine
(feedback#5). /h2-console/** is permitAll unconditionally (feedback#7).
- The list endpoint has NO institution scoping — every MSP admin sees every
customer's submissions, and QDS log in as MSP_ADMIN (feedback#19, unfixed).
- Accept never worked in prod until 2026-07-16: %2F path -> 405, then an empty
GITEA_TOKEN -> unauthenticated -> Gitea 404 on a private repo -> opaque 500.
- No Kafka, no close-sync with Gitea.
Refs #26
The module described pre-#290 behaviour: rule key resolved to a display NAME,
compared to the ATM's institution name. The code has not worked that way since
2d2dee0 — it compares rule.institution to atm.institutionKey directly,
case-insensitively, and never resolves a name. institutionKeyService is still
injected but the matcher does not call it.
This was not harmless. Investigating incident#241 (a PBNC EMV rule a customer
reported twice), the stale gotcha led to the conclusion that the rule was
silently falling through to global and that #232 was the root cause. Both wrong
— the rule fires correctly in prod (Loki: SUPPRESS 'EMV Fallback Suppression
for Peoples Bank ONLY' on NC000936, institution=PBNC). That wrong conclusion was
posted to a customer-facing ticket and had to be retracted. The Guide exists to
prevent exactly that.
- claude.md — gotcha rewritten to key-to-key; adds a do-not-re-diagnose note
- internal.md — failure-mode row + Matching gotcha section rewritten; keeps the
silent-fallback-to-global warning, which is still real when a
key genuinely mismatches, and marks the name-based version as
historical (#232/#153, fixed 2d2dee0, live 2026-07-16)
- architect.md — evaluation order step 3 corrected
claude.md's "rule institution is stored as the KEY, not a display name" was
already correct and is unchanged.
Refs #29
Cash Stats now reads devices (record of reference) with an incident fallback;
journal's CassetteInventoryService dual-writes incident + journal.cassette-baseline
to devices. Documents the mask-aware RCY parse, the removed duplicate parser, the
new topics, and the nginx /journal/ path quirk.
Refs #156
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The environment cap decided what a Guide instance serves, but defaulted to ALL
four tiers. That makes the safe configuration the one you have to remember: a
missing or misspelled GUIDE_SERVED_AUDIENCES in production would serve the
Architect and Claude tiers to any MSP_ADMIN — and MSP_ADMIN is a customer-facing
role (QDS log in as MSP_ADMIN). One forgotten env line was all it took.
Default is now `customer`. Environments opt UP explicitly:
production -> unset (customer only: modules + Customer tab)
bcos.dev/DLX -> customer,internal,dev,bcos (set in that instance's compose)
bcos.dev's compose was pinned to all four tiers first (hiveiq-openmetal-dev
b2d4e00, already live on .251) so this change cannot narrow it.
Also updates platform.guide-access, which documented the old default and told
readers the opposite of what the code now does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Task Permissions tab shipped in hiveops-fleet#119 and is live, but had no
guide module at all — the API returned 404 for fleet.task-permissions, and
fleet.tasks mentions permissions once without describing the grant model.
This is the setting that decides which commands a customer may push to their own
ATMs: deny-by-default, per-institution, per-command. Highest-consequence fleet
setting we have, least documented.
Adds the 5 canonical role tabs, grounded in hiveops-fleet source:
- Customer — grant/revoke, why only Reboot + Restart Agent are offered
- Internal — triage for "customer cannot reboot", what we never widen
- Architect — deny-by-default rationale, the two independent gates, replace-not-
merge semantics, why GRANTABLE lives in code
- Testing — role matrix + the 400-on-CONFIG_UPDATE case, against bcos.dev
- Claude — ownership, table, endpoints, gotchas
Verified: all 5 tabs parse under MarkdownGuideService.parse(); reconciler counts
126 modules with no missing role-tabs. Role matrix confirmed live on bcos.dev
(CUSTOMER 403/403/200/403, MSP_ADMIN 200/200/200/200).
Refs #23. Follow-up to hiveops-fleet#119, hiveops-fleet#100.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Behaviour changed in hiveops-auth#56 / hiveops-profile#10 (both live on
bcos.dev), so the Guide has to follow.
profile.notifications
- inboxPollSeconds added to the prefs doc, defaults and validation tables
(0-3600, 0 = off, default 60; no migration, absent key reads as default)
- corrects the "flags are stored only, no consumer" claim: it is still true
of email/push/severity/quiet-hours, but inboxPollSeconds now has a real
consumer (App.svelte arms the poll from it). It is the only setting on
that tab that does anything today
- Customer tab documents the new Messages section; testing checklist gains
rows for the interval, Off, badge appearing, and badge clearing
- internal: 400 cause, Off-by-design and silent-poll-failure rows
profile.inbox
- documents the sidebar badge: stores.ts owns unreadCount/refreshUnread,
poll interval is an auth-owned pref, unread-only semantics, #ef4444 vs
the sub-tab's #2563eb, and why it must not reuse Inbox's page-local tally
Refs hiveops-auth#56, hiveops-profile#10
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document that the flow normalises host_result (SUCCESS->approved, FAIL/TIMEOUT
->declined) and surfaces a decline Reason, matching hiveops-transactions #49.
Note the note-breakdown still needs raw_lines and a structured reason field
remain follow-ups on #48.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the msp.institution-keys module (Customer/Internal/Claude/Testing tabs)
and technical.devices to match the new authz:
- Only BCOS_ADMIN can create/delete institution keys; MSP admins edit only and
must submit a request to BCOS to add/remove an institution.
- POST/DELETE /api/institution-keys -> hasRole('BCOS_ADMIN'); PUT stays MSP+BCOS.
- Note the underlying ownership gap (devices mints institutions with no mgmt
mirror; keys have diverged) tracked in msp#29.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'dev' audience tier was gated via isInternal() (MSP_ADMIN || BCOS_ADMIN),
so MSP admins could read developer/architecture content: technical.* and
platform.* Overviews, agent-module internals, dev Internal/Overview tabs.
Remap 'dev' to isBcos() (BCOS_ADMIN only) in visible(). Retag the 66 Testing
docs from 'audience: dev' to 'audience: internal' so MSP keeps them, per
Johannes.
Result: MSP sees Customer + Internal + Testing (66 modules); loses all
Architect/Claude/architecture. BCOS unchanged. Update platform.guide-access
to the new model.
Closes#18
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 162 Architect/Claude docs were tagged 'audience: dev'. The 'dev' tier
resolves to isInternal() = ROLE_MSP_ADMIN || ROLE_BCOS_ADMIN, so every
MSP_ADMIN could read full internal architecture: source paths, DB tables,
service ports, NGINX routes. Verified live on bcos.dev as msp_a@msp1.bcos.dev.
Retag them to 'audience: bcos' (isBcos, BCOS_ADMIN only). Content-only change;
the 'dev' tier mapping is deliberately left alone so the 15 Internal, 35
Overview and 66 Testing docs stay MSP-visible.
Also add platform.guide-access documenting the audience tiers and the
per-role visibility matrix, so the access model is queryable from the Guide
instead of re-derived from source each time.
Verified: MSP module count unchanged (116 across 17 apps); no module loses
all its tabs. Guide is bcos.dev only, not deployed to production.
Closes#16
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Deploy = smoke test" only told you to curl the endpoint. An SPA returns 200 + index.html for any
path, so a curl proves nothing about a frontend feature — which is how dashboard#15 got called
"verified" on the strength of a container healthcheck and a bundle grep.
Documents hiveiq-src/hiveops-e2e and, more importantly, the trap in front of it: these SPAs cannot
be driven by a plain browser. They need X-HiveIQ-Browser: true (nginx's browser gate 302s anything
else to the marketing site) and Authorization: Bearer <jwt> (the SPAs carry no auth code; the
Electron shell injects it).
The headers must be injected at the network layer, not via Playwright's extraHTTPHeaders — those
land in the CORS preflight, nginx rejects X-HiveIQ-Browser there, and every XHR fails. The app then
renders and says "Failed to load...", which reads as a broken feature and is not.
States plainly that coverage is one spec, so nobody assumes the suite has them covered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the Total shown under each ATM name on Cash Stats, across all five
tabs.
The load-bearing facts, which are easy to get wrong:
- The total is derived client-side in CashStatus.svelte. No endpoint, DTO, or
column backs it, and none should be added.
- The asterisk means the total is a floor, not a balance: mix/retract notes
(no denomination) and cassettes with no reported count are excluded. An empty
retract cassette deliberately does not flag the row.
- An ATM with no reported inventory shows a dash, never $0, which would read as
an empty machine.
- The pills must keep wrapping: real fleets run 5+ cassettes, which under the
old nowrap layout overflowed the table and crushed the ATM column.
Also notes that the bcos.dev simulator emits no mix/retract slots, so the
partial-total path never fires there and must not be mistaken for dead code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit (8d7450d) said CI builds production images on merge to main. That is WRONG.
cd-main.yml exists in most repos but is dormant scaffolding: it targets runs-on: production, and
the only runner advertising that label (runner-02) is not an enabled service. ZERO cd-main runs
have ever executed in any repo. Merging to main builds nothing.
Only cd-develop runs. Production is built by hand, on purpose:
develop -> bcos.dev = CI builds + scans. Never hand-build.
main -> production = you build by hand. ./build.sh is CORRECT here.
Consequence now stated explicitly in both modules: the npm-audit + Trivy gate exists ONLY on
develop, so production images are unscanned by construction. That is why 'ship only what has been
verified on bcos.dev' is the entire security argument, not a nicety.
Also corrects the fleet gap to what it actually is — cd-develop never builds hiveiq-fleet — and
drops the devices cd-main 'gap', which was not a gap because that pipeline does not run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
platform.deployment and processes.release both walked the reader through ./build.sh, never
mentioning that cd-develop/cd-main already build and scan. Following either one produces an
UNSCANNED image that races CI for the mutable :dev tag. That is what happened on 2026-07-13:
a full day of hand-built images reached bcos.dev without passing Trivy.
Verified against the workflow YAML and both runners on 2026-07-14:
- cd-develop does NOT deploy for 6 of 8 repos. Only incident and devices deploy to bcos.dev;
everywhere else CI stops at push. The old 'a green run redeploys the service' claim was wrong.
- hiveops-fleet CI builds the FRONTEND ONLY — the fleet backend is built by neither cd-develop
nor cd-main. hiveops-devices cd-main builds only the frontend too. Flagged as defects.
- runner-02 IS active (incident built on it today) — the 'not the active worker' note was wrong.
- CI now pushes an immutable :<sha8> alongside :dev/:latest, so deploys can pin a scanned image.
Also records the runner failure signatures ('lease does not exist', 'network ... not found'),
the daemon-restart + prune runbook, the standing warning never to garbage-collect or delete
/opt/docker-registry, and the fact that merging a batch of PRs at once overwhelms a runner's
4 shared job slots and breaks its Docker daemon.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirrors devops.dev 'Dev Environment (bcos.dev)'. The domain is what people
actually navigate by; OpenMetal is the hosting provider, which is an
implementation detail and is already covered in the body.
Two new BCOS-only modules documenting the two environments and the rule that
separates them:
devops.prod Production Environment (OpenMetal) — hosts, repo layout, health
report + config drift, and the KNOWN DRIFT warning: main's
00-backends.conf names different active blue/green slots than prod
actually serves (devices/transactions/agent-proxy), so deploying it
as-is flips three services to the wrong slot. Do NOT bulk-copy main
over prod.
devops.dev Dev Environment (bcos.dev) — single 16G all-in-one VM at ~93-97%
RAM (recreate JVM services ONE at a time), its own postgres and
registry, the new 06:00 health report (johannes only), and the
health-check gotchas that must not be "fixed" back: SPA frontends
return 200 for /actuator/health so the body must be checked, the
probe must retry, and exited *-green/*-blue are standby slots.
Both audience: bcos.
The rule, now stated in both:
everything bcos.dev -> hiveiq-openmetal-dev
everything prod -> hiveiq-openmetal-prod
shared, env-agnostic -> hiveiq-devops
Also corrects devops.backups and devops.security, which still told readers the
backup/security scripts live in `hiveops-devops/scripts/ops/`. They were moved to
`hiveiq-openmetal-prod/scripts/ops/` today — the Guide is the source of truth, so
a stale path in it is worse than none.
Refs: hiveiq-ops/hiveiq-devops#39, #38
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hiveiq
johannes