Replaces the 'not in production' note now that 1.3.1 shipped.
Written as a do-NOT-reinvent, because the wrong design is the obvious one and was nearly shipped:
stamping the MSP's own institution key only works when that key is in the MSP's own granted list. True
for QDS in prod, FALSE on bcos.dev (msp1.bcos.dev -> msp1-bcos-dev, not granted, so absent from the
JWT). It depended on a data convention; the reporter's email domain is intrinsic to the row.
Also records: list and getScoped must stay in step via visibleTo(); every id-taking path is scoped;
out-of-tenant reads are 404 not 403; LIKE wildcards escaped because the domain is a JWT claim; the three
fail-closed directions incl. the hiveops-incident fail-open precedent; the backfill endpoint; and the
prod numbers on the day (49/50 attributed, QDS sees 47).
technical.feedback:
- status-counts + capabilities endpoints; status param is repeatable (the board sends a lane)
- new 'Triage board lanes' section with the status->lane mapping and the two rules that must survive
edits: every status belongs to exactly one lane (a status in no lane is an invisible row), and
RELOCATED is WIP not Done (19 of 50 prod rows, so it matters)
- corrected the permitAll list: /h2-console/** is no longer in the shared chain (#7/#42), it lives in
a @Profile(dev) chain and frameOptions is back to DENY. Verified 401 on prod
- gotchas: the XFF rate-limit trap (#5), label-resolution failures now warnings (#14), and the Svelte
$: reactive-timing trap that a build cannot catch
- recorded that tenant scoping (#19) is NOT in prod and why it was held back
incident.incident-management (claude):
- corrected a stale line: mode resolves from the institution KEY, not the name (#290)
- documented #347 as a do-NOT-simplify: assertWritable must use a scalar query, never
incident.getAtm().getInstitutionKey(). Includes why it failed for BCOS_ADMIN only (resolveScope()
returns null for unscoped callers, so verifyIncidentAccess never initialised the proxy), that the
pattern sits at 5 call sites, and that open-in-view comes from Spring Cloud Config so it must not
be relied on.
coverage-reconcile: 132 modules, no gaps.
Prod config drifted unnoticed for 6 days because hiveiq-openmetal-prod had
has_actions=false, so config-drift.yml never executed. Documents the check,
how to run it correctly, what it caught, and the gaps it does not cover.
Documents why AgentTokenValidationService has the shape it does, so the next
person does not simplify it back into the outage.
- three outcomes (VALID/INVALID/UNAVAILABLE) and what is cacheable
- the two caches and the bounded fail-open window
- connect/read timeouts and the circuit breaker, and why each is needed
- the revocation-lag trade-off and both kill switches
- full config table
- how to test it, including the false-pass trap that a reused token creates
A CDN-imported UPDATE_AGENT artifact 410s on the agent's chunk download
(ProcessJarUpdate has no CDN path), so every task fails instantly (agent#90).
Make the fleet-hosted-vs-CDN rule explicit in fleet.artifacts Claude tab and
the agent-release process, and note that a COMPLETED task 404s so rollouts
must be verified via /atms agentVersion, not task status.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents how to behaviourally verify a parser fix on prod (reparse-all by
atmId -> poll parse_status -> check rows), the worker-role recreate order, and
the nginx external-path gotcha (/journal/admin NOT /journal/api/journal/admin).
Notes reconcile after rolling worker recreate. Precedent: #47/#48 verified
2026-07-24.
Session: 3
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both deploy + rollback now parse the upstream from the map line and flip
only that line (address-scoped exact swap); the derived color table is
never touched. Note the failure was silent (standby healthy, traffic stays
on old container).
Session: 3
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Gotchas section explaining that journal institution is resolved once
at parse time from atm_institution_map, that a NULL means the ATM was
unmapped then (agent uploaded before device provisioned), the scope
fail-open consequence, and the scheduled self-heal added in journal #52.
Cross-reference from the Overview routing bullet.
Session: 2
Institution scope is no longer read only from the login-time JWT claim —
devices resolves it fresh from mgmt (source of truth) with fallback, and
mirrors new institutions. Documents the /api/internal/institutions endpoint
and the fresh-scope behaviour. Deployed to prod 2026-07-24.
Session: 3
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
processes.release said cd-main never runs and prod is always built by
hand. That is stale for the wave-1 cd-main rework (auth, mgmt, config,
journal, ai, remote, messaging, agent-proxy, incident), where cd-main
runs on merge to main and builds+pushes hiveiq-<svc>:<sha> (no deploy).
Split the microservice release into reworked (runs-on: ubuntu-latest,
CI builds) vs not-yet-reworked (runs-on: production, build by hand),
and note the Gitea run-status is unreliable so confirm via the registry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Incident binds bar-filters-active but its CSS is a no-op (#eef2f7 = the default
bar), so it never recolored. Devices/fleet never bound it. The blue #dbeafe was
the template's alone. No SPA sweep needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the template change: the Filters sidebar header no longer tints blue
when filters are active — the count pill + active-filter chips signal it. Aligns
with devices/fleet (which never recolored); the old behaviour was the outlier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gitea scans commits merged to main for closing keywords, so a commit SUBJECT
like `fix(x): ... close ... (#N)` closes #N on release — even when the intent
was only to reference it. This bypasses the Verify Fix -> Tristan gate exactly
like a PR-body `Closes #N`.
Happened for real: hiveops-incident#329 (carried Verify Fix, assigned for QA)
auto-closed when the fix commit landed on main via the 1.1.7 release, because
the subject read `... + close IN_PROGRESS faults back in service (#328)`.
Extends step 10 and the Do NOT list to say the Refs #N rule covers commit
messages, and to keep close/fix/resolve away from any #ref. Cross-refs the
real incidents and hiveiq-ops/hiveiq-devops#58.
Refs hiveiq-ops/hiveiq-devops#58
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hotfixes merged straight to `main` are not in `develop`. Nothing warns you
they have diverged; the bill arrives at the next release, as a conflict on
the prod branch.
hiveops-remote hit this on 2026-07-20: main had 5 commits develop had never
seen (OWASP suppressions, jackson-bom fix, jackson bump, cd-main rework,
sonar-pr removal) against develop's 9 including the hiveops-bom 1.0.2 -> 1.0.8
bumps. Both sides had fixed the SAME jackson/spring/tomcat CVEs by different
routes — main suppressed, develop bumped — so develop -> main conflicted in
pom.xml with main's hotfixes at risk of being clobbered.
Documents the two read-only probes (rev-list count, merge-tree exit code),
the back-merge-first ordering, and why resolving a divergence inside the
release PR is the wrong place for it.
Also records the second-order effect: hotfixes that bypass the develop PR
flow lose their issue paperwork. agent#62, bom#6 and devices#55/#60 all sat
open with a shipped fix, no Verify Fix label and no milestone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Profile had no technical.* module. Documents: frontend-only SPA (hiveiq-profile-frontend
:5184, no backend/DB — composes auth/mgmt/messaging), PROFILE_VERSION sha-pinning +
single-instance recreate, the profile.bcos.cloud->www.bcos.cloud root-redirect red herring,
and that Profile is the sole is_default module (browser landing page). Inbox unread badge
cross-links [profile.inbox] rather than duplicating. Captured from the 2026-07-17 prod deploy.
claude1
johannes
hiveiq