[Guide] coverage-reconcile.py cannot see a new module in an already-documented SPA #25

New Issue

2026-07-16T15:04:26-04:00

hiveiq commented

2026-07-16 15:04:26 -04:00

scripts/coverage-reconcile.py exists so nothing new stays undocumented. It has a blind spot that already cost us one: hiveops-fleet#119 shipped fleet.task-permissions — a live, customer-facing capability — with zero guide coverage, and the reconciler stayed silent. Found by hand; fixed in #23/#24.

Why it missed it

The script checks four things:

#	Check	Catches
1	backend services → `technical.<svc>`	a whole new service
2	agent modules → `agent.<slug>`	a whole new agent module
3	role-tabs on modules already in the guide	a half-documented module
4	product SPAs with no guide content at all	a whole new SPA

Check 4 (line ~82) is all-or-nothing per app:

if app in PRODUCT_APPS and app not in apps and os.path.isdir(...):
    gaps.append(("SPA", ...))

fleet already had 5 modules, so app in apps was true and the check passed. The script never enumerates SPA nav items, so a new module inside an already-documented SPA is invisible. Check 3 only validates modules that are already present — a module that doesn't exist has no tabs to be missing.

Impact

Every documented SPA (fleet, incident, devices, analytics, reports, …) can grow new tabs indefinitely with no coverage signal. This is the common case — new SPAs are rare, new tabs in existing SPAs are routine. The reconciler is strongest exactly where we need it least.

Suggested fix

Derive each SPA's module list from its nav definition and diff against guide keys. In fleet the source of truth is frontend/src/App.svelte:

const fleetTabs = [
  { id: 'tasks', label: 'Tasks' },
  ...
  { id: 'permissions', label: 'Task Permissions', adminOnly: true },   // ← was undocumented
];

Caveats worth designing around:

tab id ≠ guide key. permissions → fleet.task-permissions, patch-windows → fleet.patch-windows. Needs an alias map or a convention, or it will emit false positives immediately.
Nav shapes differ per SPA; a regex over App.svelte may not generalise. A small per-app declaration (nav key → guide key) that CI validates may beat inferring it.
Don't let it exit 1 noisily on day one — land it, see what it finds, then gate.

Alternative, cheaper: have GuidePanel report a pageKey that 404s, so a missing module surfaces at runtime instead of at CI. Complementary, not a replacement — it only fires if someone wires a Guide button.

`scripts/coverage-reconcile.py` exists so nothing new stays undocumented. It has a blind spot that already cost us one: **hiveops-fleet#119 shipped `fleet.task-permissions` — a live, customer-facing capability — with zero guide coverage, and the reconciler stayed silent.** Found by hand; fixed in #23/#24. ### Why it missed it The script checks four things: | # | Check | Catches | |---|---|---| | 1 | backend services → `technical.<svc>` | a whole new service | | 2 | agent modules → `agent.<slug>` | a whole new agent module | | 3 | role-tabs on modules **already in the guide** | a half-documented module | | 4 | product SPAs with **no** guide content at all | a whole new SPA | Check 4 (line ~82) is **all-or-nothing per app**: ```python if app in PRODUCT_APPS and app not in apps and os.path.isdir(...): gaps.append(("SPA", ...)) ``` `fleet` already had 5 modules, so `app in apps` was true and the check passed. **The script never enumerates SPA nav items**, so a *new module inside an already-documented SPA* is invisible. Check 3 only validates modules that are already present — a module that doesn't exist has no tabs to be missing. ### Impact Every documented SPA (fleet, incident, devices, analytics, reports, …) can grow new tabs indefinitely with no coverage signal. This is the common case — new SPAs are rare, new tabs in existing SPAs are routine. The reconciler is strongest exactly where we need it least. ### Suggested fix Derive each SPA's module list from its nav definition and diff against guide keys. In fleet the source of truth is `frontend/src/App.svelte`: ```js const fleetTabs = [ { id: 'tasks', label: 'Tasks' }, ... { id: 'permissions', label: 'Task Permissions', adminOnly: true }, // ← was undocumented ]; ``` Caveats worth designing around: - **tab id ≠ guide key.** `permissions` → `fleet.task-permissions`, `patch-windows` → `fleet.patch-windows`. Needs an alias map or a convention, or it will emit false positives immediately. - Nav shapes differ per SPA; a regex over `App.svelte` may not generalise. A small per-app declaration (nav key → guide key) that CI validates may beat inferring it. - Don't let it exit 1 noisily on day one — land it, see what it finds, then gate. Alternative, cheaper: have `GuidePanel` report a `pageKey` that 404s, so a missing module surfaces at runtime instead of at CI. Complementary, not a replacement — it only fires if someone wires a Guide button.

hiveiq added the Bug

Priority

Medium

labels 2026-07-16 15:04:38 -04:00

hiveiq self-assigned this 2026-07-16 15:04:38 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hiveiq-src/hiveops-guide#25