---
module: fleet.tasks
title: Fleet Task Workflow
tab: Internal
order: 20
audience: internal
---

Ops / support / admin view of Fleet Task Workflow. Service: **hiveops-fleet** (backend port 8097, DB `hiveiq_fleet`, frontend `fleet.bcos.cloud` / component `FleetManagement/FleetTasks.svelte`). Everything below is grounded in `FleetApiController`, `InternalFleetController`, and `FleetTaskService`.

## Who can do what (roles)

| Action | Required authority |
|--------|-------------------|
| View tasks / history / stats / devices | `hasAnyRole('USER','CUSTOMER','ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
| Create / cancel / retry task | `hasAnyRole('ADMIN','MSP_ADMIN','BCOS_ADMIN')` |
| Upload / import / toggle / delete artifact | `hasAnyRole('ADMIN','MSP_ADMIN','BCOS_ADMIN')` (CDN manifest browse = `MSP_ADMIN`/`BCOS_ADMIN`) |
| See pending-approval queue | `hasAuthority('FLEET_APPROVER') or hasAnyRole('MSP_ADMIN','BCOS_ADMIN')` |
| Approve / reject | `hasAuthority('FLEET_APPROVER') or hasRole('BCOS_ADMIN')` |
| Agent work loop (`/api/internal/fleet/**`) | `hasRole('INTERNAL')` — granted only via `X-Internal-Secret` header (agent-proxy) |

`FLEET_APPROVER` is not a role — it comes from the Boolean `fleetTaskApprover` JWT claim. A plain `MSP_ADMIN` **can create** tasks but **cannot approve** them unless they also carry that claim (or are `BCOS_ADMIN`). Check with `GET /api/fleet/users/me` → `fleetTaskApprover`.

## Lifecycle (what a status means)

Statuses: `PENDING_APPROVAL → PENDING → QUEUED → RUNNING → COMPLETED | FAILED`, plus `CANCELLED` / `REJECTED`.

- **PENDING_APPROVAL** — only `HOTFIX`, `HOTFIX_INSTALL`, `SOFTWARE`, `SOFTWARE_INSTALL` start here. Approve moves them to `PENDING`; reject → `REJECTED` (archived immediately).
- **PENDING** — all other kinds start here (or land here after approval). Waiting for the target device to poll. If `scheduledAt` is set, it stays PENDING until that time passes.
- **QUEUED** — the agent polled `GET /tasks/next` and claimed it (`markQueued`).
- **RUNNING** — agent posted a RUNNING status (download/verify/apply in progress); `startedAt` stamped.
- **COMPLETED** — success; **immediately archived** to `fleet_task_history` and deleted from `fleet_tasks`.
- **FAILED** — stays in `fleet_tasks` (NOT auto-archived) so it can be retried. It is *not* swept by the periodic archiver either.
- **CANCELLED / REJECTED** — terminal, archived to history.

## First things a support engineer checks

1. **"I created a task but nothing ran."** Create silently skips devices. `FleetTaskService.createTask` drops a device (logs a WARN, no error) when:
   - device id not found in `device_summary`, or
   - device `institutionKey` is outside the caller's institution scope, or
   - device is **offline**: `lastHeartbeat` is null or older than **15 minutes**.
   If *every* target is skipped the POST returns **HTTP 207 (MULTI_STATUS)** with an empty array — the UI shows no new task. Grep the fleet backend log (Loki) for `skipping task creation` / `is offline` / `out of scope`.
2. **"Task is stuck in PENDING."** It's a HOTFIX/SOFTWARE family task still in `PENDING_APPROVAL`, or it has a future `scheduledAt` (patch-window install). Check `GET /tasks/{id}` → `status` and `scheduledAt`. Approvals live in the **Approvals** tab.
3. **"Approve/Reject 409 Conflict."** The task is no longer `PENDING_APPROVAL` (already approved, cancelled, or picked up). Refresh the queue.
4. **"Task stuck in QUEUED/RUNNING."** Self-healing: any QUEUED/RUNNING task with no update for **5 minutes** (`STALE_MINUTES`) is re-dispatched — reset to `PENDING` with `downloadOffset=0` — on the next agent poll. If it never recovers, the agent isn't polling (check device heartbeat / agent-proxy).
5. **"Artifact won't attach."** Create rejects a **disabled** artifact with HTTP 400 `Artifact '<name>' is disabled`. Enable it in the Artifacts tab (`toggle-enabled`) first.

## Common failure modes & fixes

| Symptom | Cause | Fix |
|---------|-------|-----|
| POST /tasks returns 207 + empty list | all targets skipped (offline >15 min / out of scope / unknown) | verify device heartbeat & institution; wait for check-in |
| Task never leaves PENDING_APPROVAL | no approver acted | user with `FLEET_APPROVER`/`BCOS_ADMIN` approves in Approvals tab |
| Install waits, `scheduledAt` in future | `HOTFIX_INSTALL`/`SOFTWARE_INSTALL` scheduled into device's patch window | expected — install fires at window; or set explicit `scheduledAt` |
| HOTFIX/SOFTWARE 500 "Script signing failed" | `UPDATE_SCRIPT` and `SCRIPT_SIGNING_PRIVATE_KEY` missing/invalid | fix the base64 PKCS8 RSA key env var |
| FAILED task lingers in active list | FAILED is intentionally not archived | **Retry** (→ PENDING, offset reset) or **Cancel** (→ archived) |
| Dashboard counts look wrong | `getStats` only counts `AUTO_UPDATE`+`PENDING`; completed/failed hardcoded 0 | known limitation — trust the tabs/history, not the stat cards |

## Housekeeping (automatic)

- Every **5 min**: `archiveStaleTerminalTasks` sweeps stray `COMPLETED`/`CANCELLED`/`REJECTED` into history (FAILED excluded).
- Daily **00:00**: `clearOldHistory` deletes `fleet_task_history` rows older than **30 days**.
- History reads default to the last **90 days** when no `createdAfter` is given.
- **Clear history** (DELETE `/tasks/history`) is scoped to the caller's institutions.

## Cancel vs Retry

- **Cancel** — allowed unless already terminal (COMPLETED/FAILED/CANCELLED → 409); sets CANCELLED, archives.
- **Retry** — allowed only on `FAILED` or `CANCELLED` (else 409); resets to PENDING, `downloadOffset=0`, clears timestamps/message. Retrying a CANCELLED task resurrects it from history-free active state.

> **CONFIG_UPDATE caution (ops rule):** before revoking any `agent.auth.token`, push a fleet-wide `CONFIG_UPDATE` to clear it first and let tasks drain — revoking while ATMs still hold the token kills their module threads.
