[CI] cd-develop run flipped to failure ~30 min AFTER the job succeeds (act_runner v0.2.11 / Gitea status-reconciliation) — NOT the smoke window
#12
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
cd-developsmoke step false-fails after a successful deploySymptom: run #6 (task 1738, sha
11d6369, from PR #11) is marked failure, but the deploy actually succeeded — guide.bcos.dev serves the new content (QDS_ADMINcorrectly removed from all modules) andGET /guide/actuator/healthreturns200 {"status":"UP"}.Cause: the
Smoke — guide backend healthystep polls health for only 60 s (for i in $(seq 1 12)×sleep 5). The step runs immediately afterdocker compose up -drestarts the Spring Boot backend, which usually needs longer than 60 s to warm up and report healthy. The window expires →exit 1→ job fails, even though the deploy is fine. (Runtime was also ~32 min total — a slow runner build/push eats into the budget.)This is the same class of false-failure that #9 fixed for the Slack notify step, one step earlier in the job.
Fix: widen the smoke window to ~5 min (e.g. 30 × 10 s) and echo attempts for observability. No behavioural change to the deploy itself.
Impact: cosmetic today (deploys work), but a permanently-red
cd-developbadge trains people to ignore it and hides real deploy failures.[CI] cd-develop smoke health-check window too short — false 'deploy FAILED' after successful deployto [CI] cd-develop run flipped to `failure` ~30 min AFTER the job succeeds (act_runner v0.2.11 / Gitea status-reconciliation) — NOT the smoke windowCorrection — root cause is NOT the smoke window
Reopening. The smoke-window theory (fixed in PR #13, merged) was a misdiagnosis. Job logs for run #7 (
983037e, job 1771) prove the job actually succeeds — Gitea just mislabels the run ~30 min later.Evidence — run #7, runner
dlx-hiveiq-runner-01(act_runner v0.2.11), UTChealth=502(backend restarting)guide healthy✅🏁 Job succeeded— all steps greenfailure— 31 min after the job finishedJob wall-time: ~65 s, fully green. Same pattern on run #6 (#11): job done fast, run marked failed ~32 min later.
Real cause
Gitea ↔ act_runner status-reconciliation failure: the runner completes and reports success, but Gitea never records the final status and eventually times the run out to
failure(zombie/abandoned-task timeout). act_runner v0.2.11 is old; likely fixed by upgrading the runner and/or tuning Gitea[actions]ZOMBIE_TASK_TIMEOUT/ENDLESS_TASK_TIMEOUT.Notes
200 UP. Only the badge lies.dlx-hiveiq-runner-01, Gitea192.168.200.226) → owned by dlx-claude. Handoff email sent to dlx-claude for runner/Gitea investigation.Root cause found & fixed (dlx-claude, Gitea side / CT226)
Not act_runner — a Gitea MySQL connection-pool bug, fleet-wide (not guide-specific).
Mechanism: Gitea's pool
CONN_MAX_LIFETIME=3houtlived MySQL'swait_timeout=600s. MySQL server-side-killed idle connections after 10 min; Gitea's Go pool kept handing out the dead ones, so writes on them silently failed (connection reset by peer/broken pipe). When the dead write was a task's final-status UPDATE, the task stuck atRunninguntil the 30 mZOMBIE_TASK_TIMEOUTwatchdog force-failed it as stale — exactly our ~31-32 min pattern on runs #6/#7. The same failed write also lost the task's log transfer.Confirmed via Gitea
journalctlfor run #7 (task 1739): act_runner logs clean; MySQLconnection reset by peerat 17:20:11 mid-run;clear_tasks.goforce-failed at 17:45:11 (= pickup +ZOMBIE_TASK_TIMEOUT). Same daily zombie-kill hit claims, mgmt, devices, auth, recon since ~Jul 5.Fix applied (with Johannes's go-ahead):
CONN_MAX_LIFETIME=5min/etc/gitea/app.inion CT226 (below MySQL's 10 minwait_timeout, so the pool recycles proactively). app.ini backed up, gitea restarted, HTTP 200 verified. act_runner v0.2.11 unchanged.Status: monitoring the next cd-develop runs to confirm the false-red is gone. Applying
Verify Fix. PR #13 (wider smoke) stays as a harmless robustness win but was not the cause.✅ Verified — fix confirmed
Re-ran the exact failing run (same commit
983037e) after the CT226CONN_MAX_LIFETIME=5mfix:Same commit, same workflow — the only variable changed is the Gitea MySQL pool lifetime. The run now records its final status in ~1 min instead of hanging until the watchdog kills it. False-red is gone.
Deploys were healthy throughout; this was badge-only. Leaving open for the team to close per process.