feat(ci): detect fleet starvation — the failure mode is silence

[avrabe]

Asked for after today’s outages. The page is the second thing worth having; the alert is the first, because the failure mode is silence.

What went wrong today

The self-hosted fleet degraded three times on 2026-08-06/07: one wedged runner, then 4 of 7 offline, then 6 of 7. Each time CI simply stopped progressing and looked identical to a slow build — no error, no alert, no failed check. One incident burned ~3 hours with a PR sitting at 5/57 checks. Every diagnosis was a human hand-querying the runners API.

Nothing existing could have caught it, because nothing was broken: jobs queue forever against a label with no live runner.

Two signals, deliberately in this order

1 · Starvation — always on, no secret required. Any run queued past 30 minutes. Repo scope only, and it watches the symptom, so it fires for causes we haven’t thought of rather than only the ones we have.

Validated against today: #339 was queued 08:46→11:38 = 172 minutes. This alerts at the 30-minute mark — about 2½ hours before a human noticed.

2 · Fleet inventory — optional. Per-runner online/offline/busy, plus wedge detection (offline AND busy). A runner that drops while holding a job keeps that job assigned forever and no one else can take it; a naive online-count misses it entirely, which is why a required Format check once "ran" 55 minutes then cancelled.

This needs /orgs/{org}/actions/runners, which requires admin:org — the default GITHUB_TOKEN cannot read it. Set ORG_RUNNER_READ_TOKEN to enable. Without the secret the job still runs and reports starvation only: it degrades, it does not fail.

The placement rule

Runs on light, never rust-cpu. A monitor scheduled on the fleet it monitors goes down with that fleet — which is exactly how the outage stayed invisible. This is the part not to "optimise" later.

Output

• status table in the job summary (the "page")
• one rolling issue on alert — comments on the existing one rather than a new issue per tick

Verification

• No ${{ }} inside any run: block — every input bound via env:, dereferenced as a quoted shell variable. Checked with a parser over the file, not by eye.
• YAML parses; runs-on confirmed as [self-hosted, linux, x64, light].
• Detection logic tested live against the real API: found the queued runs, parsed timestamps (GNU and BSD date), computed ages, and correctly did not alert at 4 minutes.

Follow-ups, not in this PR

• If you want the fleet table, add the ORG_RUNNER_READ_TOKEN secret.
• The 30-minute threshold is a first guess; tune once we see a few weeks of real data.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG