fix(claude-action): name the cause in the exited-non-zero outage annotation - #818
Conversation
…tation
The exited branch bailed on CLAUDE_EXIT != 0 before the VERDICT jq ran, so
the most common failure path (session-limit exit) only ever emitted the
generic "see the session-logs artifact" annotation. Extract the last
assistant text from the stream and fold it into the annotation, so
report-failure.sh enrichment carries a self-diagnosing message
("You've hit your session limit · resets 8:30am (UTC)") into the
tend-outage issue for free.
Closes #816
tend-agent
left a comment
There was a problem hiding this comment.
Self-review — the change is sound. Session-limit exits non-zero and emits the <synthetic> assistant message (confirmed by #816), so the exited-non-zero branch is the right place, the extraction jq matches the shape used by the transcript-render jq earlier in this step, and it falls back to the generic annotation when there's no assistant text. One comment-accuracy nit inline.
| # your session limit · resets 8:30am (UTC)"). Surface that last | ||
| # assistant text so the annotation names the cause (session limit | ||
| # vs auth failure vs server error) instead of a generic "see the | ||
| # artifact". report-failure.sh's enrichment then carries the |
There was a problem hiding this comment.
The enrichment isn't report-failure.sh — that script only appends the run-link row. The nightly enrich-tend-outage-issues.sh is what fetches the failure annotations and posts them into the issue (as #816 itself attributes it). A maintainer chasing the enrichment path later would look in the wrong file.
| # artifact". report-failure.sh's enrichment then carries the | |
| # artifact". The nightly enrich-tend-outage-issues.sh pass then carries the |
There was a problem hiding this comment.
Fixed in 85e590d — the comment now credits the nightly enrich-tend-outage-issues.sh pass. report-failure.sh only records the run-link row; the enrich script fetches the failure annotations and posts them, so that's the path a maintainer would chase. Also corrected the same misattribution in the PR body.
The comment credited report-failure.sh with carrying the annotation into the tend-outage issue, but that script only records the run link. The nightly enrich-tend-outage-issues.sh pass fetches the failure annotations and posts them. Point the comment at the right file per self-review.
|
Real-stream validation from a production outage, in case it helps prioritize this: tend's own account hit its weekly limit at ~09:41Z on 2026-08-04 and stayed exhausted until the 00:00Z UTC reset. Every I pulled the The stream is five events ( Cost of not having it: #831 took 77+ comments over those ~15 hours, every one a bare one-row table with no cause. Working out that this was a quota reset rather than an auth break or a server error needed an artifact download and jq — for a condition that self-resolves at a known clock time and needs no code change at all. One optional belt-and-braces note if you're touching the block anyway: the trailing |
|
Fresh evidence from the 2026-08-04/05 window, since this has been open a couple of days: the failure mode this fixes recurred at 15× the scale of the incident that prompted it — 15 consecutive The text this PR would have surfaced was sitting in Two details worth noting for the implementation here:
Practical cost of not having this: outage tracker #831 accumulated 78 rows with no cause on any of them. The nightly enrichment pass ran and worked correctly — it just faithfully copied the uninformative annotation into every row. Diagnosing it meant downloading artifacts and writing jq by hand, for the second time in three days (#808 was the first). Not proposing any change to the diff — it already does the right thing. Just recording that the evidence bar is now well past one occurrence. #845 separately cuts the cadence that drives the quota exhaustion; this makes the wall legible when it's hit anyway. |
…#823) ## Problem The `Trigger` column of a `tend-outage` row is the only pointer back to the work a failed run stranded. It goes blank for the one trigger where that pointer matters most, and prints `#null` when a field is missing. **`repository_dispatch` is unhandled.** `tend-mention` relays review events through a secretless job that re-posts them as a `repository_dispatch`, so the handle job runs on that event and the PR number arrives as `client_payload.pr` rather than in a `pull_request` object. The `if`/`elif` chain has no branch for it, so every failure on the relay path records `Trigger: N/A` — and a relayed review is exactly the case a maintainer can't recover from the run alone, since `tend-review` fires only on `pull_request_target` and never retries. This path is in constant use: `gh api "repos/max-sixty/tend/actions/runs?event=repository_dispatch"` returns a steady stream of `tend-mention` runs. **`workflow_run` names no run.** The ci-fix path hardcodes `REF="CI fix for workflow run"`, discarding `workflow_run.id` — the id of the CI failure the job was dispatched to fix. **Missing fields render as `null`.** `jq -r '.issue.number'` prints the literal string `null` when the field is absent, so the cell reads `#null` rather than falling back to `N/A`. ## Solution Add a `repository_dispatch` branch reading `client_payload.pr`, link the upstream run id in the `workflow_run` branch, and give every extraction `// empty` plus a `${VAR:+…}` guard so an absent field leaves the cell blank (rendered as `N/A` by the existing `${REF:-N/A}`) instead of `#null`. ## Testing `shared/steps/` has no shell test harness — shellcheck via pre-commit is the only automated gate, and it passes on the changed file. So the block was exercised directly: the REF logic was sliced out of the script (between the `# Build a one-line reference` comment and `TIMESTAMP=`) and sourced under crafted `GITHUB_EVENT_PATH` payloads, before and after. Before, on `origin/main`: ``` repository_dispatch -> N/A workflow_run -> CI fix for workflow run issues (no number) -> #null ``` After: ``` relayed review -> #815 relay w/o pr -> N/A ci-fix -> CI fix for [run 30795510450](https://github.com/max-sixty/tend/actions/runs/30795510450) ci-fix w/o id -> CI fix for workflow run PR event -> #821 PR event w/o number -> N/A issue comment -> #808 issue w/o number -> N/A schedule -> N/A ``` Adding a real harness for `shared/steps/` is worth considering separately — three of the recent outage-path fixes have all landed in scripts nothing can test — but that is a bigger change than this fix warrants, so it is not bundled here. ## Scope Separate from the other two open changes on this path, and textually disjoint from both. #818 names the *cause* of a failure (in `claude/action.yaml`); #809 dedups *rows* across matrix legs (in the `EXISTING` branch of this same script, lines 49+). This one fixes what the row *points at*, in the REF block at lines 19–29. Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
…a silent survey (#885) This hour's Step 2 survey subagent reported **"No visible bot output in time window"** across 23 successful runs on `max-sixty/tend`. The window in fact contained 9 reviews (two `APPROVED`), 11 inline review comments, and 1 conversation comment from `tend-agent` — one of the busiest windows in the series. ## Evidence Window 2026-08-07T07:27Z → 08:16Z. The survey's verdict, verbatim: > **No visible bot output in time window.** All 23 successful runs (tend-mention, tend-review, tend-review-runs, tend-notifications) executed and completed but produced zero comments, reviews, or inline code comments between 2026-08-07T07:27:00Z and 08:16:00Z on tracked PRs/issues (809, 818, 834, 858, 863, 868, 875, 877, 878, 881, 816, 830). Two run-independent calls, taking no run ID at all, contradict it immediately: ``` gh api "repos/max-sixty/tend/issues/comments?since=2026-08-07T07:27:00Z&per_page=100" → 1 tend-agent row gh api "repos/max-sixty/tend/pulls/comments?since=2026-08-07T07:27:00Z&per_page=100" → 11 tend-agent rows ``` Plus reviews `4880965990` (COMMENTED), `4881043005` (COMMENTED) and `4881079314` (APPROVED) on [#881](#881), `4881113428` (APPROVED) on [#878](#878), and five more (all COMMENTED) on [#809](#809). The survey had listed 881, 878 and 809 among the PRs it checked, so the numbers were right and the reads came back empty anyway. The second-order cost is worse than the omission. Having established silence, the survey reasoned *from* it: it flagged the window's seven bot-PR merges as "merged by max-sixty without formal review workflow", concluding "**direct push/merge bypassing review requirement**, or **review workflow override via branch protection rule bypass**". Six of the seven (#818, #834, #858, #868, #875, #877) carry bot `COMMENTED` reviews predating the window, which is why a `since`-filtered read missed them; the seventh (#863) is the skill-authorized silence on a self-authored PR with no concerns. None reads `APPROVED` because GitHub blocks self-approval — the ordinary shape for a bot PR, not a bypassed control. Acted on, that summary is a false security finding against the maintainer. ## Root cause Every path Step 2 offers is run-keyed: run → `headBranch` → PR → endpoint. That chain is fine when it works, but it has one failure mode with no floor — break it anywhere and *every* run returns empty simultaneously. Uniform absence is exactly what a genuinely quiet hour looks like, so the summary that comes back is self-consistent and carries no signal that anything went wrong. The existing sanity-check line ("note if zero bot activity found across all runs") did fire here, and the subagent talked itself out of it in the same paragraph — a prompt to notice absence can't distinguish the two causes, because nothing in a run-keyed survey can. ## Change Adds a sweep block to the top of the Step 2 prompt that takes no run ID — the two `?since=` comment endpoints, bounded on `created_at` at both ends; a `pr list --search "updated:>"` for the candidate list the review queries need; and a loop over those candidates counting bot reviews submitted inside the window, since neither comment endpoint returns review submissions and an empty-body `APPROVE` is `tend-review`'s most common output — with instruction to report all four counts and to re-map from what they found rather than reporting those runs silent. Adds one sentence at the main-agent review point: an all-quiet report without the counts isn't usable, and absence isn't a finding to reason from. This is the check that caught the failure this run. It is four counts off two run-independent endpoints and one search, and it fails independently of the mapping it is checking. ## Relation to the other open Step 2 PRs Distinct problems, non-overlapping edits. [#864](#864) fixes *who* accepted (named non-bot actor); [#869](#869) fixes *which run* produced an output (confirm from the posting run's log). Both still start from a candidate PR list reached by run-keyed mapping — neither makes "no output at all" falsifiable, which is the failure here. ## Gate assessment - **Evidence level**: High — survey unreliability is recorded in the evidence gist across prior windows, cumulative **4 → 5** with this one. High needs 2–3. Prior occurrences were omissions of individual runs and one mislabelled silence; this is the first categorical zero-output claim, and the first to produce a fabricated inference from the absence. - **Structural**: the *specific* empty read is stochastic, but the skill's exposure is not — Step 2 offers only run-keyed paths, so any mapping break yields a plausible, uniform, unfalsifiable silence. Replay it and the summary is equally convincing every time. - **Change type**: targeted fix (one query block, one sentence) — normal bar, met. - **Passes both gates.** Evidence: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4 --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
…nst tend before filing upstream (#891) ## Problem Two dedup blocks were blind in two different ways, and the cited duplicate needed both fixed. **State filter.** `review-runs` Step 5 and `review-reviewers` Step 4 both deduped against PRs with `gh pr list --state open`. A merged PR is never returned by that query, so a finding whose fix already landed reads as undeduped and gets filed again. `running-in-ci`'s PR-creation dedup recheck already gets this right ("with `--state all` so closed and merged siblings show up"); these two recipes contradicted it. **Repo scope.** `review-runs` is a generated workflow ([`generator/src/tend/config.py:24`](https://github.com/max-sixty/tend/blob/f65f49f/generator/src/tend/config.py#L24) lists it in the enabled set), so it runs in each adopter's checkout and an unqualified `gh pr list` returns *the adopter's* PRs. Step 6 routes bundled-skill defects upstream to tend, but neither Step 5 nor any of `running-in-ci`'s dedup recipes — all local-repo — told the agent to dedup in the target repo before filing there. `--state all` alone does not close this: the adopter's PR list never contained the upstream fix at any state. `review-reviewers` is unaffected by the second half. It runs in `max-sixty/tend` and files onto tend, so its unqualified `gh pr list` already resolves to the right repo; only the state filter was wrong there. This bites hardest on tend specifically, because of the pinning model: adopters call `max-sixty/tend/<harness>@X.Y.Z`, so a merged skill fix stays dormant on their repos until the next release tags. The bug keeps reproducing after the fix merges — which is exactly the window in which the analysis legs are looking at it, and exactly when the dedup queries are blind to the fix. ## What happened `max-sixty/cargo-affected`'s `tend-review-runs` run [31160677649](https://github.com/max-sixty/cargo-affected/actions/runs/31160677649) (08:11:33Z → 08:21:41Z) hit the `| last` evidence-log mis-selection: it appended ~12 KB of run evidence into the nightly's unrelated comment on target [#73](max-sixty/cargo-affected#73), noticed on its post-verify read, restored comment `5188771252`, and re-appended to the real log `5150650688`. Good recovery. It then filed [#883](#883) upstream, whose "Proposed fix" is a `## Run ` heading predicate on the comment selector. [#875](#875) merged that exact fix at 07:34:40Z — 46 minutes before the issue was filed — as `test("^## Run [0-9]")` on the same selector, in the same file. #883 is a duplicate of a merged PR. The run made three dedup queries before filing (`gh issue list --state all --search "tracking issue comment append"`, a broader `gh issue list --state all` title regex, and a final `gh issue list --state open` recheck). All three were `gh issue list`, which never returns PRs — and all three ran against `max-sixty/cargo-affected`. Even had it run Step 5's PR line verbatim, it would not have returned #875, for both reasons: the state filter excluded merged PRs, and the query's repo was the adopter's, not tend's. ## The fix - Both skills: `gh pr list --state open` → `--state all`, projecting `state,mergedAt` so a merged hit is legible. - `review-runs` only: add the cross-repo pair (`gh pr list`/`gh issue list --repo max-sixty/tend --state all`) so a finding heading upstream under Step 6 is deduped against tend first. - `review-runs` only: the pinning note is scoped to the upstream repo, since in that skill the reader is the adopter and the local `gh pr list` above it has nothing to do with pinned refs. `review-reviewers` keeps the original wording, where tend is the reader and "on adopters" is the correct direction. Both added commands were run against this repo to confirm they parse and return the expected shape. ## Gate assessment - **Evidence level**: High. **Occurrences: 1** direct, verified end to end (session log, both dedup query sets, #875's merge time and diff, #883's body). - **Structural, not stochastic.** `gh pr list --state open` deterministically cannot return a merged PR, and a query scoped to the adopter's repo deterministically cannot return a tend PR; replayed ten times it misses #875 ten times. There is no decision point. - **Change type**: targeted fix — query lines plus one sentence naming the pinning consequence. It brings both recipes into line with a rule the same plugin already states in `running-in-ci`, rather than introducing new policy. - **Why act at one occurrence**: the six-PR batch merged at 07:34:40Z ([#875](#875), [#834](#834), [#868](#868), [#818](#818), [#877](#877), [#858](#858)) is all unreleased, so six distinct bugs remain observable on every adopter until the next release. The first analysis leg after that batch produced the first duplicate. The exposure is six-wide and standing, not one-off. ## Not covered by No open PR touches either dedup block. [#845](https://github.com/max-sixty/tend/pull/845)/[#850](https://github.com/max-sixty/tend/pull/850)/[#838](https://github.com/max-sixty/tend/pull/838) are the run-window cluster; [#849](#849), [#856](#856), [#857](#857), [#864](#864), [#869](#869), [#876](#876), [#809](#809), [#837](#837), [#821](#821), [#836](#836) are elsewhere. #883 stays open as a maintainer call — it is redundant with #875 but the option-2 half of its body (finish the gist migration for `review-runs`) is not. Evidence: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
## Problem Gap 2 of #816: a `tend-outage` row names the trigger a dead session stranded, but nothing re-runs it. `tend-review` fires only on `pull_request_target`, so a PR whose one review attempt died stays unreviewed until someone happens to push again — and the outage issue that recorded it stays open, folding the next incident into a stale one. The recovery shape has been running in worktrunk's `running-tend` overlay since the incident that filed #816, across two outages of both flavours (a 5-hour session-limit exhaustion and a weekly-limit exhaustion). It is generic — every consumer running `tend-review` on `pull_request_target` loses reviews the same way — so worktrunk's maintainer asked for it upstream rather than kept per-repo ([max-sixty/worktrunk#3742](max-sixty/worktrunk#3742 (comment))). The companion PR removes it from worktrunk's overlay once this lands. ## Solution One section in the bundled `review-runs` skill, at the end of Step 1 where failed runs are already being classified. Four rules, each a recipe: - **Find** the open `tend-outage` issue and extract its run/trigger rows. Empty on most days, so the check is a cheap no-op. - **Diagnose before re-running.** The issue body says only "The bot failed to process a request". The annotation is the cheapest next look — and #818, if it lands, makes it name the cause on the most common path — but when it doesn't, the session log carries the `<synthetic>` message. Both subscription limits are listed, because they reset on different clocks: assuming the session window's reset understates a weekly exhaustion, which can strand most of a day. - **Re-run only what won't recover on its own.** Scheduled workflows come back on their next cron tick; only event-triggered runs (`review`, `mention`, `triage`, `ci-fix`) strand. Confirm the work is still missing first — a later push often re-triggers the workflow by itself. - **Order the re-run after a clean run**, not after an assumed clock. Re-running into a still-exhausted quota just refills the outage issue with fresh rows. Closing the drained issue is part of the recipe: `report-failure.sh` auto-closes only duplicates from the create-create race, never the surviving issue. ## Testing Documentation only — no generator or action code paths change, so there is nothing for `wt test` to exercise. The commands in the section are the ones worktrunk's sweep has been running: the `grep -oE 'runs/[0-9]+|\| #[0-9]+'` extraction matches the row format `report-failure.sh` emits (`| ${TIMESTAMP} | [workflow run](${RUN_URL}) | ${REF} |`), and the workflow-trigger split is read off the generator templates — `review.yaml.j2` is `pull_request_target`, `ci-fix.yaml.j2` is `workflow_run`, `mention.yaml.j2` is `issue_comment` plus the relayed dispatch, `triage` is `issues`, and `scheduled.yaml.j2`/`notifications.yaml.j2` are `schedule`. ## Scope Skill guidance only, and deliberately not the automated recovery pass #816 sketched — that carries the design tradeoffs #818 flagged (which workflows, the quota-window ordering, confirming the work is still missing). Those are exactly the judgements a skill can make per-case and a script cannot, so the agent-driven half is worth having whether or not a deterministic pass ever lands. Textually disjoint from the other open outage-path changes: #818 is in `claude/action.yaml`, #823 and #836 and #809 are in `shared/steps/report-failure.sh`. Co-authored-by: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com>
…857) ## Problem A run that fails **before** the agent step never reaches the outage tracker. `Report failure` is gated on the agent step specifically: ```yaml - name: Report failure if: failure() && steps.claude.outcome == 'failure' ``` Twelve steps run ahead of `claude` — the security preflight, the rate-limit preflight, bot-ID resolution, auth validation, proxy/uv install, sensitive-config restore, prompt composition, the mitmproxy cache, the sandbox build, the binary and plugin installs, and the adopter's `sandbox_setup:`. When any of them fails, `steps.claude.outcome` is `skipped`, the conjunct is false, and the run goes out red with no `tend-outage` issue and no comment on an existing one. The work it stranded leaves no trace anywhere a maintainer looks. `codex/action.yaml` carries the identical gate on `steps.codex.outcome`. ## What it cost On `max-sixty/tend` between 2026-08-05T11:37Z and 2026-08-06T00:07Z, the rate-limit preflight aborted **every** agent run on the repo — 36 failed runs across four workflows: | Workflow | Failed / total | |---|---| | `tend-notifications` | 19 / 20 | | `review-reviewers` | 12 / 13 | | `tend-review` | 4 / 4 | | `tend-mention` | 1 / 1 | Every one aborted on the same line, with the burst counters at zero — the bot's daily item count had crossed the spike threshold and, since the guard runs before anything that could change that count, it stayed crossed until the UTC date rolled: ``` Rate limit: burst=0 PRs, 0 issues (20min); today=16 (limit: 15) ##[error]Rate limit: bot created 16 items today, above spike limit of 15 (baseline: 17 over past 6 days) ``` And in all 36, the outage step never ran — from the raw logs of both a [`review-reviewers` leg](https://github.com/max-sixty/tend/actions/runs/31055440109) and a [`tend-review` run](https://github.com/max-sixty/tend/actions/runs/31047860817): ``` ##[start-action display=Report failure;id=__max-sixty_tend.__run_13] ##[end-action id=__max-sixty_tend.__run_13;outcome=skipped;conclusion=skipped;duration_ms=0] ``` Zero `tend-outage` issues were filed or commented on across the whole 12.5-hour window; the label's most recent issues are still #831 and #832 from 2026-08-04, both closed. Those two got filed precisely because that outage failed *inside* the agent step. So the tracker works — it just can't see the half of the action that runs first, which is where a whole-repo, day-long stop lives. The blackout self-cleared at 00:00Z when the daily counter reset, so nothing here needs a revert; the reason a maintainer never saw it is what this PR fixes. ## Fix Gate on the job being red rather than on which step reddened it, in both harness actions, and correct the `report-failure.sh` header comment that documented the old contract. A pre-agent failure strands exactly the same work as an agent failure, so it belongs in the same tracker. The widened gate also admits post-agent failures (`Mark event notification read`, `Token usage`). Those are rarer and the agent's work has already shipped by then, but the run is still red and still worth a row — an outage issue that occasionally over-reports is the right side to err on relative to one that misses a 12-hour stop. **One exclusion: the security preflight.** `security-preflight.sh` failing means the repo isn't safely gated for the bot — an unprotected default branch, or an update ruleset the bot can bypass. That's a config refusal, not an outage, and it's persistent: it stays failing until a human fixes the repo, so under a bare `failure()` gate it would file an issue titled "Bot temporarily unavailable" and append a row on every subsequent trigger, indefinitely, while the reporter records only a run link and so never names the cause. It's also the one path where reporting has the action write to the repo (`gh issue create`, plus `gh issue close` on the reconcile path) with the bot's PAT right after the security gate refused to let it operate there. The step now carries `id: security` and the gate is `if: failure() && steps.security.outcome != 'failure'`; it's the first step in both actions, so every other failure leaves that outcome `success` and the widening is otherwise unaffected. Note that the PAT-write argument is what carves it out, not persistence: `Validate auth configured` and the adopter's `sandbox_setup:` also fail deterministically until a human edits config, and they stay in. Bounding that repeated append belongs in `report-failure.sh`, where one change covers every such step without an enumerated exclusion list — tracked in #859. **The rate-limit abort's remediation is itself a counted item.** Worth stating because the direction is counterintuitive when this is read back later: the tiers `rate-limit-preflight.sh` enforces count bot-authored issues — `RECENT_ISSUES` via `repos/$REPO/issues?creator=$BOT`, `TODAY_POSTS` via `search/issues?q=author:...` — so the first abort under the new gate creates a `tend-outage` issue and thereby nudges the very counter it tripped on. It's self-limiting rather than a loop: every later failure appends a *comment* to the now-open issue, and comments appear in neither query, so the exposure is +1 item per open-issue cycle. This is diagnosability only — it doesn't touch the rate-limit thresholds. #856, from a sibling leg of this same run, retunes the guard that caused this particular blackout by demoting its spike tier to a creation pause. The two are complementary rather than overlapping: #856 stops the spike tier from failing the run at all, and this PR makes the tiers that still abort — the two burst checks and the hard limit it keeps — plus every other pre-agent step land in the tracker when they do. ## Gate assessment - **Evidence level**: Critical — 36 failed runs, four workflows, a 12.5-hour total stop of the bot on its own repo, invisible end to end. Acts on one occurrence. - **Structural, not stochastic**: no decision point. `steps.claude.outcome` is `skipped` for every pre-agent failure, so the condition is false 100% of the time, for every consumer of both actions. - **Change type**: targeted fix — one `if:` expression per action plus a comment correction. - **Passes both gates.** Evidence log: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb Related but distinct — both assume the agent step failed and so never fire on this path: #818 (naming the cause in the exited-non-zero annotation) and #809 (deduping outage comments across matrix legs). --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
… and stop the fetch limit re-truncating it (#838) `list-recent-runs.sh`'s dropped-tick recovery anchors the window floor on the previous **completed** run of the analyzing workflow. `completed` is a status, not a conclusion, so a run that failed before its agent produced any analysis advances the anchor exactly as if it had covered its hour. After an outage the next green run therefore resumes one tick back and reports an all-clear for a window nothing ever looked at — and the gap is unrecoverable, because the following run's floor advances past it too. This is not hypothetical: it just consumed ~15 hours of `review-reviewers` coverage. ## What happened Every `review-reviewers` run from [30897445507](https://github.com/max-sixty/tend/actions/runs/30897445507) (09:41Z) through [30959115609](https://github.com/max-sixty/tend/actions/runs/30959115609) (23:10Z) failed — 16 consecutive runs, all five matrix legs, `claude -p` exiting 1 after ~4s. The `result` event in every leg's session log is identical: `is_error: true`, `num_turns: 1`, `result: "You've hit your weekly limit · resets 12am (UTC)"`, with `token-usage.json` all zeros. The agent never started, so none of those runs analyzed anything. (The outage itself is already tracked by #831 and the annotation-legibility work in #816 / #818 — this PR is only about the window the outage swallowed.) The last run that did analyze anything is [30893072876](https://github.com/max-sixty/tend/actions/runs/30893072876) at 08:42Z, which is also the last entry in the `numbagg/numbagg` evidence gist. At this run's tick (23:47Z intended), the recovery query picked `{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}` and floored the window at 22:47Z, so `list-recent-runs.sh` returned exactly one row: a `tend-notifications` pre-check no-op. Everything `numbagg-bot` actually did in the swallowed window sat outside that floor — PRs [#719](numbagg/numbagg#719) and [#720](numbagg/numbagg#720), issue [#721](numbagg/numbagg#721), three reviews and four inline comments on #719, and comments on #716, #721 and #1. That is the densest window numbagg has had in days, and per the skill's Step 1 ("If empty, record the run as all-clear … then skip to Step 6") it would have been recorded as a quiet hour. I only found it by diffing the run list against the gist's last recorded boundary by hand. ## The change Anchor on the previous **successful** run rather than any completed one, filtering server-side with `--status success` (the flag takes conclusions as well as statuses), and when the existing 6h cap clamps the recovered floor, say so on stderr so the caller records a coverage gap instead of a false all-clear. A partially-failed matrix run counts as a failure here, which only ever widens the window — overlap is re-offered work the caller dedups against its own evidence log, whereas a gap is silently unanalyzed. Widening the window then exposed a second truncation one layer down, so the fetch loop moves with it. It passed `--limit 50` while a recovered window spans up to 8h, and `gh run list` returns newest-first — so a workflow over the limit silently drops its *oldest* runs, which are exactly the ones in the gap the anchor just reached back for. On `max-sixty/tend`, `tend-mention` alone produces 57 runs in an 8h window. The limit is now 200, and a workflow returning exactly the limit warns rather than truncating in silence. Two smaller pieces: the anchor query routes through `gh_retry` and exits non-zero instead of `2>/dev/null || true`, so a transient API error can no longer masquerade as "no successful run" and emit a confident warning naming a cause that didn't happen; and `review-reviewers/SKILL.md`, which still said "if empty, record all-clear" with nothing about the stderr warnings, now treats any `WARNING:` as a coverage gap — without that, an agent following the skill literally would print the warning and record an all-clear anyway. When every tick fires and succeeds the anchor is the previous tick, `prev_intended == intended - 3600 == COMPLETED_AFTER`, and the comparison is a no-op — output is byte-identical to today's on the healthy path. <details><summary>Verification against live data</summary> Same tick, same repo, before vs. after: ``` # anchor query — before (--limit 10, no conclusion filter) {"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609} # anchor query — after (--limit 50, conclusion == "success") {"conclusion":"success","createdAt":"2026-08-04T08:42:12Z","databaseId":30893072876} ``` `TARGET_REPO=numbagg/numbagg ./list-recent-runs.sh` after the change returns 5 runs instead of 1, preceded by: ``` WARNING: the last successful 'review-reviewers' run started 2026-08-04T08:42:12Z, more than 6h back. Window floored at 2026-08-04T17:47:00Z; runs that completed before it are NOT in this list. Record a coverage gap, not an all-clear. ``` The recovered anchor (08:42:12Z) matches the gist's last recorded coverage boundary exactly. Also exercised: the no-successful-run branch (warns, floors at the cap, exits 0) and the unset-`GITHUB_WORKFLOW` branch (recovery skipped, unchanged). `shellcheck` clean; `bash -n` clean. Also exercised after the review follow-ups: the fetch-limit truncation warning (via a copy with `RUN_LIMIT=5`) and the anchor-query failure path (exits 1 rather than degrading). See [the follow-up comment](#838 (comment)) for the full branch table. </details> ## Gates - **Evidence level**: Critical — a clearly wrong outcome (the analysis records an all-clear for a window it never examined, and the window is then unreachable), which acts on 1 occurrence. Occurrences: 1 this run, 0 historical (no prior gist entry records an outage of a length that would trigger it). - **Structural, not stochastic**: no decision point. The anchor query is deterministic; replayed ten times it picks the failed run ten times. - **Change type**: targeted fix — one predicate, one query limit, two stderr warnings. Normal evidence bar under Gate 2, met. - **Dedup**: checked open issues #831, #830, #829, #828, #827, #824, #822, #817, #816, #801, #799, #750, #624 and open PRs #826, #825, #823, #821, #819, #818, #809. #816 / #818 / #823 / #809 all concern outage *reporting* — naming the cause in the annotation, the stranded trigger, per-run comment dedup — none touch window recovery. Searching issues and PRs for `list-recent-runs` surfaces only the three merged predecessors #526, #753 and #784. Evidence log: https://gist.github.com/19b5ab297bb7ac7e1e9a44d595ccde0f --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Problem
A
tend-outageissue records that a run failed but, on the most common failure path, can't say why. Inclaude/action.yamltheexitedbranch bailed onCLAUDE_EXIT -ne 0andexited before theVERDICTjq ever inspected the stream:A session-limit exhaustion — the single most common outage cause — exits non-zero and emits a
<synthetic>assistant message likeYou've hit your session limit · resets 8:30am (UTC). That text sits in the stream the action still has open, but nothing extracted it: the annotation, and the enrichment commentenrich-tend-outage-issues.shderives from it, both read only the generic "see the session-logs artifact" line. A maintainer couldn't tell session-limit from auth failure from server error without downloading artifacts and writing jq.Solution
In the exited-non-zero branch, extract the last assistant text from the stream and fold it into the annotation, keeping the generic fallback when the stream has no assistant text:
Because the nightly
enrich-tend-outage-issues.shpass posts the failure annotation into the issue, a self-diagnosing annotation carries into thetend-outageissue for free — no change to the enrichment script needed.Testing
Inline action bash, so no pytest harness. Verified the extraction jq against a realistic stream-json sample (normal turns + tool_use + a trailing
<synthetic>session-limit message): it returnsYou've hit your session limit · resets 8:30am (UTC), and returns empty for an empty stream (falls back to the generic annotation). YAML re-parses cleanly.Scope
This is Gap 1 of #816. Gap 2 — a recovery pass that re-runs the event-triggered workflow a dead run stranded (e.g. a
tend-reviewthat never retries because it fires only onpull_request_target) — is a larger design change with real tradeoffs (which workflows to re-run, the quota-window ordering constraint, confirming the work is still missing first). I've left that for a maintainer to weigh in on rather than ship a partial recovery mechanism here; details and the reporter's proposed shape are in the issue.Closes #816 — automated triage