Skip to content

fix(claude-action): name the cause in the exited-non-zero outage annotation - #818

Merged
max-sixty merged 2 commits into
mainfrom
fix/issue-816
Aug 7, 2026
Merged

fix(claude-action): name the cause in the exited-non-zero outage annotation#818
max-sixty merged 2 commits into
mainfrom
fix/issue-816

Conversation

@tend-agent

@tend-agent tend-agent commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

A tend-outage issue records that a run failed but, on the most common failure path, can't say why. In claude/action.yaml the exited branch bailed on CLAUDE_EXIT -ne 0 and exited before the VERDICT jq ever inspected the stream:

exited)
  if [ "$CLAUDE_EXIT" -ne 0 ]; then
    echo "::error::claude -p exited non-zero (exit=$CLAUDE_EXIT) — see the session-logs artifact"
    tail -20 "$STDERR_LOG" 2>/dev/null || true
    exit "$CLAUDE_EXIT"
  fi
  # VERDICT jq (names the cause) only ran when CLAUDE_EXIT == 0

A session-limit exhaustion — the single most common outage cause — exits non-zero and emits a <synthetic> assistant message like You've hit your session limit · resets 8:30am (UTC). That text sits in the stream the action still has open, but nothing extracted it: the annotation, and the enrichment comment enrich-tend-outage-issues.sh derives from it, both read only the generic "see the session-logs artifact" line. A maintainer couldn't tell session-limit from auth failure from server error without downloading artifacts and writing jq.

Solution

In the exited-non-zero branch, extract the last assistant text from the stream and fold it into the annotation, keeping the generic fallback when the stream has no assistant text:

REASON=$(jq -r '
  select(.type == "assistant") | .message.content[]?
  | select(.type == "text") | .text
' "$STREAM_JSON" 2>/dev/null | grep -v '^[[:space:]]*$' | tail -1)

Because the nightly enrich-tend-outage-issues.sh pass posts the failure annotation into the issue, a self-diagnosing annotation carries into the tend-outage issue for free — no change to the enrichment script needed.

Testing

Inline action bash, so no pytest harness. Verified the extraction jq against a realistic stream-json sample (normal turns + tool_use + a trailing <synthetic> session-limit message): it returns You've hit your session limit · resets 8:30am (UTC), and returns empty for an empty stream (falls back to the generic annotation). YAML re-parses cleanly.

Scope

This is Gap 1 of #816. Gap 2 — a recovery pass that re-runs the event-triggered workflow a dead run stranded (e.g. a tend-review that never retries because it fires only on pull_request_target) — is a larger design change with real tradeoffs (which workflows to re-run, the quota-window ordering constraint, confirming the work is still missing first). I've left that for a maintainer to weigh in on rather than ship a partial recovery mechanism here; details and the reporter's proposed shape are in the issue.


Closes #816 — automated triage

…tation

The exited branch bailed on CLAUDE_EXIT != 0 before the VERDICT jq ran, so
the most common failure path (session-limit exit) only ever emitted the
generic "see the session-logs artifact" annotation. Extract the last
assistant text from the stream and fold it into the annotation, so
report-failure.sh enrichment carries a self-diagnosing message
("You've hit your session limit · resets 8:30am (UTC)") into the
tend-outage issue for free.

Closes #816

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review — the change is sound. Session-limit exits non-zero and emits the <synthetic> assistant message (confirmed by #816), so the exited-non-zero branch is the right place, the extraction jq matches the shape used by the transcript-render jq earlier in this step, and it falls back to the generic annotation when there's no assistant text. One comment-accuracy nit inline.

Comment thread claude/action.yaml Outdated
# your session limit · resets 8:30am (UTC)"). Surface that last
# assistant text so the annotation names the cause (session limit
# vs auth failure vs server error) instead of a generic "see the
# artifact". report-failure.sh's enrichment then carries the

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The enrichment isn't report-failure.sh — that script only appends the run-link row. The nightly enrich-tend-outage-issues.sh is what fetches the failure annotations and posts them into the issue (as #816 itself attributes it). A maintainer chasing the enrichment path later would look in the wrong file.

Suggested change
# artifact". report-failure.sh's enrichment then carries the
# artifact". The nightly enrich-tend-outage-issues.sh pass then carries the

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 85e590d — the comment now credits the nightly enrich-tend-outage-issues.sh pass. report-failure.sh only records the run-link row; the enrich script fetches the failure annotations and posts them, so that's the path a maintainer would chase. Also corrected the same misattribution in the PR body.

The comment credited report-failure.sh with carrying the annotation into
the tend-outage issue, but that script only records the run link. The
nightly enrich-tend-outage-issues.sh pass fetches the failure annotations
and posts them. Point the comment at the right file per self-review.
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Real-stream validation from a production outage, in case it helps prioritize this: tend's own account hit its weekly limit at ~09:41Z on 2026-08-04 and stayed exhausted until the 00:00Z UTC reset. Every review-reviewers run in that window failed on exactly this path — 14 consecutive hourly runs × 5 matrix legs, all Supervisor: status=exited elapsed=2s claude_exit=1 → the generic see the session-logs artifact annotation.

I pulled the claude-stream.json from one of those legs (run 30959115609) and ran this PR's extraction against it verbatim. It returns the right string:

$ jq -r 'select(.type == "assistant") | .message.content[]? | select(.type == "text") | .text' \
    claude-stream.json | grep -v '^[[:space:]]*$' | tail -1
You've hit your weekly limit · resets 12am (UTC)

The stream is five events (active_goal, system/init, rate_limit_event, one <synthetic> assistant message with error: "rate_limit" and is_api_error_message: true, then result), so the tail -1 picks the synthetic message with nothing else competing — the same shape as the session-limit case the PR describes, just the weekly variant. Zero tokens billed, num_turns: 1.

Cost of not having it: #831 took 77+ comments over those ~15 hours, every one a bare one-row table with no cause. Working out that this was a quota reset rather than an auth break or a server error needed an artifact download and jq — for a condition that self-resolves at a known clock time and needs no code change at all.

One optional belt-and-braces note if you're touching the block anyway: the trailing result event carries api_error_status: 429 and duplicates the same text in .result, so it's an independent source for the same fact if the synthetic-assistant shape ever changes. Not needed for this fix to work — the extraction as written is correct on the real artifact.

@tend-agent

Copy link
Copy Markdown
Collaborator Author

Fresh evidence from the 2026-08-04/05 window, since this has been open a couple of days: the failure mode this fixes recurred at 15× the scale of the incident that prompted it — 15 consecutive review-reviewers runs × 5 matrix legs = 75 job failures over 14.5 continuous hours, every one annotated claude -p exited non-zero (exit=1) — see the session-logs artifact with an empty $STDERR_LOG.

The text this PR would have surfaced was sitting in $STREAM_JSON the whole time:

You've hit your weekly limit · resets 12am (UTC)

Two details worth noting for the implementation here:

  • The weekly limit produces the same synthetic-assistant shape as the session limit — a <synthetic> model message whose last text block carries the message — so tail -1 of the assistant text picks it up unchanged. No change needed.
  • The stream also carries a structured rate_limit_event (rateLimitType: "seven_day", resetsAt, overageStatus) and the result event has api_error_status: 429. Not needed for the annotation, but they're there if a future pass wants to distinguish quota class or report a reset time.

Practical cost of not having this: outage tracker #831 accumulated 78 rows with no cause on any of them. The nightly enrichment pass ran and worked correctly — it just faithfully copied the uninformative annotation into every row. Diagnosing it meant downloading artifacts and writing jq by hand, for the second time in three days (#808 was the first).

Not proposing any change to the diff — it already does the right thing. Just recording that the evidence bar is now well past one occurrence. #845 separately cuts the cadence that drives the quota exhaustion; this makes the wall legible when it's hit anyway.

max-sixty pushed a commit that referenced this pull request Aug 5, 2026
…#823)

## Problem

The `Trigger` column of a `tend-outage` row is the only pointer back to
the work a failed run stranded. It goes blank for the one trigger where
that pointer matters most, and prints `#null` when a field is missing.

**`repository_dispatch` is unhandled.** `tend-mention` relays review
events through a secretless job that re-posts them as a
`repository_dispatch`, so the handle job runs on that event and the PR
number arrives as `client_payload.pr` rather than in a `pull_request`
object. The `if`/`elif` chain has no branch for it, so every failure on
the relay path records `Trigger: N/A` — and a relayed review is exactly
the case a maintainer can't recover from the run alone, since
`tend-review` fires only on `pull_request_target` and never retries.
This path is in constant use: `gh api
"repos/max-sixty/tend/actions/runs?event=repository_dispatch"` returns a
steady stream of `tend-mention` runs.

**`workflow_run` names no run.** The ci-fix path hardcodes `REF="CI fix
for workflow run"`, discarding `workflow_run.id` — the id of the CI
failure the job was dispatched to fix.

**Missing fields render as `null`.** `jq -r '.issue.number'` prints the
literal string `null` when the field is absent, so the cell reads
`#null` rather than falling back to `N/A`.

## Solution

Add a `repository_dispatch` branch reading `client_payload.pr`, link the
upstream run id in the `workflow_run` branch, and give every extraction
`// empty` plus a `${VAR:+…}` guard so an absent field leaves the cell
blank (rendered as `N/A` by the existing `${REF:-N/A}`) instead of
`#null`.

## Testing

`shared/steps/` has no shell test harness — shellcheck via pre-commit is
the only automated gate, and it passes on the changed file. So the block
was exercised directly: the REF logic was sliced out of the script
(between the `# Build a one-line reference` comment and `TIMESTAMP=`)
and sourced under crafted `GITHUB_EVENT_PATH` payloads, before and
after.

Before, on `origin/main`:

```
repository_dispatch  -> N/A
workflow_run         -> CI fix for workflow run
issues (no number)   -> #null
```

After:

```
relayed review               -> #815
relay w/o pr                 -> N/A
ci-fix                       -> CI fix for [run 30795510450](https://github.com/max-sixty/tend/actions/runs/30795510450)
ci-fix w/o id                -> CI fix for workflow run
PR event                     -> #821
PR event w/o number          -> N/A
issue comment                -> #808
issue w/o number             -> N/A
schedule                     -> N/A
```

Adding a real harness for `shared/steps/` is worth considering
separately — three of the recent outage-path fixes have all landed in
scripts nothing can test — but that is a bigger change than this fix
warrants, so it is not bundled here.

## Scope

Separate from the other two open changes on this path, and textually
disjoint from both. #818 names the *cause* of a failure (in
`claude/action.yaml`); #809 dedups *rows* across matrix legs (in the
`EXISTING` branch of this same script, lines 49+). This one fixes what
the row *points at*, in the REF block at lines 19–29.

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
@max-sixty
max-sixty merged commit c18885a into main Aug 7, 2026
7 checks passed
@max-sixty
max-sixty deleted the fix/issue-816 branch August 7, 2026 07:34
max-sixty pushed a commit that referenced this pull request Aug 7, 2026
…a silent survey (#885)

This hour's Step 2 survey subagent reported **"No visible bot output in
time window"** across 23 successful runs on `max-sixty/tend`. The window
in fact contained 9 reviews (two `APPROVED`), 11 inline review comments,
and 1 conversation comment from `tend-agent` — one of the busiest
windows in the series.

## Evidence

Window 2026-08-07T07:27Z → 08:16Z. The survey's verdict, verbatim:

> **No visible bot output in time window.** All 23 successful runs
(tend-mention, tend-review, tend-review-runs, tend-notifications)
executed and completed but produced zero comments, reviews, or inline
code comments between 2026-08-07T07:27:00Z and 08:16:00Z on tracked
PRs/issues (809, 818, 834, 858, 863, 868, 875, 877, 878, 881, 816, 830).

Two run-independent calls, taking no run ID at all, contradict it
immediately:

```
gh api "repos/max-sixty/tend/issues/comments?since=2026-08-07T07:27:00Z&per_page=100" → 1 tend-agent row
gh api "repos/max-sixty/tend/pulls/comments?since=2026-08-07T07:27:00Z&per_page=100"  → 11 tend-agent rows
```

Plus reviews `4880965990` (COMMENTED), `4881043005` (COMMENTED) and
`4881079314` (APPROVED) on
[#881](#881), `4881113428`
(APPROVED) on [#878](#878), and
five more (all COMMENTED) on
[#809](#809). The survey had
listed 881, 878 and 809 among the PRs it checked, so the numbers were
right and the reads came back empty anyway.

The second-order cost is worse than the omission. Having established
silence, the survey reasoned *from* it: it flagged the window's seven
bot-PR merges as "merged by max-sixty without formal review workflow",
concluding "**direct push/merge bypassing review requirement**, or
**review workflow override via branch protection rule bypass**". Six of
the seven (#818, #834, #858, #868, #875, #877) carry bot `COMMENTED`
reviews predating the window, which is why a `since`-filtered read
missed them; the seventh (#863) is the skill-authorized silence on a
self-authored PR with no concerns. None reads `APPROVED` because GitHub
blocks self-approval — the ordinary shape for a bot PR, not a bypassed
control. Acted on, that summary is a false security finding against the
maintainer.

## Root cause

Every path Step 2 offers is run-keyed: run → `headBranch` → PR →
endpoint. That chain is fine when it works, but it has one failure mode
with no floor — break it anywhere and *every* run returns empty
simultaneously. Uniform absence is exactly what a genuinely quiet hour
looks like, so the summary that comes back is self-consistent and
carries no signal that anything went wrong. The existing sanity-check
line ("note if zero bot activity found across all runs") did fire here,
and the subagent talked itself out of it in the same paragraph — a
prompt to notice absence can't distinguish the two causes, because
nothing in a run-keyed survey can.

## Change

Adds a sweep block to the top of the Step 2 prompt that takes no run ID
— the two `?since=` comment endpoints, bounded on `created_at` at both
ends; a `pr list --search "updated:>"` for the candidate list the review
queries need; and a loop over those candidates counting bot reviews
submitted inside the window, since neither comment endpoint returns
review submissions and an empty-body `APPROVE` is `tend-review`'s most
common output — with instruction to report all four counts and to re-map
from what they found rather than reporting those runs silent. Adds one
sentence at the main-agent review point: an all-quiet report without the
counts isn't usable, and absence isn't a finding to reason from.

This is the check that caught the failure this run. It is four counts
off two run-independent endpoints and one search, and it fails
independently of the mapping it is checking.

## Relation to the other open Step 2 PRs

Distinct problems, non-overlapping edits.
[#864](#864) fixes *who* accepted
(named non-bot actor);
[#869](#869) fixes *which run*
produced an output (confirm from the posting run's log). Both still
start from a candidate PR list reached by run-keyed mapping — neither
makes "no output at all" falsifiable, which is the failure here.

## Gate assessment

- **Evidence level**: High — survey unreliability is recorded in the
evidence gist across prior windows, cumulative **4 → 5** with this one.
High needs 2–3. Prior occurrences were omissions of individual runs and
one mislabelled silence; this is the first categorical zero-output
claim, and the first to produce a fabricated inference from the absence.
- **Structural**: the *specific* empty read is stochastic, but the
skill's exposure is not — Step 2 offers only run-keyed paths, so any
mapping break yields a plausible, uniform, unfalsifiable silence. Replay
it and the summary is equally convincing every time.
- **Change type**: targeted fix (one query block, one sentence) — normal
bar, met.
- **Passes both gates.**

Evidence: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
max-sixty pushed a commit that referenced this pull request Aug 7, 2026
…nst tend before filing upstream (#891)

## Problem

Two dedup blocks were blind in two different ways, and the cited
duplicate needed both fixed.

**State filter.** `review-runs` Step 5 and `review-reviewers` Step 4
both deduped against PRs with `gh pr list --state open`. A merged PR is
never returned by that query, so a finding whose fix already landed
reads as undeduped and gets filed again. `running-in-ci`'s PR-creation
dedup recheck already gets this right ("with `--state all` so closed and
merged siblings show up"); these two recipes contradicted it.

**Repo scope.** `review-runs` is a generated workflow
([`generator/src/tend/config.py:24`](https://github.com/max-sixty/tend/blob/f65f49f/generator/src/tend/config.py#L24)
lists it in the enabled set), so it runs in each adopter's checkout and
an unqualified `gh pr list` returns *the adopter's* PRs. Step 6 routes
bundled-skill defects upstream to tend, but neither Step 5 nor any of
`running-in-ci`'s dedup recipes — all local-repo — told the agent to
dedup in the target repo before filing there. `--state all` alone does
not close this: the adopter's PR list never contained the upstream fix
at any state.

`review-reviewers` is unaffected by the second half. It runs in
`max-sixty/tend` and files onto tend, so its unqualified `gh pr list`
already resolves to the right repo; only the state filter was wrong
there.

This bites hardest on tend specifically, because of the pinning model:
adopters call `max-sixty/tend/<harness>@X.Y.Z`, so a merged skill fix
stays dormant on their repos until the next release tags. The bug keeps
reproducing after the fix merges — which is exactly the window in which
the analysis legs are looking at it, and exactly when the dedup queries
are blind to the fix.

## What happened

`max-sixty/cargo-affected`'s `tend-review-runs` run
[31160677649](https://github.com/max-sixty/cargo-affected/actions/runs/31160677649)
(08:11:33Z → 08:21:41Z) hit the `| last` evidence-log mis-selection: it
appended ~12 KB of run evidence into the nightly's unrelated comment on
target [#73](max-sixty/cargo-affected#73),
noticed on its post-verify read, restored comment `5188771252`, and
re-appended to the real log `5150650688`. Good recovery. It then filed
[#883](#883) upstream, whose
"Proposed fix" is a `## Run ` heading predicate on the comment selector.

[#875](#875) merged that exact fix
at 07:34:40Z — 46 minutes before the issue was filed — as `test("^## Run
[0-9]")` on the same selector, in the same file. #883 is a duplicate of
a merged PR.

The run made three dedup queries before filing (`gh issue list --state
all --search "tracking issue comment append"`, a broader `gh issue list
--state all` title regex, and a final `gh issue list --state open`
recheck). All three were `gh issue list`, which never returns PRs — and
all three ran against `max-sixty/cargo-affected`. Even had it run Step
5's PR line verbatim, it would not have returned #875, for both reasons:
the state filter excluded merged PRs, and the query's repo was the
adopter's, not tend's.

## The fix

- Both skills: `gh pr list --state open` → `--state all`, projecting
`state,mergedAt` so a merged hit is legible.
- `review-runs` only: add the cross-repo pair (`gh pr list`/`gh issue
list --repo max-sixty/tend --state all`) so a finding heading upstream
under Step 6 is deduped against tend first.
- `review-runs` only: the pinning note is scoped to the upstream repo,
since in that skill the reader is the adopter and the local `gh pr list`
above it has nothing to do with pinned refs. `review-reviewers` keeps
the original wording, where tend is the reader and "on adopters" is the
correct direction.

Both added commands were run against this repo to confirm they parse and
return the expected shape.

## Gate assessment

- **Evidence level**: High. **Occurrences: 1** direct, verified end to
end (session log, both dedup query sets, #875's merge time and diff,
#883's body).
- **Structural, not stochastic.** `gh pr list --state open`
deterministically cannot return a merged PR, and a query scoped to the
adopter's repo deterministically cannot return a tend PR; replayed ten
times it misses #875 ten times. There is no decision point.
- **Change type**: targeted fix — query lines plus one sentence naming
the pinning consequence. It brings both recipes into line with a rule
the same plugin already states in `running-in-ci`, rather than
introducing new policy.
- **Why act at one occurrence**: the six-PR batch merged at 07:34:40Z
([#875](#875),
[#834](#834),
[#868](#868),
[#818](#818),
[#877](#877),
[#858](#858)) is all unreleased,
so six distinct bugs remain observable on every adopter until the next
release. The first analysis leg after that batch produced the first
duplicate. The exposure is six-wide and standing, not one-off.

## Not covered by

No open PR touches either dedup block.
[#845](https://github.com/max-sixty/tend/pull/845)/[#850](https://github.com/max-sixty/tend/pull/850)/[#838](https://github.com/max-sixty/tend/pull/838)
are the run-window cluster;
[#849](#849),
[#856](#856),
[#857](#857),
[#864](#864),
[#869](#869),
[#876](#876),
[#809](#809),
[#837](#837),
[#821](#821),
[#836](#836) are elsewhere.

#883 stays open as a maintainer call — it is redundant with #875 but the
option-2 half of its body (finish the gist migration for `review-runs`)
is not.

Evidence: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
max-sixty pushed a commit that referenced this pull request Aug 11, 2026
## Problem

Gap 2 of #816: a `tend-outage` row names the trigger a dead session
stranded, but nothing re-runs it. `tend-review` fires only on
`pull_request_target`, so a PR whose one review attempt died stays
unreviewed until someone happens to push again — and the outage issue
that recorded it stays open, folding the next incident into a stale one.

The recovery shape has been running in worktrunk's `running-tend`
overlay since the incident that filed #816, across two outages of both
flavours (a 5-hour session-limit exhaustion and a weekly-limit
exhaustion). It is generic — every consumer running `tend-review` on
`pull_request_target` loses reviews the same way — so worktrunk's
maintainer asked for it upstream rather than kept per-repo
([max-sixty/worktrunk#3742](max-sixty/worktrunk#3742 (comment))).
The companion PR removes it from worktrunk's overlay once this lands.

## Solution

One section in the bundled `review-runs` skill, at the end of Step 1
where failed runs are already being classified. Four rules, each a
recipe:

- **Find** the open `tend-outage` issue and extract its run/trigger
rows. Empty on most days, so the check is a cheap no-op.
- **Diagnose before re-running.** The issue body says only "The bot
failed to process a request". The annotation is the cheapest next look —
and #818, if it lands, makes it name the cause on the most common path —
but when it doesn't, the session log carries the `<synthetic>` message.
Both subscription limits are listed, because they reset on different
clocks: assuming the session window's reset understates a weekly
exhaustion, which can strand most of a day.
- **Re-run only what won't recover on its own.** Scheduled workflows
come back on their next cron tick; only event-triggered runs (`review`,
`mention`, `triage`, `ci-fix`) strand. Confirm the work is still missing
first — a later push often re-triggers the workflow by itself.
- **Order the re-run after a clean run**, not after an assumed clock.
Re-running into a still-exhausted quota just refills the outage issue
with fresh rows.

Closing the drained issue is part of the recipe: `report-failure.sh`
auto-closes only duplicates from the create-create race, never the
surviving issue.

## Testing

Documentation only — no generator or action code paths change, so there
is nothing for `wt test` to exercise. The commands in the section are
the ones worktrunk's sweep has been running: the `grep -oE
'runs/[0-9]+|\| #[0-9]+'` extraction matches the row format
`report-failure.sh` emits (`| ${TIMESTAMP} | [workflow run](${RUN_URL})
| ${REF} |`), and the workflow-trigger split is read off the generator
templates — `review.yaml.j2` is `pull_request_target`, `ci-fix.yaml.j2`
is `workflow_run`, `mention.yaml.j2` is `issue_comment` plus the relayed
dispatch, `triage` is `issues`, and
`scheduled.yaml.j2`/`notifications.yaml.j2` are `schedule`.

## Scope

Skill guidance only, and deliberately not the automated recovery pass
#816 sketched — that carries the design tradeoffs #818 flagged (which
workflows, the quota-window ordering, confirming the work is still
missing). Those are exactly the judgements a skill can make per-case and
a script cannot, so the agent-driven half is worth having whether or not
a deterministic pass ever lands. Textually disjoint from the other open
outage-path changes: #818 is in `claude/action.yaml`, #823 and #836 and
#809 are in `shared/steps/report-failure.sh`.

Co-authored-by: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com>
max-sixty pushed a commit that referenced this pull request Aug 12, 2026
…857)

## Problem

A run that fails **before** the agent step never reaches the outage
tracker. `Report failure` is gated on the agent step specifically:

```yaml
- name: Report failure
  if: failure() && steps.claude.outcome == 'failure'
```

Twelve steps run ahead of `claude` — the security preflight, the
rate-limit preflight, bot-ID resolution, auth validation, proxy/uv
install, sensitive-config restore, prompt composition, the mitmproxy
cache, the sandbox build, the binary and plugin installs, and the
adopter's `sandbox_setup:`. When any of them fails,
`steps.claude.outcome` is `skipped`, the conjunct is false, and the run
goes out red with no `tend-outage` issue and no comment on an existing
one. The work it stranded leaves no trace anywhere a maintainer looks.
`codex/action.yaml` carries the identical gate on `steps.codex.outcome`.

## What it cost

On `max-sixty/tend` between 2026-08-05T11:37Z and 2026-08-06T00:07Z, the
rate-limit preflight aborted **every** agent run on the repo — 36 failed
runs across four workflows:

| Workflow | Failed / total |
|---|---|
| `tend-notifications` | 19 / 20 |
| `review-reviewers` | 12 / 13 |
| `tend-review` | 4 / 4 |
| `tend-mention` | 1 / 1 |

Every one aborted on the same line, with the burst counters at zero —
the bot's daily item count had crossed the spike threshold and, since
the guard runs before anything that could change that count, it stayed
crossed until the UTC date rolled:

```
Rate limit: burst=0 PRs, 0 issues (20min); today=16 (limit: 15)
##[error]Rate limit: bot created 16 items today, above spike limit of 15 (baseline: 17 over past 6 days)
```

And in all 36, the outage step never ran — from the raw logs of both a
[`review-reviewers`
leg](https://github.com/max-sixty/tend/actions/runs/31055440109) and a
[`tend-review`
run](https://github.com/max-sixty/tend/actions/runs/31047860817):

```
##[start-action display=Report failure;id=__max-sixty_tend.__run_13]
##[end-action id=__max-sixty_tend.__run_13;outcome=skipped;conclusion=skipped;duration_ms=0]
```

Zero `tend-outage` issues were filed or commented on across the whole
12.5-hour window; the label's most recent issues are still #831 and #832
from 2026-08-04, both closed. Those two got filed precisely because that
outage failed *inside* the agent step. So the tracker works — it just
can't see the half of the action that runs first, which is where a
whole-repo, day-long stop lives.

The blackout self-cleared at 00:00Z when the daily counter reset, so
nothing here needs a revert; the reason a maintainer never saw it is
what this PR fixes.

## Fix

Gate on the job being red rather than on which step reddened it, in both
harness actions, and correct the `report-failure.sh` header comment that
documented the old contract. A pre-agent failure strands exactly the
same work as an agent failure, so it belongs in the same tracker.

The widened gate also admits post-agent failures (`Mark event
notification read`, `Token usage`). Those are rarer and the agent's work
has already shipped by then, but the run is still red and still worth a
row — an outage issue that occasionally over-reports is the right side
to err on relative to one that misses a 12-hour stop.

**One exclusion: the security preflight.** `security-preflight.sh`
failing means the repo isn't safely gated for the bot — an unprotected
default branch, or an update ruleset the bot can bypass. That's a config
refusal, not an outage, and it's persistent: it stays failing until a
human fixes the repo, so under a bare `failure()` gate it would file an
issue titled "Bot temporarily unavailable" and append a row on every
subsequent trigger, indefinitely, while the reporter records only a run
link and so never names the cause. It's also the one path where
reporting has the action write to the repo (`gh issue create`, plus `gh
issue close` on the reconcile path) with the bot's PAT right after the
security gate refused to let it operate there. The step now carries `id:
security` and the gate is `if: failure() && steps.security.outcome !=
'failure'`; it's the first step in both actions, so every other failure
leaves that outcome `success` and the widening is otherwise unaffected.
Note that the PAT-write argument is what carves it out, not persistence:
`Validate auth configured` and the adopter's `sandbox_setup:` also fail
deterministically until a human edits config, and they stay in. Bounding
that repeated append belongs in `report-failure.sh`, where one change
covers every such step without an enumerated exclusion list — tracked in
#859.

**The rate-limit abort's remediation is itself a counted item.** Worth
stating because the direction is counterintuitive when this is read back
later: the tiers `rate-limit-preflight.sh` enforces count bot-authored
issues — `RECENT_ISSUES` via `repos/$REPO/issues?creator=$BOT`,
`TODAY_POSTS` via `search/issues?q=author:...` — so the first abort
under the new gate creates a `tend-outage` issue and thereby nudges the
very counter it tripped on. It's self-limiting rather than a loop: every
later failure appends a *comment* to the now-open issue, and comments
appear in neither query, so the exposure is +1 item per open-issue
cycle.

This is diagnosability only — it doesn't touch the rate-limit
thresholds. #856, from a sibling leg of this same run, retunes the guard
that caused this particular blackout by demoting its spike tier to a
creation pause. The two are complementary rather than overlapping: #856
stops the spike tier from failing the run at all, and this PR makes the
tiers that still abort — the two burst checks and the hard limit it
keeps — plus every other pre-agent step land in the tracker when they
do.

## Gate assessment

- **Evidence level**: Critical — 36 failed runs, four workflows, a
12.5-hour total stop of the bot on its own repo, invisible end to end.
Acts on one occurrence.
- **Structural, not stochastic**: no decision point.
`steps.claude.outcome` is `skipped` for every pre-agent failure, so the
condition is false 100% of the time, for every consumer of both actions.
- **Change type**: targeted fix — one `if:` expression per action plus a
comment correction.
- **Passes both gates.**

Evidence log: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb

Related but distinct — both assume the agent step failed and so never
fire on this path: #818 (naming the cause in the exited-non-zero
annotation) and #809 (deduping outage comments across matrix legs).

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
max-sixty pushed a commit that referenced this pull request Aug 12, 2026
… and stop the fetch limit re-truncating it (#838)

`list-recent-runs.sh`'s dropped-tick recovery anchors the window floor
on the previous **completed** run of the analyzing workflow. `completed`
is a status, not a conclusion, so a run that failed before its agent
produced any analysis advances the anchor exactly as if it had covered
its hour. After an outage the next green run therefore resumes one tick
back and reports an all-clear for a window nothing ever looked at — and
the gap is unrecoverable, because the following run's floor advances
past it too.

This is not hypothetical: it just consumed ~15 hours of
`review-reviewers` coverage.

## What happened

Every `review-reviewers` run from
[30897445507](https://github.com/max-sixty/tend/actions/runs/30897445507)
(09:41Z) through
[30959115609](https://github.com/max-sixty/tend/actions/runs/30959115609)
(23:10Z) failed — 16 consecutive runs, all five matrix legs, `claude -p`
exiting 1 after ~4s. The `result` event in every leg's session log is
identical: `is_error: true`, `num_turns: 1`, `result: "You've hit your
weekly limit · resets 12am (UTC)"`, with `token-usage.json` all zeros.
The agent never started, so none of those runs analyzed anything. (The
outage itself is already tracked by #831 and the annotation-legibility
work in #816 / #818 — this PR is only about the window the outage
swallowed.)

The last run that did analyze anything is
[30893072876](https://github.com/max-sixty/tend/actions/runs/30893072876)
at 08:42Z, which is also the last entry in the `numbagg/numbagg`
evidence gist. At this run's tick (23:47Z intended), the recovery query
picked
`{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}`
and floored the window at 22:47Z, so `list-recent-runs.sh` returned
exactly one row: a `tend-notifications` pre-check no-op.

Everything `numbagg-bot` actually did in the swallowed window sat
outside that floor — PRs
[#719](numbagg/numbagg#719) and
[#720](numbagg/numbagg#720), issue
[#721](numbagg/numbagg#721), three reviews and
four inline comments on #719, and comments on #716, #721 and #1. That is
the densest window numbagg has had in days, and per the skill's Step 1
("If empty, record the run as all-clear … then skip to Step 6") it would
have been recorded as a quiet hour. I only found it by diffing the run
list against the gist's last recorded boundary by hand.

## The change

Anchor on the previous **successful** run rather than any completed one,
filtering server-side with `--status success` (the flag takes
conclusions as well as statuses), and when the existing 6h cap clamps
the recovered floor, say so on stderr so the caller records a coverage
gap instead of a false all-clear. A partially-failed matrix run counts
as a failure here, which only ever widens the window — overlap is
re-offered work the caller dedups against its own evidence log, whereas
a gap is silently unanalyzed.

Widening the window then exposed a second truncation one layer down, so
the fetch loop moves with it. It passed `--limit 50` while a recovered
window spans up to 8h, and `gh run list` returns newest-first — so a
workflow over the limit silently drops its *oldest* runs, which are
exactly the ones in the gap the anchor just reached back for. On
`max-sixty/tend`, `tend-mention` alone produces 57 runs in an 8h window.
The limit is now 200, and a workflow returning exactly the limit warns
rather than truncating in silence.

Two smaller pieces: the anchor query routes through `gh_retry` and exits
non-zero instead of `2>/dev/null || true`, so a transient API error can
no longer masquerade as "no successful run" and emit a confident warning
naming a cause that didn't happen; and `review-reviewers/SKILL.md`,
which still said "if empty, record all-clear" with nothing about the
stderr warnings, now treats any `WARNING:` as a coverage gap — without
that, an agent following the skill literally would print the warning and
record an all-clear anyway.

When every tick fires and succeeds the anchor is the previous tick,
`prev_intended == intended - 3600 == COMPLETED_AFTER`, and the
comparison is a no-op — output is byte-identical to today's on the
healthy path.

<details><summary>Verification against live data</summary>

Same tick, same repo, before vs. after:

```
# anchor query — before (--limit 10, no conclusion filter)
{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}

# anchor query — after (--limit 50, conclusion == "success")
{"conclusion":"success","createdAt":"2026-08-04T08:42:12Z","databaseId":30893072876}
```

`TARGET_REPO=numbagg/numbagg ./list-recent-runs.sh` after the change
returns 5 runs instead of 1, preceded by:

```
WARNING: the last successful 'review-reviewers' run started 2026-08-04T08:42:12Z, more than 6h back. Window floored at 2026-08-04T17:47:00Z; runs that completed before it are NOT in this list. Record a coverage gap, not an all-clear.
```

The recovered anchor (08:42:12Z) matches the gist's last recorded
coverage boundary exactly. Also exercised: the no-successful-run branch
(warns, floors at the cap, exits 0) and the unset-`GITHUB_WORKFLOW`
branch (recovery skipped, unchanged). `shellcheck` clean; `bash -n`
clean. Also exercised after the review follow-ups: the fetch-limit
truncation warning (via a copy with `RUN_LIMIT=5`) and the anchor-query
failure path (exits 1 rather than degrading). See [the follow-up
comment](#838 (comment))
for the full branch table.

</details>

## Gates

- **Evidence level**: Critical — a clearly wrong outcome (the analysis
records an all-clear for a window it never examined, and the window is
then unreachable), which acts on 1 occurrence. Occurrences: 1 this run,
0 historical (no prior gist entry records an outage of a length that
would trigger it).
- **Structural, not stochastic**: no decision point. The anchor query is
deterministic; replayed ten times it picks the failed run ten times.
- **Change type**: targeted fix — one predicate, one query limit, two
stderr warnings. Normal evidence bar under Gate 2, met.
- **Dedup**: checked open issues #831, #830, #829, #828, #827, #824,
#822, #817, #816, #801, #799, #750, #624 and open PRs #826, #825, #823,
#821, #819, #818, #809. #816 / #818 / #823 / #809 all concern outage
*reporting* — naming the cause in the annotation, the stranded trigger,
per-run comment dedup — none touch window recovery. Searching issues and
PRs for `list-recent-runs` surfaces only the three merged predecessors
#526, #753 and #784.

Evidence log: https://gist.github.com/19b5ab297bb7ac7e1e9a44d595ccde0f

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tend-outage issues can't name the cause, and nothing re-runs the trigger they strand

2 participants