fix(running-in-ci): pin the CI-monitor poll to the SHA the run pushed

[tend-agent]

Problem

The CI Monitoring loop in running-in-ci gated on gh pr view <number> --json statusCheckRollup, which describes the PR's current head rather than the commit the session pushed. Any other actor pushing while the loop slept retargeted the gate: the loop began waiting on the new commit, exited when those checks settled, and the closing gh pr checks <number> printed the new commit's results. The run then reported its own push as green without ever observing a terminal state for it — and the gated follow-up the recipe exists to drive (fix the failure, dismiss the approval) never fired.

The loop was also blind to an early failure: pending() counted only in-flight items, so a check reaching failure while siblings ran didn't break the loop, and nothing in the exit path named it.

Reported in #829, with a concrete instance on max-sixty/cargo-affected PR #54.

Solution

Query the same rollup by commit OID (GraphQL repository.object(oid:).statusCheckRollup) instead of through the PR. One call keeps everything the PR-scoped version had — check-runs and legacy status contexts, the own-run URL filter, and the same-workflow filter (the workflow name lives on checkSuite.workflowRun.workflow, since a CheckRun's name is the job name) — while staying pinned to the SHA the run is accountable for.

The helper now returns {pending, failed} rather than a bare count, so the loop breaks as soon as its own SHA is red and the exit path names the failing checks with their run URLs. After the loop, headRefOid is re-read: a moved head is reported as a distinct outcome rather than silently absorbed.

PINNED_SHA covers both gated cases, and the block spells out a derivation for each rather than naming one. After your own push it's git rev-parse HEAD. In a review session it is not: tend-review checks out refs/pull/N/merge, and that ephemeral merge commit carries no statusCheckRollup at all — the query returns null, the loop breaks on iteration 1 with {"pending":0,"failed":[]}, and the approval is reported green without the commit it sits on ever being observed. That is this PR's own bug relocated onto the approve-then-dismiss gate, so the review-session derivation (gh pr view <number> --json headRefOid) ships alongside, commented.

Testing

Replayed the reported instance against the live API. 551e059 on max-sixty/cargo-affected PR #54, whose head has since advanced to 6955352:

$ # old recipe, PR-scoped
{"total":6,"failed":[]}

$ # new recipe, pinned to 551e059
{"pending":0,"failed":["lint https://github.com/max-sixty/cargo-affected/actions/runs/30889116309/job/91926839301"]}
branch advanced to 6955352192882134d0fb0df302e849fff27ab234 — the result above is still 551e059798afbdfe7d458e29484c0faa4049250c's, which is what this run is accountable for

The old form reports zero failures; the pinned form surfaces the lint failure that the session missed, and flags the moved head.

The merge-commit case was then confirmed against this PR's own review run: 4897ea37 (its git rev-parse HEAD) returns statusCheckRollup: null, while the PR head 8582f53 carries eight contexts — so the merge commit's {"pending":0,"failed":[]} is indistinguishable from settled green. Bucketing was exercised against synthetic nodes covering every conclusion: STARTUP_FAILURE and ACTION_REQUIRED now count as failed (they previously landed in neither bucket, so a job that never started read as green), CANCELLED stays excluded, and both the own-run and same-workflow filters still hold. The three-way outcome line was exercised across all of red, cap-hit, and green. The own-run and same-workflow filters were exercised in that replay too (GITHUB_RUN_ID=30889113519, GITHUB_WORKFLOW=tend-review — the review check-run was correctly excluded). The block as committed passes bash -n.

Review rounds

Five rounds of review on this branch found the same false-green class surviving inside the fix itself; all are addressed:

• PINNED_SHA derivation. git rev-parse HEAD is wrong in a review session — tend-review checks out refs/pull/N/merge, and that merge commit's statusCheckRollup is null, so the loop broke on iteration 1 with a false green on the approve-then-dismiss gate. Both derivations are now spelled out, one commented.

• Failure buckets. STARTUP_FAILURE and ACTION_REQUIRED are terminal CheckConclusionState values that landed in neither bucket, so a job that never started read as green. CANCELLED stays excluded.

• Cap signal. The single trailing echo "$R" exited 0 whether the loop settled or timed out. Replaced with an explicit outcome line naming red, cap-hit, no-rollup, and green.

• Break-on-red removed. review's pre-APPROVE path routes a FAILURE co-existing with in-flight checks back through this loop to be waited out, because that shape is usually a stale cancellation-cascade artifact. An early break returned ~60 s in with the same mid-flight snapshot, so the caller re-peeked an identical rollup.

• Every no-rollup response now reaches the guard. // [] collapsed a GraphQL errors envelope, an unresolvable OID, and a nullable statusCheckRollup all to {"pending":0,"failed":[]} — byte-identical to settled green. // empty emits nothing so they route to no rollup returned … unverified, not green. Both in-loop queries also guard on an empty $R: the [ tests errored on empty strings and fired neither continue nor break, costing 90 s per iteration — 810 s against the 600 s Bash cap, which killed the call before the verdict printed. Now 540 s.

• Superseded check runs. The rollup was read un-reduced. A concurrency-cancelled run's check runs stay on the commit forever, and an if: always() merge-gate omnibus whose dependency was cancelled concludes FAILURE rather than CANCELLED, so the bucket doesn't exclude it — once the replacement settled the loop broke with PENDING == 0, read the stale red, and printed red on $PINNED_SHA, dismissing an approval over a check whose replacement went green. Now reduced to the latest entry per (check name, workflow), matching the block review's pre-APPROVE path got in fix(review): dedupe pre-APPROVE rollup by name and decide cap-expiry on provenance #821 — which is what delegates here for exactly that shape.

• Outcome ordering. Removing the break left FAILED > 0 with PENDING > 0 reachable only at cap expiry — the same unsettled shape the loop is told to wait out — yet it printed red. PENDING is now tested first, so an unsettled FAILURE stays unverified; a settled red still reports red because the loop only breaks once PENDING is 0.

Round four's reduction was verified by extracting the committed block and running its jq filter verbatim against a synthetic rollup: a superseded check-ok-to-merge FAILURE (run 111) drops out while a same-named FAILURE from a different workflow (run 333) survives, a settled FAILURE superseded by a still-QUEUED replacement moves from failed into pending, and STARTUP_FAILURE, CANCELLED-exclusion, the own-run filter, the same-workflow filter, and the codecov/patch status context are all unchanged. That queued case rests on startedAt being set at queue time, checked against live QUEUED check runs (pytorch/pytorch) via both REST and GraphQL rather than assumed. Live commits still answer as before, and an unresolvable OID still emits nothing and routes to no rollup returned.

• Where latest-wins doesn't hold. The reduction presumes the later entry supersedes the earlier one, which is true for the cancellation cascade it targets and false for two independent runs of a workflow with no concurrency group — tend-mention's group-less relay / verify / handle are that shape, and this branch's head carries two relay entries tied to the same second, where max_by falls back to input order. Kept the reduction (the recurring stale red is the case worth fixing) and documented the limit, pointing at the per-run conclusion check for when the distinction has to be exact.

Loop behaviour was simulated against a stubbed rollup() with sleep counting seconds rather than sleeping: settled green 90 s / 2 calls, transient blip then green 150 s / 3, stale cascade red then green 210 s / 4, genuine red 210 s / 4, never settles 540 s / 9 (cap hit), persistent API failure 540 s / 9 (no rollup returned).

Not included

review's pre-APPROVE bullet still says to re-peek the PR rollup after polling, which has the same head-move exposure. #821 is currently rewriting that exact line, so the one-line change is left out to avoid a conflict; it should adopt PINNED_SHA=$HEAD_SHA once #821 lands.

---

Closes #829 — automated triage

[tend-agent]

Querying the rollup by commit OID is the right shape, and the own-run / same-workflow filters carry over faithfully (the checkSuite.workflowRun.workflow.name hop is correct — a CheckRun's name really is the job name). One correctness issue in how PINNED_SHA is derived, plus two smaller gaps; all three have inline suggestions.

git rev-parse HEAD is the wrong SHA in exactly the session the comment above it names. The review workflow checks out refs/pull/N/merge — see the Resolve PR checkout ref step in .github/workflows/tend-review.yaml, which only falls back to refs/pull/N/head when the merge ref 404s on a conflicting PR. So in a review session HEAD is an ephemeral merge commit, and a merge commit carries no check rollup at all. The loop then breaks on iteration 1 with {"pending":0,"failed":[]} and reports green without ever observing the commit the approval sits on. That's the same false-green this PR is fixing, relocated from "the PR head moved" to "the pinned SHA was never the right one" — and it lands on the approve-then-dismiss gate, which is the one case the PR body says PINNED_SHA covers.

Since shell state doesn't persist between Bash calls, ${HEAD_SHA:-…} won't rescue it either — HEAD_SHA from the review skill's step 1 is gone by the time this block runs. The derivation has to be spelled out per session type.

Verified from inside this review run

git rev-parse HEAD here is 4897ea376798028919212298b4138991f0df896f, a merge of the PR head 8582f53499b8bbce01391f91c237e50a53cfdacd into 4672f809cedd2f206e109761072641798aa0184e. Running the PR's own rollup() query against each:

--- 4897ea37 (git rev-parse HEAD) ---
statusCheckRollup: null, 0 contexts   ->  {"pending":0,"failed":[]}
--- 8582f534 (PR head) ---
5 contexts: review IN_PROGRESS, test/test-proxy/test-worker/lint SUCCESS

The merge commit's {"pending":0,"failed":[]} is indistinguishable from a settled-green result, so nothing downstream can tell the gate never ran.

Smaller, both inline: STARTUP_FAILURE and ACTION_REQUIRED are terminal CheckConclusionState values that land in neither bucket (not pending, not failed), so a job that fails to start reads as green; and the cap-expiry signal is gone — the old block ended echo "CI still running after 9 minutes"; exit 1, while the new one falls through to the same echo "$R" whether the loop settled or timed out, always exiting 0. Numbered item 4 still asks the run to report still-pending checks as unverified, so it's worth keeping something that says which case it's in.

Not blocking, and leaving review's pre-APPROVE peek to #821 reads right — it's still open and touching that line.

[tend-agent]

The three fixes from the last round hold up: the two-derivation PINNED_SHA block is applied verbatim, STARTUP_FAILURE/ACTION_REQUIRED are in the live CheckConclusionState enum and belong in the failed bucket, and the three-way outcome line is a better answer than the suggestion it replaced — [ "$PENDING" -gt 0 ] alone really would have fired on the break-on-red path. Resolving those threads.

Two things left, both about the loop reporting a state it didn't observe.

The break-on-red exit makes review's settled-rollup path unreachable. That path names this loop as the mechanism for waiting out a mid-flight FAILURE, and #821 adds a second branch that presumes the loop can reach its cap with a FAILURE present. The early break returns after one iteration instead. Inline on the break line, with the tradeoff.

The else branch reports green when the rollup query returned nothing at all. Inline, with the reproduction.

Both are inline suggestions. No approval — self-authored.

[tend-agent]

The break-on-red removal is right, and the comment above it now says why the loop has to outlast a mid-flight FAILURE — review's settled-rollup path is reachable again. Resolving that thread.

The empty-$R fix is the half of the false-green it can reach, but the guard it added can't see the larger half. rollup()'s // [] turns every "this response carries no rollup" shape into {"pending":0,"failed":[]} — byte-identical to a settled green, and non-empty, so [ -z "$R" ] never fires. Three ways in, all verified against the live API from this run: a GraphQL errors envelope (rate limit, transient server error — gh writes the body to stdout and jq happily reduces it), an OID GraphQL can't resolve, and a commit whose checks haven't registered yet. Each one breaks the loop on iteration 1 and prints green on $PINNED_SHA for a rollup the run never received. That's the PR's own thesis — reporting a state you didn't observe — surviving in the branch meant to catch it.

The second thread stays open rather than resolved, because the truly-empty case isn't fully closed either: the in-loop [ "$PENDING" -gt 0 ] / -eq 0 tests still error on an empty string, so neither continue nor break fires and each iteration costs 90 s instead of 60 s. Nine of those is 810 s against the 600 s Bash cap this block is explicitly sized for — the call is killed mid-loop and the verdict block never runs, so a persistent API failure still ends with no verdict at all rather than "unverified, not green".

Three inline suggestions; together they route all of it into the guard you already wrote. No approval — self-authored.

Verified: the three no-rollup shapes are indistinguishable from green

Live responses, this run:

$ gh api graphql -f query='... object(oid:"deadbeef…") { ... statusCheckRollup … }'
{"data":{"repository":{"object":null}}}                       # exit 0

$ gh api graphql -f query='query { nonexistentField }'
{"errors":[{"message":"Field 'nonexistentField' doesn't exist on type 'Query'"…}]}   # exit 1, body on stdout

statusCheckRollup is schema-nullable ({"name":"statusCheckRollup","type":{"kind":"OBJECT","name":"StatusCheckRollup"}} — no NON_NULL wrapper), so a commit with no checks yet is the same shape.

Each fed through the PR's own jq filter:

errors envelope     -> {"pending":0,"failed":[]}
unresolvable OID    -> {"pending":0,"failed":[]}
statusCheckRollup:null -> {"pending":0,"failed":[]}
settled green       -> {"pending":0,"failed":[]}     <- same bytes

With the suggestions applied, simulating the loop against a stubbed rollup():

persistent API failure          calls=9  R=[]  -> no rollup returned — unverified, not green
settled green                   calls=2        -> green
settled red                     calls=2        -> red
pending -> green                calls=3        -> green
red while pending -> green      calls=3        -> green      (stale cascade waited out)
one blip -> green               calls=3        -> green
never settles                   calls=9        -> cap hit — unverified

The failure case is 9 x 60 s = 540 s, inside the 600 s cap, so the verdict block still prints.

[tend-agent]

The // empty rewrite and both [ -z "$R" ] guards close the no-rollup half — I re-ran the three shapes through the committed filter (statusCheckRollup: null, object: null, errors envelope) and all three now emit nothing, so they route to no rollup returned … unverified, not green instead of the byte-identical green. Resolving those threads.

One finding in the new code, and it's the mirror of this PR's own thesis rather than a repeat of it. The outcome line tests FAILED before PENDING. That ordering was right when it was written in e6362ee, because the loop could still break on red with checks in flight and the disposition there was "go diagnose". 0ca2c53 removed that break — so FAILED > 0 alongside PENDING > 0 is now reachable only on cap expiry, which is precisely the shape the comment three lines above says not to judge ("a FAILURE alongside in-flight checks is often a stale cancellation-cascade artifact … Judge the settled rollup"). A rollup that never settled now prints red on $PINNED_SHA — diagnose the failures above, indistinguishable from a settled red.

That lands on the gate review delegates here. Its pre-APPROVE path routes $FAILED set + $PENDING > 0 back through this loop specifically to wait the cascade out; if the replacement matrix outlasts the 9-iteration cap, the caller reads "red", skips the approve or dismisses one — for a commit whose checks are still running and will likely go green. Item 4 of the numbered list ("if the cap hits with checks still running, comment the still-pending checks as unverified") also has nothing left to fire on, since the red branch has already claimed the case.

Swapping the two tests is safe: the loop only breaks when PENDING is 0, so a settled red always has PENDING == 0 and still reports red. Inline suggestion below. No approval — self-authored.

Simulated against a stubbed rollup(), sleeps counted rather than slept

Committed ordering:

stale cascade: FAILURE + in-flight, never settles    -> red                    (540s)
plain never-settles, no failure                      -> cap hit / unverified   (540s)
settled red                                          -> red                    (90s)
settled green                                        -> green                  (90s)
persistent API failure                               -> no rollup / unverified (540s)

With PENDING tested first:

stale cascade: FAILURE + in-flight, never settles    -> cap hit / unverified   (540s)
plain never-settles, no failure                      -> cap hit / unverified   (540s)
settled red                                          -> red                    (90s)
settled green                                        -> green                  (90s)
persistent API failure                               -> no rollup / unverified (540s)

Only the first row moves; the failing check names and run URLs are still printed by the echo "$R" immediately above, so nothing is lost from the cap-hit report.

The // empty check, same run, each shape fed through the committed filter:

statusCheckRollup: null   -> (no output)
object: null              -> (no output)
errors envelope           -> (no output)
empty nodes array         -> {"pending":0,"failed":[]}
mixed real nodes          -> {"pending":1,"failed":["lint …/runs/222/job/1","never-started …/runs/444/job/4"]}

The last row also confirms STARTUP_FAILURE still buckets as failed, CANCELLED stays excluded, and both the own-run and same-workflow filters hold.

[tend-agent]

The ordering swap in 9c87a82 is right, and it's the last of the four rounds' findings — every thread on this branch is now resolved.

One new finding, and it arrived with the base merge rather than with your commit. c5b7977 brought #821 in, which established that a concurrency-cancelled run leaves both check runs on the commit, so a rollup has to be reduced to the latest entry per check name and workflow before it is read. review's pre-APPROVE block now does that reduction; the loop in this PR — the one review delegates to for exactly that case — does not.

It lands harder here than it did there, because this PR is what introduces the failed bucket and the red on $PINNED_SHA verdict. A superseded FAILURE never leaves the commit, so once the replacement run settles the loop breaks with PENDING == 0, reads the stale red out of failed, and prints red on $PINNED_SHA — diagnose the failures above. Step 6's follow-up then dismisses an approval over a check whose replacement went green — the PR's own thesis, reporting a state you didn't observe, reaching the one gate review hands to this loop.

Two inline suggestions. startedAt / createdAt have to be in the selection set before anything can sort on them, then the same latest-per-(name, workflow) reduction #821 uses. Keying on the workflow name as well as the check name matters for the same reason it does there: two workflows can register the same check name, and collapsing those would hide a genuine red behind an unrelated green.

No approval — self-authored.

Verified: the OID-pinned query returns duplicates un-collapsed; the reduction drops only the superseded entry

GitHub does not collapse same-named check runs in statusCheckRollup.contexts. This PR's own commits carry live examples — e6362ee returns 17 contexts of which four are handle / tend-mention, three are relay, three are verify:

--- e6362eea2313065817a85cb215c29b4db59d3acb ---
{"total":17,"dupes":[{"n":"handle","w":"tend-mention",...},{"n":"handle","w":"tend-mention",...},
                     {"n":"handle","w":"tend-mention",...},{"n":"handle","w":"tend-mention",...},
                     {"n":"relay","w":"tend-mention",...} x3, {"n":"verify","w":"tend-mention",...} x3]}

Those all conclude SKIPPED / SUCCESS, so they cost nothing today; the point is that the duplicates arrive. Fed a synthetic rollup where the superseded entry is the red one — check-ok-to-merge FAILURE at 10:00 from run 111, replaced by SUCCESS at 10:05 from run 222, plus a genuine check-ok-to-merge FAILURE from a different workflow, a CANCELLED/SUCCESS pair, a STARTUP_FAILURE, and a codecov/patch status context:

committed: {"pending":0,"failed":["check-ok-to-merge","lint","codecov/patch","check-ok-to-merge"]}
deduped:   {"pending":0,"failed":["check-ok-to-merge","codecov/patch","lint"]}

The committed filter reports the superseded red. The deduped one drops it, keeps the other workflow's genuine check-ok-to-merge red (which is what the workflowName half of the key buys), and leaves STARTUP_FAILURE, CANCELLED-excluded, and the status-context handling exactly as they were.

startedAt and createdAt were confirmed by introspection rather than assumed — CheckRun exposes startedAt (and completedAt), StatusContext exposes createdAt. startedAt is populated while a run is still going, not only at completion: the live rollup on this PR's head shows review IN_PROGRESS with startedAt: 2026-08-12T06:49:06Z, so an in-flight replacement still wins max_by against a settled predecessor.

With both suggestions applied, the block passes bash -n and runs live:

e6362eea (17 contexts, 10 of them duplicates) -> {"pending":0,"failed":[]}
c5b79778 (live head, this run in flight)      -> {"pending":1,"failed":[]}
06b57475 (base tip)                           -> {"pending":0,"failed":[]}

The // empty guard still fires — group_by sits after the array construction, so an unresolvable OID and an all-zeros OID both still emit nothing and route to no rollup returned … unverified, not green. pending is unaffected either way: a superseded check run is COMPLETED, so it was never in that bucket.

[tend-agent]

Both suggestions applied verbatim, and the reduced filter holds up when I run it rather than read it: fed a rollup carrying a superseded check-ok-to-merge FAILURE alongside its replacement's SUCCESS, plus a genuine same-named FAILURE from a second workflow, a codecov/patch status context, and a CANCELLED leg whose replacement is still QUEUED, it returns {"pending":1,"failed":["check-ok-to-merge …/runs/333/…","codecov/patch …"]} — superseded red dropped, the other workflow's red kept, and the queued replacement still holding the loop open. The startedAt-at-queue-time claim checks out against a live QUEUED check run too (started_at non-null while status is still queued), so the pending signal survives the reduction. Both threads resolved.

One inline note on the block's framing, not on its behaviour — a judgment call for you rather than a finding, and fine to land as-is.

No approval — self-authored.

[max-sixty]

Superseded by #965, which lands the same fix as a tested script rather than a markdown recipe: plugins/tend-ci-runner/scripts/poll-pr-checks.sh queries the rollup by commit OID, so a concurrent push can no longer make the poll report another commit's green as this run's. #829 closed with it.

The edge cases this PR's review rounds surfaced are carried over into the script and pinned by pytest: null rollups on ephemeral merge-ref commits, superseded check runs from a cancellation cascade, and the STARTUP_FAILURE / ACTION_REQUIRED conclusions that fall into neither the pending nor the failed bucket if forgotten. Cases the recipe didn't reach are covered too — a commit with more contexts than the query's one 100-node page now routes to UNVERIFIED instead of reading green.

Closing rather than rebasing: the recipe this branch edits no longer exists.

This was written by Claude Code on behalf of max-sixty