Five verification and surface gaps: OOM outcomes, the judge-alone rate, diff coverage, an answer key, and the goal loop over the API - #1331
Conversation
…ff coverage (#1294, #1295, #1291) Three verification-ladder gaps, each closed the same way: state what was observed, never let the absence of an observation read as a finding. **#1294 — an out-of-memory kill is its own outcome.** `stella_pipeline::oom` classifies a run the machine killed for memory (SIGKILL, exit 137, or a runtime's own allocation-failure message on a run that failed); `CmdKind::OutOfMemory` carries it, `assertion_result()` reports it as no observation at all, and the label a judge reads is `out_of_memory` rather than `infra_failure`. Every test run in the crate now goes through one retrying entry point (`run_test_observed`, `test_oom_retries`, one retry by default) — retry rather than revise, because the run observed nothing and telling a worker its work failed asks it to "fix" code no test ever judged. The kernel log is deliberately not read; the module says why. **#1295 — the judge-alone rate is measurable, and the send-back is gated.** `stella calibration` now reports how often a verdict had no flip and no green test behind it, folded off ladder snapshots already persisted. The "send it back for evidence" behaviour ships behind `require_evidence_for_lone_judge_pass` (settings: `pipeline_evidence_for_lone_judge_pass`), off by default with the Terminal-Bench measurement recorded beside it. On, it spends at most one revision and still lands on the same UNVERIFIED relabel — asking for a check never becomes a way to fail a run for not having one. **#1291 — a passing test that never ran the changed lines is unproven.** A `CoverageProbe` port runs the tracked command under `cargo llvm-cov` or `pytest --cov` in the pre-submit audit; `verify::coverage` intersects the executed lines with the diff's added ones. A measured non-overlap withholds the deterministic credit and escalates — never a failure, never a deterministic red. `unmeasured` is stated in the verdict and on the wire (`LadderSnapshot::diff_coverage`) without withholding, because a gate that fires on every workspace without coverage tooling is a tax rather than a check; `require_diff_coverage` turns the strict reading on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
… a session ends (#1293) The judge is measured badly lenient, and every threshold built on it is a guess until something independent says which verdicts were wrong. #871 built the first source — a terminal CI verdict in the same session stream — and left two gaps. Both close here. **Reverts are now a source.** `replay::ground_truth::parse_revert_targets` reads git's own generated `This reverts commit <sha>` marker (not the retypeable `Revert "…"` subject), and `stella calibration` gathers it from the workspace history. A revert is counted apart from red CI: CI failing can be a flake or a neighbouring change, while a human reverting is a person deciding later, with more information, that the work should not have landed. **Reconciliation outlives the session.** `calibration_pending` carries a stream's unsettled passes out of the fold with the commits and PRs they cover; `ground_truth::reconcile` settles them against evidence keyed on those artifacts, so a terminal verdict recorded in another session — or a revert that lands weeks later — still reaches the judgement it contradicts. The asymmetry is deliberate and enforced by test: a revert settles a pass as a false positive, while the ABSENCE of a revert confirms nothing and leaves the pass unreconciled and out of every denominator. Counting un-reverted commits as confirmations would improve every measured rate by construction, which is the one direction a calibration instrument must never drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
The two capabilities an agent-app host is most likely to be paying for
existed only on the command line — not restricted, not disabled, simply
unrequestable over the wire. Both are now blocks on the turn request, on
`/v1/turns` and `/v1/sessions/{id}/turns` alike.
**`goal` — judged multi-round runs.** `src/goal.rs` drives rounds as a loop
over this crate's own step driver, exactly as `drive_turn` is a loop over
`run_step`, so a round is cancellable *between* reverse requests and reaches
the checkpoint seam at every step boundary. The three questions the issue
asked, answered: progress arrives on the SSE stream that already exists (one
`goal_verdict` event per round — nothing new to poll); stopping is the
existing cancel, which unwinds the round in flight and keeps the completed
rounds' work; and an unmet run ends `aborted` with a reason naming which
backstop stopped it, never a silent success.
A mode on a turn rather than a `/v1/goals` resource, which is the second
shape `stella-parity`'s own row named: every transport concern a goal run has
is a turn concern that already exists and is already witnessed, and a
parallel resource would restate each with its own id space and its own bugs.
**`sub_agents` — a turn that can delegate.** `src/subagents.rs` layers the
`task` tool over the host's remoted stack and runs children on the same
reverse-RPC ports, so containment is unchanged: the sidecar still executes
nothing. A child cannot write (`ReadOnlyTools`) and cannot spawn a child
(`task` is not in its view), both structurally rather than by prompt. The
caller sets pool, per-child steps and provider; `ServeConfig::sub_agents`
bounds all three and is **off by default**, because children spend money on
the host's account and that must not be reachable from a request body alone.
`provider_request` gains `provider_id` and `role` — additive, ignorable by a
single-model host — because a turn stopped meaning one model. Without them a
host cannot route a judge to a different family, which is the entire point of
an independent judge. The same change made per-provider request-id namespaces
necessary: a worker and a child both minting `prov-0` into one `Pending`
registry would collide, so each provider carries an instance tag.
Both rows in `stella-parity` move from Deferred to Shipped with named
witnesses; `docs/design/serve-surface.md` documents the shapes with a worked
example.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Reviewer's GuideImplements OOM-aware test retry and classification, a diff-coverage audit, judge-alone rate measurement with late reconciliation via reverts/CI, and exposes goal-loop + sub-agent capabilities on the HTTP API, with corresponding config, settings, wire, and test updates. Sequence diagram for OOM-aware test runs with diff-coverage and judge-alone gatingsequenceDiagram
participant Pipeline
participant TestRunner
participant CmdOutcome
participant CoverageProbe
participant Coverage
participant Ladder
Pipeline->>TestRunner: run_test_observed(invocation)
TestRunner-->>Pipeline: CmdOutcome
Pipeline->>CmdOutcome: is_out_of_memory()
alt outcome is OutOfMemory
Pipeline->>TestRunner: run_test(invocation) // retry
TestRunner-->>Pipeline: CmdOutcome
end
Pipeline->>CmdOutcome: assertion_result()
alt Some(passed)
Pipeline->>CoverageProbe: covered_lines(cwd, invocation)
CoverageProbe-->>Pipeline: Option<CoverageReport>
Pipeline->>Coverage: overlap(changed_lines, report)
Coverage-->>Pipeline: DiffCoverage
Pipeline->>Ladder: ladder_decision(LadderInputs{diff_coverage, require_diff_coverage})
alt LadderDecision::SubmitFast
Pipeline->>Pipeline: deterministic_pass_evidence(tracked_cmd, diff_lines, diff_coverage)
else LadderDecision::SubmitJudge
Pipeline->>Pipeline: judge_pass_stands_alone()
alt require_evidence_for_lone_judge_pass && revisions < max_revisions
Pipeline->>Pipeline: revise_candidate(... LONE_JUDGE_PASS_EVIDENCE_REQUEST ...)
else
Pipeline->>Pipeline: relabel UNVERIFIED
end
end
else None (timeout/oom/infra)
Pipeline->>Pipeline: handle non-observation (no assertion_result)
end
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Sorry @macanderson, your pull request is larger than the review limit of 150000 diff characters
…rified (#1291) The first cut stated `diff_coverage=unmeasured` in the verdict but still credited a `DeterministicPass`, which the issue's fourth point rejects: "the honest answer is also 'unproven' rather than silently passing". It was raised as a deliberate deviation and is now closed the way the issue asks. What changes is the CLAIM, not the cost. An unmeasured overlap still takes the fast-submit — no judge call, no extra turn, the run completes exactly as before — but it scores `Unverified` rather than `DeterministicPass`, and the verdict summary now LEADS with `UNPROVEN` instead of appending the status at the end of a sentence. Paying for honesty with a ranking position instead of a model call is what makes it affordable to be honest by default, in every workspace, rather than only where coverage tooling happens to be installed. Escalating instead stays available behind `require_diff_coverage`, whose docs now say what it adds (a reviewer) rather than what it fixes (the claim, which is unconditional). `evidence.deterministic` deliberately stays `true`: the flip WAS a real test observation, and the calibration cohorts (#871) partition by evidence kind, not by coverage. Only the score moves. `PipelineOutcome::score` is surfaced so the distinction is legible to a host — a caller reading `verdict.deterministic` alone would see a pass and nothing else, which is the silence the issue objects to. Two witnesses pin both directions: an unmeasured overlap scores `Unverified` while spending no judge call, and a measured overlap still earns the deterministic badge in full (without which the downgrade would be indistinguishable from "never score a deterministic pass again"). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
|
Pushed An unmeasurable coverage overlap now scores
Paying for honesty with a ranking position instead of a model call is what makes it affordable by default in every workspace, rather than only where coverage tooling happens to be installed. Escalating stays available behind
Two witnesses pin both directions:
On the other ❌ — #1293's "drive threshold tuning from measured data" — that is deliberately not in scope, and the issue says so itself: "Once there is real data, tune the thresholds against it — not before." This PR builds the instrument that produces the data (reverts as ground truth, cross-session reconciliation); there is none yet to tune against. Recorded in Gate is green on the new commit: One note on CI: no Generated by Claude Code |
What & why
Five issues, three commits. Four of them share one theme — the absence of an observation must never read as a finding — and the fifth closes the last two CLI-only capabilities on the API surface.
Closes #1294
Closes #1295
Closes #1291
Closes #1293
Closes #1297
#1294 — tell "out of memory" apart from "the test failed"
They mean opposite things: a failing test says fix the code, an OOM kill says nothing was learned.
stella_pipeline::oomclassifies a memory kill from the three observable traces (SIGKILL, exit 137, or a runtime's own allocation-failure message on a run that failed — a passing run is never an OOM whatever it printed).CmdKind::OutOfMemorycarries it as its own outcome,assertion_result()reports no observation, and the label a judge reads isout_of_memory, neverinfra_failure.Every test run in the pipeline now goes through one retrying entry point (
run_test_observed,test_oom_retries, one retry by default) — retry rather than revise, because re-running is what a human does by hand and it is the only response that can produce the observation the pipeline wanted. It also protects the pre-submit confirmation run, where an OOM used to silently demote a real flip toUnstable.The kernel log is deliberately not read; the module documents why attribution there is unsafe (restricted
dmesg, no namespacing, kills belonging to a grandchild the runner never knew about).#1295 — measure the judge-alone rate first, then decide
The "send it back for more evidence" behaviour was built and measured off.
stella calibrationnow reports the current rate, folded off ladder snapshots already persisted (no new write path): how often a verdict had no flip and no green test behind it, with its denominator, plus how many of those were model-judge PASSes.The behaviour ships behind
require_evidence_for_lone_judge_pass(settings keypipeline_evidence_for_lone_judge_pass), off by default with the Terminal-Bench measurement recorded beside it. Switched on it spends at most one revision and still lands on the same UNVERIFIED relabel — asking for a check never becomes a way to fail a run for not having one.#1291 — check the passing test ran the changed lines
A
CoverageProbeport runs the tracked command under the workspace's own coverage tool in the pre-submit audit (cargo llvm-cov→ LCOV,pytest --cov→ coverage.py JSON — the two dialectsverify::fingerprintcan already read test output for; everything else is explicitly unmeasured, never guessed).verify::coverageintersects the executed lines with the diff's added ones.Three-valued by design. A measured non-overlap withholds the deterministic credit and escalates — unproven, never a failure and never a deterministic red.
unmeasuredis stated in the verdict and on the wire (LadderSnapshot::diff_coverage) without withholding.#1293 — an answer key for the judge
Two gaps in #871's first slice. Reverts are now a ground-truth source:
replay::ground_truth::parse_revert_targetsreads git's own generatedThis reverts commit <sha>marker (not the retypeableRevert "…"subject), counted apart from red CI because a human reverting is a later and better-informed statement than a CI run. Reconciliation outlives the session:calibration_pendingcarries a stream's unsettled passes out of the fold with the commits and PRs they cover, andground_truth::reconcilesettles them against evidence keyed on those artifacts — another session's stream, or the git history.The asymmetry is deliberate and test-enforced: a revert settles a pass as a false positive, while the absence of one confirms nothing and leaves the pass out of every denominator. Counting un-reverted commits as confirmations would improve every measured rate by construction.
#1297 — the goal loop and sub-agents over the API
goalandsub_agentsblocks onPOST /v1/turns(and/v1/sessions/{id}/turns).goal_verdictevent per round on the SSE endpoint that already exists. Nothing new to poll.abortedwith its real cost.taskover the host's remoted stack; children run on the same reverse-RPC ports (containment unchanged), cannot write, and cannot spawn a child — both structurally. The caller sets pool / steps / provider;ServeConfig::sub_agentsbounds all three and is off by default.provider_requestgainsprovider_idandrole(additive, ignorable by a single-model host) so a judge or a child is addressable as a different model — the property the goal loop exists for and the one the engine cannot enforce, since the host owns the calls.A mode on a turn rather than a
/v1/goalsresource — the second shapestella-parity's own row named. Both matrix rows move Deferred → Shipped with named witnesses.The witness
main, passes here)Each issue has acceptance tests that fail without its change:
an_out_of_memory_test_run_is_retried_instead_of_revised,a_persistent_memory_kill_is_never_a_deterministic_test_failure,an_out_of_memory_kill_is_its_own_outcome, plus 5 classifier teststhe_judge_alone_rate_is_measured_from_recorded_snapshots,a_lone_judge_pass_is_relabelled_unverified_by_default,requesting_evidence_costs_a_revision_and_still_never_fails_the_runa_flip_whose_test_never_ran_the_changed_lines_is_unproven_not_failed,an_unmeasurable_overlap_is_stated_and_only_gates_when_asked, plus 6 pure overlap/parser tests and 2 host-probe testsa_pass_after_the_last_ci_verdict_survives_its_session_and_a_revert_settles_it,a_terminal_verdict_in_a_later_session_settles_an_earlier_ones_pass,a_revert_outranks_a_green_ci_verdict,the_in_stream_fold_is_unchangeda_goal_run_is_requestable_over_the_wire_and_streams_its_rounds,an_unmet_goal_stops_at_the_round_cap_with_a_named_reason,a_served_turn_can_delegate_to_a_sub_agent,a_deployment_that_has_not_opted_in_advertises_no_task_toolThe gate
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspacecargo doc --workspace --no-deps(no warnings)check-file-size(baseline ratchet updated, diff in the PR),check-doc-citations,check-invariants,check-wire-schema(docs/wire/regenerated)ROADMAP.md§1/§2/§4,docs/design/serve-surface.md(route table + a worked example),stella-serve/README.md, and settings-key doc commentsGround-rule check
stella-core; no new dependencies at allLadderSnapshot::diff_coverageand the two newprovider_requestfields are additive (serde(default), absent-when-none) and covered by the existing round-trip and wire-contract testsAnything reviewers should know?
unmeasuredcoverage default is the one place I did not take the issue literally — reasoning and the escape hatch are above. It is a one-line default flip if you disagree.prov-0into onePendingregistry — unreachable before a turn could hold two providers, guaranteed after); and a session thread that hung after its finalCompleteevent because the sub-agent dispatcher held a clone of the event sender past the turn. The second is whyrun_sessionnow drops engine → tool view → dispatcher in that order before closing the channel.calibration_pendingis deliberately generous: a session with two passes and one commit attributes that commit's fate to both. Sessions that adopt one change are the ordinary case, and over-attributing a revert errs toward reporting the judge as worse than it is — the safe direction for an instrument whose measured failure is leniency. Documented at the function.Generated by Claude Code
Summary by Sourcery
Add OOM-aware test retry and classification, introduce diff coverage instrumentation and controls, expose goal-loop and sub-agent capabilities over the serve API, and extend calibration to use snapshots, late CI, and git reverts for judge metrics.
Bug Fixes:
Enhancements:
Documentation:
Tests: