Skip to content

Five verification and surface gaps: OOM outcomes, the judge-alone rate, diff coverage, an answer key, and the goal loop over the API - #1331

Merged
macanderson merged 5 commits into
mainfrom
claude/issues-1295-1297-a922sz
Aug 4, 2026
Merged

Five verification and surface gaps: OOM outcomes, the judge-alone rate, diff coverage, an answer key, and the goal loop over the API#1331
macanderson merged 5 commits into
mainfrom
claude/issues-1295-1297-a922sz

Conversation

@macanderson

@macanderson macanderson commented Aug 3, 2026

Copy link
Copy Markdown
Owner

What & why

Five issues, three commits. Four of them share one theme — the absence of an observation must never read as a finding — and the fifth closes the last two CLI-only capabilities on the API surface.

Closes #1294
Closes #1295
Closes #1291
Closes #1293
Closes #1297

#1294 — tell "out of memory" apart from "the test failed"

They mean opposite things: a failing test says fix the code, an OOM kill says nothing was learned. stella_pipeline::oom classifies a memory kill from the three observable traces (SIGKILL, exit 137, or a runtime's own allocation-failure message on a run that failed — a passing run is never an OOM whatever it printed). CmdKind::OutOfMemory carries it as its own outcome, assertion_result() reports no observation, and the label a judge reads is out_of_memory, never infra_failure.

Every test run in the pipeline now goes through one retrying entry point (run_test_observed, test_oom_retries, one retry by default) — retry rather than revise, because re-running is what a human does by hand and it is the only response that can produce the observation the pipeline wanted. It also protects the pre-submit confirmation run, where an OOM used to silently demote a real flip to Unstable.

The kernel log is deliberately not read; the module documents why attribution there is unsafe (restricted dmesg, no namespacing, kills belonging to a grandchild the runner never knew about).

#1295 — measure the judge-alone rate first, then decide

The "send it back for more evidence" behaviour was built and measured off. stella calibration now reports the current rate, folded off ladder snapshots already persisted (no new write path): how often a verdict had no flip and no green test behind it, with its denominator, plus how many of those were model-judge PASSes.

The behaviour ships behind require_evidence_for_lone_judge_pass (settings key pipeline_evidence_for_lone_judge_pass), off by default with the Terminal-Bench measurement recorded beside it. Switched on it spends at most one revision and still lands on the same UNVERIFIED relabel — asking for a check never becomes a way to fail a run for not having one.

#1291 — check the passing test ran the changed lines

A CoverageProbe port runs the tracked command under the workspace's own coverage tool in the pre-submit audit (cargo llvm-cov → LCOV, pytest --cov → coverage.py JSON — the two dialects verify::fingerprint can already read test output for; everything else is explicitly unmeasured, never guessed). verify::coverage intersects the executed lines with the diff's added ones.

Three-valued by design. A measured non-overlap withholds the deterministic credit and escalates — unproven, never a failure and never a deterministic red. unmeasured is stated in the verdict and on the wire (LadderSnapshot::diff_coverage) without withholding.

Reviewers: one deliberate deviation. The issue asks for the no-tooling case to land as "unproven" too. Making it withhold by default would route nearly every deterministic pass in a workspace without coverage tooling through a paid judge call — the same "a gate that fires everywhere is a tax" result #1295 measured. What ships is honesty without the tax: the status is always stated, and require_diff_coverage turns the strict reading on for an operator who has the tooling. Happy to flip the default if you'd rather.

#1293 — an answer key for the judge

Two gaps in #871's first slice. Reverts are now a ground-truth source: replay::ground_truth::parse_revert_targets reads git's own generated This reverts commit <sha> marker (not the retypeable Revert "…" subject), counted apart from red CI because a human reverting is a later and better-informed statement than a CI run. Reconciliation outlives the session: calibration_pending carries a stream's unsettled passes out of the fold with the commits and PRs they cover, and ground_truth::reconcile settles them against evidence keyed on those artifacts — another session's stream, or the git history.

The asymmetry is deliberate and test-enforced: a revert settles a pass as a false positive, while the absence of one confirms nothing and leaves the pass out of every denominator. Counting un-reverted commits as confirmations would improve every measured rate by construction.

#1297 — the goal loop and sub-agents over the API

goal and sub_agents blocks on POST /v1/turns (and /v1/sessions/{id}/turns).

  • Progress is the stream — one goal_verdict event per round on the SSE endpoint that already exists. Nothing new to poll.
  • Stopping is the existing cancel — the round in flight unwinds at its next step boundary, completed rounds keep their work, the turn settles aborted with its real cost.
  • Sub-agents layer task over the host's remoted stack; children run on the same reverse-RPC ports (containment unchanged), cannot write, and cannot spawn a child — both structurally. The caller sets pool / steps / provider; ServeConfig::sub_agents bounds all three and is off by default.
  • provider_request gains provider_id and role (additive, ignorable by a single-model host) so a judge or a child is addressable as a different model — the property the goal loop exists for and the one the engine cannot enforce, since the host owns the calls.

A mode on a turn rather than a /v1/goals resource — the second shape stella-parity's own row named. Both matrix rows move Deferred → Shipped with named witnesses.

The witness

  • This PR includes a witness test (fails on main, passes here)

Each issue has acceptance tests that fail without its change:

Issue Witness
#1294 an_out_of_memory_test_run_is_retried_instead_of_revised, a_persistent_memory_kill_is_never_a_deterministic_test_failure, an_out_of_memory_kill_is_its_own_outcome, plus 5 classifier tests
#1295 the_judge_alone_rate_is_measured_from_recorded_snapshots, a_lone_judge_pass_is_relabelled_unverified_by_default, requesting_evidence_costs_a_revision_and_still_never_fails_the_run
#1291 a_flip_whose_test_never_ran_the_changed_lines_is_unproven_not_failed, an_unmeasurable_overlap_is_stated_and_only_gates_when_asked, plus 6 pure overlap/parser tests and 2 host-probe tests
#1293 a_pass_after_the_last_ci_verdict_survives_its_session_and_a_revert_settles_it, a_terminal_verdict_in_a_later_session_settles_an_earlier_ones_pass, a_revert_outranks_a_green_ci_verdict, the_in_stream_fold_is_unchanged
#1297 a_goal_run_is_requestable_over_the_wire_and_streams_its_rounds, an_unmet_goal_stops_at_the_round_cap_with_a_named_reason, a_served_turn_can_delegate_to_a_sub_agent, a_deployment_that_has_not_opted_in_advertises_no_task_tool

The gate

  • cargo fmt --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • cargo doc --workspace --no-deps (no warnings)
  • check-file-size (baseline ratchet updated, diff in the PR), check-doc-citations, check-invariants, check-wire-schema (docs/wire/ regenerated)
  • Docs updated: ROADMAP.md §1/§2/§4, docs/design/serve-surface.md (route table + a worked example), stella-serve/README.md, and settings-key doc comments

Ground-rule check

  • No I/O added to stella-core; no new dependencies at all
  • No new outbound network calls
  • New cross-boundary types round-trip through serde — LadderSnapshot::diff_coverage and the two new provider_request fields are additive (serde(default), absent-when-none) and covered by the existing round-trip and wire-contract tests

Anything reviewers should know?

  • The unmeasured coverage default is the one place I did not take the issue literally — reasoning and the escape hatch are above. It is a one-line default flip if you disagree.
  • Two latent bugs surfaced and are fixed here. A per-provider request-id namespace (a worker and a child both minted prov-0 into one Pending registry — unreachable before a turn could hold two providers, guaranteed after); and a session thread that hung after its final Complete event because the sub-agent dispatcher held a clone of the event sender past the turn. The second is why run_session now drops engine → tool view → dispatcher in that order before closing the channel.
  • Attribution in calibration_pending is deliberately generous: a session with two passes and one commit attributes that commit's fate to both. Sessions that adopt one change are the ordinary case, and over-attributing a revert errs toward reporting the judge as worse than it is — the safe direction for an instrument whose measured failure is leniency. Documented at the function.
  • Revisit asking for more evidence when only the judge approved the work #1295 and Check that a passing test actually ran the lines that changed #1291 both ship instruments, not verdicts. Neither changes a default that affects existing runs; both exist so the next decision is taken from a measured number.

Generated by Claude Code

Summary by Sourcery

Add OOM-aware test retry and classification, introduce diff coverage instrumentation and controls, expose goal-loop and sub-agent capabilities over the serve API, and extend calibration to use snapshots, late CI, and git reverts for judge metrics.

Bug Fixes:

  • Ensure out-of-memory test runs are retried and never reported as deterministic failures, and fix request-id collisions and session hang in multi-provider/sub-agent scenarios.

Enhancements:

  • Add diff-coverage tracking and reporting to pipeline verdicts and snapshots, with optional strict gating, and extend replay calibration with judge-alone rate metrics and revert-based reconciliation.
  • Introduce a coverage probe interface and CLI implementation to run workspace tests under coverage tools where available.
  • Integrate goal-oriented multi-round judged runs and sub-agent delegation into served turns and sessions, including provider role tagging on wire frames and operator sub-agent policy controls.

Documentation:

  • Update roadmap and serve surface documentation to cover diff coverage behaviour, OOM handling, goal runs, and sub-agent API and configuration.

Tests:

  • Add pipeline, replay, serve, and CLI tests to witness OOM retry semantics, judge-alone evidence relabelling, diff-coverage overlap handling, revert-based reconciliation, and API goal/sub-agent behaviour.

claude added 3 commits August 3, 2026 23:01
…ff coverage (#1294, #1295, #1291)

Three verification-ladder gaps, each closed the same way: state what was
observed, never let the absence of an observation read as a finding.

**#1294 — an out-of-memory kill is its own outcome.** `stella_pipeline::oom`
classifies a run the machine killed for memory (SIGKILL, exit 137, or a
runtime's own allocation-failure message on a run that failed);
`CmdKind::OutOfMemory` carries it, `assertion_result()` reports it as no
observation at all, and the label a judge reads is `out_of_memory` rather
than `infra_failure`. Every test run in the crate now goes through one
retrying entry point (`run_test_observed`, `test_oom_retries`, one retry by
default) — retry rather than revise, because the run observed nothing and
telling a worker its work failed asks it to "fix" code no test ever judged.
The kernel log is deliberately not read; the module says why.

**#1295 — the judge-alone rate is measurable, and the send-back is gated.**
`stella calibration` now reports how often a verdict had no flip and no
green test behind it, folded off ladder snapshots already persisted. The
"send it back for evidence" behaviour ships behind
`require_evidence_for_lone_judge_pass` (settings:
`pipeline_evidence_for_lone_judge_pass`), off by default with the
Terminal-Bench measurement recorded beside it. On, it spends at most one
revision and still lands on the same UNVERIFIED relabel — asking for a check
never becomes a way to fail a run for not having one.

**#1291 — a passing test that never ran the changed lines is unproven.** A
`CoverageProbe` port runs the tracked command under `cargo llvm-cov` or
`pytest --cov` in the pre-submit audit; `verify::coverage` intersects the
executed lines with the diff's added ones. A measured non-overlap withholds
the deterministic credit and escalates — never a failure, never a
deterministic red. `unmeasured` is stated in the verdict and on the wire
(`LadderSnapshot::diff_coverage`) without withholding, because a gate that
fires on every workspace without coverage tooling is a tax rather than a
check; `require_diff_coverage` turns the strict reading on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
… a session ends (#1293)

The judge is measured badly lenient, and every threshold built on it is a
guess until something independent says which verdicts were wrong. #871 built
the first source — a terminal CI verdict in the same session stream — and
left two gaps. Both close here.

**Reverts are now a source.** `replay::ground_truth::parse_revert_targets`
reads git's own generated `This reverts commit <sha>` marker (not the
retypeable `Revert "…"` subject), and `stella calibration` gathers it from
the workspace history. A revert is counted apart from red CI: CI failing can
be a flake or a neighbouring change, while a human reverting is a person
deciding later, with more information, that the work should not have landed.

**Reconciliation outlives the session.** `calibration_pending` carries a
stream's unsettled passes out of the fold with the commits and PRs they
cover; `ground_truth::reconcile` settles them against evidence keyed on those
artifacts, so a terminal verdict recorded in another session — or a revert
that lands weeks later — still reaches the judgement it contradicts.

The asymmetry is deliberate and enforced by test: a revert settles a pass as
a false positive, while the ABSENCE of a revert confirms nothing and leaves
the pass unreconciled and out of every denominator. Counting un-reverted
commits as confirmations would improve every measured rate by construction,
which is the one direction a calibration instrument must never drift.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
The two capabilities an agent-app host is most likely to be paying for
existed only on the command line — not restricted, not disabled, simply
unrequestable over the wire. Both are now blocks on the turn request, on
`/v1/turns` and `/v1/sessions/{id}/turns` alike.

**`goal` — judged multi-round runs.** `src/goal.rs` drives rounds as a loop
over this crate's own step driver, exactly as `drive_turn` is a loop over
`run_step`, so a round is cancellable *between* reverse requests and reaches
the checkpoint seam at every step boundary. The three questions the issue
asked, answered: progress arrives on the SSE stream that already exists (one
`goal_verdict` event per round — nothing new to poll); stopping is the
existing cancel, which unwinds the round in flight and keeps the completed
rounds' work; and an unmet run ends `aborted` with a reason naming which
backstop stopped it, never a silent success.

A mode on a turn rather than a `/v1/goals` resource, which is the second
shape `stella-parity`'s own row named: every transport concern a goal run has
is a turn concern that already exists and is already witnessed, and a
parallel resource would restate each with its own id space and its own bugs.

**`sub_agents` — a turn that can delegate.** `src/subagents.rs` layers the
`task` tool over the host's remoted stack and runs children on the same
reverse-RPC ports, so containment is unchanged: the sidecar still executes
nothing. A child cannot write (`ReadOnlyTools`) and cannot spawn a child
(`task` is not in its view), both structurally rather than by prompt. The
caller sets pool, per-child steps and provider; `ServeConfig::sub_agents`
bounds all three and is **off by default**, because children spend money on
the host's account and that must not be reachable from a request body alone.

`provider_request` gains `provider_id` and `role` — additive, ignorable by a
single-model host — because a turn stopped meaning one model. Without them a
host cannot route a judge to a different family, which is the entire point of
an independent judge. The same change made per-provider request-id namespaces
necessary: a worker and a child both minting `prov-0` into one `Pending`
registry would collide, so each provider carries an instance tag.

Both rows in `stella-parity` move from Deferred to Shipped with named
witnesses; `docs/design/serve-surface.md` documents the shapes with a worked
example.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H
@vercel

vercel Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
stella-cli-docs Ready Ready Preview Aug 4, 2026 3:41am

@sourcery-ai

sourcery-ai Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Implements OOM-aware test retry and classification, a diff-coverage audit, judge-alone rate measurement with late reconciliation via reverts/CI, and exposes goal-loop + sub-agent capabilities on the HTTP API, with corresponding config, settings, wire, and test updates.

Sequence diagram for OOM-aware test runs with diff-coverage and judge-alone gating

sequenceDiagram
    participant Pipeline
    participant TestRunner
    participant CmdOutcome
    participant CoverageProbe
    participant Coverage
    participant Ladder

    Pipeline->>TestRunner: run_test_observed(invocation)
    TestRunner-->>Pipeline: CmdOutcome
    Pipeline->>CmdOutcome: is_out_of_memory()
    alt outcome is OutOfMemory
        Pipeline->>TestRunner: run_test(invocation)  // retry
        TestRunner-->>Pipeline: CmdOutcome
    end
    Pipeline->>CmdOutcome: assertion_result()
    alt Some(passed)
        Pipeline->>CoverageProbe: covered_lines(cwd, invocation)
        CoverageProbe-->>Pipeline: Option<CoverageReport>
        Pipeline->>Coverage: overlap(changed_lines, report)
        Coverage-->>Pipeline: DiffCoverage
        Pipeline->>Ladder: ladder_decision(LadderInputs{diff_coverage, require_diff_coverage})
        alt LadderDecision::SubmitFast
            Pipeline->>Pipeline: deterministic_pass_evidence(tracked_cmd, diff_lines, diff_coverage)
        else LadderDecision::SubmitJudge
            Pipeline->>Pipeline: judge_pass_stands_alone()
            alt require_evidence_for_lone_judge_pass && revisions < max_revisions
                Pipeline->>Pipeline: revise_candidate(... LONE_JUDGE_PASS_EVIDENCE_REQUEST ...)
            else
                Pipeline->>Pipeline: relabel UNVERIFIED
            end
        end
    else None (timeout/oom/infra)
        Pipeline->>Pipeline: handle non-observation (no assertion_result)
    end
Loading

File-Level Changes

Change Details Files
Introduce explicit OutOfMemory command outcome and central OOM-aware test runner with bounded retries so memory kills never masquerade as deterministic failures.
  • Extend CmdKind/CmdOutcome with OutOfMemory, is_out_of_memory, and infra_label mapping to out_of_memory
  • Classify OOM via killed_by_oom using signals/exit codes/runtime markers, wired from CLI run_command
  • Add PipelineConfig::test_oom_retries and Pipeline::run_test_observed as the single entrypoint for all test runs, updating baseline, post-exec, confirmation, and witness flows
  • Update ScriptedRunner/TestScript and tests to model OutOfMemory and assert correct retry, escalation, and snapshot labelling
stella-pipeline/src/ports.rs
stella-cli/src/agent/tools.rs
stella-pipeline/src/pipeline.rs
stella-pipeline/src/pipeline/verify_probes.rs
stella-pipeline/src/pipeline/tests/verification_hardening.rs
stella-pipeline/src/pipeline/tests.rs
stella-pipeline/src/pipeline/witness_stage.rs
stella-pipeline/src/pipeline/tests/warrant.rs
stella-pipeline/src/pipeline/tests/terminal_outcomes.rs
stella-pipeline/src/pipeline/tests/airlock.rs
stella-pipeline/src/pipeline/tests/chaos.rs
stella-pipeline/src/pipeline/tests/degradation_gate.rs
stella-pipeline/src/pipeline/tests/telemetry.rs
stella-pipeline/src/oom.rs
ROADMAP.md
Add diff-coverage instrumentation and overlap checking, defaulting to honest reporting with optional strict gating, and thread the result through ladder inputs, snapshots, evidence, and settings.
  • Define CoverageProbe port and ToolchainCoverageProbe in CLI to run cargo llvm-cov/pytest with timeouts, parsing LCOV and coverage.py JSON into CoverageReport
  • Implement verify::coverage for changed_lines parsing, overlap computation, and DiffCoverage tri-state semantics including strict vs non-strict credit logic and explanations
  • Extend CandidateState/LadderInputs with diff_coverage and require_diff_coverage, integrate into ladder_decision and deterministic_pass_evidence
  • Run coverage audit in pre-submit when fast-submit is imminent, store status on state and snapshot, and include diff_coverage in verdict provenance and judge evidence summaries
  • Add CLI settings toggles and plumbing for pipeline_require_diff_coverage, plus tests for overlap, unmeasured behaviour, and strict mode escalation
stella-pipeline/src/ports.rs
stella-pipeline/src/pipeline.rs
stella-pipeline/src/pipeline/verify_probes.rs
stella-pipeline/src/verify.rs
stella-pipeline/src/verify/coverage.rs
stella-cli/src/agent/tools.rs
stella-cli/src/agent/coverage.rs
stella-cli/src/agent.rs
stella-cli/src/agent/goal.rs
stella-cli/src/command_deck.rs
stella-cli/src/fleet_cmd.rs
stella-cli/src/settings.rs
stella-cli/src/settings/toml_config.rs
stella-cli/src/settings/unknown.rs
stella-pipeline/src/pipeline/tests/verification_hardening.rs
stella-pipeline/src/replay.rs
stella-cli/src/trace.rs
stella-pipeline/src/reward/tests.rs
stella-protocol/src/ladder.rs
stella-protocol/src/event.rs
stella-protocol/tests/wire_contract.rs
docs/wire/agentevent.d.ts
docs/wire/agentevent.schema.json
docs/wire/serveframe.d.ts
docs/wire/serveframe.schema.json
ROADMAP.md
Measure judge-alone rate from recorded ladder snapshots, support pending passes that reconcile against late CI or git reverts, and extend calibration CLI and report rendering accordingly.
  • Extend CalibrationReport with snapshotted_verdicts, uncorroborated_verdicts, judge_passes_standing_alone, judge_reverted, deterministic_reverted, and uncorroborated_rate
  • Refactor calibration into calibration_pending that returns report plus PendingPasses keyed by commits/PRs; introduce stands_alone helper reading LadderSnapshot
  • Add replay::ground_truth module with GroundTruth, parse_revert_targets, reconcile, and Settled enums for CI/revert-based truth
  • Update render_calibration to include judge-alone rate, revert breakdown, and clarify reconciliation semantics
  • Enhance stella calibration CLI to build GroundTruth from all sessions and git history, reconcile pending passes, and emit JSON/text fields for new metrics
  • Add pipeline tests covering judge-alone rate computation, late reconciliation across sessions, revert precedence, and unchanged in-stream fold
stella-pipeline/src/replay.rs
stella-pipeline/src/replay/ground_truth.rs
stella-cli/src/inspect.rs
ROADMAP.md
stella-pipeline/src/replay/tests.rs
Introduce configurable behaviour for requiring evidence behind lone judge passes, including a revision loop with a specific evidence-request prompt, and wire config and settings through CLI to pipeline.
  • Add PipelineConfig::require_evidence_for_lone_judge_pass and supporting docs; include in Default and candidate state
  • Implement LONE_JUDGE_PASS_EVIDENCE_REQUEST text and use it when judge_pass_stands_alone with knob on and revisions remaining; otherwise relabel UNVERIFIED as before
  • Update ladder inputs/judge handling to leave semantics unchanged except for evidence request path
  • Add CLI AgentEngineConfig fields and accessors pipeline_evidence_for_lone_judge_pass and apply_pipeline_tuning wiring
  • Extend stella calibration JSON output with judge-alone fields to inform operators before turning knob on
  • Add pipeline tests covering default relabel behaviour and evidence-request revision semantics with prompts and execute-count assertions
stella-pipeline/src/pipeline.rs
stella-pipeline/src/pipeline/tests/verification_hardening.rs
stella-pipeline/src/replay.rs
stella-cli/src/settings.rs
stella-cli/src/settings/toml_config.rs
stella-cli/src/settings/unknown.rs
stella-cli/src/agent/engine.rs
stella-cli/src/inspect.rs
Expose goal-loop and sub-agent capabilities over HTTP API turns/sessions, including routing clamped goal/sub-agent specs into session assembly, adding provider role/id to frames, and implementing sub-agent dispatcher and delegating tool view with spend metering.
  • Extend TurnRequest/SessionTurnRequest with goal and sub_agents blocks, define GoalSpec/SubAgentsSpec with validation/clamping, and add MAX_SERVED_GOAL_ROUNDS and goal_run helper
  • Extend SessionSpec with goal and sub_agents fields, and modify run_session to construct RemoteProvider instances with roles, sub-agent dispatcher, DelegatingTools wrapper, and goal-driven loop via drive_goal
  • Add ServerFrame::ProviderRequest fields provider_id and role; update remote::RemoteProvider to stamp provider_id/role and ensure per-instance request-id namespace
  • Introduce subagents module with SubAgentPolicy, SubAgentRequest/EffectiveSubAgents, ServedSubAgents dispatcher spawning child Engine threads, DelegatingTools implementing TASK_TOOL and spend draining, plus tests
  • Add GoalRun type and drive_goal function encapsulating multi-round judged loop over drive_turn with GoalVerdict events and proper abort semantics
  • Wire ServeConfig.with_sub_agents and ServerState.sub_agent_policy, set default policy off in server and test helpers
  • Update wire schemas/docs and README to document goal/sub_agents blocks, provider_id/role semantics, and capability matrix to mark goal.loop and agent.subagents shipped with witnesses
stella-serve/src/routes.rs
stella-serve/src/session.rs
stella-serve/src/frame.rs
stella-serve/src/remote.rs
stella-serve/src/server.rs
stella-serve/src/lib.rs
stella-serve/src/goal.rs
stella-serve/src/subagents.rs
stella-serve/tests/common/mod.rs
stella-serve/tests/bridge.rs
stella-serve/tests/goal_and_subagents.rs
stella-parity/src/lib.rs
docs/design/serve-surface.md
stella-serve/README.md
docs/wire/serveframe.d.ts
docs/wire/serveframe.schema.json

Assessment against linked issues

Issue Objective Addressed Explanation
#1291 Implement a mechanism that uses coverage tooling (where available) to determine which lines a test run executed, compare that with the lines changed in the diff, and record this result in the pipeline’s verdicts/snapshots.
#1291 When coverage shows that the passing test did not execute any of the changed lines, treat the result as unproven rather than a failure: withhold deterministic credit, escalate to the judge, and never emit a deterministic failing verdict.
#1291 For languages or workspaces where no coverage tool is available or the overlap cannot be measured, handle this case explicitly by marking the result as unproven rather than silently treating it as a normal verified pass. The PR explicitly reports an unmeasurable overlap via diff_coverage=unmeasured in the verdict and ladder snapshot, but by default it still credits deterministic passes and does not mark them as unproven or escalate; only an opt-in require_diff_coverage setting turns unmeasured into an escalation. The issue requested that the no-tooling case land as unproven by default.
#1293 Record reverts as ground-truth signals and connect reverted Stella-produced commits back to the judge verdicts that approved them.
#1293 Reconcile verdicts (e.g., CI results) that arrive after a session ends so they still update earlier pass records instead of remaining unreconciled.
#1293 Drive judge threshold tuning from measured data, including the newly collected answer key, rather than from hard-coded guesses. The PR instruments and reports new measured data (reverts and cross-session CI via CalibrationReport, calibration_pending, GroundTruth, and stella calibration), but it does not change any thresholds to be auto-tuned or dynamically adjusted based on that data. The ROADMAP note explicitly says threshold auto-tuning still waits on this data, so tuning behavior remains unchanged.
#1294 Recognize and classify out-of-memory kills as a distinct outcome from test failures and other infra errors, and surface this classification through the pipeline (including labels/snapshots).
#1294 Ensure the pipeline treats out-of-memory test runs as "we learned nothing" rather than "the test failed", specifically retrying OOM runs (with a bounded retry count) instead of revising code or emitting deterministic test failures.
#1294 Add automated tests that cover the OOM behavior, including classification, retry behavior, and outcome reporting, to prevent regressions.
#1295 Measure and report, from recorded ladder snapshots with the git baseline on, how often a verdict has no flip and no green test behind it (the judge-alone rate).
#1295 Implement a configurable pipeline behavior that, when enabled, sends back lone judge PASS verdicts for one revision to seek deterministic evidence, without ever converting such runs from pass to fail and with bounded extra revisions.
#1295 Persist and expose the judge-alone measurements and the decision about enabling the evidence requirement so future operators can see the current rate and rationale without re-discovering it.
#1297 Expose a judged multi-round goal run over the API so that a goal run can be started, its rounds followed, and it can be stopped via existing turn endpoints.
#1297 Expose sub-agents over the API so callers can request delegation (including pool/steps/provider settings) with operator-side limits enforced.
#1297 Document the new goal-loop and sub-agent API capabilities with a working example covering behavior when stopping a run and how completed work is preserved.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@macanderson
macanderson marked this pull request as ready for review August 4, 2026 00:05

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @macanderson, your pull request is larger than the review limit of 150000 diff characters

…rified (#1291)

The first cut stated `diff_coverage=unmeasured` in the verdict but still
credited a `DeterministicPass`, which the issue's fourth point rejects: "the
honest answer is also 'unproven' rather than silently passing". It was
raised as a deliberate deviation and is now closed the way the issue asks.

What changes is the CLAIM, not the cost. An unmeasured overlap still takes
the fast-submit — no judge call, no extra turn, the run completes exactly as
before — but it scores `Unverified` rather than `DeterministicPass`, and the
verdict summary now LEADS with `UNPROVEN` instead of appending the status at
the end of a sentence. Paying for honesty with a ranking position instead of
a model call is what makes it affordable to be honest by default, in every
workspace, rather than only where coverage tooling happens to be installed.
Escalating instead stays available behind `require_diff_coverage`, whose
docs now say what it adds (a reviewer) rather than what it fixes (the claim,
which is unconditional).

`evidence.deterministic` deliberately stays `true`: the flip WAS a real test
observation, and the calibration cohorts (#871) partition by evidence kind,
not by coverage. Only the score moves.

`PipelineOutcome::score` is surfaced so the distinction is legible to a host
— a caller reading `verdict.deterministic` alone would see a pass and
nothing else, which is the silence the issue objects to. Two witnesses pin
both directions: an unmeasured overlap scores `Unverified` while spending no
judge call, and a measured overlap still earns the deterministic badge in
full (without which the downgrade would be indistinguishable from "never
score a deterministic pass again").

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013G7b8qy1My98MVivTtZs5H

Copy link
Copy Markdown
Owner Author

Pushed a2974ba, closing the one deviation I flagged in the description — the same item Sourcery's assessment marked ❌ for #1291.

An unmeasurable coverage overlap now scores Unverified, not DeterministicPass. What changed is the claim, not the cost:

Paying for honesty with a ranking position instead of a model call is what makes it affordable by default in every workspace, rather than only where coverage tooling happens to be installed. Escalating stays available behind require_diff_coverage, whose docs now describe what it adds (a reviewer) rather than what it fixes — the claim is unconditional now.

PipelineOutcome::score is surfaced alongside verdict, because a host reading verdict.deterministic alone would see a pass and nothing else — which is the silence the issue objects to.

Two witnesses pin both directions:

Witness Pins
an_unmeasurable_overlap_is_scored_unproven_without_costing_a_judge_call Unverified + UNPROVEN summary + no Judge stage
a_measured_overlap_still_earns_the_deterministic_badge a proven overlap is still DeterministicPass — without this the downgrade would be indistinguishable from "never score a deterministic pass again"

On the other ❌ — #1293's "drive threshold tuning from measured data" — that is deliberately not in scope, and the issue says so itself: "Once there is real data, tune the thresholds against it — not before." This PR builds the instrument that produces the data (reverts as ground truth, cross-session reconciliation); there is none yet to tune against. Recorded in ROADMAP.md §4.

Gate is green on the new commit: fmt --check, clippy --workspace --all-targets, test --workspace, cargo doc (no warnings), plus check-file-size (baseline updated), check-doc-citations, check-invariants and check-wire-schema.

One note on CI: no ci.yml run exists for this branch. pull_request workflows are not triggered by events an app token creates, and workflow_dispatch is refused for this integration (403 Resource not accessible by integration), so the Rust gate needs a maintainer to kick it — a re-run from the Actions tab, or any push from a user account.


Generated by Claude Code

@macanderson
macanderson merged commit c3c8cfd into main Aug 4, 2026
8 of 12 checks passed
@macanderson
macanderson deleted the claude/issues-1295-1297-a922sz branch August 4, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants