storyboard: declare grading mode (controlled-test vs live-sandbox); test controller presence shouldn't silently change rubric

[bokelley]

The structural gap

comply_test_controller presence currently flips the storyboard's grading rubric implicitly. That conflates two different compliance contexts under one summary, and operators have no way to declare which one they're in.

Mode A: controlled-test (cert / onboarding)

The seller exposes comply_test_controller. The SDK uses it to seed deterministic state — force a buy into the submitted arm, inject delivery metrics, plant governance fixtures. This is the rubric for certifying a new seller: every scenario in the bundle must grade, including the ones that need seeded state.

In this mode, missing the test controller is a hard fail. The operator has signed up to be certified; they're asserting their stack supports the full compliance surface.

Mode B: live-sandbox / verified-live (production monitoring)

The seller is a real production agent. No comply_test_controller (you don't ship test seeding skills to prod). The runner grades what's naturally observable on the wire — happy paths, real get_products queries, real create_media_buy against real inventory. Controlled-only scenarios should be marked not-applicable, not skipped — they're a different rubric.

In this mode, missing the test controller is expected. The verdict should be "live agent passed N/M live-applicable scenarios", not "passed N/M with 28 skipped for unspecified reasons."

What's broken today

Today the runner's behavior is "if test controller present → grade everything; if absent → silently skip controller-driven scenarios with skip_reason: missing_test_controller." The operator never declared intent. Downstream tooling can't tell:

• a real production agent that's correctly in mode B (28 N/A is fine, agent is healthy), from
• a stack that should be in mode A but is missing seeding skills (28 skipped means it can't be certified)

These have opposite operator-action implications — "ship it" vs "block release" — but the same wire signal today.

Proposal

1. Operator declares grading mode

Add a --grading-mode flag (or env var ADCP_GRADING_MODE) with values:

• controlled-test (default for npx storyboard run against localhost?)
• live-sandbox (default for runs against public HTTPS URLs?)
• auto (current behavior; flagged as deprecated)

The mode picks which rubric is loaded and what counts as a fail.

2. Scenarios declare their applicable mode(s)

Each storyboard scenario YAML grows an applicable_modes list:

id: media_buy_seller/create_media_buy_async/force_submitted_arm
applicable_modes: [controlled-test]   # only graded when --grading-mode=controlled-test

id: media_buy_seller/refine_products/setup
applicable_modes: [controlled-test, live-sandbox]   # graded in both, but
required_tools_one_of:                              # under different rules
  - [sync_accounts, list_accounts]                  # one of these is required

When mode mismatch → scenario is not_applicable (distinct from skipped), and counted in a new bucket.

3. Mode-aware verdict

── Controlled-test mode ──
  Steps: 31 passed, 28 failed (missing test controller), 0 skipped
  STORYBOARD-FAIL: cannot certify without comply_test_controller

── Live-sandbox mode ──
  Steps: 31 passed, 0 failed, 28 not-applicable (controlled-test only)
  STORYBOARD-OK: agent passes all 31 live-applicable scenarios

Same agent, same wire behavior, different verdict — because the operator declared intent.

Why this matters more than #1623 / #1624

#1623 (skip-cause aggregation) is presentation. #1624 (required-tool fail) is rubric for one specific gap. This issue is the architecture both should land into: the rubric isn't fixed — it depends on the mode the operator is grading in. Once mode is explicit:

• storyboard: surface skip causes in always-on summary #1623's skip-cause block gets organized by mode-applicability.
• storyboard: missing required-account tool should fail, not skip #1624's "required tool missing → fail" only applies in controlled-test; in live-sandbox the operator may legitimately not have account discovery wired (different deployment).
• Future "verified live" badge work has a clean place to plug in: it's just a published artifact of live-sandbox mode.

Strawman migration

1. Land the mode flag with auto as the default (no behavior change).
2. Tag scenarios with applicable_modes over the next few releases.
3. Flip the default to require explicit --grading-mode once tagging is complete.
4. Verified-live infrastructure (badges, dashboards) starts emitting from live-sandbox-mode runs only.

Happy to spec this out further or send a PR for step 1 — wanted to surface the framing before going deeper.

Related: #1623, #1624.

[bokelley]

Triage

Classification: Feature request (architectural)
Bucket(s): conformance (also cli, cross-repo — labels not found in repo)
Status: ready-for-human

What the experts said:

• ad-tech-protocol-expert: Flag — applicable_modes must be defined in the spec repo before the client ships --grading-mode; auto as a deprecated-from-birth default with no YAML corpus to key on produces a cosmetic-only flag; spec-first ordering is required per runner-output-contract precedent.
• adtech-product-expert: Flag — two-mode framing misses a third case (cert candidate seeding via HTTP admin, not controller); caller-declared mode is gameable for badge infrastructure vs agent-declared detection; cleaner path is splitting skipped_count into skipped_controller_required + skipped_other (zero spec dependency); storyboard: surface skip causes in always-on summary #1623 and storyboard: missing required-account tool should fail, not skip #1624 are independently unblocked now.

My take: Both experts agree on design discussion before implementation. Three open questions for @bokelley: (1) should mode be caller-declared (--grading-mode) or agent-declared via controller detection in capabilities, (2) does applicable_modes require a spec-repo PR before the client ships the flag, and (3) does the cert-without-controller use case invalidate the two-mode framing? Experts also note #1623 and #1624 don't need to wait on this.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01UH1bZRCYRtWejTeLfVXD2r

---

Generated by Claude Code

[bokelley]

Design pass — mode framing dissolves into per-scenario requires

Working through the three open questions surfaced in triage produced a cleaner shape than the original mode proposal. Posting for a second triage read.

Why the mode framing collapses

controlled-other doesn't carry its weight. Out-of-band seeding (HTTP admin, pre-test script, staging fixture) covers initial state but cannot cover runtime forcing — branch selection, delivery-metric injection, async task time-travel, error-arm forcing. Most controlled-test scenarios need runtime forcing, so controlled-other ends up grading a much narrower slice than the framing suggested. That undermines the three-mode taxonomy.

Pulling the requirement vocab down to the scenario level gets the same separation without inventing modes that don't compose.

Proposed shape: scenario-level requires

Each storyboard scenario declares what it needs:

id: media_buy_seller/create_media_buy_async/force_submitted_arm
requires: [controller]                  # runtime forcing — needs comply_test_controller
phases: [...]

id: media_buy_seller/refine_products/setup
requires: [seeded_state]                # initial state — controller OR out-of-band seeded
phases: [...]

id: media_buy_seller/get_products/happy_path
# requires: [real_wire] (default when omitted) — observable on production wire
phases: [...]

Runner detects what's available:

• controller → from agent capabilities (comply_test_controller advertised)
• seeded_state → operator asserts via --asserts-seeded-state flag (default off)
• real_wire → always available

Each scenario runs if requires ⊆ available; otherwise marks not_applicable with a requirement_unmet skip reason naming the missing requirement.

Verdict shape

── A2A ──
  Steps: 26 passed, 0 failed, 28 not-applicable
  Not-applicable causes:
    [22] requires: controller — agent does not advertise comply_test_controller
    [ 6] requires: seeded_state — pass --asserts-seeded-state to grade

Cause is per-scenario, not per-mode. Operator reads "you're 22 scenarios away from controlled coverage because of the controller, and 6 more if you assert seeded state."

Answers to the original triage questions

• Q1 (caller-declared vs agent-declared mode): dissolved. controller is agent-declared (capability detection, not gameable by caller). seeded_state is operator-asserted, but only makes scenarios run more not fewer — and those scenarios fail naturally if the seed isn't actually there. Self-correcting; nothing to game.
• Q2 (spec-first ordering for applicable_modes): narrower than originally framed. Spec-first applies to the YAML schema if it becomes a published contract for third-party storyboard suites. SDK-internal tagging needs no spec PR — runner reads its own YAML. Propose the schema upstream once it's bedded in; for now, ship as an SDK-internal additive extension.
• Q3 (HTTP-admin cert seeding case): the third case caused the mode framing to collapse. Under requires, it's just an operator flag (--asserts-seeded-state) that flips one requirement available — no third mode needed.

Anti-gaming for badge infrastructure

"Verified live" badges key on "all scenarios with requires: [real_wire] (or no requires declared) passed" — not on a mode label the operator declared. Operator can't game; the requirement vocab is honest about what each scenario needs.

Cross-language friendliness

requires lives in storyboard YAML and the runner. The agent under test never sees it. Python / Go / Rust adopters running the Node CLI against their server get identical behavior. No language-specific assumptions, and the schema is plain YAML when it lands as upstream contract.

Default when requires is absent

[real_wire] — matches existing pre-tagging behavior; storyboards stay working untagged. Tagging is additive opt-in. Migrate controlled-test-only scenarios first, then seeded-state-only.

Cheap interim — folds into #1623

The skip-cause aggregation in #1623 becomes the per-requirement break-down naturally. Lands first as presentation over current skip_reason values; rewires to the unified requirement_unmet shape once #1626 implements it.

Implementation order

1. storyboard: surface skip causes in always-on summary #1623 — skip-cause block (presentation, current shape).
2. storyboard: missing required-account tool should fail, not skip #1624 — required_any_of_tools YAML key + runner.ts:936 fix (universal account-discovery rule, depends on spec: account discovery (list_accounts | sync_accounts) is required, not optional adcp#4302 merging upstream).
3. storyboard: declare grading mode (controlled-test vs live-sandbox); test controller presence shouldn't silently change rubric #1626 — requires schema, detection logic, --asserts-seeded-state flag, unified requirement_unmet skip reason, SDK storyboard tagging.
4. Later — propose requires schema upstream; badge infra reads requires: [real_wire] pass rate.

cc @bokelley for triage re-read.

[bokelley]

Triage (re-read)

Classification: Feature request (architectural)
Bucket(s): conformance, cli
Status: ready-for-human

What the experts said:

• code-reviewer: Flag — design mechanism is sound and the three prior questions are well answered, but two implementation decisions need to be explicit before a PR: (1) skip_reason: missing_test_controller is a canonical exported type (types.ts:1137, emitted in seeding.ts:320) — renaming to requirement_unmet is a potentially breaking surface change that needs a deprecation path or explicit minor/major bump in the changeset; (2) unknown requires tags silently skip scenarios forever — the parser should warn at runtime rather than silently drop coverage.
• code-reviewer (additional): requires: [] (empty array) semantics are undefined — parser should either reject it or document that it means "run always" (which differs from the absent-field default of [real_wire]).

My take: The requires-based design resolves all three prior questions cleanly. The anti-gaming argument holds (agent-declared controller, self-correcting seeded_state), the spec-first concern is correctly narrowed to "propose upstream once bedded in," and the third seeding case collapses to --asserts-seeded-state. Two concrete decisions still needed before a PR can open: the migration strategy for the missing_test_controller string rename, and the parse-time behavior for unknown/empty requires tags. @bokelley, once those two are settled the change looks executable.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01RkHrRn1iMT55VuuvK97KaX

---

Generated by Claude Code

[bokelley]

Triage addendum — protocol expert findings (async)

The ad-tech-protocol-expert reviewed the codebase and diverges from the code-reviewer above. Both positions for @bokelley:

code-reviewer: Flag — rename of skip_reason: missing_test_controller → requirement_unmet is a potentially breaking surface change; unknown requires tags silently skip scenarios.

ad-tech-protocol-expert: Execute (in sequence) — both items are nits, not blockers, because:

• The RunnerDetailedSkipReason / DETAILED_SKIP_TO_CANONICAL map in types.ts:1205-1217 already provides the pattern; adding requirement_unmet there (mapping to unsatisfied_contract) follows the existing capability_unsupported, missing_test_kit_contract precedent.
• Unknown-tag validation has a direct model in validateContextOutputs (loader.ts:153-184) — add validateRequiresTags in validateStoryboardShape, same shape.
• The requires_tool gate at runner.ts:2504-2528 is the implementation model for per-scenario requires evaluation.
• The design is sound; controller detection reuses seeding.ts:227; --asserts-seeded-state mirrors the existing skip_controller_seeding option pattern.

Sequencing confirmed: #1623 → #1624 → #1626. The [22] requires: controller verdict format in #1626 depends on the skip-cause block from #1623 being present first.

Expert disagreement is on severity, not direction. Both experts agree the mechanism is correct and the design is sound. The code-reviewer worked from description without codebase context; the protocol expert confirms the existing type map and loader patterns reduce the "breaking" concern to an additive nit. @bokelley is best placed to decide whether to treat the rename as a nit (add to DETAILED_SKIP_TO_CANONICAL) or a minor-bump deprecation cycle.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01RkHrRn1iMT55VuuvK97KaX

---

Generated by Claude Code

[bokelley]

Draft PR #1635 opened with the locked design (per-storyboard requires: gate, --asserts-seeded-state flag, no-rename of missing_test_controller, fail-load on unknown/empty requires).

Scope of this PR (3 of 5 follow-ups in this issue):

• Schema, runner gate, CLI flag, loader validation, tests + changeset.

Deferred to follow-ups:

• Tagging upstream storyboards in compliance/cache/ — those are auto-synced from adcontextprotocol/adcp and local edits get clobbered. Will propose schema upstream once it has bedded in across SDK adopters.
• Skip-cause aggregator picking up requirement_unmet — PR feat(comply): surface skip causes in always-on storyboard summary #1629 (storyboard: surface skip causes in always-on summary #1623) hasn't merged yet; requires: [controller] unmet emits the existing missing_test_controller skip_reason which feat(comply): surface skip causes in always-on storyboard summary #1629's ACTIONABLE_SKIP_REASONS already covers, so the immediate use case is unblocked. Adding requirement_unmet to the aggregator is a small follow-up after feat(comply): surface skip causes in always-on storyboard summary #1629 lands.