docs: explain how compliance grading works end-to-end (manifest → scenarios → capability gates)

[bokelley]

What's missing

There's no single doc that walks through the compliance grading system as a system. The pieces individually exist, but the relationship between them is not documented anywhere I could find:

1. Specialism manifests — static/compliance/source/specialisms/<name>/index.yaml declares requires_scenarios: listing protocol/scenario IDs the adopter is graded on.
2. Per-scenario YAMLs — static/compliance/source/protocols/<protocol>/scenarios/<name>.yaml define the actual graded steps and may declare a requires_capability: block (e.g. path: media_buy.supports_proposals, equals: true).
3. Capability gates — get_adcp_capabilities flags like media_buy.supports_proposals toggle scenarios on/off across specialisms that reference them.

Today an adopter who claims sales-guaranteed cannot answer the basic question "which storyboards will run against my agent?" without grepping yaml. The narrative blurbs inside sales-proposal-mode/index.yaml and sales-guaranteed/index.yaml hint at it, but the mechanism is never named.

Concrete ask

Add a new page (suggested location: docs/building/verification/how-grading-works.mdx, linked from the existing Compliance Catalog) that walks through:

• The four-layer model: protocol baseline → specialism manifest → required scenarios → capability gates.
• A worked example: "I claim sales-guaranteed and declare media_buy.supports_proposals: true. Here's the full set of storyboards I'll be graded on, and which capability flags would change that set."
• The grading verdicts an adopter will see, including capability_unsupported (when a requires_capability gate evaluates false) vs an actual fail.
• A pointer to where each piece lives in source: manifest path, scenario path, the universal storyboard list already in compliance-catalog.mdx.

Why this matters

I went looking for this in the JS-side TS for over an hour before realizing the source of truth is the YAML manifests + per-scenario requires_capability blocks. A single page describing the system would have closed that loop in five minutes.

Related (filing separately)

• Specialism decision tree
• Capability-flag → scenario cross-reference index

[bokelley]

Triage

Classification: Feature request (new docs page)
Bucket(s): web / site / docs, compliance suite
Status: drafting-pr

What the experts said:

• docs-expert: genuine gap — no existing page explains the manifest→scenario→capability-gate evaluation algorithm; new page is the right approach (not extending conformance.mdx or compliance-catalog.mdx)
• ad-tech-protocol-expert: non-breaking docs addition confirmed; identified two accuracy blockers: (1) capability_unsupported is not a formal runner verdict code — the canonical term from runner-output-contract.yaml is skip_result.reason: not_applicable; (2) the specialism narrative's "omit the field → grade it" directly contradicts the schema's "default": false for supports_proposals (schema says "When false or absent, conformance runners skip proposal-lifecycle storyboards")

My take: Drafting PR #4044 with the new page, a fix to the specialism narrative error, and a cross-link from the Compliance Catalog. The "absence = true" claim in the specialism narrative was wrong per the schema; the new page documents the correct behavior (absent = false = skip, matching the schema description).

Drafted in #4044 — new page at docs/building/verification/how-grading-works.mdx.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01M71brR4fRwi8w1CdjZUXXR

---

Generated by Claude Code