Skip to content

docs(compliance): explain how grading works end-to-end (#4037)#4044

Draft
bokelley wants to merge 1 commit intomainfrom
claude/issue-4037-compliance-grading-docs
Draft

docs(compliance): explain how grading works end-to-end (#4037)#4044
bokelley wants to merge 1 commit intomainfrom
claude/issue-4037-compliance-grading-docs

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

@bokelley bokelley commented May 4, 2026

Closes #4037

Adds docs/building/verification/how-grading-works.mdx — a new page explaining how the compliance runner translates a specialism declaration into a concrete set of graded scenarios, and how per-scenario requires_capability gates narrow that set. Also corrects a factual error in the sales-guaranteed specialism narrative and adds a cross-link from the Compliance Catalog.

Non-breaking justification: New docs page (additive), one-sentence cross-link addition to compliance-catalog.mdx, narrative-prose correction in a specialism YAML (no schema or wire changes), nav entry in docs.json. Existing consumers unaffected.

Factual correction: The sales-guaranteed specialism narrative previously stated "sellers that declare true (or omit the field) are graded against proposal_finalize." The capabilities schema has "default": false for supports_proposals with description "When false or absent, conformance runners skip proposal-lifecycle storyboards." The narrative conflicted with the schema. The corrected narrative reflects the schema: omit = false = skip. A previous triage commit on this branch documented absence as true; this commit supersedes it with the schema-accurate behavior.

Key content:

  • Four-step evaluation algorithm: specialism manifest → requires_scenarios list → per-scenario requires_capability gate → skip or grade
  • Worked example: sales-guaranteed + media_buy.supports_proposals: true (all 8 scenarios run) vs false (7 run, proposal_finalize skipped)
  • skip.reason: not_applicable is the canonical runner code for skipped scenarios — capability_unsupported only appears in narrative prose and is not a runner output contract value
  • Grading verdicts table sourced from runner-output-contract.yaml
  • Artifact location table (URL paths + source paths, per issue request)

Nits not fixed (surfaced for reviewer):

  • The runner-output-contract.yaml does not formally define the requires_capability field (it's used in one scenario, proposal_finalize, but not in storyboard-schema.yaml). The page notes this is a nascent mechanism.
  • /schemas/v3/protocol/get-adcp-capabilities-response.json uses the major-version alias per playbook convention; $id in the schema file uses a versionless path — both are correct for their respective purposes.

Pre-PR review:

  • code-reviewer: approved — fixed passed: null (not in contract spec) → removed in favor of skip block as authoritative signal; fixed not_applicable table to consolidate cases with skip.detail as the distinguisher
  • docs-expert: approved — genuine gap filled, correct audience level, conformance.mdx layer table linked not duplicated, Priya/StreamHaus canonical per character bible, <Note> component correct

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout <num>
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See #3121
for context.

Session: https://claude.ai/code/session_01M71brR4fRwi8w1CdjZUXXR


Generated by Claude Code

Add docs/building/verification/how-grading-works.mdx explaining how
the runner translates a specialism declaration into a concrete set of
graded scenarios and evaluates per-scenario requires_capability gates.

Also corrects the sales-guaranteed specialism narrative which incorrectly
stated that omitting supports_proposals would trigger proposal grading;
the schema default is false so absent = skip (not grade).

https://claude.ai/code/session_01M71brR4fRwi8w1CdjZUXXR
@bokelley bokelley added the claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage. label May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: explain how compliance grading works end-to-end (manifest → scenarios → capability gates)

2 participants