Schema URLs and Zod validators are baked at SDK build time — drift from agent's advertised `adcp_version`

[bokelley]

Summary

The runner cites schema URLs and runs Zod validation against schemas pinned to the SDK's build-time ADCP_VERSION rather than the agent's advertised version. When the two are skewed (which happens any time an adopter installs an @adcp/sdk version older than the agent they're probing), the runner reports schema URLs that don't reflect what's actually on the wire and validates responses against a schema set the agent has already moved past.

📌 Update 2026-05-12: scope correction after BidMachine investigation

Three findings shift the urgency and shape of this issue:

1. The BidMachine symptom (adcp#4419) has a different root cause. Verified the published @adcp/sdk tarballs 5.25.1, 6.12.0, and current all use .passthrough() on SyncAccountsResponseSchema.accounts[]. Zod silently accepts authorization. The actual rejection is context.no_secret_echo's name-based dragnet flagging the spec-legitimate authorization field. Fix landed via adcp-client#1713 / PR fix(invariant): no_secret_echo only fails on string-valued suspect-named fields (#1713) #1714.

2. Deliverables 1 (.strict() → .passthrough()) and 2 (codegen regen) are not needed. The published codegen already uses passthrough on every additionalProperties: true schema. fgranata's "Zod .strict() codegen lag" diagnosis (per adcp-client#1711) doesn't match what's actually in the tarballs. Removed from scope.

3. The agent doesn't advertise a full semver. get_adcp_capabilities exposes only adcp.major_versions: integer[] (e.g. [3]) — no adcp_version patch-level string. So the "cite the agent's version in schemas_used" framing (deliverable 7 as originally written) has no agent-side signal to consult at full-semver granularity. The honest cite today is the SDK's build-time pin (current behavior).

What remains real: adopters who install a stale @adcp/sdk validate against stale Zod schemas regardless of what the agent emits. That's a real correctness drift, but its blast radius is limited to: (a) typed-shape rejection of newer fields the codegen doesn't know about, and (b) report-time URL drift. Both are bounded by additionalProperties: true permissiveness on most response schemas. There's no urgent unblocker; this is architectural cleanup, not an emergency.

Revised recommendation: hold #1707 for adopter demand to surface. The two empirical symptoms it would address are:

• Cosmetic: schemas_used URLs cite SDK-pin instead of agent-version
• Behavioral: SDK Zod codegen could reject legitimate newer fields IF a future schema tightens additionalProperties (currently almost always true)

Neither is breaking adopters today. The dynamic-fetch architecture (deliverables 3-8 below) is the right design when we tackle this, but it's a real lift (200-300 LOC + tests + design pass on offline/cache semantics) that wants a clear "we need this now" trigger rather than speculative implementation.

This is the deeper root-cause class behind adcontextprotocol/adcp#4419 (BidMachine's summary.schemas_used shows /schemas/3.0.1/ against a 3.0.11 seller). The Zod .strict() → .passthrough() flip that triage isolated as #4419's surface fix is part of this issue's scope, not a separate deliverable — see Deliverables below.

Evidence

Sampled tarballs across the recent SDK release history:

@adcp/sdk published	ADCP_VERSION in tarball
5.23.0 – 6.6.0 (~12 releases over 4 days)	3.0.1
6.7.0 – 6.8.0	3.0.5
6.9.0 – 6.12.0	3.0.6
6.13.0 – 6.15.x	3.0.6
6.16.0 – 6.17.0	3.0.8
6.18.0	3.0.9
6.19.1 / 7.0.0	3.0.11

Every published version pins schemas at whatever ADCP_VERSION was current at release time. An adopter who installed any version in the 5.23–6.6 band — a four-day window with twelve published releases — has their runner permanently citing /schemas/3.0.1/ until they upgrade. BidMachine's report shows exactly that symptom.

Today's behavior (the bug)

src/lib/testing/storyboard/validations.ts:343-356:

const SCHEMA_URL_BASE = 'https://adcontextprotocol.org';

function resolveSchemaIdentity(schemaRef: string | undefined) {
  if (!schemaRef) return { schema_id: null, schema_url: null };
  const trimmed = schemaRef.replace(/^\/+/, '');
  const schemaId = `/schemas/${ADCP_VERSION}/${trimmed}`;  // ← build-time pin
  const schemaUrl = `${SCHEMA_URL_BASE}${schemaId}`;
  return { schema_id: schemaId, schema_url: schemaUrl };
}

The cited URLs go straight into ComplianceResult.schemas_used — they're what consumers (dashboards, JUnit, the AdCP Verified renderer) display to the seller as "what we validated against." When ADCP_VERSION is stale relative to the agent, the URLs are misleading at best and actively wrong at worst (the agent legitimately added fields per additionalProperties: true that the cited 3.0.1 schema doesn't document).

The Zod validators are baked the same way — codegen produces typed schemas for ADCP_VERSION at SDK build time and ships them inside the tarball.

Recommended approach: dynamic fetch with bundled fallback

The three options I originally enumerated reduce to one principled answer:

• Honest-URLs-only is dishonest. The SDK would cite URLs from a schema version it hasn't actually validated against. Same "SDK as translator" anti-pattern (Tracking: ship 1676–1680 as one stance — the SDK is a witness, not a translator #1685) we've been killing. Reject.
• Bundle-every-version is overkill. Tarball size cost without offline being a real adopter ask today.
• Dynamic fetch with bundled fallback is the right shape.

Specifically:

1. Compliance validation uses AJV against schemas fetched from adcontextprotocol.org/schemas/<agent_version>/, where <agent_version> is profile.raw_capabilities.adcp_version (already captured at discovery). In-memory cache keyed by version; existing ssrfSafeFetch primitive bounds the network surface.
2. If fetch fails (404 for an unknown version, offline CI, network error), fall back to the bundled SDK-build-time schemas. That's what every adopter ships today, so no regression.
3. When the fallback fires, emit a notices entry via the surface landing in feat(runner): structured RunnerNotice advisory surface on StoryboardResult / ComplianceResult #1705 — code schema_version_fetch_failed, fields requested_version / used_version / reason. The report carries an honest machine-readable signal that the validation used a different version than the agent claimed.
4. Zod codegen stays as the developer-ergonomics typing layer for SDK API surfaces. Compliance validation moves to JSON-schema-driven AJV against the fetched (or fallback) schema set.
5. schemas_used URLs always reflect the version actually used — never lies, never cites a URL the runner didn't validate against.

Deliverables (scope of this issue)

1. Zod strict-mode fix. Switch .strict() → .passthrough() on every codegen-emitted schema whose source JSON declares additionalProperties: true. This is the #4419 surface fix; it lands here because it's tightly coupled to (2) — regenerating against a current ADCP_VERSION is a no-op if the strict-mode shape is wrong.
2. Codegen regeneration against current ADCP_VERSION. Sibling to (1); the existing generated artifacts in src/lib/types/*.generated.ts were emitted at whatever pin was current when they last regenerated, which is a separate drift surface from runtime validation.
3. AgentProfile.raw_capabilities.adcp_version plumbed into the validation path. Today the runner already captures this at discovery; threading it into resolveSchemaIdentity and into a new AJV-backed validator dispatch.
4. Schema fetcher + cache. fetchSchemaForVersion(adcpVersion: string): Promise<JSONSchema> using ssrfSafeFetch (origin already locked to adcontextprotocol.org), in-memory cache keyed by adcp_version, body-size and time bounds matching the existing probe defaults.
5. Bundled fallback. When fetch fails or returns non-2xx, fall back to bundled schemas. Same bundled set that's already in the tarball under dist/lib/schemas-data/.
6. schema_version_fetch_failed notice via feat(runner): structured RunnerNotice advisory surface on StoryboardResult / ComplianceResult #1705. Stable code, fields requested_version / used_version / reason. Lands in ComplianceResult.notices and per-StoryboardResult.notices so the report carries the signal.
7. schemas_used URL fix. Compute schemaUrl from the version actually used, not from build-time ADCP_VERSION.
8. Tests covering: happy path (agent v matches SDK pin) / agent v ahead of SDK pin / agent v behind / fetch 404 fallback / network-error fallback / notice emission on fallback / strict→passthrough doesn't regress schemas legitimately closed in the spec.

Validation hook

fgranata (BidMachine) volunteered as a retest target once deliverables 1–7 land:

We (bm-agentic-seller / BidMachine) are happy to be a guinea-pig retest target once a fix lands. Our prod endpoint is https://adcp.bidmachine.io/adcp/mcp; current comply-suite score is 63/128 (49%) and the CLI runner score on the same agent is 45/59 (76%) — the 27-point delta is the diagnostic signal both scopes above should close.

The 27-point delta between the two evaluators is also the canonical evidence for adcp-client#1708 (parity smoke test) — once #1707 lands and BidMachine retests, the delta should collapse. If it doesn't, #1708's parity check earns its keep immediately.

Sequencing

The full coordinated stance covers five issues; this one is in the middle:

1. adcp-client#1706 (auto-derived COMPATIBLE_ADCP_VERSIONS) — merged ✓ (05ce456f).
2. adcp-client#1705 (structured RunnerNotice surface) — merged ✓ (e1ec3ef3). Notice channel deliverable 6 rides on is in place.
3. adcp-client#1709 (harness error attribution: Zod rejects misattribute to the next downstream assertion) — PR fix(runner): attribute Zod schema rejects to response_schema (#1709) #1712 open. Lands before Schema URLs and Zod validators are baked at SDK build time — drift from agent's advertised adcp_version #1707 so any remaining schema rejects during Schema URLs and Zod validators are baked at SDK build time — drift from agent's advertised adcp_version #1707's rollout produce honest signal, rather than misattributing to no_secret_echo etc. and masking the strict→passthrough flip's behavior.
4. adcp-client#1707 (this) — implements deliverables 1-8 once feat(runner): structured RunnerNotice advisory surface on StoryboardResult / ComplianceResult #1705 and Harness error attribution: Zod validation rejects surface as failures on unrelated downstream assertions #1709 have landed.
5. adcp-client#1708 (cross-evaluator parity smoke test in CI) — durable safety net catching future divergence between the comply suite and the CLI runner as either surface evolves. Lands after Schema URLs and Zod validators are baked at SDK build time — drift from agent's advertised adcp_version #1707 so the parity baseline reflects the fixed state.

Related work

• comply runner: storyboard verdicts contradict spec-valid prod probes across 10+ runs adcp#4419 — BidMachine's "63/128 stuck score" report. Triage isolated strict→passthrough + codegen regen as the surface fix; this issue's deliverables 1 and 2 close it.
• fix(compliance): add normative additionalProperties validator clause to runner-output-contract adcp#4422 — upstream draft adding a normative additionalProperties: true MUST clause to runner-output-contract.yaml.
• Standardize notices field on runner-output-contract (warn-on-skip / forward-readiness surface) adcp#4418 — upstream notices field standardization on runner-output-contract.yaml.
• adcp-client#1706 — auto-derived COMPATIBLE_ADCP_VERSIONS (sibling).
• adcp-client#1705 / Structured notices on ComplianceResult / StoryboardResult (warn-on-skip surface) #1704 — structured notices surface (this issue's deliverable 6 rides on it).
• adcp-client#1709 — Zod-reject error attribution (the misattribution that made #4419 hard to diagnose; sequencing dependency above).
• adcp-client#1708 — cross-evaluator parity smoke test (the durable safety net once this issue lands).
• adcp-client#1685 — "the SDK is a witness, not a translator" stance. This bug is the same anti-pattern: the SDK fabricates schema URLs that don't reflect what the agent advertises.

Refs

• src/lib/testing/storyboard/validations.ts:343-356 — the build-time pinning surface
• src/lib/version.ts — ADCP_VERSION source of truth
• src/lib/types/*.generated.ts — Zod codegen output frozen at build time
• npm release timeline above — empirical evidence of the freeze pattern