AAO crawler/API: persist managerdomain discovery provenance and reverse index

[bokelley]

Blocked on: #4173 (merging the managerdomain fallback in AdAgentsManager)
Refs: #4175 (RFC)

Context

PR #4173 adds an ads.txt managerdomain one-hop fallback to AdAgentsManager.validateDomain(). Functionally the AAO crawler picks up delegated publishers transparently — validation.raw_data is the manager's manifest, agents and properties get recorded under each publisher. So the bootstrap-6K-publishers value is captured.

But the fallback discards provenance at the seam between the validator and everything downstream. That creates concrete operational problems.

Problem

1. No reverse index from manager → delegating publishers. When manager.example updates their adagents.json, we have no way to know which publishers delegated to them. Today the crawler is per-publisher; without a reverse index, change-propagation is "wait for the next full sweep" — which negates a chunk of the bootstrap value at scale.

2. cacheAdagentsManifest(publisherDomain, raw_data) stores the manager's manifest under the publisher's row with no marker that it was discovered via managerdomain. We can't display "via Raptive" in admin/member surfaces, and we can't audit how authorization was discovered after the fact.

3. Events (publisher.adagents_discovered, publisher.adagents_changed) don't carry discovery method or manager domain — downstream consumers (analytics, member dashboards, audit) can't distinguish direct from delegated.

4. API surface (/api/adagents/validate, publisher resolution endpoint, the property source enum that's currently adagents_json | agent_claim) — external consumers will reasonably want to know when authorization is one-hop-via-manager. Same discovery_method ask raised on the upstream PR thread.

Proposed scope

1. Once the upstream discovery_method field lands on AdAgentsValidationResult (pushed for in RFC 4175: Add ads.txt managerdomain fallback for adagents lookup #4173):
   • Persist discovery_method + manager_domain columns on the publisher / adagents cache table.
2. Reverse index: when discovery_method = 'ads_txt_managerdomain', persist a manager_domain → publisher_domain mapping. Use it to fan out re-validation when the manager's adagents.json changes.
3. Surface discovery_method in the /api/adagents/validate response and in publisher / property resolution responses. Consider extending the property source enum or adding a sub-discriminator.
4. Include discovery_method and manager_domain in publisher.adagents_discovered and publisher.adagents_changed events.
5. Optional: /api/registry/managers/:domain/recrawl endpoint that invalidates and re-validates all delegating publishers in one shot (rate-limited; mirrors the existing per-publisher recrawl).

Why this matters now

Without (2), the operational story for "Raptive rotates an agent" is "wait for the next full crawl of every delegated publisher." Without (3) and (4), buyers and members can't distinguish direct attestation from one-hop delegation, which is exactly the trust-tier distinction the upstream RFC depends on.

Triage should pick this up when #4173 merges.

[bokelley]

Triage

Classification: Feature request (follow-up)
Bucket(s): registry / discovery, admin / ops tools
Status: deferred

My take: Well-scoped follow-up to #4173; the entire proposed scope — discovery_method + manager_domain persistence, reverse index, API surface extension, and event enrichment — depends on AdAgentsValidationResult carrying a discovery_method field first. Nothing here is actionable until that upstream lands.

Blocked-on: #4173 — resurfaces on merge.

---

Triaged by Claude Code. Session: https://claude.ai/code/cse_014PYWXRRjXXM1SkxQPxxN8o

---

Generated by Claude Code

[bokelley]

Unblock: #4173 merged in 50a01eb with discovery_method: 'direct' | 'authoritative_location' | 'ads_txt_managerdomain' and manager_domain now required on AdAgentsValidationResult. The proposed scope here is fully unblocked.

Removing claude-triaged to re-enter the triage queue. Highest-impact item is the manager → publishers reverse index — without it, "Raptive rotates an agent" means waiting for the next full crawl sweep over every delegated publisher to propagate.

[bokelley]

Triage

Classification: Feature request (follow-up)
Bucket(s): registry / discovery, admin / ops tools
Status: drafting-pr

What the experts said:

• ad-tech-protocol-expert: source enum is server-API-only (not AdCP wire spec); flat adagents_json_via_manager value is cleaner than sub-discriminator; direct + authoritative_location both remain adagents_json, only ads_txt_managerdomain gets the new value. Events are server-internal, adding fields is non-breaking additive.
• adtech-product-expert: Items 1, 3, 4 are mechanical and cohesive as one PR. Reverse index fan-out (item 2) must be queue-backed — inline crawlSingleDomain fan-out at Raptive scale would saturate the crawler concurrency budget. Sequence it separately with the queue primitive.
• internal-tools-strategist: Ship items 1, 3, 4 together; splitting them leaves the DB columns populated but nothing reading or emitting them. Item 2's fan-out needs a bounded work queue before it ships. Item 5 is a thin wrapper that only makes sense after the reverse index exists.

My take: Items 1, 3, and 4 are a single atomic change: persist discovery_method + manager_domain on the publishers row, surface them in /api/validate-publisher and /api/registry/publisher, add to event payloads. Item 2 (reverse index + fan-out) sequences separately with a queue primitive so a large managed network rotating its adagents.json doesn't trigger unbounded parallel crawlSingleDomain calls.

Drafting a PR for items 1, 3, 4. Item 2 remains open as the next tracked piece; item 5 defers until 2 lands.

---

Triaged by Claude Code. Session: https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt

---

Generated by Claude Code

[bokelley]

Shipped in #4204: items 1, 3, 4 — migration 470 (discovery_method + manager_domain columns on publishers), provenance threading through all three cacheAdagentsManifest call sites, event payload enrichment, and API surface exposure on /api/validate-publisher, /api/public/validate-publisher, and /api/registry/publisher. PublisherLookupResultSchema and the OpenAPI schema for the public validate endpoint updated.

Remaining in this issue:

• Item 2 (reverse index + fan-out): manager_domain → publisher_domain mapping table + queue-backed re-validation trigger when a manager's adagents.json changes. Needs a bounded work queue primitive before shipping — inline crawlSingleDomain fan-out at managed-network scale would saturate crawler concurrency. Issue stays open as the tracker.
• Item 5 (manager recrawl endpoint): deferred until item 2 lands.
• Follow-up: per-agent source enum extension to adagents_json_via_manager (requires threading discoveryMethod through recordAgentFromAdagentsJson → upsertAuthorization).

---

Generated by Claude Code