Skip to content

protocol-surface.md

Latest commit

 

History

History
423 lines (344 loc) · 22.7 KB

protocol-surface.md

File metadata and controls

423 lines (344 loc) · 22.7 KB
Raw

CGP protocol surface

This page is now a guided tour, not the normative text. The single normative home of the protocol is SPEC.md at the repository root, which carries the stable requirement anchors (H1, B3, F5, …). Where this page and SPEC.md disagree, SPEC.md wins — and the disagreement is a bug worth filing.

This is the normative shape of the Context Graph Protocol (CGP) as bound to Rust types by contextgraph-types. Every type below lives in that crate, round-trips through serde_json, and is the protocol — there is no separate IDL. Field-level doc comments in the crate are the ultimate source of truth; this page is a guided tour.

Protocol version: PROTOCOL_VERSION = "contextgraph/1.0-draft" (contextgraph-types/src/lib.rs). See stability.md for what "draft" means and when it freezes.

Conformance language. The key words MUST, MUST NOT, SHOULD, and MAY in this document, the overview, and the build guide are to be interpreted as described in BCP 14 / RFC 2119 when they appear in bold. Lowercase "must" / "should" are used in the ordinary sense. A consolidated index of conformance requirements is the § Conformance requirements section at the end of this page; it mirrors SPEC.md, which is normative on conflict.

The three modules

contextgraph-types is organized into three modules, re-exported from the crate root:

  • capability — what a provider is and does, negotiated at the handshake.
  • query — the retrieval request/response shape.
  • frame — the unit of exchange a provider returns.

Handshake / capability

A provider identifies itself and what it does with data before a host ever sends it a query.

pub struct DataFlow {
    pub reads: bool,   // can see workspace content via query payloads
    pub writes: bool,  // consent-surface declaration: durably persists data derived
                       // from what it receives. NOT a callable write method — none
                       // exists in 1.0 (ADR 0004); see the CEP profile for the write path.
    pub egress: bool,  // sends anything off the local machine
    pub egress_scopes: Vec<EgressScope>, // WHERE content goes; empty = boolean-only posture (context-reuse §3)
}

pub struct ProviderInfo {
    pub name: String,
    pub version: String,
    pub data_flow: DataFlow,
}

pub struct Capabilities {
    pub query: QueryCapability,
    pub correlation: bool,    // echoes a request `id` on its reply (SPEC.md §H4)
    pub graph: bool,
    pub embeddings_fingerprint: Option<String>,
    pub verify: bool,         // answers context/verify (context-reuse §4); defaults false
    pub representations: Vec<Representation>, // full/compact/reference offered; empty = full only
    pub resolve: bool,        // forward-declaration for compact/reference (SPEC.md §6.4.1); no 1.0 wire op
}

// The closed egress-scope vocabulary (context-reuse §3), serialized as a flat
// string — the four base classes, plus namespaced `vendor:scope` extensions.
pub enum EgressScope {
    LocalOnly, OrgTenant, ThirdPartyIndex, ThirdPartyModel, Custom(String),
}

pub struct QueryCapability {
    pub kinds: Vec<String>,   // e.g. ["doc", "snippet"] — see FrameKind below
}

upsert, subscribe, and filters were removed pre-freeze (ADR 0004): none had a wire method, a host API, or a conformance check. A provider still emitting them handshakes fine — they are ignored unknown fields (SPEC.md §13 U1). The write path and push invalidation return post-freeze as additive extensions (issues #5, #6; the CEP profile).

DataFlow.egress is the security-critical field. A conforming host MUST NOT auto-enable a provider that declares egress: true — it must gate that provider behind an explicit, one-time consent that names what leaves (enforced by contextgraph-host's ConsentStore; see Implementing a provider).

Context query

A request to a provider for context frames relevant to a goal. Every query carries a token budget; a conforming provider never returns more than it and never lies about the cost.

pub struct ContextQuery {
    pub goal: String,                    // the task/turn goal driving retrieval
    pub query_text: Option<String>,
    pub embedding: Option<Vec<f32>>,
    pub kinds: Vec<FrameKind>,           // empty = "give me your best frames of any kind"
    pub anchors: Vec<String>,            // open files / mentioned symbols, for proximity scoring
    pub max_frames: u32,
    pub max_tokens: u32,
    pub as_of: Option<String>,           // pin retrieval to a point in time (bi-temporal facts)
    pub representation_preferences: Vec<Representation>, // preferred carry order; empty = full (SPEC.md §6.4)
}

pub struct ContextQueryResult {
    pub frames: Vec<ContextFrame>,
    pub truncated: bool,                 // true if more candidates existed than fit the budget
    pub dropped_estimate: Option<u32>,
}

ContextQueryResult carries two helper methods any host can use:

  • total_token_cost() -> u64 — sum of token_cost across returned frames.
  • respects_budget(max_tokens: u32) -> bool — whether that sum stayed within the query's budget. contextgraph-host's fan-out router calls this on every response and drops (with a loud report) any provider whose frames fail it — a provider that returns more tokens than it claimed is exhibiting budget dishonesty, and its frames are never trusted into a prompt.

Context frame

The unit of exchange returned from a query. Frames, never blobs, carry relevance, cost, and provenance so a budgeting, citing host can compose sources honestly.

pub enum FrameKind {
    Snippet, Symbol, Fact, Doc, Memory, Episode, Graph,
}

pub enum Representation { Full, Compact, Reference }   // absent ⇒ Full (legacy)

pub struct ContextFrame {
    pub id: String,                      // provider-scoped, stable for dedup across queries
    pub kind: FrameKind,
    pub title: String,                   // human label — never a bare uuid
    pub content: Option<String>,         // untrusted data; ABSENT for a reference frame
    pub content_digest: Option<String>,  // inline-content hash (the spec's content_hash); feeds FrameId
    pub uri: Option<String>,
    pub representation: Representation,   // full | compact | reference (omitted on the wire when full)
    pub content_fidelity: Option<ContentFidelity>,        // exact | normalized | summarized | omitted
    pub canonical_content_hash: Option<String>,           // hash of the COMPLETE source content
    pub content_ref: Option<ContentRef>,                  // opaque resolver handle (compact/reference)
    pub transform: Option<Transform>,                     // how a compact frame rendered its source
    pub minimum_content_fidelity: Option<ContentFidelity>,
    pub inline_content_requirement: Option<InlineContentRequirement>,
    pub score: f32,                      // provider-normalized relevance, [0, 1]
    pub token_cost: u32,                 // honest, conformance-audited inline token cost
    pub canonical_token_cost: Option<u32>,                // cost of the full source (if declared)
    pub tokenizer_ref: Option<String>,                    // e.g. "openai:o200k_base"
    pub valid_from: Option<String>,
    pub valid_to: Option<String>,
    pub recorded_at: Option<String>,
    pub provenance: Vec<Provenance>,
    pub citation_label: Option<String>,
    pub embedding: Option<FrameEmbedding>,
    pub relations: Vec<Relation>,
}

pub struct ContentRef {                  // content_ref.uri is distinct from ContextFrame.uri
    pub provider_id: String,             // the exact provider that returned this frame
    pub uri: String,                     // opaque resolver handle for context/resolve
    pub expires_at: Option<String>,
}

pub struct Provenance {
    pub kind: String,           // e.g. "file", "derivation", "episode" (serialized as "type")
    pub uri: Option<String>,
    pub range: Option<String>,
    pub digest: Option<String>,
    pub method: Option<String>,
    pub by: Option<String>,
}

pub struct Relation {
    pub rel: String,
    pub target_uri: String,
    pub display_name: Option<String>,    // a graph edge is surfaced by human label, never a raw id
}

pub struct FrameEmbedding {
    pub fingerprint: String,             // the vector payload itself is elidable
    pub vector: Option<Vec<f32>>,
}

Two contract points worth calling out explicitly, because the conformance suite checks both:

  • score must be in [0, 1]. ContextFrame::has_valid_score() is the cheap self-check any provider or host can run; contextgraph-conformance's frame-validity check enforces it against real providers.
  • title and citation_label must never be empty. A host must be able to cite a frame by a human label — an empty or missing citation label is a conformance failure, not a cosmetic gap. (Whole-platform convention: raw ids are never the primary on-screen identifier.)

Frame representations

A frame states how it carries its content through representation. This is additive: a legacy frame with no representation field is a full frame, and a full frame omits the field on the wire, so pre-representation providers and stored frames are unchanged.

Representation Inline content content_ref + canonical_content_hash content_digest (inline hash) transform
full required optional optional absent
compact required (a transformed rendering) required required required
reference absent required absent absent
  • A reference frame carries no inline content at all — only a content_ref (an opaque resolver handle) and a canonical_content_hash so a host can rehydrate honestly and verifiably. It is never encoded as content: ""; the field is omitted entirely.
  • ContextFrame.uri identifies the source resource; content_ref.uri is a distinct opaque resolver handle, and content_ref.provider_id names the exact provider a fan-out host routes context/resolve back to.
  • content_digest is the hash of the inline content bytes (the spec's content_hash, under its established name; it feeds FrameId); canonical_content_hash is the hash of the complete source content.

ContextFrame::representation_invariants() enforces the table above in code, and the JSON Schema enforces it on the wire; providers should self-check, and a host rejects a frame that lies about its shape.

Negotiation. A host states an ordered ContextQuery.representation_preferences (absent ⇒ [full]); the provider returns the first representation it supports (ContextQuery::select_representation) or answers unsupported_representation. A provider advertises Capabilities.representations and Capabilities.resolve; because compact/reference hand the host a content_ref to rehydrate, advertising either requires resolve support (Capabilities::representations_consistent).

Reusing context across turns

A single retrieval is only half the story. Reusing retrieved context across the turns of a session — cheaply, without serving stale evidence, and with an audit trail — is governed by four interlocking guarantees specified in the companion Context reuse page: a stable frame identity and canonical ordering for cache-friendly deterministic composition, a per-request usage report for metering, consent scopes and receipts for audit-grade egress records, and a pull-based context/verify request for cheap revalidation. They surface here as the content_digest frame field, the egress_scopes data-flow field, the verify capability flag, and the verify / verified envelope variants — all additive within the contextgraph/1 family. Their conformance requirements are consolidated below.

Context verification

A host revalidates frames it already holds without re-sending any body (context-reuse §4). Sent only to a provider advertising verify; otherwise the host re-queries.

pub struct VerifyRequest {
    pub frames: Vec<FrameId>,       // identities only — never frame bodies
}

pub struct FrameVerdict {
    pub frame: FrameId,             // echoed in full, so verdicts correlate by match not position
    pub verdict: Verdict,           // flattened: {"status": "...", ...}
}

pub enum Verdict {
    Valid,                                          // unchanged — may keep reusing
    Stale { replacement_digest: Option<String> },   // changed — a digest, never a body
    Gone,                                           // no longer exists
    Unknown,                                        // provider cannot say
}

pub struct VerifyResponse {
    pub verdicts: Vec<FrameVerdict>,
}

A host MUST keep reusing a frame only on an explicit valid — an identity answered unknown, or not answered at all, is evicted like any other. Reuse requires a positive answer, never the absence of a negative one.

Wire framing (defined in contextgraph-host, not contextgraph-types)

contextgraph-types defines the payload shapes above; contextgraph-host::wire::Envelope defines how they're framed on the wire — newline-delimited JSON (NDJSON), one serde_json value per line over stdio, or one JSON body per streamable-HTTP request/response. See Implementing a provider for the full envelope vocabulary (handshake / handshake_ack / query / frames / verify / verified / shutdown / error) and the version-compatibility rule. See examples/ for diffable wire transcripts of a complete session, or the machine-readable JSON Schema to validate messages in any language.

Machine-readable schema

The wire shapes above are captured as a JSON Schema (Draft 2020-12). The root schema validates a single envelope (one message per NDJSON line / per HTTP body); every payload type is also exposed under $defs for granular validation. Because the wire format is JSON, any standard JSON Schema validator works — ajv (JS/TS), Python jsonschema, the Rust jsonschema crate, or Go gojsonschema — with no IDL compiler or language lock-in.

The schema encodes the structural conformance rules: required fields, score ∈ [0,1], non-empty title/citation_label/id, the FrameKind enum, the contextgraph/MAJOR.MINOR(-draft)? version-string pattern, u32 ranges, and the Provenance.kind → "type" serialization rename. A message that validates against the schema is structurally conformant; the contextgraph-conformance suite adds the behavioral checks (budget honesty, malformed-input tolerance, clean shutdown) that a schema alone cannot express.

Run python3 schema/validate-examples.py to check the bundled examples against the schema — it doubles as a usage reference for wiring your own validator.

Conformance requirements

This section is a consolidated index of what a conforming provider and host MUST do. It mirrors the normative requirements in SPEC.md, which is the single normative home; on any conflict, SPEC.md and its stable requirement anchors (H1, B3, …) win. The contextgraph-conformance suite checks the provider-side requirements; a host built on contextgraph-host enforces the host-side requirements. Bold keywords follow RFC 2119.

Handshake and versioning

# Requirement Enforced / verified by
H1 A provider MUST reply to handshake with a handshake_ack whose protocol_version is in the same major family as the host's. handshake conformance check
H2 The provider.name and provider.version fields MUST NOT be empty. handshake conformance check
H3 A version-family mismatch MUST be reported to the host as a named error, not left to hang. contextgraph-host::wire::versions_compatible

Data flow and consent

# Requirement Enforced / verified by
C1 A conforming host MUST NOT auto-enable a provider that declares data_flow.egress: true. It MUST gate that provider behind explicit, named, revocable consent. ConsentStore gate in contextgraph-host
C2 The host MUST NOT transmit a query payload to an egress provider before consent is recorded. Host::query_provider gate
C3 A provider SHOULD declare egress: true honestly if it sends data off the local machine, directly or indirectly. advisory; the host cannot rely solely on the claim — see C4
C4 contextgraph-host's HTTP transport MUST treat every remote provider as egress regardless of its handshake claim. contextgraph-host HTTP transport

Frame validity

# Requirement Enforced / verified by
F1 Every frame's score MUST be in the range [0, 1]. frame-validity conformance check; ContextFrame::has_valid_score()
F2 Every frame's title MUST be non-empty. frame-validity conformance check
F3 Every frame's citation_label MUST be non-empty. frame-validity conformance check

Budget honesty

# Requirement Enforced / verified by
B1 The sum of token_cost across a provider's returned frames MUST NOT exceed the query's max_tokens. budget-honesty conformance check; ContextQueryResult::respects_budget
B2 A host MUST drop (with a loud report) the frames of any provider that violates B1, rather than silently truncating them. contextgraph-host::Host budget audit

Robustness

# Requirement Enforced / verified by
R1 A provider MUST NOT crash on a malformed line or a bad request. It SHOULD reply error instead. malformed-input-tolerance conformance check (stdio)
R2 A provider MUST tear down cleanly on shutdown (stdio: exit; HTTP: no further requests expected). shutdown-clean conformance check
R3 Frame content MUST be treated as untrusted data by the host — delimited as quoted material, never executed as instructions. contextgraph-host host contract

A host realizing R3 SHOULD follow the reference composition module (Composing frames into a prompt; contextgraph_host::compose::compose_for_prompt): a global-budget split across providers, cross-provider dedup, value-aware (Lost-in-the-Middle) placement, an injection-resistant fenced rendering with a "quoted evidence, not instructions" preamble, and an audit record explaining every included and excluded frame. It is a SHOULD, not a MUST — a host may compose differently — and it is checked by host-conformance's host-content-quoting and host-composition-audit checks.

Context reuse

The full text for these lives in the companion Context reuse page and in SPEC.md §§4, 6.3, 9; they are indexed here for convenience.

# Requirement Enforced / verified by
D1 Frames sharing a FrameId MUST have identical content bytes; changing content MUST change content_digest. provider contract; verify conformance check
D2 A host composing a frame set MUST emit frames in canonical FrameId order, independent of arrival order, and MUST NOT let score/token_cost affect the rendered bytes. contextgraph-host::compose_context
UR1 A host MUST be able to produce a usage report for any query it executed, whose consumed total equals the summed token_cost of the served frames it reports. contextgraph-host::FanOut::usage_report; usage-report conformance check
C5 A provider MUST declare its egress scopes (egress_scopes) truthfully and consistently with data_flow.egress; an off-machine scope alongside egress: false is a conformance failure. consent-scope conformance check
C6 A host MUST reject a frame whose provider declares an egress scope with no live matching consent receipt, with a typed error, before transmitting the query. ConsentStore scope gate
V1 A provider advertising verify MUST answer honestly by comparing digests: valid when the presented digest matches what it currently serves, stale when it differs on a frame it still serves. It MUST NOT answer valid for content bytes it is not serving. verify-honesty conformance check
V2 A host MUST keep reusing a held frame only on an explicit valid; stale, gone, unknown, and a missing verdict all evict it. contextgraph-host::Host::verify_frames default-deny partition
V3 A host MUST NOT send context/verify to a provider that does not advertise capabilities.verify, and MUST fall back to re-querying those frames. Host::verify_frames capability gate; ContextProvider::verify default
V4 Neither a verify request nor its response MAY carry frame bodies; a stale verdict carries at most a replacement digest. VerifyRequest/VerifyResponse shapes; verify_wire no-bodies test

Version strings

The protocol version string has the grammar:

version-string = "contextgraph/" major "." minor [ "-draft" ]
major          = 1*DIGIT
minor          = 1*DIGIT

The major family is the substring up to (but not including) the first . — e.g. the family of contextgraph/1.0-draft is contextgraph/1.

Two version strings interoperate if and only if they share a major family. contextgraph/1.0-draft and contextgraph/1.0 both belong to family contextgraph/1 and interoperate; contextgraph/2.0 does not interoperate with either. The -draft suffix marks a not-yet-frozen version within a family and does not affect interoperability. This rule is implemented by contextgraph-host::wire::versions_compatible; an out-of-Rust implementation SHOULD compare major families directly rather than hardcoding a specific version string.

See stability.md for the crate-version vs. protocol-version distinction and the draft-to-freeze model, and GOVERNANCE.md for who decides the freeze.