Skip to content

Latest commit

 

History

History
176 lines (147 loc) · 10 KB

ingest.md

File metadata and controls

176 lines (147 loc) · 10 KB

Record Ingest

Submitting a provenance record to the registry validates it synchronously: POST /records runs the validator chain (the same plugins behind the /validation endpoint, see validation.md), stores the record in Apache Atlas — byte-identical, keyed by its record IRI — and answers with the verdict.

Record types

The spec defines more than one record type, and the record's own type term decides everything downstream — the schema it is judged by, the entity type it is stored under, and the attributes a VALIDATED record gets mapped to:

Record type Atlas entity type Mapped attributes (VALIDATED)
ProvenanceRecord sourcelume_dataset name, description, sourceUri, licenseId
TrainingProvenanceRecord sourcelume_model name, description, modelUri, modelVersion, disclosure

A TrainingProvenanceRecord is the model producer's claim of which datasets a model was trained on; its trainingData list references dataset records by IRI. Per the spec design those referenced records are resolved when the model record is read — nothing about them is stored on the model entity — and the dataset records themselves stay untouched. GET /records/{id} on a VALIDATED model record lists each trainingData entry with the referenced dataset record's current recordStatus, absent when the registry holds no record for that IRI (lookupFailed: true instead when the backend could not be asked). The distinct datasets are resolved together through the storage SPI. The Atlas plugin reads the graph in batches, so a long list does not become one call per dataset. A failed lookup marks every dataset in that read and does not try again.

A ProvenanceRecord's custodyChain agents are resolved the same way. An agent may be the id of any record — a dataset, or a model that rewrote the data — or an external identifier such as an organization's homepage. An IRI denotes exactly one record, of one type, so the lookup answers with that one record whatever store holds it. GET /records/{id} on a VALIDATED dataset record lists each custodyChain event with that record's current recordStatus, absent when the IRI denotes no record (lookupFailed: true when the backend could not be asked). The lookup goes one hop and reports status only; nothing is stored on either record. The distinct agents use the same batch-read contract; Atlas sends batches of requests per entity type. A failed lookup marks every agent in that read.

A submission whose type is absent or unknown still gets its verdict: it fails validation (there is no schema for it) and lands as INCOMPLETE on the dataset entity, where it stays curatable like any other failed record. Its id remains assigned to the dataset record type; correct it as a ProvenanceRecord, or use a new id for a training record.

The flow

POST /records ──► validate ──► store with verdict ──► answer
                    │                                   │
                    ├─ conforms ──► VALIDATED       201 / 200
                    └─ issues     ─► INCOMPLETE     + issues in the body

There is no intermediate state: the record reaches the store only with its verdict. That is a stronger guarantee than "nothing is published without validation" — no verdict-less record ever sits in Atlas awaiting judgment.

Curating provenance data usually takes several rounds (a missing license, an incomplete custody chain, ...), so an invalid record is not rejected and thrown away. It is stored as INCOMPLETE with its issues attached, the owner fixes the document and resubmits it under the same id and record type, and the new submission gets a new verdict. An id's record type is fixed from the first submission, even while the record is INCOMPLETE. Submitting another record type under that id returns 409 without changing or deleting the original record.

The record state model

The verdict lives on the entity as the recordStatus attribute, an enum deliberately independent of the backend's own lifecycle (Atlas' ACTIVE/DELETED/PURGED say whether an entity exists; this says what the validation concluded):

  • VALIDATED — the record conforms to the spec. Consumers should filter on this status. The mapped provenance attributes are written from the record (see the record-types table above).
  • INCOMPLETE — the record carries its validation issues and stays curatable. A corrected resubmission is validated in place.

Every verdict carries its context: sha256 over the received request bytes (the endpoint hashes the body before decoding — "byte for byte" for real), validatedBy (the validator chain, comma-separated ids), and validatedAt (ISO-8601 UTC). Without them a stored verdict loses its meaning as the spec and the chain evolve. The POST answer and the GET status view both report the context. The document itself comes back unchanged from GET /records/{id}/document — the bytes a signature or the digest is checked against; nothing is mapped or reserialized on the way out.

REST semantics

An id denotes one record. Records are written only via POST /records; /api/v1/datasets is read-only (GET by qualifiedName, path or GUID).

Request Result
POST /records (JSON-LD body) 201 new record, 200 resubmission under the same record type — body: qualifiedName, type, recordStatus, sha256, validatedBy, validatedAt, and for INCOMPLETE the validationIssues
POST /records for an id that is already VALIDATED or stored under another entity type 409 — the stored record is unchanged; body: qualifiedName, recordStatus (absent for a plain dataset), entityType (the store holding the id), reason
POST /records without a usable @id, unparseable, or not valid UTF-8 400
POST /records with an unsupported content type 415
GET /records/{id} 200 record view: type, status, the type's mapped attributes, trainingData with each dataset's status for a VALIDATED model record, custodyChain with each agent's record status for a VALIDATED dataset record, and for INCOMPLETE the issues + verdict context / 404
GET /records/{id}/document 200 — the stored JSON-LD document, byte-identical to the submission, as application/ld+json; the record's sha256 repeats as ETag. For any verdict, VALIDATED or INCOMPLETE / 404 — no record for the id, or the entity under it carries no document
POST /records, a GET on /records or a /api/v1/datasets view when the backend can't be reached 503 — retry later; body: message (error on /api/v1/datasets, like its other errors)
POST /records, a GET on /records or a /api/v1/datasets view when the backend answers with an error 502 — body as for 503; the backend's answer is only logged

The id of the JSON-LD document doubles as the record's qualified name; the Location header of the answers points at the (URL-encoded) status view.

Design notes

  • Synchronous on purpose. Validating a provenance record is fast (JSON Schema and SHACL over a KB-sized document), so the simplest design wins: no second application, no polling, no race window, no poison records — and the issues are in the response instead of a poll away. This follows the dev-list discussion on the ingest proposal.
  • The seam for a queue. IngestService.process() is the pipeline; the REST resource calls it inline. If a validation step ever actually gets slow (attestation over large payloads, say), a queue and a worker slot in behind the same SPI without touching the pipeline or the status model. Until then, a queue would pay async's price without earning it: records are independent (no ordering to preserve), volumes are small, the raw document on the entity is the retention, and the write hits Atlas either way.
  • 409 instead of update. A published (VALIDATED) record never changes — legitimate updates (a new custody event, a corrected license) mean a new id linked to the previous record. The linking is deferred until the spec covers it; the rejection is deliberate.
  • Known limit: the 409 check is check-then-write, and Atlas has no compare-and-set. Two concurrent submissions for the same id race; serialization comes with ownership and auth.

Testing

Unit and resource tests run against stubs by design; the end-to-end round runs against a real Apache Atlas via Testcontainers (mvn verify -Pintegration-tests, see dev-support/README.md): submit, verdict, resubmission cycle, duplicate rejection, and the bootstrap's idempotency on an existing type.

Metadata storage

The common ingest path now calls RecordStore through the selected metadata plugin. DefaultIngestService owns validation and revision selection; the plugin owns the authoritative condition check and persistence. See storage-spi.md for the call flow and Atlas mapping.

The HTTP API retains its validation status codes, top-level recordStatus, revisionless correction behavior and document retrieval route. A document above the selected store's size limit returns 413. An ambiguous Atlas write returns 503 with X-Sourcelume-Error: UNKNOWN_WRITE; it requires reconciliation rather than an automatic retry. Atlas deployments require a single active writer.