Submitting a provenance record to the registry validates it synchronously:
POST /records runs the validator chain (the same plugins behind the
/validation endpoint, see validation.md), stores the
record in Apache Atlas — byte-identical, keyed by its record IRI — and
answers with the verdict.
The spec defines more than one record type, and the record's own type
term decides everything downstream — the schema it is judged by, the
entity type it is stored under, and the attributes a VALIDATED record
gets mapped to:
Record type |
Atlas entity type | Mapped attributes (VALIDATED) |
|---|---|---|
ProvenanceRecord |
sourcelume_dataset |
name, description, sourceUri, licenseId |
TrainingProvenanceRecord |
sourcelume_model |
name, description, modelUri, modelVersion, disclosure |
A TrainingProvenanceRecord is the model producer's claim of which
datasets a model was trained on; its trainingData list references
dataset records by IRI. Per the spec design those referenced records are
resolved when the model record is read — nothing about them is stored
on the model entity — and the dataset records themselves stay untouched.
GET /records/{id} on a VALIDATED model record lists each trainingData
entry with the referenced dataset record's current recordStatus, absent
when the registry holds no record for that IRI (lookupFailed: true
instead when the backend could not be asked). The distinct datasets are
resolved together through the storage SPI. The Atlas plugin reads the
graph in batches, so a long list does not become one call per dataset. A failed lookup marks
every dataset in that read and does not try again.
A ProvenanceRecord's custodyChain agents are resolved the same way.
An agent may be the id of any record — a dataset, or a model that
rewrote the data — or an external identifier such as an organization's
homepage. An IRI denotes exactly one record, of one type, so the lookup
answers with that one record whatever store holds it. GET /records/{id} on a VALIDATED dataset record lists each custodyChain
event with that record's current recordStatus, absent when the IRI
denotes no record (lookupFailed: true when the backend could not be
asked). The lookup goes one hop and reports status only; nothing is
stored on either record. The distinct agents use the same batch-read
contract; Atlas sends batches of requests per entity type. A failed
lookup marks every agent in that read.
A submission whose type is absent or unknown still gets its verdict:
it fails validation (there is no schema for it) and lands as
INCOMPLETE on the dataset entity, where it stays curatable like any
other failed record. Its id remains assigned to the dataset record type;
correct it as a ProvenanceRecord, or use a new id for a training record.
POST /records ──► validate ──► store with verdict ──► answer
│ │
├─ conforms ──► VALIDATED 201 / 200
└─ issues ─► INCOMPLETE + issues in the body
There is no intermediate state: the record reaches the store only with its verdict. That is a stronger guarantee than "nothing is published without validation" — no verdict-less record ever sits in Atlas awaiting judgment.
Curating provenance data usually takes several rounds (a missing license,
an incomplete custody chain, ...), so an invalid record is not rejected
and thrown away. It is stored as INCOMPLETE with its issues attached,
the owner fixes the document and resubmits it under the same id and record
type, and the new submission gets a new verdict. An id's record type is
fixed from the first submission, even while the record is INCOMPLETE.
Submitting another record type under that id returns 409 without
changing or deleting the original record.
The verdict lives on the entity as the recordStatus attribute, an enum
deliberately independent of the backend's own lifecycle (Atlas'
ACTIVE/DELETED/PURGED say whether an entity exists; this says what the
validation concluded):
- VALIDATED — the record conforms to the spec. Consumers should filter on this status. The mapped provenance attributes are written from the record (see the record-types table above).
- INCOMPLETE — the record carries its validation issues and stays curatable. A corrected resubmission is validated in place.
Every verdict carries its context: sha256 over the received request
bytes (the endpoint hashes the body before decoding — "byte for byte"
for real), validatedBy (the validator chain, comma-separated ids),
and validatedAt (ISO-8601 UTC). Without them a stored verdict loses
its meaning as the spec and the chain evolve. The POST answer and the
GET status view both report the context. The document itself comes back
unchanged from GET /records/{id}/document — the bytes a signature or
the digest is checked against; nothing is mapped or reserialized on the
way out.
An id denotes one record. Records are written only via POST /records; /api/v1/datasets is read-only (GET by qualifiedName, path or GUID).
| Request | Result |
|---|---|
POST /records (JSON-LD body) |
201 new record, 200 resubmission under the same record type — body: qualifiedName, type, recordStatus, sha256, validatedBy, validatedAt, and for INCOMPLETE the validationIssues |
POST /records for an id that is already VALIDATED or stored under another entity type |
409 — the stored record is unchanged; body: qualifiedName, recordStatus (absent for a plain dataset), entityType (the store holding the id), reason |
POST /records without a usable @id, unparseable, or not valid UTF-8 |
400 |
POST /records with an unsupported content type |
415 |
GET /records/{id} |
200 record view: type, status, the type's mapped attributes, trainingData with each dataset's status for a VALIDATED model record, custodyChain with each agent's record status for a VALIDATED dataset record, and for INCOMPLETE the issues + verdict context / 404 |
GET /records/{id}/document |
200 — the stored JSON-LD document, byte-identical to the submission, as application/ld+json; the record's sha256 repeats as ETag. For any verdict, VALIDATED or INCOMPLETE / 404 — no record for the id, or the entity under it carries no document |
POST /records, a GET on /records or a /api/v1/datasets view when the backend can't be reached |
503 — retry later; body: message (error on /api/v1/datasets, like its other errors) |
POST /records, a GET on /records or a /api/v1/datasets view when the backend answers with an error |
502 — body as for 503; the backend's answer is only logged |
The id of the JSON-LD document doubles as the record's qualified name;
the Location header of the answers points at the (URL-encoded)
status view.
- Synchronous on purpose. Validating a provenance record is fast (JSON Schema and SHACL over a KB-sized document), so the simplest design wins: no second application, no polling, no race window, no poison records — and the issues are in the response instead of a poll away. This follows the dev-list discussion on the ingest proposal.
- The seam for a queue.
IngestService.process()is the pipeline; the REST resource calls it inline. If a validation step ever actually gets slow (attestation over large payloads, say), a queue and a worker slot in behind the same SPI without touching the pipeline or the status model. Until then, a queue would pay async's price without earning it: records are independent (no ordering to preserve), volumes are small, the raw document on the entity is the retention, and the write hits Atlas either way. - 409 instead of update. A published (VALIDATED) record never changes — legitimate updates (a new custody event, a corrected license) mean a new id linked to the previous record. The linking is deferred until the spec covers it; the rejection is deliberate.
- Known limit: the 409 check is check-then-write, and Atlas has no compare-and-set. Two concurrent submissions for the same id race; serialization comes with ownership and auth.
Unit and resource tests run against stubs by design; the end-to-end
round runs against a real Apache Atlas via Testcontainers
(mvn verify -Pintegration-tests, see dev-support/README.md): submit,
verdict, resubmission cycle, duplicate rejection, and the bootstrap's
idempotency on an existing type.
The common ingest path now calls RecordStore through the selected metadata
plugin. DefaultIngestService owns validation and revision selection; the plugin
owns the authoritative condition check and persistence. See
storage-spi.md for the call flow and Atlas mapping.
The HTTP API retains its validation status codes, top-level recordStatus,
revisionless correction behavior and document retrieval route. A document above
the selected store's size limit returns 413. An ambiguous Atlas write returns
503 with X-Sourcelume-Error: UNKNOWN_WRITE; it requires reconciliation rather
than an automatic retry. Atlas deployments require a single active writer.