validation.md

Record Validation

This document describes the record validation architecture of the Apache Sourcelume Registry: the RecordValidator SPI in the core, the shipped validation plugins, and the design decisions behind them.

The goal is Java-side parity with the Python reference validator (sourcelume-spec tools/validate.py, jsonschema[format] + pyshacl), so records can be validated at ingest time inside the JVM — while the spec artifacts (JSON Schema, SHACL shapes, JSON-LD context) remain the single source of truth, packaged in the sourcelume-spec artifact.

Plugin architecture

Validation is a chain of plugins, not a built-in feature:

• The core (sourcelume-registry-core) defines only the contract: RecordValidator (stable id(), execution order(), and validate(String jsonLd) that reports failures as results, never as exceptions), the immutable ValidationResult/ValidationIssue types, and the ValidatorChain.
• Concrete validators live in separate plugin modules, each owning its validation machinery. They register themselves through the JDK ServiceLoader (META-INF/services); the chain discovers whatever is on the classpath.

Chain semantics (mirroring validate.py, which runs the JSON Schema stage before SHACL and only invokes the latter when the former passes):

• Validators run by ascending order().
• The chain is fail-fast: the first non-conforming validator ends the chain; its issues (plus everything collected so far) are returned.
• An empty chain is rejected (IllegalStateException): a chain that validates nothing would silently accept every document.
• Duplicate ids are rejected (IllegalStateException): two plugins claiming the same stage is a deployment error. Swapping a plugin means swapping the jar, not running both.

Validators take the raw JSON-LD string, not a DTO: the JSON-LD semantics (term-to-IRI expansion) must be preserved, and validation must be possible before any DTO mapping happens.

Shipped plugins

Module	Stage id	Order	Machinery	Weight
sourcelume-registry-validation-jsonschema	json-schema	100	networknt json-schema-validator 2.x (draft 2020-12)	light
sourcelume-registry-validation-shacl	shacl	200	Apache Jena SHACL 6.x (JSON-LD via Titanium)	~18 MB

Deployments pick the stages they want by adding or removing plugin jars; the discovery mechanism does the rest. The core stays free of both networknt and Jena dependencies.

Both stages are record-type aware — the spec defines more than one record type (ProvenanceRecord, TrainingProvenanceRecord):

• The json-schema plugin bundles one schema per record type and picks it by the document's type term — the same selection validate.py makes via its SCHEMA_FILES map. A document with no usable type, or one no bundled schema knows, fails the stage.
• The shacl plugin's shapes target each record class (sl:ProvenanceRecord, sl:TrainingProvenanceRecord); its trivial-conformance guard (below) accepts a node of any known record type as the focus.

Usage

ValidatorChain chain = ValidatorChain.discover();
ValidationResult result = chain.validate(jsonLd);
if (!result.conforms()) {
    result.issues().forEach(issue ->
        System.out.println(issue.severity() + " " + issue.path() + ": " + issue.message()));
}

The "no record without a verdict" policy is enforced by the core's IngestService, which the REST layer calls synchronously on every submission: it runs the chain and stores the record with its verdict — VALIDATED for a conforming record, INCOMPLETE with the structured issues attached otherwise (see ingest.md). The throwing ValidationService facade with its SourcelumeValidationException stays reserved for callers that want validation as a gate; the ingest pipeline deliberately treats a failed validation as data, not an exception — an INCOMPLETE record is a normal, expected outcome of curation.

Runtime wiring and the validation endpoint

The Quarkus runtime depends on both plugin modules and wires the chain once at startup (ValidatorChainProducer): ValidatorChain.discover() throws when no plugins are found, so a mispackaged registry fails to boot instead of silently accepting unvalidated records. There is no switch to disable validation — a registry that skips it contradicts its purpose. Because CDI producers of @ApplicationScoped scope are created lazily on first access, the producer observes the Quarkus StartupEvent and touches the (proxied) chain there — discovery and shape/schema parsing happen exactly once, during boot, guaranteed.

That wiring backs a pre-flight endpoint, POST /records/validate:

• Request: the raw JSON-LD record (Content-Type: application/json).
• 200 with {"conforms": true, "issues": []} if the record passes both stages.
• 422 with the issues of the first failing validator (the chain is fail-fast) otherwise; malformed JSON is reported through the same contract, never a 500.

The response is a wire DTO, not the core ValidationResult — the HTTP contract stays stable while core evolves. The endpoint touches no backend (Atlas stays out of it on purpose), so producers can pre-flight records before submitting them; the same chain will enforce itself during ingest.

Why two stages? The alias gap

The two stages overlap by roughly 90%: required properties, string types, cardinality. One class of errors, however, is invisible to JSON Schema:

The sourcelume schema allows additional properties (a deliberate 0.0.1 decision). Term aliases therefore cannot be constrained at the JSON level.

A record can claim an Apache-2.0 license under "license" and a GPL-3.0 license under the full IRI "http://purl.org/dc/terms/license". The JSON Schema passes (additionalProperties is unconstrained). After JSON-LD expansion both claims map to dct:license on the same node — and the SHACL shapes reject it (maxCount 1 and nodeKind sh:IRI).

Both stages together see what the registry actually stores and queries: the JSON as submitted, and the RDF graph after expansion. The shared fixtures were run against the Python oracle (tools/validate.py) during development with matching verdicts; keeping that parity honest is exactly why the fixtures live next to the plugin tests. Each plugin module deliberately carries its own copies of the shared fixtures (module-local test resources, no cross-module test coupling) — when a fixture changes, update both copies and re-run the oracle.

SHACL plugin: design decisions and findings

Offline context policy (SSRF)

Documents reference the JSON-LD context by IRI; the spec fixes it as https://sourcelume.apache.org/context/0.0.1/sourcelume.jsonld. Fetching whatever IRI an ingested document names is still not acceptable:

• validation would depend on network access, and
• ingest payloads would direct the validator at arbitrary hosts (SSRF).

The JsonLdContextInliner therefore replaces every reference ending in the bundled context resource path with the context shipped in the sourcelume-spec artifact and refuses documents with any other string reference. Inline context objects are passed through, but scanned recursively for every construct that would make the RDF parser retrieve a remote document — @import and term-scoped context references (in string and array form, and at any depth inside term definitions) are refused, so the offline guarantee is structural (a JSON-LD processor only ever retrieves documents for exactly those constructs). Both amplification channels are capped: the number of replacements (a document made of repeated context references cannot amplify into the serialized form) and the number of reported refusals (a document made of many distinct unknown references cannot amplify into the issue list and the HTTP response). Documents without any @context at all, or with a @context that is neither a string, an array, nor an object, are refused as well: their terms would not expand, the graph would be empty of sourcelume types, and an empty graph trivially conforms to the shapes — silently accepting garbage would be worse than rejecting the document.

The same trivial-conformance concern applies to context games: an inline context can redefine the type term mapping (or be simply empty), in which case the document expands to no node of a known record type and the shapes would conform vacuously. The plugin therefore refuses any graph that, after expansion, contains no node typed sl:ProvenanceRecord or sl:TrainingProvenanceRecord — validation must never succeed by dropping the focus node.

Format assertions (parity with jsonschema[format])

Since JSON Schema draft 2019-09, format only annotates by default. The Python reference validator enforces date-time/uri via jsonschema[format] (rfc3339-validator, rfc3987), so the json-schema plugin explicitly enables networknt's format assertions and pins the message locale to Locale.ROOT for deterministic output. Bad timestamps and malformed URIs are violations, not annotations.

Apache Jena as the SHACL engine

SHACL validates RDF graphs, not JSON trees: the document must be expanded to RDF first (JSON-LD → RDF via Titanium in Jena 6.x), which is why a SHACL stage cannot be built from a JSON library alone — the same reason the Python side pairs pyshacl with rdflib. Jena executes the spec's sourcelume.shacl.ttl directly (SHACL Core only: targetClass, datatype, min/maxCount, nodeKind, sh:in), keeping the shapes file the single source of truth for both implementations.

ASF licensing note (category B)

jena-shacl transitively depends on org.glassfish:jakarta.json (EPL-2.0 / GPL-2.0 with Classpath Exception), an ASF category B dependency. That is acceptable as a binary dependency, but it must be reflected in LICENSE/NOTICE when this plugin is bundled in a distribution. All other Jena transitives are category A (Apache/MIT/BSD).

Known spec follow-ups

These are open questions for dev@sourcelume.apache.org, not blockers:

• SHACL normativity: the spec prose describes RDF-level validation but does not yet state normatively that SHACL shapes must pass. The registry treats them as normative for ingest.

License

Apache License, Version 2.0 — see the repository root.