design review: shared sandbox, evaluation and training boundaries

[Sam123ben]

Parent: #1. Related: #7, #10, tesserix/devai#392, #393, tesserix/ai-agents#26 and tesserix/australis#21.

Outcome

Define product-neutral GCS, CNPG and identity boundaries for Document Intelligence sandbox execution, evaluation corpora/results, training/calibration data and candidate model artefacts. The OCR service is reusable by every authorized product and agent through versioned service endpoints and Australis tools; it is not owned by Kora.

Kora is one consumer. Its production document buckets and Langfuse credentials remain product-specific and cannot become shared service training/evaluation infrastructure.

This is a design-review issue. It does not authorize live bucket, IAM, KMS, CNPG, Kubernetes, dataset-copy, training, model-promotion or rollout changes.

Proposed shared logical storage classes

• document-intelligence sandbox: ephemeral synthetic/public/de-identified documents only.
• document-intelligence dataset candidates: quarantined, governed nominations from any eligible product.
• document-intelligence training corpus: only approved train/calibration objects; never held-out test objects.
• document-intelligence development evaluation corpus: immutable approved fixtures for ordinary non-production evaluation.
• document-intelligence evaluation results: redacted scores, reports and safe trace correlation metadata.
• document-intelligence candidate models: signed model/profile bundles awaiting qualification.
• protected document-intelligence golden corpus: sealed held-out test set, unavailable to developers, sandboxes, training and ordinary CI identities.

Physical names and projects remain placeholders until GitOps review. A dedicated evaluation/training GCP project is preferred. Dataset manifests carry product source, tenant-policy class, allowed purpose, residency and consent/licence without granting one product access to another products raw data.

Reusable integration boundary

• Products upload/read only through authenticated v1 service endpoints, never shared bucket credentials.
• AI agents call the provider-neutral Australis extract_document tool or generated service SDK.
• Tenant and product identity come from verified workload/user identity, never request-controlled fields.
• API/result/event schemas are additive and versioned independently from OCR engines/models.
• Product-specific agents, prompts, policies and Langfuse projects remain outside the OCR service.
• Service model/evaluation promotion publishes a signed compatibility manifest usable by all approved consumers.

Review checklist

• Approve data classification and allowed provenance for every shared storage class.
• Approve globally unique physical names, owning GCP project, region, encryption and lifecycle rules.
• Prove every product runtime, including Kora, cannot directly read shared training/golden buckets.
• Prove sandbox/training identities cannot read any production product document bucket or the held-out test corpus.
• Prove the protected evaluator cannot mutate frozen corpus versions.
• Define immutable manifest promotion without in-place edits or cross-product data leakage.
• Define poisoning, licence/consent, de-identification, duplicate/leakage and human-adjudication gates.
• Define global CNPG registry states and roles; bytes remain in GCS.
• Define DevAI PR, nightly, release, shadow and canary execution with separate non-production credentials.
• Define reproducibility tuple: dataset, labels, code, model, preprocessing, calibration, schema, runtime and hardware digests.
• Define TTL cleanup, cost quotas, deletion/withdrawal, audit, backup/restore and incident response.
• Require zero production credentials and zero corpus bytes in PR/fork workflows or CI artefacts.
• Add contract tests proving two different products can use the service without data, cache, trace or policy crossover.

Exit criteria

A reviewer-approved matrix shows which shared service identity can read/write each bucket and CNPG relation; product identities use endpoints/tools only; datasets cannot enter training/evaluation without immutable governed versions; test leakage is mechanically prevented; DevAI compares candidates reproducibly; and promotion/rollback operate on a signed service compatibility manifest.

[Sam123ben]

Sandbox, evaluation and training data design

Outcome

Kora needs a safe place to exercise the OCR service and document agents, a
governed corpus for improving models and preprocessing, and a sealed benchmark
that proves those changes generalize. These are separate trust and data-use
boundaries. A convenient shared bucket would allow accidental production-data
use, benchmark leakage and unrepeatable evaluations.

This document defines logical placeholders. Physical GCS names, projects,
regions, identities, KMS policy and budgets are approved and delivered through
GitOps in tesserix/tesserix-k8s#914.

Initial operating guardrails

These are starting safety and cost limits, not product forecasts:

Dimension	Initial design point
Sandbox admission	2 jobs/s and 10 pages/s globally; at most 5 active jobs per user
Sandbox input	Same 100 MiB/300-page hard parser bounds as production
Sandbox object lifetime	24 hours by default, 72 hours maximum by approved run policy
Evaluation scratch lifetime	24 hours; redacted aggregate reports retained 30 days
Nightly evaluation throughput	Capped at 25 pages/s so it cannot starve shared workloads
Training concurrency	One run per approved accelerator pool until capacity is measured
CI from forks	No cloud credentials, corpus access, model registry access or Langfuse secrets

Every run has page, accelerator-hour, storage-byte, provider-spend and wall-time
budgets. A missing budget fails admission. These limits must be replaced with
measured values before production qualification.

Logical GCS storage classes

Do not create one bucket per run, agent or document. Use buckets by trust and
lifecycle class, then opaque run/dataset/case paths inside them.

Logical placeholder	Content	Default lifecycle	Allowed provenance
nonprod-kora-di-sandbox	Ephemeral inputs, derived pages and agent/OCR outputs	Delete after 24 hours; bounded lease up to 72 hours	Synthetic, public or explicitly de-identified non-production fixtures
nonprod-kora-di-dataset-candidates	Quarantined samples nominated for possible corpus use	Delete rejected candidates within 30 days	Consented nomination plus provenance record; not yet trusted for training
nonprod-kora-di-training	Immutable train and calibration dataset versions	Dataset-governance retention	Approved, licensed, de-identified where required and human-reviewed cases only
nonprod-kora-di-eval-corpus	Immutable development/regression evaluation versions	Dataset-governance retention	Synthetic, public or approved non-sensitive cases suitable for non-production runners
nonprod-kora-di-eval-results	Redacted predictions, metric inputs, reports and scratch	Scratch 24 hours; redacted reports 30 days unless promoted as evidence	Outputs only; no unrestricted source-document copies
nonprod-kora-di-model-staging	Signed candidate model and compatibility bundles	Keep active candidate plus bounded rollback history	Reproducible training/build outputs that passed supply-chain checks
prod-kora-ocr-golden	Sealed held-out golden test corpus	Separate protected governance policy	Approved benchmark cases; never a training input

A separate non-production GCP project is preferred. If platform constraints keep
these buckets in a shared project, separate service accounts, Workload Identity,
KMS keys, bucket IAM and automated deny tests are mandatory. Production Kora
document buckets and production Langfuse secrets are outside every sandbox,
training and ordinary CI identity.

Example non-sensitive object names are:

runs/{run_id}/inputs/{case_id}/{source_digest}
runs/{run_id}/artifacts/{step}/{artifact_digest}
candidates/{candidate_id}/objects/{object_digest}
datasets/{dataset_id}/versions/{manifest_digest}/cases/{case_id}/{media_role}
training-runs/{training_run_id}/outputs/{artifact_digest}
models/{model_family}/candidates/{compatibility_digest}/{artifact_role}
evaluations/{experiment_id}/results/{result_digest}

Names contain no tenant name, filename, email, patient/customer name, label value
or other business content. Every reference includes exact GCS generation and
SHA-256 digest in the CNPG registry.

Agent-scoped dataset namespaces

Dataset ownership is per agent and capability. One universal score would hide
different tools, risks and success criteria. A logical manifest view is:

datasets/
├── customer-support-agent/
│   ├── normal-queries/
│   ├── tool-selection/
│   ├── policy-compliance/
│   └── adversarial/
├── ocr-agent/
│   ├── invoices/
│   ├── receipts/
│   ├── handwriting/
│   └── poor-quality-scans/
└── sre-agent/
    ├── kubernetes/
    ├── incident-analysis/
    ├── safe-remediation/
    └── permission-boundaries/

This tree is a registry/manifest namespace, not a requirement to expose raw GCS
prefix listing. Each case is addressed by opaque ID and immutable manifest. An
agent evaluation case has a constrained expected outcome rather than requiring
one exact natural-language answer:

{
  "schema_version": "agent-eval-case/v1",
  "agent": "ocr-agent",
  "case_id": "opaque-case-id",
  "input_ref": {
    "object_digest": "sha256:...",
    "generation": "immutable-generation"
  },
  "expected_outcome": {
    "required_facts": [],
    "acceptable_ranges": {},
    "required_citations": [],
    "output_schema_digest": "sha256:..."
  },
  "required_tools": [],
  "forbidden_actions": [],
  "forbidden_claims": [],
  "evaluation_rules": [],
  "safety_boundaries": [],
  "tags": ["invoice", "low-resolution", "english"]
}

Sensitive values stay in referenced encrypted objects. The registry record adds
provenance, classification, consent/licence, residency, retention, split,
document-family/duplicate cluster, label/adjudication version and allowed-use
policy. Case schemas are versioned and bounded; arbitrary evaluator code cannot
be supplied inside a dataset case.

Identity and access matrix

Identity	Sandbox	Candidates	Training	Dev eval corpus	Eval results	Model staging	Sealed golden
Developer/local	bounded API only	nominate metadata	none	none	own safe summary	none	none
PR workflow	none	none	none	none; repository fixtures only	CI summary only	none	none
Sandbox runner	read/write leased prefix	nominate opaque reference	none	read approved version	write run prefix	read approved candidate	none
Dataset curator	bounded reviewed read	read/adjudicate	publish frozen versions	publish approved versions	none	none	proposal only
Training runner	none	none	read frozen train/calibration	none	write training metrics	write candidate prefix	none
Nightly evaluator	none	none	none	read one frozen version	write experiment prefix	read candidate/baseline	none
Protected evaluator	none	none	none	optional read	write protected experiment prefix	read candidate/baseline	read one frozen version
Promotion controller	none	none	manifests only	manifests only	read signed decision	promote signed alias	manifests only
Lifecycle controller	delete expired	delete rejected/expired	policy delete	policy delete	delete expired	policy delete	governed deletion only

No identity has storage.admin. The training runner cannot read either evaluation
corpus. The protected evaluator can read but never mutate a frozen golden version.
DevAI asks runners to execute opaque run specifications; it does not receive raw
corpus bytes or permanent/signed object URLs.

Dataset admission and split sealing

flowchart LR
    Signal[Review correction or failure signal] --> Eligibility{Consent and policy eligible?}
    Eligibility -->|no| Reject[Reject and expire]
    Eligibility -->|yes| Candidate[(Candidate bucket)]
    Candidate --> Scan[Malware, classification and provenance]
    Scan --> DeID[De-identification where required]
    DeID --> Label[Human label and adjudication]
    Label --> Dedupe[Family and near-duplicate clustering]
    Dedupe --> Split{Governed split assignment}
    Split --> Train[(Train version)]
    Split --> Calibration[(Calibration version)]
    Split --> DevEval[(Development evaluation version)]
    Split --> Golden[(Sealed held-out golden version)]

Loading

Admission never moves a mutable object in place. The curator writes a new
content-addressed object, verifies its digest/generation and freezes an immutable
manifest in CNPG. Split assignment is by document family, template and
near-duplicate cluster, not individual page, so related samples cannot cross
train/calibration/evaluation boundaries.

The test split is sealed before tuning begins. Engineers, training jobs and
ordinary evaluation jobs cannot list or read it. The protected evaluator returns
only approved aggregate/per-cohort scores and safe case IDs for failures. A test
case cannot be moved into training without creating a new dataset lineage and a
new independently reviewed held-out test version.

CNPG registry additions

Global CNPG remains authoritative for governance and experiment state; GCS owns
bytes. Extend the ocr_evaluation model with:

• storage_classes and purpose_policies for allowed provenance, use and TTL;
• dataset_access_grants binding an identity, frozen version, purpose and expiry;
• dataset_lineage connecting candidates, labels, split decisions and derived versions;
• training_runs with code, dataset, hyperparameter, seed, runtime and hardware digests;
• model_candidates with artefact digest, SBOM, signature and compatibility manifest;
• evaluation_runs with candidate/baseline, metric implementation and environment digests;
• budget_reservations for pages, accelerator time, provider cost and storage;
• deletion_operations and append-only audit/outbox events.

The registry grants a frozen version, not a mutable bucket prefix. Creating an
access grant and budget reservation is one short transaction. Runner execution,
GCS access and Langfuse export happen after commit and are idempotently driven by
the outbox.

Agent and OCR engineering loop

flowchart TD
    Observe[Safe production metrics and consented review signals] --> Curate[Candidate governance]
    Curate --> Version[Freeze train, calibration and eval versions]
    Version --> Train[Train or optimize preprocessing/model]
    Train --> Stage[Sign candidate compatibility manifest]
    Stage --> DevAI[DevAI deterministic and development eval]
    DevAI --> Protected[Protected held-out golden evaluation]
    Protected --> Gate{Quality, safety, latency and cost pass?}
    Gate -->|no| Diagnose[Use train/dev-eval evidence only]
    Diagnose --> Train
    Gate -->|yes| Shadow[GitOps shadow]
    Shadow --> Canary[Bounded canary]
    Canary --> Promote[Promote signed alias]
    Promote --> Monitor[Monitor drift and review rate]
    Monitor --> Observe

Loading

The deployable compatibility manifest binds the Rust engine image, models,
preprocessing transforms, confidence calibration, extraction/validation schemas,
route policy, agent version, Australis tool contract, result schema, runtime and
minimum hardware. DevAI compares candidate and baseline on the same frozen
dataset, metrics, runtime and hardware. Rollback moves the alias back to the
previous signed compatibility manifest.

Training uses only train data. Threshold selection and confidence calibration
use only calibration data. Developers diagnose against development evaluation
data. The held-out golden set is evaluated only by the protected release gate;
its results cannot be used repeatedly as a tuning dashboard.

CI/CD and Langfuse

Stage	Storage access	Langfuse	Required result
Pull request/fork	Repository synthetic/public fixtures only	None or local mock	Contract, unit, property, fuzz, security and deterministic agent assertions
Internal merge/nightly	One frozen development-eval version	Dedicated non-production evaluation project	Candidate/baseline cohort quality, latency, memory and cost
Training	One frozen train/calibration grant	Dedicated non-production training/evaluation project if traces are needed	Signed candidate plus reproducibility record
Protected release	One frozen sealed-golden grant	Dedicated protected evaluation project/credentials	Release decision and redacted evidence
Shadow/canary	Consented policy-eligible traffic only	Environment-specific safe traces	SLO, unsafe-accept, review, cost and rollback decision

The two Kora production Langfuse credentials are never used by sandbox, training,
DevAI, local development, PR or fork workflows. Langfuse receives opaque run/case
IDs, version digests, cohort tags, safe metrics, latency, cost and trace links.
It receives no corpus bytes, ground-truth values, signed URLs or unrestricted OCR
text. CNPG remains authoritative if Langfuse is unavailable.

Failure, deletion and abuse handling

• Sandbox or evaluation outage does not affect production document processing.
• Training outage leaves the last signed model active and produces no partial promotion.
• A budget reservation prevents runaway retries or accelerator/provider spend.
• Duplicate run requests resolve to one run ID and result through a unique idempotency key.
• A crash leaves TTL-bound scratch or an unreferenced immutable object for the sweeper.
• Dataset withdrawal creates a new manifest, revokes future grants and evaluates which models/promotions are affected; history is never silently rewritten.
• Poisoning controls include source diversity caps, tenant/cohort sampling limits, human adjudication, anomaly review and reproducible label provenance.
• Malware, decompression bombs, prompt injection and hostile PDFs stay untrusted even after admission; every execution repeats bounded parsing in a sandbox.

Review gates

• Approve physical bucket names, owning projects, regions, KMS and lifecycle rules.
• Approve every Workload Identity and prove cross-environment and held-out-test denies.
• Approve corpus licences, consent, residency, de-identification and withdrawal policy.
• Approve CNPG schema/roles, budget model, backup/restore and audit retention.
• Establish baseline cohort thresholds for OCR, agent grounding/safety, latency and cost.
• Exercise dataset freeze, test-leakage denial, scratch expiry, credential denial,
  model signing, promotion and one-action rollback before enabling protected data.

[Sam123ben]

ADR-0007: Separate sandbox, training and held-out evaluation boundaries

Status: proposed

Context

Kora needs repeatable OCR and AI-agent experimentation. Sharing one bucket or
identity between sandbox, training and evaluation would expose sensitive data,
allow benchmark leakage and make a reported improvement impossible to trust.
Ordinary CI and contributor pull requests are an especially weak trust boundary.

Decision

Use separate logical GCS storage classes and Workload Identities for ephemeral
sandbox data, quarantined dataset candidates, frozen train/calibration versions,
development evaluation versions, evaluation outputs and staged model bundles.
Keep the held-out golden test corpus in its existing protected bucket and deny it
to developers, sandbox runners, training jobs and ordinary CI.

Global CNPG owns manifests, lineage, grants, budgets, experiments, promotion and
audit state. GCS owns bytes. DevAI orchestrates opaque evaluation requests and
receives redacted metrics/trace references. Langfuse is a non-authoritative safe
projection with separate non-production/protected credentials; Kora production
credentials are never reused.

Promote only a signed compatibility manifest binding OCR code/models,
preprocessing, calibration, extraction schemas, routing, agent, Australis tool
contract, result schema, runtime and hardware requirements. Dataset versions are
immutable. Training, calibration, development evaluation and held-out test have
separate purposes and access paths.

Alternatives

• One non-production bucket: rejected because one IAM/lifecycle boundary permits
  accidental training on evaluation data and broadens sandbox access.
• Copy the production golden corpus into DevAI: rejected because it duplicates
  sensitive data and gives CI a direct corpus path.
• Store datasets in Langfuse: rejected because Langfuse is experiment telemetry,
  not the canonical sensitive-data or governance store.
• Let training jobs evaluate the held-out test set: rejected because repeated
  visibility turns the test set into training feedback.
• Online learning from production corrections: rejected because consent,
  poisoning, label quality and rollback cannot be guaranteed.

Consequences

Experiments are reproducible and test leakage is mechanically constrained, at
the cost of more buckets, identities, manifests and lifecycle policies. Protected
evaluation can be unavailable without affecting production OCR. Physical names,
projects, regions, identities, quotas and retention remain review gates owned by
GitOps rather than defaults embedded in application code.

[Sam123ben]

Scope correction: this issue and all shared sandbox/training/evaluation/model assets are product-neutral Document Intelligence infrastructure. The earlier Kora-prefixed design comments are superseded by the current issue body and the design below. Kora remains one endpoint/tool consumer with separate runtime storage and credentials.

[Sam123ben]

Sandbox, evaluation and training data design

Outcome

Document Intelligence needs a safe place to exercise the OCR service and its
agent integrations, a governed corpus for improving models and preprocessing,
and a sealed benchmark that proves those changes generalize. These are separate
trust and data-use boundaries. A convenient shared bucket would allow accidental
production-data use, benchmark leakage and unrepeatable evaluations.

This document defines logical placeholders. Physical GCS names, projects,
regions, identities, KMS policy and budgets are approved and delivered through
GitOps in tesserix/tesserix-k8s#916.

Products never receive these bucket identities. They integrate with Document
Intelligence through the same versioned service API, generated client libraries
or the provider-neutral Australis tool. Dataset manifests retain product/source
provenance and policy without turning any one product into the owner of the
shared engine or evaluation platform.

Initial operating guardrails

These are starting safety and cost limits, not product forecasts:

Dimension	Initial design point
Sandbox admission	2 jobs/s and 10 pages/s globally; at most 5 active jobs per user
Sandbox input	Same 100 MiB/300-page hard parser bounds as production
Sandbox object lifetime	24 hours by default, 72 hours maximum by approved run policy
Evaluation scratch lifetime	24 hours; redacted aggregate reports retained 30 days
Nightly evaluation throughput	Capped at 25 pages/s so it cannot starve shared workloads
Training concurrency	One run per approved accelerator pool until capacity is measured
CI from forks	No cloud credentials, corpus access, model registry access or Langfuse secrets

Every run has page, accelerator-hour, storage-byte, provider-spend and wall-time
budgets. A missing budget fails admission. These limits must be replaced with
measured values before production qualification.

Logical GCS storage classes

Do not create one bucket per run, agent or document. Use buckets by trust and
lifecycle class, then opaque run/dataset/case paths inside them.

Logical placeholder	Content	Default lifecycle	Allowed provenance
nonprod-document-intelligence-sandbox	Ephemeral inputs, derived pages and agent/OCR outputs	Delete after 24 hours; bounded lease up to 72 hours	Synthetic, public or explicitly de-identified non-production fixtures
nonprod-document-intelligence-dataset-candidates	Quarantined samples nominated for possible corpus use	Delete rejected candidates within 30 days	Consented nomination plus provenance record; not yet trusted for training
nonprod-document-intelligence-training	Immutable train and calibration dataset versions	Dataset-governance retention	Approved, licensed, de-identified where required and human-reviewed cases only
nonprod-document-intelligence-eval-corpus	Immutable development/regression evaluation versions	Dataset-governance retention	Synthetic, public or approved non-sensitive cases suitable for non-production runners
nonprod-document-intelligence-eval-results	Redacted predictions, metric inputs, reports and scratch	Scratch 24 hours; redacted reports 30 days unless promoted as evidence	Outputs only; no unrestricted source-document copies
nonprod-document-intelligence-model-staging	Signed candidate model and compatibility bundles	Keep active candidate plus bounded rollback history	Reproducible training/build outputs that passed supply-chain checks
protected-document-intelligence-golden	Sealed held-out golden test corpus	Separate protected governance policy	Approved benchmark cases; never a training input

A separate non-production GCP project is preferred. If platform constraints keep
these buckets in a shared project, separate service accounts, Workload Identity,
KMS keys, bucket IAM and automated deny tests are mandatory. Every product's
production document buckets and Langfuse credentials are outside all sandbox,
training and ordinary CI identities.

Example non-sensitive object names are:

runs/{run_id}/inputs/{case_id}/{source_digest}
runs/{run_id}/artifacts/{step}/{artifact_digest}
candidates/{candidate_id}/objects/{object_digest}
datasets/{dataset_id}/versions/{manifest_digest}/cases/{case_id}/{media_role}
training-runs/{training_run_id}/outputs/{artifact_digest}
models/{model_family}/candidates/{compatibility_digest}/{artifact_role}
evaluations/{experiment_id}/results/{result_digest}

Names contain no tenant name, filename, email, patient/customer name, label value
or other business content. Every reference includes exact GCS generation and
SHA-256 digest in the CNPG registry.

Agent-scoped dataset namespaces

Dataset ownership is per agent and capability. One universal score would hide
different tools, risks and success criteria. A logical manifest view is:

datasets/
├── customer-support-agent/
│   ├── normal-queries/
│   ├── tool-selection/
│   ├── policy-compliance/
│   └── adversarial/
├── ocr-agent/
│   ├── invoices/
│   ├── receipts/
│   ├── handwriting/
│   └── poor-quality-scans/
└── sre-agent/
    ├── kubernetes/
    ├── incident-analysis/
    ├── safe-remediation/
    └── permission-boundaries/

This tree is a registry/manifest namespace, not a requirement to expose raw GCS
prefix listing. Each case is addressed by opaque ID and immutable manifest. An
agent evaluation case has a constrained expected outcome rather than requiring
one exact natural-language answer:

{
  "schema_version": "agent-eval-case/v1",
  "agent": "ocr-agent",
  "case_id": "opaque-case-id",
  "input_ref": {
    "object_digest": "sha256:...",
    "generation": "immutable-generation"
  },
  "expected_outcome": {
    "required_facts": [],
    "acceptable_ranges": {},
    "required_citations": [],
    "output_schema_digest": "sha256:..."
  },
  "required_tools": [],
  "forbidden_actions": [],
  "forbidden_claims": [],
  "evaluation_rules": [],
  "safety_boundaries": [],
  "tags": ["invoice", "low-resolution", "english"]
}

Sensitive values stay in referenced encrypted objects. The registry record adds
provenance, classification, consent/licence, residency, retention, split,
document-family/duplicate cluster, label/adjudication version and allowed-use
policy. Case schemas are versioned and bounded; arbitrary evaluator code cannot
be supplied inside a dataset case.

Identity and access matrix

Identity	Sandbox	Candidates	Training	Dev eval corpus	Eval results	Model staging	Sealed golden
Developer/local	bounded API only	nominate metadata	none	none	own safe summary	none	none
PR workflow	none	none	none	none; repository fixtures only	CI summary only	none	none
Sandbox runner	read/write leased prefix	nominate opaque reference	none	read approved version	write run prefix	read approved candidate	none
Dataset curator	bounded reviewed read	read/adjudicate	publish frozen versions	publish approved versions	none	none	proposal only
Training runner	none	none	read frozen train/calibration	none	write training metrics	write candidate prefix	none
Nightly evaluator	none	none	none	read one frozen version	write experiment prefix	read candidate/baseline	none
Protected evaluator	none	none	none	optional read	write protected experiment prefix	read candidate/baseline	read one frozen version
Promotion controller	none	none	manifests only	manifests only	read signed decision	promote signed alias	manifests only
Lifecycle controller	delete expired	delete rejected/expired	policy delete	policy delete	delete expired	policy delete	governed deletion only

No identity has storage.admin. The training runner cannot read either evaluation
corpus. The protected evaluator can read but never mutate a frozen golden version.
DevAI asks runners to execute opaque run specifications; it does not receive raw
corpus bytes or permanent/signed object URLs.

Dataset admission and split sealing

flowchart LR
    Signal[Review correction or failure signal] --> Eligibility{Consent and policy eligible?}
    Eligibility -->|no| Reject[Reject and expire]
    Eligibility -->|yes| Candidate[(Candidate bucket)]
    Candidate --> Scan[Malware, classification and provenance]
    Scan --> DeID[De-identification where required]
    DeID --> Label[Human label and adjudication]
    Label --> Dedupe[Family and near-duplicate clustering]
    Dedupe --> Split{Governed split assignment}
    Split --> Train[(Train version)]
    Split --> Calibration[(Calibration version)]
    Split --> DevEval[(Development evaluation version)]
    Split --> Golden[(Sealed held-out golden version)]

Loading

Admission never moves a mutable object in place. The curator writes a new
content-addressed object, verifies its digest/generation and freezes an immutable
manifest in CNPG. Split assignment is by document family, template and
near-duplicate cluster, not individual page, so related samples cannot cross
train/calibration/evaluation boundaries.

The test split is sealed before tuning begins. Engineers, training jobs and
ordinary evaluation jobs cannot list or read it. The protected evaluator returns
only approved aggregate/per-cohort scores and safe case IDs for failures. A test
case cannot be moved into training without creating a new dataset lineage and a
new independently reviewed held-out test version.

CNPG registry additions

Global CNPG remains authoritative for governance and experiment state; GCS owns
bytes. Extend the ocr_evaluation model with:

• storage_classes and purpose_policies for allowed provenance, use and TTL;
• dataset_access_grants binding an identity, frozen version, purpose and expiry;
• dataset_lineage connecting candidates, labels, split decisions and derived versions;
• training_runs with code, dataset, hyperparameter, seed, runtime and hardware digests;
• model_candidates with artefact digest, SBOM, signature and compatibility manifest;
• evaluation_runs with candidate/baseline, metric implementation and environment digests;
• budget_reservations for pages, accelerator time, provider cost and storage;
• deletion_operations and append-only audit/outbox events.

The registry grants a frozen version, not a mutable bucket prefix. Creating an
access grant and budget reservation is one short transaction. Runner execution,
GCS access and Langfuse export happen after commit and are idempotently driven by
the outbox.

Agent and OCR engineering loop

flowchart TD
    Observe[Safe production metrics and consented review signals] --> Curate[Candidate governance]
    Curate --> Version[Freeze train, calibration and eval versions]
    Version --> Train[Train or optimize preprocessing/model]
    Train --> Stage[Sign candidate compatibility manifest]
    Stage --> DevAI[DevAI deterministic and development eval]
    DevAI --> Protected[Protected held-out golden evaluation]
    Protected --> Gate{Quality, safety, latency and cost pass?}
    Gate -->|no| Diagnose[Use train/dev-eval evidence only]
    Diagnose --> Train
    Gate -->|yes| Shadow[GitOps shadow]
    Shadow --> Canary[Bounded canary]
    Canary --> Promote[Promote signed alias]
    Promote --> Monitor[Monitor drift and review rate]
    Monitor --> Observe

Loading

The deployable compatibility manifest binds the Rust engine image, models,
preprocessing transforms, confidence calibration, extraction/validation schemas,
route policy, agent version, Australis tool contract, result schema, runtime and
minimum hardware. DevAI compares candidate and baseline on the same frozen
dataset, metrics, runtime and hardware. Rollback moves the alias back to the
previous signed compatibility manifest.

Training uses only train data. Threshold selection and confidence calibration
use only calibration data. Developers diagnose against development evaluation
data. The held-out golden set is evaluated only by the protected release gate;
its results cannot be used repeatedly as a tuning dashboard.

CI/CD and Langfuse

Stage	Storage access	Langfuse	Required result
Pull request/fork	Repository synthetic/public fixtures only	None or local mock	Contract, unit, property, fuzz, security and deterministic agent assertions
Internal merge/nightly	One frozen development-eval version	Dedicated non-production evaluation project	Candidate/baseline cohort quality, latency, memory and cost
Training	One frozen train/calibration grant	Dedicated non-production training/evaluation project if traces are needed	Signed candidate plus reproducibility record
Protected release	One frozen sealed-golden grant	Dedicated protected evaluation project/credentials	Release decision and redacted evidence
Shadow/canary	Consented policy-eligible traffic only	Environment-specific safe traces	SLO, unsafe-accept, review, cost and rollback decision

No product's production Langfuse credentials are used by sandbox, training,
DevAI, local development, PR or fork workflows. Langfuse receives opaque run/case
IDs, version digests, cohort tags, safe metrics, latency, cost and trace links.
It receives no corpus bytes, ground-truth values, signed URLs or unrestricted OCR
text. CNPG remains authoritative if Langfuse is unavailable.

Failure, deletion and abuse handling

• Sandbox or evaluation outage does not affect production document processing.
• Training outage leaves the last signed model active and produces no partial promotion.
• A budget reservation prevents runaway retries or accelerator/provider spend.
• Duplicate run requests resolve to one run ID and result through a unique idempotency key.
• A crash leaves TTL-bound scratch or an unreferenced immutable object for the sweeper.
• Dataset withdrawal creates a new manifest, revokes future grants and evaluates which models/promotions are affected; history is never silently rewritten.
• Poisoning controls include source diversity caps, tenant/cohort sampling limits, human adjudication, anomaly review and reproducible label provenance.
• Malware, decompression bombs, prompt injection and hostile PDFs stay untrusted even after admission; every execution repeats bounded parsing in a sandbox.

Review gates

• Approve physical bucket names, owning projects, regions, KMS and lifecycle rules.
• Approve every Workload Identity and prove cross-environment and held-out-test denies.
• Approve corpus licences, consent, residency, de-identification and withdrawal policy.
• Approve CNPG schema/roles, budget model, backup/restore and audit retention.
• Establish baseline cohort thresholds for OCR, agent grounding/safety, latency and cost.
• Exercise dataset freeze, test-leakage denial, scratch expiry, credential denial,
  model signing, promotion and one-action rollback before enabling protected data.

[Sam123ben]

ADR-0007: Separate sandbox, training and held-out evaluation boundaries

Status: proposed

Context

The reusable Document Intelligence service needs repeatable OCR and AI-agent
experimentation. Sharing one bucket or identity between sandbox, training and
evaluation would expose sensitive data, allow benchmark leakage and make a
reported improvement impossible to trust. Ordinary CI and contributor pull
requests are an especially weak trust boundary.

Decision

Use separate logical GCS storage classes and Workload Identities for ephemeral
sandbox data, quarantined dataset candidates, frozen train/calibration versions,
development evaluation versions, evaluation outputs and staged model bundles.
Keep the held-out golden test corpus in its existing protected bucket and deny it
to developers, sandbox runners, training jobs and ordinary CI.

Global CNPG owns manifests, lineage, grants, budgets, experiments, promotion and
audit state. GCS owns bytes. DevAI orchestrates opaque evaluation requests and
receives redacted metrics/trace references. Langfuse is a non-authoritative safe
projection with separate non-production/protected credentials; no consuming
product's production credentials are reused.

Promote only a signed compatibility manifest binding OCR code/models,
preprocessing, calibration, extraction schemas, routing, agent, Australis tool
contract, result schema, runtime and hardware requirements. Dataset versions are
immutable. Training, calibration, development evaluation and held-out test have
separate purposes and access paths.

Alternatives

• One non-production bucket: rejected because one IAM/lifecycle boundary permits
  accidental training on evaluation data and broadens sandbox access.
• Copy the production golden corpus into DevAI: rejected because it duplicates
  sensitive data and gives CI a direct corpus path.
• Store datasets in Langfuse: rejected because Langfuse is experiment telemetry,
  not the canonical sensitive-data or governance store.
• Let training jobs evaluate the held-out test set: rejected because repeated
  visibility turns the test set into training feedback.
• Online learning from production corrections: rejected because consent,
  poisoning, label quality and rollback cannot be guaranteed.

Consequences

Experiments are reproducible and test leakage is mechanically constrained, at
the cost of more buckets, identities, manifests and lifecycle policies. Protected
evaluation can be unavailable without affecting production OCR. Physical names,
projects, regions, identities, quotas and retention remain review gates owned by
GitOps rather than defaults embedded in application code.

[Sam123ben]

Storage implementation evidence (2026-09-04): the shared sandbox, dataset, candidate, evaluation-result, model-staging, and protected-golden boundary is now provisioned through the reviewed Terraform state. Read-back confirms seven regional buckets with public-access prevention, uniform bucket-level access, object versioning, shared evaluation CMEK, and the designed lifecycle rules where required. The only CMEK principal added is the Cloud Storage service agent. No production corpus, training artifact, document content, or secret value was introduced. Dedicated least-privilege workload identities remain the next separately gated step.

[Sam123ben]

Identity implementation evidence (2026-09-04): the reviewed, saved Terraform plan completed with seven dedicated shared-platform Google service accounts and their one-to-one Kubernetes Workload Identity bindings, plus only dataset-bucket-scoped object roles. The protected golden corpus remains read-only to its dedicated evaluator; Kora identities receive no shared-corpus access. No project-wide role, secret accessor, runtime deployment, dataset object, document content, or secret value was changed. Live Kubernetes/Argo read-back is healthy; the focused infrastructure policy suite passes.