design review: Kora runtime document storage integration

[Sam123ben]

Parent: #1. Shared evaluation/training design: #12. Infrastructure: tesserix/tesserix-k8s#914 and #916. Kora agent integration: tesserix/ai-agents#25.

Scope correction

Kora is one consumer of the reusable Document Intelligence service. This issue owns only Kora production document storage/integration policy. Shared OCR models, sandbox, training, evaluation datasets, held-out golden corpus, candidate models and DevAI release gates are product-neutral and owned by #12 and tesserix/tesserix-k8s#916.

Historical comments on this issue that describe a Kora-owned golden/evaluation platform are superseded by this body and #12.

Kora runtime decision

Use Kora-specific GCS buckets separated by runtime data class: quarantine, accepted source, temporary derived pages and normalized results. Do not create one bucket per document; isolate documents through opaque tenant/document/version paths, immutable generations and object authorization.

Kora products and agents submit/read through authenticated v1 service endpoints or the provider-neutral Australis extract_document tool. They receive no bucket credentials. Kora-specific Langfuse credentials remain accessible only to the Kora AI workload and are not used by the OCR service, DevAI, training or shared evaluation.

Review checklist

• Approve Kora physical bucket names, owning project, regional placement, encryption and lifecycle per runtime data class.
• Approve Workload Identities for upload signing, scanning, OCR processing, result access and lifecycle deletion.
• Prove Kora workloads cannot access another product documents or shared training/golden buckets.
• Prove shared training/evaluation workloads cannot browse Kora production documents.
• Verify signed URLs are exact-object/method/size/type bound, short-lived and issued after authorization.
• Verify object names contain no filename or PII.
• Verify deletion across source, derived, results, CNPG metadata, Qdrant and Valkey.
• Verify safe trace correlation to Kora Langfuse without raw OCR content or production credentials crossing boundaries.
• Run reusable API/tool contract tests with Kora and at least one second product fixture.

No GCS, CNPG, Langfuse, IAM, Kubernetes or production mutation is authorized by this design issue. Infrastructure must be delivered through reviewed GitOps changes.

[Sam123ben]

ADR-0006: Kora-isolated document storage and governed evaluation loop

Status: proposed

Context

Kora documents can contain sensitive personal, health or financial data. Production processing and model evaluation require different identities, retention and access. A bucket per document is operationally expensive and does not replace application authorization. Storing raw golden data in global Postgres or Langfuse would widen the sensitive-data boundary.

Decision

Use separate Kora buckets for quarantine, accepted source, temporary derived pages, normalized results and the governed golden corpus. Isolate documents by opaque tenant/document/version object paths and object-level authorization, not a bucket per file.

Store golden dataset registry, labels, consent, splits, metrics and promotion history in a dedicated database/schema on global CNPG. Store raw samples and ground truth in the restricted Kora golden bucket. Langfuse stores safe experiment/tracing metadata and metrics; sensitive cases remain opaque references resolved by a dedicated evaluation runner.

Production corrections enter only a candidate queue. Consent, de-identification, human labelling, duplicate/leakage checks and dataset-version review precede admission. Training, calibration and held-out test splits have separate purposes. CI, nightly, release, shadow and canary gates promote signed model/profile versions through GitOps.

Alternatives

• Bucket per document: rejected because bucket quotas, IAM, lifecycle and operational overhead do not provide better application-level object authorization.
• One Kora bucket for every stage: rejected because quarantine, production results and golden data require different identities and retention.
• Raw golden corpus in CNPG: rejected because large binaries inflate WAL/backups and widen database access.
• Full golden cases in Langfuse: rejected for sensitive documents; Langfuse is experiment/trace evidence, not the canonical corpus.
• Automatic online learning from corrections: rejected because consent, poisoning, label quality and benchmark leakage cannot be guaranteed.

Consequences

Kora gains strong product/data-class isolation and a measurable improvement loop. It requires multiple lifecycle policies, identities and deletion checks. Evaluation cannot run when the global registry or golden bucket is unavailable, but production OCR remains independent.

[Sam123ben]

Kora storage, evaluation and improvement loop

Storage boundary

Kora receives product-specific buckets separated by data class, not one bucket per document. Bucket-level separation gives meaningful IAM, lifecycle, retention and blast-radius boundaries; document isolation is enforced through generated object identities, immutable generations and application authorization.

Candidate logical buckets are:

Bucket class	Purpose	Typical retention
prod-kora-ocr-quarantine	Untrusted direct uploads before validation	Hours to a few days
prod-kora-ocr-source	Accepted immutable originals	Tenant policy, initially 30 days
prod-kora-ocr-derived	Rendered and preprocessed pages	Short, normally 24–72 hours
prod-kora-ocr-results	Full normalized result artefacts and evidence maps	Tenant policy, initially 30 days
prod-kora-ocr-golden	Approved evaluation samples and ground truth	Dataset-governance policy

Actual globally unique names will include the project/environment convention selected in tesserix-k8s. All buckets use uniform bucket-level access, public-access prevention, regional placement matching policy, Workload Identity, encryption, access logging and lifecycle rules. A retention lock is enabled only where legal policy requires it because it prevents early erasure.

Object names contain no filename, email, patient/customer name or other business data:

tenants/{tenant_id}/documents/{document_id}/versions/{source_digest}/original
tenants/{tenant_id}/documents/{document_id}/versions/{source_digest}/pages/{page_id}/{profile_digest}
tenants/{tenant_id}/documents/{document_id}/results/{result_version}/document.json.zst
datasets/{dataset_id}/versions/{manifest_digest}/cases/{case_id}/source

The tenant ID in a path is routing metadata, not authorization. Every signed URL is created only after object-level authorization and is bound to one method, object, content length/type and short expiry. Agents never receive direct bucket credentials.

Workload access matrix

Workload	Quarantine	Source	Derived	Results	Golden
Upload signer	create-only signed intent	none	none	none	none
Scanner/importer	read/delete	create	none	none	none
OCR engine worker	none	read approved object	read/write	write	none
API/result service	metadata/sign bounded read	bounded sign	bounded sign	bounded sign/read	none
Review service	none	bounded page tile via API	bounded page tile via API	authorized result read	none
Lifecycle deleter	delete	delete	delete	delete	policy-controlled delete
Evaluation runner	none	none	temporary scratch only	benchmark output only	read approved dataset version
Dataset curator	none	none	none	candidate reference through approval	versioned write/read
Kora agent	none	none	none	through document API/tool only	none

Separate Kubernetes and GCP service identities implement these rows. No identity receives storage.admin. The scanner cannot read results; production OCR cannot read the golden corpus; the evaluation runner cannot browse production source buckets.

Connected processing flow

flowchart TD
    Client[Kora application] -->|upload intent| API[Document API]
    API -->|short-lived create URL| Quarantine[(Kora quarantine bucket)]
    Client -->|document bytes| Quarantine
    Quarantine --> Scan[Sandboxed MIME, malware and limits]
    Scan -->|accepted immutable generation| Source[(Kora source bucket)]
    Scan -->|typed rejection| API
    Source --> Workflow[Durable document workflow]
    Workflow --> Engine[Rust OCR engine]
    Engine --> Derived[(Short-lived derived pages)]
    Engine --> Results[(Kora result bucket)]
    Workflow --> PG[(CNPG authoritative metadata)]
    PG --> Outbox[Transactional outbox]
    Outbox --> Qdrant[(Qdrant semantic index)]
    Outbox --> Valkey[(Valkey cache invalidation)]
    Results --> Tool[Australis extract_document tool]
    Tool --> Agent[Kora AI agent]
    Agent --> Langfuse[Langfuse traces and experiment scores]
    Workflow --> Review[Human review]
    Review --> PG
    Review --> Candidate[Dataset candidate gate]

Loading

Golden dataset ownership

The global CNPG platform stores a dedicated ocr_evaluation database or schema containing dataset registry metadata, case IDs, product/tenant scope, consent manifests, labels, consent/legal basis, residency, retention, splits, metric definitions, experiment summaries, reviewer decisions and promotion history. It does not store raw image/PDF bytes.

The Kora golden bucket stores encrypted source samples, page renders where needed and machine-readable ground truth. Every dataset version has an immutable manifest digest linking exact object generations, label schema, split membership and provenance. Qdrant and Valkey are never the golden source of truth.

Production Kora data reaches a dataset-candidate table only after tenant consent and policy eligibility. It then passes malware/data classification, de-identification where required, human review and licence/consent verification. Admission creates a new dataset version; it never silently modifies the current baseline.

Train, calibration and test splits are grouped by source document family, template and near-duplicate cluster to prevent the same template or scan appearing across splits. The held-out golden test split is not used to tune preprocessing parameters, train weights, select thresholds or write prompts.

Langfuse boundary

Langfuse at https://langfuse.tesserix.app records Kora agent traces, experiment IDs, candidate/baseline versions, safe case IDs, metric values, latency, cost and promotion evidence. It is not the canonical dataset store and receives no permanent signed GCS URL, Secret Manager value or unrestricted raw sensitive document.

The production Kora runtime uses the existing Kora-only Langfuse secrets. CI and DevAI must use a separate Langfuse evaluation project and dedicated non-production credentials delivered through Workload Identity/Secret Manager. Pull requests never receive production Langfuse keys.

For legally approved non-sensitive fixtures, Langfuse may hold the case content. Sensitive cases remain opaque references resolved only inside the authorized evaluation runner. Traces retain identifiers and metrics, not raw OCR text or ground-truth field values.

CI/CD evaluation ladder

Stage	Data and execution	Gate
Pull request	Synthetic/public tiny corpus, unit/property/fuzz tests, recorded tensors/provider responses; no production credentials	Contract, safety, deterministic metrics and performance smoke
Merge/nightly	Dedicated evaluation Workload Identity reads an approved stratified golden subset	Drift, CER/WER, evidence IoU, latency, memory and cost comparison
Release candidate	Protected environment runs the full permitted corpus on pinned hardware	Cohort quality, calibration, security, load, failure and cost gates
Shadow	Consented production-like inputs; candidate results cannot reach users	Candidate versus current result and resource profile
Canary	Small tenant/page percentage with one-action rollback	SLO burn, unsafe accept, error, review, latency and cost
Promotion	Signed model/profile plus compatibility manifest	Immutable production version and recorded approver

CI emits redacted JSON/summary artefacts and a Langfuse experiment reference. It never uploads the golden corpus as a workflow artefact. Dataset and implementation changes are evaluated independently; a pull request cannot change its test data and use that same run to self-approve.

Engineering improvement loop

1. Observe quality, latency, memory, cost, review and drift signals by document/language/quality cohort.
2. Admit only consented corrections and failures into the candidate queue.
3. Curate, de-identify, label and near-duplicate-check candidates.
4. Freeze a new dataset manifest and preserve the previous version.
5. Optimize preprocessing/model code against training data.
6. Set confidence calibration and review thresholds on the calibration split.
7. Evaluate once against the held-out golden split and compare with the production baseline.
8. Reject regression or sign the model/profile and attach dataset, code, runtime and hardware provenance.
9. Run shadow and canary deployment through GitOps.
10. Promote or roll back as one versioned compatibility set.
11. Monitor post-promotion drift and start the next candidate cycle.

Image cleanup parameters are versioned processing profiles. Each experiment reports whether a transform improved recognition without damaging geometry, tables, handwriting or faint text. The original image always remains the evidence source, and aggressive cleanup never hides an unusable-input decision.

Required metrics

• CER/WER and exact match by cohort;
• detector/word/evidence polygon IoU;
• reading-order, table and field accuracy;
• confidence calibration error and unsafe auto-accept rate;
• review/correction rate and reviewer agreement;
• preprocessing benefit/regression by defect type;
• p50/p95/p99 latency, pages/s, peak memory and GPU utilization;
• cost per successful page/document and cache/index lag;
• dataset age, cohort coverage, leakage/duplicate checks and drift;
• canary SLO burn and rollback rate.

Failure and recovery

• Quarantine outage prevents new uploads but does not affect existing result reads.
• Source/results outage pauses affected work; CNPG cannot mark a missing object complete.
• Global evaluation CNPG or golden bucket outage pauses evaluation only; production OCR continues.
• Langfuse outage loses no canonical dataset/result; telemetry buffers boundedly or records loss.
• Qdrant outage queues indexing and degrades semantic retrieval; OCR and golden evaluation truth remain intact.
• Valkey outage falls through to canonical stores and uses conservative local admission limits.

[Sam123ben]

Kora storage, evaluation and improvement loop

Design envelope and objectives

This design uses the launch assumptions in REQUIREMENTS.md: 100 active
tenants, 20 jobs/s and 100 pages/s peak, 100,000 documents per month at 12
months and 500,000 per month at 36 months. At the typical 2 MiB/document and
30-day source retention, Kora holds about 200 GiB of accepted source at the
12-month point and 1 TiB at the 36-month point, before versioning and safety
margin. At 250 KiB/page, uncompressed normalized results add about 100 GiB and
500 GiB respectively. Provisioning and cost review must use at least 2x those
working-set estimates until measured compression, rejection and version rates
replace the assumptions.

The storage control plane follows the job API's 99.9% monthly availability
target. An acknowledged job is not lost. The metadata target is RPO at most
five minutes and RTO at most 60 minutes. Quarantine and derived data are
re-creatable; accepted source, normalized results, audit records, consent and
dataset manifests are not. Cross-tenant disclosure and promotion of a critical
field without evidence have zero tolerance.

Storage boundary

Kora receives product-specific buckets separated by data class, not one bucket per document. Bucket-level separation gives meaningful IAM, lifecycle, retention and blast-radius boundaries; document isolation is enforced through generated object identities, immutable generations and application authorization.

Candidate logical buckets are:

Bucket class	Purpose	Initial lifecycle	Recovery class
prod-kora-ocr-quarantine	Untrusted direct uploads before validation	Delete rejected/incomplete objects after 24 hours; accepted objects are deleted after verified promotion	Re-upload required
prod-kora-ocr-source	Accepted immutable originals	Tenant policy, initially 30 days; generation-specific deletion	Authoritative until retention/deletion completes
prod-kora-ocr-derived	Rendered and preprocessed pages	Delete after 48 hours unless an active job/review lease exists	Rebuild from accepted source and processing profile
prod-kora-ocr-results	Full normalized result artefacts and evidence maps	Tenant policy, initially 30 days	Authoritative result artefact while retained
prod-kora-ocr-golden	Approved evaluation samples and ground truth	Versioned governance policy; no general lifecycle deletion	Authoritative governed corpus

Actual globally unique names will include the project/environment convention selected in tesserix-k8s. All buckets use uniform bucket-level access, public-access prevention, regional placement matching policy, Workload Identity, encryption, access logging and lifecycle rules. A retention lock is enabled only where legal policy requires it because it prevents early erasure.

The initial placement candidate is asia-south1, matching the current shared
GCP/GKE platform, but it is not approved until the Kora privacy owner accepts
the residency and backup path. Each bucket is separately declared in GitOps,
labelled with environment=prod, product=kora, component=document-intelligence,
and its data class. Default Google-managed encryption may be used only after the
data-class review; a CMEK decision records key ring, rotation, decrypting
identities and behaviour when KMS is unavailable. Object versioning is not a
substitute for backup and is enabled only with a lifecycle rule that bounds old
generations and honours deletion policy.

Object names contain no filename, email, patient/customer name or other business data:

tenants/{tenant_id}/documents/{document_id}/versions/{source_digest}/original
tenants/{tenant_id}/documents/{document_id}/versions/{source_digest}/pages/{page_id}/{profile_digest}
tenants/{tenant_id}/documents/{document_id}/results/{result_version}/document.json.zst
datasets/{dataset_id}/versions/{manifest_digest}/cases/{case_id}/source

The tenant ID in a path is routing metadata, not authorization. Every signed URL is created only after object-level authorization and is bound to one method, object, content length/type and short expiry. Agents never receive direct bucket credentials.

Workload access matrix

Workload	Quarantine	Source	Derived	Results	Golden
Upload signer	create-only signed intent	none	none	none	none
Scanner/importer	read/delete	create	none	none	none
OCR engine worker	none	read approved object	read/write	write	none
API/result service	metadata/sign bounded read	bounded sign	bounded sign	bounded sign/read	none
Review service	none	bounded page tile via API	bounded page tile via API	authorized result read	none
Lifecycle deleter	delete	delete	delete	delete	policy-controlled delete
Evaluation runner	none	none	temporary scratch only	benchmark output only	read approved dataset version
Dataset curator	none	none	none	candidate reference through approval	versioned write/read
Kora agent	none	none	none	through document API/tool only	none

Separate Kubernetes and GCP service identities implement these rows. No identity receives storage.admin. The scanner cannot read results; production OCR cannot read the golden corpus; the evaluation runner cannot browse production source buckets.

Bucket IAM is granted to GCP service accounts bound to dedicated Kubernetes
ServiceAccounts through Workload Identity. No service-account key files exist.
Signed upload URLs expire within 15 minutes; signed result downloads expire
within five minutes by default. Exact expiries are policy inputs, not caller
controlled. A signer may authorize only a Postgres-recorded object generation
belonging to the authenticated tenant. All cross-tenant object lookups return
the same not-found response as a missing object.

Connected processing flow

flowchart TD
    Client[Kora application] -->|upload intent| API[Document API]
    API -->|short-lived create URL| Quarantine[(Kora quarantine bucket)]
    Client -->|document bytes| Quarantine
    Quarantine --> Scan[Sandboxed MIME, malware and limits]
    Scan -->|accepted immutable generation| Source[(Kora source bucket)]
    Scan -->|typed rejection| API
    Source --> Workflow[Durable document workflow]
    Workflow --> Engine[Rust OCR engine]
    Engine --> Derived[(Short-lived derived pages)]
    Engine --> Results[(Kora result bucket)]
    Workflow --> PG[(CNPG authoritative metadata)]
    PG --> Outbox[Transactional outbox]
    Outbox --> Qdrant[(Qdrant semantic index)]
    Outbox --> Valkey[(Valkey cache invalidation)]
    Results --> Tool[Australis extract_document tool]
    Tool --> Agent[Kora AI agent]
    Agent --> Langfuse[Langfuse traces and experiment scores]
    Workflow --> Review[Human review]
    Review --> PG
    Review --> Candidate[Dataset candidate gate]

Loading

Upload-to-result workflow

1. The Kora client requests an upload intent. The API derives the tenant from
   the verified principal, reserves an opaque document/version ID in CNPG and
   returns a create-only, content-length/content-type-bound quarantine URL.
2. GCS receives bytes directly. Finalization delivers only an opaque bucket,
   object and generation reference; the API also supports a reconciler so a
   missed or duplicate notification cannot lose or duplicate an upload.
3. The scanner claims that exact generation, sniffs the real MIME type, scans
   for malware and enforces byte, pixel, page, decompression and parser-time
   limits in a sandbox. It never follows arbitrary embedded URLs.
4. Acceptance copies the object to the source bucket with generation-match and
   destination-does-not-exist preconditions. CNPG commits the digest, source
   generation, inspection outcome, job record and outbox event in one short
   transaction. Only then may quarantine cleanup run.
5. The durable workflow renders and preprocesses bounded page groups, writes
   content-addressed derived objects and records the transform/profile digest.
   Page retries address only failed pages and cannot overwrite another result.
6. OCR, extraction and validation produce an immutable result object first.
   CNPG then atomically records its generation/digest, summaries, terminal job
   state, audit event and downstream outbox intents. Metadata never points to a
   result object that was not durably written.
7. Outbox consumers invalidate Valkey, index tenant-scoped canonical chunks in
   Qdrant and deliver signed webhooks. All are at-least-once and idempotent;
   their failure cannot roll back a completed OCR result.
8. Australis resolves the job/result through the API, not through GCS. The Kora
   agent receives untrusted text plus evidence IDs and bounded result locators,
   never bucket credentials or unrestricted signed URLs.

An object write followed by a failed database commit leaves an unreferenced
object, not a false completed job. A bounded sweeper deletes only generations
that have no live CNPG reference and are older than its safety window.

Golden dataset ownership

The global CNPG platform stores a dedicated ocr_evaluation database or schema containing dataset registry metadata, case IDs, product/tenant scope, consent manifests, labels, consent/legal basis, residency, retention, splits, metric definitions, experiment summaries, reviewer decisions and promotion history. It does not store raw image/PDF bytes.

The Kora golden bucket stores encrypted source samples, page renders where needed and machine-readable ground truth. Every dataset version has an immutable manifest digest linking exact object generations, label schema, split membership and provenance. Qdrant and Valkey are never the golden source of truth.

Global CNPG evaluation registry

Use a dedicated ocr_evaluation database when CNPG ownership and backup policy
permit it; otherwise use an ocr_evaluation schema with the same isolated
roles. The final database-versus-schema choice belongs to the CNPG review. The
minimum relational model is:

Relation	Authoritative contents
datasets	owner, purpose, product scope, classification and lifecycle state
dataset_versions	immutable manifest digest, label-schema version, curator, approval and freeze time
cases	opaque case ID, provenance class, consent/licence record, residency, retention and near-duplicate cluster
case_objects	exact golden-bucket object generation, digest, media role and encryption key reference—not signed URLs
case_labels	versioned label object digest, reviewer provenance and adjudication state
split_memberships	train/calibration/test split plus grouping key and leakage-check version
metric_definitions	versioned metric name, implementation digest, direction, cohort and threshold policy
experiments	code/model/profile/runtime/hardware/dataset versions and terminal state
experiment_metrics	aggregate and per-cohort results, sample count and confidence interval
promotion_decisions	candidate/baseline, gate results, approver, reason, expiry and rollback target
candidate_admissions	consented production correction reference and every governance decision
audit_events / outbox_events	append-only governance history and reliable publication intent

Dataset-version freezing is one CNPG transaction: validate every case and split,
insert the immutable manifest digest, mark the version frozen and append its
audit/outbox records. No HTTP, GCS or Langfuse call runs inside the transaction.
The manifest object is written and digest-verified first. A failed transaction
leaves a sweepable unreferenced manifest rather than a registry row pointing at
missing bytes.

Roles are default-deny: ocr_eval_registry_reader, ocr_eval_runner,
ocr_eval_curator, ocr_eval_promoter and ocr_eval_migrator. The runner can
read only frozen versions and write experiment results; it cannot mutate cases,
splits or promotion decisions. The promoter cannot read raw object content.
RLS scopes product/tenant-sensitive metadata, while golden object access remains
separately enforced by GCS IAM.

Production Kora data reaches a dataset-candidate table only after tenant consent and policy eligibility. It then passes malware/data classification, de-identification where required, human review and licence/consent verification. Admission creates a new dataset version; it never silently modifies the current baseline.

Train, calibration and test splits are grouped by source document family, template and near-duplicate cluster to prevent the same template or scan appearing across splits. The held-out golden test split is not used to tune preprocessing parameters, train weights, select thresholds or write prompts.

Langfuse boundary

Langfuse at https://langfuse.tesserix.app records Kora agent traces, experiment IDs, candidate/baseline versions, safe case IDs, metric values, latency, cost and promotion evidence. It is not the canonical dataset store and receives no permanent signed GCS URL, Secret Manager value or unrestricted raw sensitive document.

The production Kora runtime uses the existing Kora-only Langfuse secrets. CI and DevAI must use a separate Langfuse evaluation project and dedicated non-production credentials delivered through Workload Identity/Secret Manager. Pull requests never receive production Langfuse keys.

For legally approved non-sensitive fixtures, Langfuse may hold the case content. Sensitive cases remain opaque references resolved only inside the authorized evaluation runner. Traces retain identifiers and metrics, not raw OCR text or ground-truth field values.

Each evaluation creates one CNPG experiment_id and one Langfuse experiment
carrying the same opaque ID. Allowed Langfuse fields are dataset-version digest,
case ID, cohort tags, candidate/baseline version digests, safe metric values,
latency, resource use, decimal cost, route decision and trace links. The CNPG
record remains authoritative if Langfuse is delayed or unavailable. Export is
outbox-driven, bounded and idempotent; after retry exhaustion it is dead-lettered
and alerted without failing OCR or losing the evaluation result.

CI/CD evaluation ladder

Stage	Data and execution	Gate
Pull request	Synthetic/public tiny corpus, unit/property/fuzz tests, recorded tensors/provider responses; no production credentials	Contract, safety, deterministic metrics and performance smoke
Merge/nightly	Dedicated evaluation Workload Identity reads an approved stratified golden subset	Drift, CER/WER, evidence IoU, latency, memory and cost comparison
Release candidate	Protected environment runs the full permitted corpus on pinned hardware	Cohort quality, calibration, security, load, failure and cost gates
Shadow	Consented production-like inputs; candidate results cannot reach users	Candidate versus current result and resource profile
Canary	Small tenant/page percentage with one-action rollback	SLO burn, unsafe accept, error, review, latency and cost
Promotion	Signed model/profile plus compatibility manifest	Immutable production version and recorded approver

CI emits redacted JSON/summary artefacts and a Langfuse experiment reference. It never uploads the golden corpus as a workflow artefact. Dataset and implementation changes are evaluated independently; a pull request cannot change its test data and use that same run to self-approve.

The protected evaluation runner executes inside GKE with no public ingress,
read-only access to one approved golden dataset version, write access only to an
evaluation scratch prefix, and narrowly allowlisted egress. CI sends an opaque
run request and receives scores; it never receives corpus bytes or signed URLs.
The runner wipes scratch objects on success and through a TTL lifecycle on crash.
PRs from forks and untrusted contributors cannot invoke the protected runner or
receive Secret Manager credentials.

Promotion compares candidate and baseline on the identical frozen dataset,
metric implementation, hardware class and runtime profile. A candidate fails
when a zero-tolerance security/evidence assertion fails, a required cohort is
missing, an agreed quality floor regresses, latency/SLO budget regresses, or
cost exceeds policy. Thresholds start as unapproved values in issue #7 and become
release gates only after a baseline and owner approval; missing measurements are
failures, never zeros.

Engineering improvement loop

1. Observe quality, latency, memory, cost, review and drift signals by document/language/quality cohort.
2. Admit only consented corrections and failures into the candidate queue.
3. Curate, de-identify, label and near-duplicate-check candidates.
4. Freeze a new dataset manifest and preserve the previous version.
5. Optimize preprocessing/model code against training data.
6. Set confidence calibration and review thresholds on the calibration split.
7. Evaluate once against the held-out golden split and compare with the production baseline.
8. Reject regression or sign the model/profile and attach dataset, code, runtime and hardware provenance.
9. Run shadow and canary deployment through GitOps.
10. Promote or roll back as one versioned compatibility set.
11. Monitor post-promotion drift and start the next candidate cycle.

Image cleanup parameters are versioned processing profiles. Each experiment reports whether a transform improved recognition without damaging geometry, tables, handwriting or faint text. The original image always remains the evidence source, and aggressive cleanup never hides an unusable-input decision.

The deployable unit is a signed compatibility manifest containing the Rust
engine image digest, detector/recognizer/layout model digests, preprocessing
profile, confidence calibration, extraction/validation schemas, route policy,
canonical result schema and minimum hardware/runtime versions. Shadow, canary,
promotion and rollback move this unit as one version so an improved image filter
cannot be paired accidentally with stale calibration or evidence transforms.

Reviewer corrections improve the system only after governance admission. They
are never directly consumed by an online learner. Candidate selection is sampled
and rate-limited by cohort to resist poisoning and prevent one large tenant from
dominating. Human adjudication records disagreement; near-duplicate detection
runs before splitting; the held-out test remains sealed from engineers tuning
models, transforms, prompts or thresholds.

Deletion and legal holds

Deletion is an orchestrated, idempotent workflow tracked in CNPG. It marks the
document deletion_pending, revokes new URL issuance, cancels active work, deletes
exact GCS generations, removes Qdrant points by deterministic tenant/version IDs,
invalidates Valkey namespaces and records acknowledgements. It becomes
deleted only after every required store confirms removal. Retried requests
return the same operation; failures remain visible and alertable.

Backups and locked retention can delay physical erasure. The API and audit record
must report the applicable expiry window rather than claiming immediate deletion.
A legal hold blocks lifecycle deletion with actor, authority, scope and expiry in
the audit trail. A production document admitted into a golden version is governed
by a separate explicit consent/legal basis; deleting the product copy does not
silently decide golden-corpus retention, and any withdrawal path is recorded as
a new dataset version and promotion-impact review.

Required metrics

• CER/WER and exact match by cohort;
• detector/word/evidence polygon IoU;
• reading-order, table and field accuracy;
• confidence calibration error and unsafe auto-accept rate;
• review/correction rate and reviewer agreement;
• preprocessing benefit/regression by defect type;
• p50/p95/p99 latency, pages/s, peak memory and GPU utilization;
• cost per successful page/document and cache/index lag;
• dataset age, cohort coverage, leakage/duplicate checks and drift;
• canary SLO burn and rollback rate.

Failure and recovery

• Quarantine outage prevents new uploads but does not affect existing result reads.
• Source/results outage pauses affected work; CNPG cannot mark a missing object complete.
• Global evaluation CNPG or golden bucket outage pauses evaluation only; production OCR continues.
• Langfuse outage loses no canonical dataset/result; telemetry buffers boundedly or records loss.
• Qdrant outage queues indexing and degrades semantic retrieval; OCR and golden evaluation truth remain intact.
• Valkey outage falls through to canonical stores and uses conservative local admission limits.

Open production decisions

• Confirm asia-south1 or select another launch/residency region, including the
  golden corpus and CNPG backup locations.
• Approve globally unique physical bucket names and the project that owns them.
• Approve lifecycle durations, versioning and whether each data class requires
  CMEK; identify key administrators separately from data readers.
• Name the Kubernetes/GCP service accounts and approve the least-privilege IAM
  matrix through tesserix-k8s#913.
• Choose dedicated database versus schema for ocr_evaluation, then approve
  roles, backup/restore and connection budgets.
• Create a non-production Langfuse evaluation project and its own Secret Manager
  credentials; production Kora credentials are not reused.
• Establish initial per-cohort quality, unsafe-accept, latency and cost gates from
  a reviewed baseline, and name the promotion/rollback approvers.

[Sam123ben]

Implementation ownership is now split without broadening this design approval:

• design Kora runtime document storage infrastructure tesserix-k8s#914 owns GitOps design for the five Kora data-class buckets, Workload Identity/IAM, global-CNPG evaluation registry, protected evaluation runner, lifecycle, recovery and cost evidence.
• evaluate reusable document agents and OCR compatibility set devai#392 owns the agent/tool/OCR compatibility evaluation, protected golden runs, trace review and promotion gates.
• scope Kora Langfuse Secret Manager access to its workload identity tesserix-k8s#913 remains limited to Kora Langfuse Secret Manager access/projection.

All are assigned to Sam123ben. No live production mutation is authorized; the unchecked decisions in this issue remain review gates.

[Sam123ben]

Follow-up design #12 now owns the non-production sandbox, candidate, train/calibration, development-evaluation, evaluation-results and candidate-model boundaries. The sealed held-out golden corpus remains under this issue and cannot be read by sandbox, training, developer or ordinary CI identities.

[Sam123ben]

Kora identity/routing clarification: the gateway derives product_id=kora, environment and tenant scope from verified workload/OIDC identity. Kora dev calls non-production service policy and uses only dev-kora Langfuse resources; Kora production calls production policy and uses only prod-kora resources. Request fields, prompts, OCR text and tool output cannot select product, environment, endpoint, project or secrets. The shared OCR service receives neither credential pair.

[Sam123ben]

Storage implementation evidence (2026-09-04): the Kora document boundary is now provisioned through the reviewed Terraform state. Read-back confirms four regional buckets with public-access prevention, uniform bucket-level access, object versioning, product-scoped CMEK, and the designed lifecycle rules. The only CMEK principal added is the Cloud Storage service agent. No runtime workload identity, service deployment, raw document content, or secret value was changed. The next gated step is the separately planned dedicated workload identities and bucket-only grants.