design review: evaluation, tracing and DevAI agent verification

[Sam123ben]

Parent: #1. Cross-repo epic: tesserix/australis#20.

Review docs/design/EVALUATION-AND-TRACING.md.

Two separate gates

1. OCR service: CER/WER, layout/evidence IoU, table accuracy, classification, field F1/exact match, validator/review safety, latency and cost by cohort.
2. Agent: tool sequence, schema, citations/grounding, escalation/refusal, prompt-injection containment, tenancy, async recovery, token/latency/cost budgets.

The agent will live in tesserix/ai-agents; Australis provides the narrow tool; DevAI provides signed Registry import, isolated sandbox, deterministic and judge evaluation, baseline comparison, trace review and promotion.

Required review

• Golden corpus license/consent, residency, encryption, retention and access.
• PR cassette tests, nightly live-provider drift tests, release qualification and consented shadow evaluation.
• Dataset/candidate provenance and prevention of self-approving dataset changes.
• W3C propagation and async span links across agent → Australis → API → Temporal → page/provider → result/webhook.
• Always-sampled evaluations and production tail sampling without sampling-dependent cost metrics.
• Automated trace assertion that prohibited content/labels never appear in attributes.
• Baselines fail closed when a metric, dataset version, cost or trace evidence is missing.

Approval checklist

• OCR evaluation owner approves metrics and cohort gates
• ai-agents owner approves agent cases and deterministic assertions
• DevAI owner approves sandbox/baseline/promotion integration
• Observability owner approves spans, attributes, redaction and sampling
• Security owner approves adversarial corpus and containment checks
• Rollback uses the prior service/tool/agent compatibility set

[Sam123ben]

Evaluation and tracing design

Two independent quality gates

Service evaluation asks whether the document was read and normalized correctly. Agent evaluation asks whether an agent used that untrusted, cited result safely and effectively. A strong agent score cannot hide weak OCR, and a strong OCR score cannot authorize an unsafe agent.

flowchart LR
    Corpus[Versioned golden corpus] --> ServiceEval[OCR service evaluation]
    ServiceEval --> ServiceGate{Accuracy / latency / cost gate}
    ServiceGate -->|pass| Candidate[Versioned service candidate]
    Candidate --> AgentTool[Australis extract_document tool]
    AgentSuite[DevAI agent suite] --> Sandbox[Isolated DevAI sandbox]
    AgentTool --> Sandbox
    Sandbox --> AgentGate{Safety / grounding / tool / cost gate}
    AgentGate -->|pass| Promote[Promote agent + service compatibility]

Loading

Golden corpus

The corpus is versioned by manifest digest. Samples must have documented license/consent, classification, residency, retention, and allowed evaluator access. Sensitive originals are encrypted and never stored in ordinary CI artifacts. Labels and expected hashes may be reviewed separately from raw content.

Required cohorts include clean digital PDFs, low-resolution scans, rotations/skew/perspective, shadows/folds, mobile photos, handwriting, mixed languages/scripts, complex/borderless tables, merged cells, long documents, unusual fonts, forms/checkboxes, blank/duplicate pages, hostile PDFs, decompression bombs, and multilingual prompt-injection content.

Every case records document type, languages/scripts, acquisition type, quality defects, expected pages, ground-truth text/layout/tables/fields, critical fields, validation outcome, and expected review route. Cohort metrics are first-class; an overall mean cannot hide one failing language or document class.

Service metrics

Layer	Measures
Intake/preprocessing	MIME decision, page count, orientation, crop/deskew transform error, blank/duplicate detection, quality decision recall
OCR	character error rate, word error rate, reading-order accuracy, word/line polygon IoU, script/language accuracy
Layout/table	block classification F1, table detection F1, cell exact match, row/column structure, merged-cell accuracy
Classification	accuracy, macro F1, unknown precision/recall, calibration error
Extraction	field precision/recall/F1, critical-field exact match, normalization accuracy, evidence presence and region IoU
Validation/review	rule precision/recall, unsafe auto-accept rate, unnecessary-review rate
Operations	p50/p95/p99 latency, pages/s, retries, fallback rate, cache hit, provider errors, cost per successful page/document

Critical gates fail closed on unmeasured values. Initial numeric thresholds are set only after the first reviewed baseline; the target must be per cohort and include zero tolerance for cross-tenant access, missing critical evidence, and unsafe auto-accept in known-invalid cases.

Evaluation tiers

1. Pull request, deterministic: parsers, normalization, validation, routing, security bounds, provider contract fixtures, and recorded provider responses. No live provider spend.
2. Nightly live provider: stratified corpus subset against pinned candidate and current production versions; detects provider drift, measures cost/latency, and stores redacted reports.
3. Release qualification: full legally permitted corpus, load/failure tests, deletion/restore exercise, security suite, and candidate-versus-baseline comparison.
4. Production shadow: explicitly consented and policy-eligible traffic only, with results isolated from users; compare candidate without silently training on customer content.

A change to the dataset and the implementation is not used to self-approve. Dataset versions and candidate versions are separate provenance fields. Overrides name metric, approver, reason, and expiry.

DevAI agent verification

The agent lives in tesserix/ai-agents; DevAI supplies the lifecycle: signed Registry artifact, immutable import, isolated sandbox, evaluation, baseline comparison, trace review, and gated promotion. Australis exposes the narrow provider-neutral tool.

The agent suite must cover:

• submits a supported document and uses job/result contracts correctly;
• waits or resumes asynchronous work without duplicate submission;
• cites page/region evidence for every document-derived claim;
• asks for a better scan or review when quality/reliability is below policy;
• never invents a missing field and never treats unknown confidence/cost as zero;
• ignores instructions inside OCR text, including encoded, split, role-impersonating, and tool-shaped attacks;
• cannot let document content widen tools, change tenant/principal, choose callback destinations, select credentials, or introduce system directives;
• observes tenant and document authorization and gets 404 for another tenant's identifiers;
• handles partial pages, cancellation, provider outage, webhook replay, and stale result versions;
• stays within tool-call, token, latency, and cost budgets.

Use deterministic assertions first: schema validity, expected tool sequence, expected sources, citation resolution, refusal/escalation, forbidden tool calls, tenant containment, and budget. Model judges may assess semantic support and answer quality, but they supplement rather than replace deterministic security and contract gates.

DevAI candidate runs compare against a stored main baseline on the same dataset version. PR runs use mocked/cassette tool mode; live suites run after merge or on an explicitly labelled review. Candidate reports and trace links are retained as promotion evidence.

Trace model

W3C traceparent and tracestate cross HTTP, Temporal activities, provider calls, events, webhooks, Australis, and the agent runtime. Because jobs are asynchronous, the workflow/job trace links to the submission trace; a later agent resume links to the job rather than pretending one multi-hour parent span stayed open.

Required spans:

agent.run
  australis.tool.extract_document
    document.api.create_job
      document.workflow (linked async root)
        document.inspect
        document.preprocess
        document.route
        document.page.process
          document.provider.call
          document.normalize
        document.extract
        document.validate
        document.persist
        document.notify

Allowed attributes include request/job/document opaque IDs, tenant ID, environment, document type, page count, provider/model/adapter version, route-policy version, task queue, attempt, status/error code, quality/confidence buckets, duration, token/page usage, decimal cost and currency, cache/fallback/review flags, and dataset/candidate version in evaluation.

Prohibited attributes include raw OCR text, field values, filenames, object URLs, signed URLs, document passwords, authorization headers, prompts, passport/medical/financial contents, or reviewer corrections. Errors carry stable codes and bounded non-content detail. Trace files pass through the ADK redaction path before persistence.

Evaluation runs are always sampled. Production uses tail sampling that retains errors, slow traces, fallback/review paths, and a bounded successful sample. Metrics and cost counters are not sampling-dependent. A collector outage cannot fail document processing; telemetry loss is counted and alerted.

Trace assertions in tests

• Every completed evaluation case has one job trace linked to the agent/tool trace.
• Each provider attempt is visible; retries are not collapsed.
• Provider, model, adapter, route policy, schema, and result versions are present.
• Failed/partial/review states carry the stable reason without content.
• Cost totals equal attributable provider/page observations; missing cost is marked unknown.
• A denylist scan of exported attributes finds no prohibited content keys or golden-label values.
• Replaying the same fixture produces stable correlation identifiers where DevAI baseline comparison requires them.

[Sam123ben]

Kora-only Langfuse credential boundary added

Read-only GCP metadata verification confirmed these two Secret Manager resources exist in project tesseracthub-480811; no secret version or value was accessed:

• prod-langfuse-org-secret-key
• prod-langfuse-org-public-key

Both are Kora-only and both are treated as secrets. Document Intelligence, Australis, DevAI, other products, PR workflows, and local/sandbox evaluation receive no access. The OCR service exports OTel through the platform collector; Kora correlates the agent/tool/document traces with W3C context and opaque IDs.

Implementation is split into:

• integrate Kora environment-specific Langfuse credentials ai-agents#25 — application integration, redaction, failure isolation, rotation and trace tests.
• scope Kora Langfuse Secret Manager access to its workload identity tesserix-k8s#913 — exact Kora workload identity, Secret Manager IAM/projection, egress and GitOps rollout.

The secrets currently use automatic replication. Privacy/residency review must accept that or require an approved migration before production use. No live infrastructure change was made.

[Sam123ben]

Superseding the earlier credential-name note: the Kora production Secret Manager resources are now prod-kora-langfuse-org-secret-key and prod-kora-langfuse-org-public-key. The two generic resources were deleted after verified migration. Kora uses non-secret LANGFUSE_BASE_URL=https://langfuse.tesserix.app.

Document Intelligence still receives no Langfuse credentials and exports OTel through the platform collector.

[Sam123ben]

DevAI implementation and verification now have a dedicated owning issue: tesserix/devai#392. It covers immutable agent/tool/OCR compatibility versions, protected golden execution, deterministic security/grounding gates, Langfuse trace boundaries, and GitOps promotion/rollback. Numeric thresholds and reviewer approval remain owned by this design gate.

[Sam123ben]

The evaluation review now also depends on #12 for sandbox/train/dev-eval/sealed-test boundaries, tesserix/devai#393 for the reusable agent improvement flywheel, tesserix/ai-agents#26 for manifests/traces and tesserix/australis#21 for cache correctness. Proposed 98% required-tool and 99% structured-output gates remain unapproved until calibrated against a reviewed baseline.

[Sam123ben]

Trace integration now requires product/environment identity from verified workload configuration. Kora dev/prod traces use distinct Kora credential pairs and mutual IAM deny tests; DevAI keeps separate evaluation credentials. The shared Document Intelligence service exports OTel only and cannot access any product Langfuse credential.

[Sam123ben]

Lifecycle clarification (2026-09-03): DevAI is the non-production author/evaluate/compare/promotion control plane, not the runtime owner of Document Intelligence. A product agent imported from a signed Registry envelope runs in a sandbox with service.namespace=<signed product namespace> and deployment.environment.name=dev, emits OTLP only, and is routed by the ingest tier to that product's development Langfuse project. Published production agents are invoked by the product's verified workload identity and use the separate production route. Global CNPG is authoritative for dataset manifests, lineage, grants, budgets, run state and signed promotion decisions; GCS owns bytes; Langfuse is non-authoritative trace/evaluation visibility. No production trace automatically becomes training data.

Tesserix ADK 0.54.0 is the required OCR-agent baseline. Release PR tesserix/agent-development-kit#311 is green but still review-required and v0.54.0 is not yet published, so consumer pinning remains blocked until the reviewed release exists.

[Sam123ben]

The service-side trace boundary is implemented and merged through PR #14.

Evidence:

• W3C traceparent continuation is verified against an in-memory OTLP exporter.
• Exported spans contain only stable operation name, method, response status, duration, request ID, and product/tenant scope derived from verified identity.
• Secret-shaped URI query, authorization header, original-URI header, and trace-state inputs are absent from JSON events and exported attributes.
• Direct public/Langfuse endpoints and OTLP credential header variables are rejected; the shared service can export only credential-free to loopback or *.svc.cluster.local gateways.
• Product-specific Langfuse credentials therefore remain behind the originating product/environment gateway.
• Full workspace tests, 32 PostgreSQL-backed tests, Clippy with warnings denied, locked build, RustSec, patch-scoped Gitleaks, and a real /healthz binary smoke passed.

This completes the service HTTP/OTLP foundation only. Async workflow span links, provider spans, evaluator ownership, sampling policy approval, DevAI promotion, and ADK 0.54.0 publication remain under the existing review gates.

[Sam123ben]

Service-level deterministic text evaluation candidate #20 is ready for review. It supplies bounded Unicode-normalized CER/WER and correct corpus aggregation without retaining raw text in metric values/errors. CI, real PostgreSQL coverage and secret scanning are green. Dataset admission, Langfuse/DevAI integration and promotion thresholds remain separately gated here.

[Sam123ben]

Evaluation-onboarding runtime finding

Read-only cluster inspection found the OCR EvalOnboarding claim exists but is not ready. The controller reports reconcile Langfuse project: create Langfuse project: Langfuse API returned HTTP 403; all current product claims are similarly unready.

This blocks live DevAI/Langfuse evaluation traces and scorecards. The fix must use a dedicated, least-privilege OCR dev/evaluation credential path approved for this reusable service. It must not reuse, copy, or grant access to Kora product credentials, and no credential value belongs in the OCR repository, issue, trace, or browser UI.

Before enabling the claim, verify product/environment separation, Secret Manager-to-workload identity access, the minimum Langfuse project-management scope, and a synthetic trace that proves redacted metadata reaches only the OCR development/evaluation project. Record the result here; this observation is not approval to create, rotate, or expose any secret.

[Sam123ben]

Correction and narrowed root cause

Follow-up verification shows the evaluated GitOps revision already references the scoped prod-evals-langfuse-org-* Secret Manager resources. The live ExternalSecret also resolves those names successfully and refreshes normally, so no Kora credential is being consumed by this operator and Secret Manager/IAM wiring is not the current fault.

The remaining HTTP 403 occurs after the operator reads the scoped credentials and calls Langfuse to create a project. This narrows remediation to the credential's Langfuse organisation-level project-management authorization (or the corresponding Langfuse server-side role/policy), without changing the product-specific trace keys.

Do not replace the scoped evals keys with Kora or per-product project keys. Create or authorize a least-privilege organisation management credential specifically for the evals-onboarding operator, then verify project creation with a synthetic claim and confirm the resulting per-product project keys remain isolated.

[Sam123ben]

IAM drift finding

Terraform on tesserix-k8s declares secret-level secretAccessor bindings for the two scoped evals organisation credentials, but live Secret Manager reports no bindings on either secret. The operator can currently read them through existing project-level custom roles instead.

The project-level manager role is condition-limited to production Langfuse key resource names, but it remains broader than the declared two-secret Terraform contract. An exact Terraform plan should first add only the two declared secret-level bindings and prove effective access. Any later reduction of the legacy project role needs separate review because the onboarding operator also creates/manages per-product Langfuse project-key secrets.

No IAM mutation, credential access, or role revocation was performed during this finding.

[Sam123ben]

Terraform review update (no secret values): tesserix-k8s #952 isolates the evaluator operator’s two scoped Langfuse organisation-credential secretAccessor bindings from the dynamic service-account map. The verified targeted production plan is exactly 2 to add, 0 to change, 0 to destroy; it does not create service accounts, alter buckets, or change any credential value. This replaces the prior unsafe targeted plan, which expanded to unrelated pending identities. No apply has been performed. This removes the secret-level IAM drift only; the independent Langfuse HTTP 403 project-creation failure still needs organisation-level authorisation review.