phase 1: bridge Temporal orchestration to durable CNPG page execution

[Sam123ben]

Outcome

Connect the qualified Temporal Rust workflow to the existing tenant/product-scoped CheckpointedPageRunner and DocumentFinalizer, preserving durable page checkpoints and publishing a provider-neutral complete or partial result without placing document content in Temporal history.

Capacity and SLO assumptions

• 20 job starts/s peak per region, burstable to 50/s.
• 1–300 pages/job; 10 MiB/page decoded ceiling; page artifacts remain in object storage, never workflow history.
• At 20 starts/s and an assumed 10 pages/job, the worker tier must sustain 200 page attempts/s; page concurrency remains a bounded deployment setting.
• Job-control API availability target: 99.9% monthly.
• Accepted-job durability target: no acknowledged job loss; RPO 0 for CNPG workflow metadata, provider artifacts governed by object-storage durability.
• Control-plane p99 target: 500 ms excluding OCR execution; page and document completion latency are measured separately by document class.

These assumptions must be validated by the qualification soak before production sizing.

Architectural decision to review

CNPG PageWorkflow is the sole authority for page-attempt count and terminal page state. Temporal owns durable orchestration, cancellation, timers and worker recovery, but does not count a provider/page failure as a Temporal retry.

• One identifiers-only Temporal activity calls CheckpointedPageRunner::run_once.
• A recorded retryable page failure returns activity success with metadata-only running state; the workflow schedules the next bounded runner iteration and CNPG advances the attempt.
• Temporal activity retries are reserved for failures where the durable CNPG transition is unknown or unavailable (connection loss, timeout, optimistic revision conflict). Duplicate delivery is safe through CNPG revision checks and deterministic page activity keys.
• Validation, tenant/product scope, missing workflow and permanent-input failures are non-retryable activity failures.
• Once CNPG reports completed or partial, a metadata-only finalizer activity invokes DocumentFinalizer; result bytes move only between object storage and the service process.
• Cancellation records durable cancellation before workflow closure; no new page work is claimed afterward.

This separates transport recovery from business retry policy and avoids two competing page retry counters.

Security and trust boundaries

Assets: tenant documents, extracted PII, object locators and product identity. Threats: cross-tenant callers, malicious uploads/OCR prompt injection, compromised worker/provider, and accidental telemetry leakage.

Every activity input is bounded and contains only validated tenant_id, product_id, job_id, document identity/version and correlation metadata. Every CNPG lookup is scoped by tenant and product under RLS. Activity outputs, errors, logs, traces and Temporal search/history attributes contain no OCR text, page bytes, signed URLs, secret values or extracted fields.

Acceptance criteria

• ADR records the retry-authority decision, alternatives, dependency failure behaviour, rollback and cost implications.
• Red tests prove identifiers-only activity serialization and metadata-only results/errors.
• Real Temporal + real PostgreSQL test proves page attempt exhaustion through the SDK produces the existing durable partial result and preserves successful page artifacts.
• Duplicate activity delivery does not reprocess a successfully checkpointed page or double-publish a result.
• Cross-tenant and cross-product activity identifiers cannot load or mutate a job.
• Cancellation before start, during page work, between runner iterations and during finalization is deterministic and leaves a recoverable durable state.
• Dependency timeout/revision conflict is retryable; scope/validation/not-found is non-retryable; page-provider failure is recorded by CNPG and returned as activity success.
• Finalizer delivery is idempotent and returns only result status/locator metadata allowed by the public contract.
• 1/50/51/300-page live cases preserve continue-as-new and bounded history.
• Existing replay, worker-loss, database/RLS, formatting, Clippy, build, test, audit and secret-scan checks remain green.
• PR remains draft until the design decision and Temporal Rust qualification are human-reviewed.

Failure behaviour

• CNPG unavailable: activity fails retryably with bounded backoff; no provider work starts without a durable claim.
• Object storage/provider unavailable: the page runner records the bounded page failure; surviving pages continue and the final result may be partial.
• Temporal unavailable: accepted histories remain durable and workers reconnect; API starts fail fast after the bounded RPC deadline.
• Final result publication interrupted: idempotent publication resumes from durable page artifacts.
• Valkey/Qdrant/Langfuse unavailable: not part of this activity correctness path; processing continues with degraded cache, memory or telemetry.

Delivery and rollback

Ship behind a separate worker build/version and task-queue route. Run sandbox, replay, shadow and canary qualification before switching new workflows. Rollback routes new starts to the prior build while candidate histories remain replayable by their compatible worker. No schema deletion or production rollout is part of this issue.

Explicitly out of scope

• Production provider credentials or provider selection policy.
• Kubernetes/GitOps rollout and secret mutation.
• Human review UI.
• Claiming completion of the required 24-hour soak before it has actually run.

Parent: #1
Qualification dependency: #25

[Sam123ben]

Implemented and verified the durable runner workflow in stacked draft PR #28 (96ba471).

Evidence added:

• CNPG maximum page count and attempt limits are now exported from the domain, so the Temporal runner bound is 3,000 possible page attempts rather than the transport retry count.
• DurableDocumentWorkflow invokes run_checkpointed_pages_v1, continues-as-new every 50 runner iterations, and returns metadata-only terminal status.
• Live Temporal CLI 1.5.0/server 1.29.0 tests cover 51 durable iterations, continue-as-new and replay without document content in history.
• Live Temporal + disposable PostgreSQL test runs a 3-page document through the SDK activity boundary: page 2 exhausts, the workflow returns partial, and artifacts for pages 1 and 3 remain addressable.
• Full ordinary tests, full ignored PostgreSQL/RLS suite, Temporal harness, Clippy, build, audit and secret scan pass locally.

Still open in this issue: durable finalization/result publication, durable cancellation propagation, duplicate-delivery proof, cross-version worker routing, and the real 24-hour soak. The PR remains draft and unmerged.

[Sam123ben]

Security hardening update (commit d8e1841): parser subprocesses now run in a dedicated Unix process group. Any timeout, I/O/protocol failure, or dropped inspection terminates that group, preventing helper descendants from escaping cancellation.

The parser boundary continues to accept bounded input/output only and returns generic availability errors; raw documents, OCR text, paths, and credentials are not emitted to logs, traces, or workflow history. A dedicated regression test proves a spawned helper child no longer survives a deadline.

Verification: workspace tests; database-inclusive llvm coverage at 78.40% (70% gate); Rustfmt; Clippy with warnings denied; build; cargo audit; Gitleaks; and prohibited-reference scan all passed. Draft PR #28 remains unmerged for review.

[Sam123ben]

Durable finalization persistence update (commit 146ee0d): DocumentFinalizer now resolves result identity under the existing CNPG tenant/product RLS scope. It derives a validated opaque document_id from the job identifier and uses the accepted source artifact digest as the document_version; neither value is supplied by a caller or added to Temporal history.

The database-backed finalizer test now proves: a partial result receives the expected source version, and duplicate finalizer delivery returns Existing rather than making another terminal transition. The affected ocr-service/ocr-store suite, including all ignored database/RLS tests, passes locally.

Next in this issue: invoke this idempotent finalizer through a separate metadata-only Temporal activity, then prove cancellation and duplicate delivery across the SDK boundary.

[Sam123ben]

Temporal finalization orchestration update (commit d557686): terminal completed and partial durable workflows now invoke a distinct finalize_document_v1 activity. Its input remains the validated tenant/product/job identifier envelope and its result is empty; document bytes, OCR text, fields, signed URLs and result locators remain outside Temporal history. Cancelled workflows do not invoke finalization.

Qualification: Temporal contract tests passed; the checksum-verified live Temporal harness passed all six workflow scenarios; and the real PostgreSQL bridge passed both page-exhaustion/RLS scenarios. The finalizer activity is injected as its own worker capability, ready for the concrete GCS/CNPG adapter next.

Remaining work in this issue: concrete worker binding to DocumentFinalizer, SDK duplicate-finalizer delivery, cancellation during finalization, worker version routing and the 24-hour soak.

[Sam123ben]

Concrete finalizer adapter update (commit 7b75d89): DocumentFinalizerExecutor now implements the durable finalization capability. It invokes DocumentFinalizer::finalize_stored, so document identity/version are resolved inside CNPG scope. Only Committed and idempotent Existing outcomes succeed; Conflict, non-committable state, missing scope, invalid artifacts and cancellation are not silently treated as a published result. Storage, artifact-read and publication interruptions are surfaced as bounded retryable dependency failures.

Focused formatting, Clippy and Temporal contract tests pass. Hosted PR checks have been retriggered. Remaining proof work is an SDK-level duplicate finalizer delivery test using the concrete adapter, cancellation during finalization, worker-version routing and the required 24-hour soak.