spike: qualify Temporal Rust orchestration boundary

[Sam123ben]

Outcome

Produce a non-production, reviewable qualification candidate for the official Temporal Rust SDK v0.8.0 against the OCR document/page workflow. This is evidence for design issue #9, not production selection or deployment.

Scope

• Pin the official SDK source to the immutable v0.8.0 commit and record its preview status and supply-chain implications.
• Derive the Temporal workflow ID deterministically from validated product, tenant and job IDs; no document text, filenames, URLs, storage locators, credentials, or result payloads enter workflow search attributes or logs.
• Define a versioned, bounded workflow input and activity protocol using opaque identifiers only.
• Map accepted and cancellation outbox events to idempotent workflow start/signal behavior; acknowledge the CNPG outbox only after Temporal accepts the command.
• Keep all Postgres, GCS, OCR and provider I/O in activities; workflow code must remain replay-deterministic.
• Configure explicit activity timeouts, heartbeats, bounded retries with jitter, non-retryable validation errors, and cancellation.
• Exercise replay, duplicate start, cancellation, crash/retry, worker-version compatibility, page-level recovery, and a bounded 300-page history strategy.
• Document the 24-hour upgrade-soak procedure and the stable Go runner fallback required by design review: Rust-native evidence-first OCR engine #9 if the preview SDK fails qualification.

Acceptance criteria

• Red-first deterministic tests cover workflow ID/input bounds, duplicate delivery, cancellation, failed-page-only retry, and history rollover/continue-as-new decisions.
• An integration harness runs against a disposable local Temporal development server when available; ordinary unit tests require no network.
• No custom durable state machine is introduced; CNPG remains the lifecycle projection and transactional outbox authority.
• Rust format, all-target/all-feature Clippy, locked build/tests, RustSec, secret scan, prohibited-reference scan, and existing PostgreSQL coverage pass.
• No cloud, IAM, Kubernetes, secret, deployment, or production mutation is included.

Review gate

Do not merge, deploy, or select the Rust SDK for production until #9 records Temporal/platform, security, and SRE approval.

[Sam123ben]

Qualification candidate PR #26 is open as a draft and assigned to @Sam123ben.\n\nLocal evidence is green: exact SDK pins/checksums, deterministic scoped IDs, strict 512-byte/300-page inputs, duplicate start and cancellation mapping, five-second RPC deadline, 50-page rollover planning, bounded page activity policy, full locked workspace build/tests, real PostgreSQL coverage, RustSec and secret scans.\n\nThe PR intentionally remains draft. The disposable Temporal server harness, replay/crash/worker-version matrix and 24-hour upgrade soak remain required before this issue can close or design #9 can approve production selection. No cloud or production mutation was made.

[Sam123ben]

Hosted CI is now green after pinning shared-workflows PR tesserix/tesserix-workflows#11 at commit 299a1ee2ced322a825cc0f0838235c6c3269610f and enabling its boolean protoc gate.\n\nPR #26 checks:\n- Rust quality: passed\n- Rust PostgreSQL coverage: passed\n- repository secret scan: passed\n- GitGuardian: passed\n\nThe PR remains draft because the live Temporal server replay/crash/worker-versioning matrix and 24-hour upgrade soak are qualification evidence still to be produced; no production approval is implied.

[Sam123ben]

Temporal SDK execution qualification update (commit 54a227a):

• Added the real ocr_document_v1 SDK workflow with one stable activity ID per page.
• Continue-as-new carries a versioned, strictly validated content-free cursor every 50 pages.
• Activity policy is enforced in SDK options: 120s start-to-close, 10s heartbeat, 3 attempts, 1–30s backoff, non-retryable validation/scope failures, and cancellation propagation.
• Added a checksum-verified, opt-in Temporal CLI 1.5.0 harness using disposable local storage and no cloud credentials.
• Live result on 2026-09-04: 51 pages completed across two runs against server 1.29.0; first-run history replay passed.

Verification passed:

• cargo fmt --all --check
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo build --workspace --all-targets --all-features --locked
• cargo test --workspace --all-targets --all-features --locked
• ./scripts/test-temporal-qualification.sh
• cargo audit
• gitleaks detect --no-banner --redact --source .
• prohibited-reference scan and git diff --check

This does not claim the required worker-crash/cancellation matrix, worker-version test, or 24-hour soak. PR #26 remains draft and unmerged pending those gates and human approval.

[Sam123ben]

Cancellation qualification follow-up (c30cdb8):

• Added an observed in-flight activity cancellation test against Temporal CLI 1.5.0 / server 1.29.0.
• The test waits for the qualification activity to begin, cancels through the workflow handle, and verifies history contains the workflow cancellation request and activity cancellation request, no following page is scheduled, and the terminal event is WorkflowExecutionCanceled.
• The qualification activity emits content-free heartbeat progress and exposes an observation hook only in the non-production qualification implementation.
• Live continue-as-new/replay and cancellation cases both pass via scripts/test-temporal-qualification.sh.
• Fixed the PostgreSQL coverage interaction: reusable CI deliberately runs ignored database tests with --include-ignored; live Temporal tests now no-op when the checksum-verified CLI path is absent while the dedicated harness always supplies it. This reproduces and fixes the prior hosted failure.

Worker process-loss/restart remains unproven. An in-process task abort cannot model it because SDK 0.8.0 retains worker registration until orderly finalization; the next qualification must use process isolation rather than weakening that test.

[Sam123ben]

Worker-loss qualification completed locally (bae7a64):

• Runs the workflow worker and activity worker in separate OS processes.
• Starts a 300-page workflow and receives a one-byte, loopback-only content-free readiness signal from page 1.
• Force-terminates the activity-worker process, starts a replacement process, and waits through the 10-second heartbeat timeout.
• Temporal history records page 1's final start as attempt 2 with a prior failure; the first 50-page history contains exactly 50 completions.
• The workflow completes all 300 pages across six continue-as-new runs.
• The helper accepts only loopback HTTP Temporal endpoints, loopback observation endpoints, bounded task-queue names and fixed modes. External metadata endpoints and malformed inputs fail before connection.
• The verified CLI harness now serializes the three local-server cases to remove resource/race coupling.

Verification passed: dedicated live harness, full workspace tests, Clippy with warnings denied, formatting, RustSec audit, gitleaks, prohibited-reference scan and diff check. This closes integration-matrix item 1. Cross-version replay/routing, remaining cancellation timings, bounded-exhaustion behavior and the 24-hour soak remain open; PR #26 stays draft and unmerged.

[Sam123ben]

Cancellation matrix completed locally (323edbe):

• cancellation before workflow start is idempotent;
• cancellation while page 1 is executing emits an activity-cancel request, schedules no next page and terminates cancelled;
• cancellation after continue-as-new reaches page 51 preserves exactly 50 completed first-run pages and schedules no page 52;
• repeated cancellation after successful completion is idempotent.

The dedicated checksum-verified harness now passes five serialized live-server cases, including the 300-page process-loss recovery. Full workspace tests and Clippy remain green. Integration-matrix item 3 is complete. The PR remains draft pending cross-version replay/routing, bounded-exhaustion semantics, the 24-hour soak and human approvals.

[Sam123ben]

CNPG bounded-exhaustion evidence (170e289):

• page 2 returns a retryable failure exactly three times;
• durable attempts are 1, 2 and 3, then the page becomes exhausted;
• pages 1 and 3 remain successful and their immutable artifact locators remain present;
• the page workflow becomes partial;
• the finalizer reads the two surviving page artifacts, publishes a provider-neutral partial result and atomically commits the job as partial with a ready result locator.

The full real-PostgreSQL/RLS ignored suite passed locally (including 6 page-runner tests and 17 store contracts). This proves the CNPG/GCS-locator side of matrix item 4. Injecting the same exhaustion through the Temporal SDK activity boundary remains open.

[Sam123ben]

Qualification promotion update (2026-09-04): PR #26 conflict with the merged evaluation workspace was resolved in 6d90ae2 by retaining both ocr-eval and ocr-temporal plus their required dependencies.

Evidence: local cargo fmt --all --check, Clippy with warnings denied, locked build, full workspace tests, and cargo audit pass. Hosted Rust quality, CNPG-backed database coverage, secret scan and GitGuardian checks are all successful; the PR is CLEAN.

This does not select Temporal Rust for production. PR #26 remains draft pending the recorded Temporal/platform, security and SRE review; live replay/crash/worker-versioning evidence and the required 24-hour soak are also still open.

[Sam123ben]

Current-main Temporal qualification re-run (2026-09-04): ./scripts/test-temporal-qualification.sh passed all six real loopback cases against checksum-verified Temporal CLI 1.5.0 / server 1.29.0: 300-page activity-worker loss/recovery, cancellation before/during/after continue-as-new, 51-page replay, and durable runner replay.\n\nI also verified the remaining worker-versioning gap is substantive: qualification_deployment_options() supplies a stable build ID but WorkerDeploymentOptions::from_build_id leaves use_worker_versioning=false. A red-first experimental opt-in caused the live activity-worker-loss case to time out (5/6 cases passed), because no current deployment version had been established for the local server. The SDK exposes SetWorkerDeploymentCurrentVersion, whose protocol requires waiting for all expected task-queue pollers and must not bypass missing-poller protection.\n\nThe experiment was reverted; no partial versioning configuration, deployment selection, cloud mutation, or production claim was retained. The next qualification slice must add an isolated local-server deployment manager that registers the workflow and activity pollers, establishes current v1 without allow_no_pollers, then proves a v1 in-flight history is replayed/routed correctly while v2 is introduced. This remains a #9 human-review and 24-hour-soak gate.

[Sam123ben]

Merged qualification-document update: #44 (8644670288541839d492ea86c507933d569e3751) records the observed versioning scheduling failure and required current-version/poller-readiness protocol. All seven PR checks passed: Rust quality, CNPG database coverage, both container builds, image contract, secret scan and GitGuardian. This improves review evidence only; it does not opt in a worker, select Temporal for production, or satisfy worker-version routing/24-hour-soak gates.