design review: CNPG, Qdrant and Valkey data boundaries

[Sam123ben]

Parent: #1. Related reviews: #2, #3, #4, #5, #7 and #9.

Decision

• CloudNativePG Postgres is the authoritative transactional database.
• GCS stores immutable source, page and full-result objects.
• Qdrant stores rebuildable semantic-memory embeddings and bounded retrieval metadata.
• Valkey stores TTL-bound cache entries, atomic admission counters and non-authoritative coordination hints.

No business fact exists only in Qdrant or Valkey. Canonical memory, chunks and evidence exist in Postgres/GCS before asynchronous indexing.

Consistency

Postgres business writes and outbox intent commit in one transaction. Idempotent consumers update Qdrant and invalidate/refill Valkey. Vector indexing and caches are eventually consistent and expose version/lag. Agent citations resolve each vector hit back to the authorized canonical document version.

Review checklist

• Review tenant/RLS boundary for every Postgres table and Qdrant query.
• Review CNPG three-instance multi-zone topology and Pooler transaction mode.
• Verify SQLx pool sizing and disabled statement cache behind PgBouncer.
• Approve WAL archive, base backup, PITR and restore targets.
• Prove automatic failover and quarterly restore/failover exercises.
• Approve Qdrant collection, tenant partition and embedding-version strategy.
• Prove deterministic idempotent vector IDs and collection-alias reindex rollback.
• Require TTL and schema/tenant namespace on every Valkey key.
• Verify cache outage falls through and admission remains conservatively bounded.
• Verify cross-store deletion remains pending until GCS, Qdrant and Valkey confirm removal.
• Define storage, backup and cross-zone cost envelope.

Failure contract

• CNPG outage pauses authoritative lifecycle work; no state is invented elsewhere.
• Qdrant outage does not block OCR; indexing queues and semantic retrieval degrades explicitly.
• Valkey outage increases latency and applies local admission ceilings; it never loses work.
• GCS outage prevents result commit; Postgres cannot claim a missing result object.

Production manifests and IAM are GitOps-only through tesserix-k8s; this issue authorizes design review, not a live cluster mutation.

[Sam123ben]

ADR-0005: CNPG Postgres, Qdrant semantic memory and Valkey cache

Status: proposed

Context

OCR job state, document provenance, review corrections and audit records require transactional correctness. Agent retrieval needs vector search. Hot status/result reads, admission counters and deduplication hints need low latency. Binary documents and large normalized results do not belong in a relational row or vector payload.

Using all three datastores without strict ownership would create conflicting truth, incomplete deletion and outage coupling.

Decision

• CloudNativePG Postgres is the authoritative transactional database.
• GCS stores immutable source, derived page and full result objects.
• Qdrant stores rebuildable embeddings and bounded searchable metadata for semantic document/agent memory.
• Valkey stores expiring cache entries, distributed admission counters and non-authoritative coordination hints.

No business fact exists only in Qdrant or Valkey. A canonical memory/chunk record and its source evidence exist in Postgres/GCS before indexing. Postgres transactions write business state and an outbox event together; idempotent consumers update Qdrant and invalidate/refill Valkey after commit.

CNPG topology

Start with one three-instance CNPG cluster in the approved region, spread across zones: one primary and two streaming replicas. Use a dedicated database and least-privilege roles for API, worker, outbox/indexer and migration identities. Application traffic uses a CNPG Pooler in transaction mode; SQLx disables statement caching behind PgBouncer.

Continuous WAL archiving and scheduled base backups write to a separate versioned GCS backup bucket through Workload Identity. Initial targets are automatic failover within two minutes for a pod/node failure, backup RPO at most five minutes and full restore RTO at most 60 minutes. These are accepted only after quarterly restore and failover exercises.

Migrations are forward-only and GitOps delivered, with lock/statement timeouts and expand-migrate-contract sequencing. Backups do not make an incompatible schema rollback safe; the previous application version must remain compatible.

Qdrant ownership

Use collections partitioned by region and embedding model major version. Every point carries tenant ID, canonical record ID, document/version, chunk/span/evidence IDs, schema version, retention deadline and embedding version. Queries require an authorized tenant filter at the repository boundary. Regulated or outsized tenants may receive a dedicated collection only by policy.

Point IDs derive deterministically from tenant, canonical record version and embedding version, making at-least-once indexing idempotent. Re-indexing builds a new versioned collection, validates recall/latency, switches an alias, and retains the prior collection for bounded rollback.

Qdrant is eventually consistent with Postgres. Index lag is observable. A Qdrant outage does not block OCR completion; indexing remains queued and semantic search returns a typed unavailable/degraded result rather than silently incomplete answers.

Valkey ownership

All keys are tenant- and schema-version namespaced and have explicit jittered TTLs. Cache-aside reads fall through to Postgres/GCS. Terminal job states may use longer TTLs because they are immutable; active status is short lived. Writes invalidate through the outbox so a database commit is never dependent on Valkey.

Rate limits and quotas use atomic scripts or single-round-trip increment/expiry operations. If Valkey is unavailable, the service bypasses cached reads and applies conservative per-instance admission ceilings so the outage increases latency without allowing unbounded work.

Valkey is not used as the durable job queue, workflow history, document store or authoritative idempotency ledger.

Deletion

Deletion begins with an authorized Postgres state transition and outbox event. Workers remove source/results from GCS, points from every Qdrant collection/version and related Valkey keys. The job becomes deleted only after required stores confirm removal; otherwise it remains deletion_pending with retries and alerts. A content-free tombstone/audit record remains according to policy, and backup expiry is disclosed separately.

Alternatives

• Postgres with pgvector only: simplest and retained as a fallback, but Qdrant is selected for the requested semantic-memory specialization and independent vector scaling.
• Qdrant as memory truth: rejected because transactional audit, corrections, erasure and rebuildability would be weaker.
• Valkey as a queue or authoritative idempotency store: rejected because eviction/outage must not lose accepted work.
• Store OCR JSON and documents in Postgres: rejected because large payloads inflate WAL, backup and replication cost.

Consequences

Each store has one clear failure mode and recovery path, but the platform operates three stateful systems. The added Qdrant and Valkey cost is justified only by measured retrieval and latency needs. GitOps, backup, restore, tenancy and observability must cover each system before production.

[Sam123ben]

Data architecture

Ownership map

Data	Authoritative store	Derived/cache store
Tenant policy and quotas	CNPG Postgres	Valkey counters/cache
Upload, job, page and attempt state	CNPG Postgres	Valkey status cache
Idempotency and outbox	CNPG Postgres	none
Original document and page artefacts	GCS	none
Full normalized OCR result	GCS with digest/version in Postgres	bounded Valkey result metadata
Extracted fields, validation and review summary	CNPG Postgres	Valkey cache
Canonical chunks/memory and evidence	CNPG Postgres plus GCS text artefact	Qdrant embedding/index
Reviewer corrections	CNPG Postgres	Qdrant re-index after commit
Audit and deletion state	CNPG Postgres	none
Workflow history	qualified durable workflow engine	Postgres status projection

Write and index flow

flowchart LR
    Worker[OCR / extraction worker] -->|one transaction| PG[(CNPG Postgres)]
    Worker -->|immutable object first| GCS[(GCS)]
    PG --> Outbox[Transactional outbox]
    Outbox --> Indexer[Idempotent memory indexer]
    Indexer --> Qdrant[(Qdrant)]
    Outbox --> Invalidator[Cache invalidator]
    Invalidator --> Valkey[(Valkey)]
    Agent[Authorized agent retrieval] --> Qdrant
    Agent -->|resolve canonical evidence| PG
    Agent -->|resolve text/version| GCS

Loading

Large object upload completes before the Postgres row references its digest and generation. A failed metadata transaction leaves an unreferenced object for a bounded sweeper; it never creates a result pointing at missing bytes. Postgres plus outbox is the only commit boundary for events, indexing intent and cache invalidation.

Postgres schema groups

• tenants, tenant_policies, retention_policies;
• uploads, documents, document_versions, pages;
• jobs, job_pages, attempts, idempotency_keys;
• result_versions, field_summaries, validation_findings;
• review_tasks, correction_versions;
• memory_records, chunks, embedding_intents;
• outbox_events, consumer_receipts, audit_events, deletion_operations;
• model_manifests, processing_profiles, calibration_versions.

Every tenant-owned table has tenant_id NOT NULL, composite uniqueness that includes tenant identity where appropriate and row-level security as defense in depth. UUIDv7 identifies externally visible objects. Times are timestamptz; monetary cost is numeric or integer minor units; queryable lifecycle fields are columns rather than opaque JSONB.

Hot list indexes begin with tenant equality, then state/range and stable cursor ordering, for example (tenant_id, created_at DESC, id DESC). Pending-work subsets use partial indexes. Exact indexes and pool sizes are accepted only after query plans and concurrency arithmetic exist.

CNPG operations

• Three instances with topology spread and disruption budget.
• Primary writes; replicas serve only explicitly stale-tolerant reads after measurement.
• CNPG Pooler/PgBouncer transaction mode with total SQLx pool connections below database limits and administrative/migration headroom.
• WAL archive plus scheduled base backups to a separate GCS bucket using Workload Identity.
• Anti-affinity, storage alerts, replication-lag alerts and SLO-facing database availability alerts.
• Quarterly failover, PITR and full restore exercises; recorded recovery time is the evidence for RTO.
• GitOps owns Cluster, Pooler, backup, monitoring and migration configuration. No imperative production database changes.

Qdrant semantic memory

Qdrant contains vectors, sparse/dense retrieval fields and bounded metadata—not raw documents, unrestricted OCR payloads or the only copy of a memory. The repository method requires a verified tenant scope, collection/model version and result limit; callers cannot pass arbitrary filters.

Indexing is asynchronous and idempotent. Read-your-own-write is not promised for semantic memory; the API exposes index status/lag. Agent answers resolve Qdrant hits back to the canonical Postgres/GCS version before citation, preventing stale or cross-tenant vector payloads from becoming evidence.

Snapshots protect recovery time, but canonical records allow a full rebuild. Collection alias promotion supports zero-downtime embedding/model migrations and rollback.

Valkey keys

Example namespaces:

ocr:job:v1:{tenant_id}:{job_id}
ocr:result-meta:v1:{tenant_id}:{document_version}
ocr:admission:v1:{tenant_id}:{window}
ocr:model-manifest:v1:{model_digest}

Every key has a TTL. Active job cache entries are short lived; immutable terminal metadata may be longer. Hot-key refill uses single-flight. Rate-limit scripts set expiry atomically. Valkey loss never loses a job, event, correction or deletion request.

Failure behavior

Dependency	Behavior
CNPG primary unavailable	Writes fail fast; failover promotes a replica; workers retry before committing state
CNPG cluster unavailable	Create/read lifecycle endpoints are unavailable; accepted workflow work pauses safely
GCS unavailable	Document processing retries boundedly; metadata cannot claim a result whose object was not committed
Qdrant unavailable	OCR completes; index intents queue; semantic retrieval returns typed unavailable/degraded status
Valkey unavailable	Reads fall through to Postgres/GCS; conservative local admission limits replace distributed counters

Consistency and deletion

Job state, review corrections and deletion authorization are strongly consistent in Postgres. Cache state and vector indexing are eventually consistent and expose version/lag. An agent citation is valid only after resolving the canonical document and evidence version.

Deletion is a workflow across stores and is not reported complete until GCS, Qdrant and Valkey removal succeeds. Retried messages are safe because object generations and deterministic vector IDs make operations idempotent.

[Sam123ben]

Kora infrastructure implementation now has a dedicated GitOps issue: tesserix/tesserix-k8s#914. It owns the five data-class buckets, exact-generation workflow, CNPG evaluation registry/roles, protected runner, lifecycle/deletion, recovery and cost evidence. This issue remains authoritative for the CNPG/Qdrant/Valkey data boundaries.

[Sam123ben]

2026-09-04 Rust client admission evidence: crates.io currently exposes tokio-valkey 0.0.0-alpha2 (async, unknown MSRV) and valkey 0.0.0-alpha5 (synchronous), while fred 10.1.0 is a mature async client explicitly supporting Valkey, MIT licensed, MSRV 1.75. preview-glide-rust 0.2.0 is also preview-branded and defaults to synchronous behavior.

Recommendation for review: qualify digest/lockfile-pinned fred 10.1.0 with minimal features against the GitOps Valkey version, TLS/auth, cluster failover, cancellation and timeout tests before application integration. Do not admit the alpha/preview clients merely for naming. Any adapter must remain cache-aside, TTL-only, tenant/version namespaced and fall through to CNPG/GCS on timeout. No dependency or infrastructure change was made by this assessment.

[Sam123ben]

Semantic-memory domain candidate #22 is ready for review. It makes deterministic point IDs, tenant-scoped query construction, version-derived collection aliases, bounded limits and content-free evidence metadata enforceable before a Qdrant client is selected. CI, real PostgreSQL coverage and secret scanning are green. Client/cluster admission, alias rollout, snapshots, deletion and production indexing remain open checklist items.

[Sam123ben]

Valkey status-cache candidate #24 is ready for design review. It implements the previously recorded boundary: schema/product/tenant/job namespacing, mandatory bounded TTLs, bounded GETRANGE reads, scope verification on every hit, short timeouts, and PostgreSQL fallback on miss/error/invalid data. It stores no OCR text, filenames, URLs, storage locators, credentials, idempotency state, or result payloads. All PR checks are green; no cloud or production mutation was made.