Skip to content

Add retry-safe projection batch claims - #342

Open
dills122 wants to merge 4 commits into
masterfrom
codex/projection-batch-claim-slice-1a
Open

Add retry-safe projection batch claims#342
dills122 wants to merge 4 commits into
masterfrom
codex/projection-batch-claim-slice-1a

Conversation

@dills122

Copy link
Copy Markdown
Owner

Summary

  • add deterministic projection-batch identities, durable claims, member frontiers, and cleanup guards
  • establish database-clock retry authority before candidate selection in both projection topologies
  • retain completed authority for the full retry horizon so staggered stale selectors cannot repeat effects
  • keep claim, read-model effects, watermark advancement, and completion atomic; injected pre-watermark failures now roll everything back
  • strengthen ambiguous-commit coverage so repeated lifecycle enqueues cannot hide behind dirty-table conflict coalescing
  • repair the live Redpanda crash gate for Kafka transaction semantics and wire its materializer/projector fault hooks through Compose

Why

An ambiguous database commit or delayed stale selector must not repeat non-idempotent projection effects. This slice makes the claimed canonical membership the durable authority and fails closed when that authority is incomplete, expired, or conflicts with immutable projection semantics.

Verification

  • ./gradlew check — passed, including JaCoCo coverage verification
  • focused migrated-PostgreSQL claim/materialization integration suite — passed
  • node --test scripts/dev/db/migrate.test.mjs — 25/25 passed
  • make dev-smoke-venue-event-crash-gate — passed on a fresh isolated Docker stack
    • all four partitions covered
    • engine publish and transaction-offset failures recovered
    • materializer commit-before-ack failure recovered
    • projector failure left zero rows and zero watermarks before restart
    • zero replay gaps, checksum mismatches, missing outcomes, or duplicate replay inserts
  • Compose configuration, script syntax, and git diff --check — passed

Scope

This completes the Slice 1A correctness foundation. It does not claim the final projection throughput target; DigitalOcean scaling gates remain the next evidence step.

Claim canonical projection batches before effects so rollback and ambiguous-commit retries cannot repeat projection work. Add a stable identity contract, deadline/watermark-safe cleanup, guards for both persistence paths, and PostgreSQL regression coverage.
Drive same-store effects from the exact claimed member set, apply the configured retry horizon on both persistence paths, and restore the frozen measurement-before-tuning gates. Add PostgreSQL concurrency, duplicate-claim, and horizon regressions.
Start separated-store authority before candidate acquisition so stale paused batches expire before cleanup or claim. Enforce completed result counts against claimed membership in migrated and compatibility contracts, with PostgreSQL regressions.
Retain completed batch authority for the full database-clock retry horizon, keep stale selectors fenced, and make projector failure injection roll back atomically. Harden ambiguity coverage and repair the live transactional crash gate so these guarantees are exercised end to end.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant