fix(ingestion): Horizon replay safety and idempotency (#1261)

[Nwoyemartha]

Summary

• Implemented durable replay/idempotency protection at the Horizon bond-creation ingestion boundary (issue [Quality][High] Horizon ingestion and reconciliation: replay and idempotency — QE-2026-08 #1261).
• Bound every chain operation to its durable request key (stream_name, operation_id) as a row of the horizon_events ledger, written in the same transaction as the business effect (identity/bond mutation) and the cursor checkpoint.
• Added deterministic rejection of conflicting request-key reuse (EVENT_ID_CONFLICT) and stale/behind-checkpoint operations (STALE_INGESTION_EVENT), both without any state write.
• Made the cursor checkpoint forward-only (upsertCursorMonotonic) so replay storms and reordered deliveries converge on the furthest checkpoint instead of regressing it.
• onEvent now fires only for newly applied events, so a duplicate delivery can never surface a second business effect to subscribers.

Correctness / Invariants

• The same logical operation may be retried safely and produces a deterministic result without applying the business effect more than once.
• Duplicate processing results in one committed business effect (verified duplicates are no-ops; the UNIQUE (stream_name, event_id) index is the concurrency gate).
• Conflicting request-key reuse (same operation id, materially different payload) is rejected deterministically; the committed record and its state are left untouched.
• Failed/rejected operations (provider fault, stale, conflict, crash before COMMIT) leave no unauthorized or partially applied state — everything rolls back together.
• Replay/reordering converges to authoritative state: committed replays are verified duplicates, the checkpoint never moves backwards, and parity reconciliation (horizon-events-parity.md) remains unchanged.

Tests

New integration-boundary tests (src/listeners/__tests__/horizonBondIngestion.test.ts, 10 tests) at the actual business-effect boundary:

• Success path — valid operation applies exactly once; state, ledger record, and cursor persist.
• Duplicate — identical redelivery is a deterministic no-op; no second business effect.
• Conflicting key — same operation id with a different payload is rejected (EVENT_ID_CONFLICT); committed state intact.
• Timeout/retry — transient connection failure followed by retry commits exactly one effect.
• Lost provider response — committed-but-response-lost retry converges without a second effect.
• Mid-transaction failure — any write failure rolls back the whole transaction; no partial state.
• Stale/reordered — uncommitted operation behind the checkpoint is rejected (STALE_INGESTION_EVENT); no writes.
• Replay — replaying a committed range yields verified duplicates and converges.
• Concurrent — two overlapping deliveries commit a single effect.
• No partial state — stale rejection leaves nothing behind.

Repository-level tests (src/db/repositories/horizonEventRepository.test.ts) cover claim() insert/duplicate/conflict semantics, canonical payload fingerprinting (key-order invariance), per-stream scoping, and the client-scoped idempotent INSERT.

Validation

• npx vitest run src/listeners/__tests__/horizonBondIngestion.test.ts src/db/repositories/horizonEventRepository.test.ts src/listeners/__tests__/horizonBondEvents.test.ts — PASS (3 files, 47 tests)
• npx tsc --noEmit — no new errors vs origin/main (identical 40-error pre-existing baseline; zero errors in changed files)
• ESLint on all changed files — no new errors (7 pre-existing no-console in horizonBondEvents.ts unchanged; test files are eslint-ignored)
• Pre-existing failures verified identical on origin/main via worktree (horizonBondEvents.atomic.test.ts, horizonBondEvents.validation.test.ts, src/__tests__/horizonBondEvents.test.ts — all fail the same way on main due to a workerPool metrics-mock issue unrelated to this change)

Security / Correctness

• Every decision is made inside the transaction that would carry the effect: no code path can apply a business effect without its durable ledger record or with a stale/conflicting request key.
• Payload fingerprints are canonical (recursively sorted keys), so semantically identical payloads are never misclassified as conflicts.
• Timeout/uncertain-completion semantics: rolled-back retries take the applied path exactly once; committed-but-lost responses take the replayed path — both safe by construction.

Compatibility / Migration

• subscribeBondCreationEvents(dlqRouter, onEvent?, pool?) signature unchanged.
• One observable behavior change: onEvent fires only for newly applied events (previously it fired for every delivery). Strictly safer; covered by updated listener tests.
• New typed errors HorizonEventConflictError (EVENT_ID_CONFLICT) and HorizonEventStaleError (STALE_INGESTION_EVENT) abort the transaction and flow through the existing DLQ routing.
• No schema change / migration: reuses existing horizon_events (033) and horizon_cursors (007) tables. No rollback steps required.

Documentation

Added docs/horizon-replay-idempotency.md covering the invariant, durable request identity, deterministic outcomes, failure-mode handling, compatibility, and security assumptions.

Issue

Closes #1261

[drips-wave]

@Nwoyemartha Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits