Skip to content

feat(events): quarantine events with unknown contract schema versions - #1250

Merged
mikewheeleer merged 5 commits into
Talenttrust:mainfrom
maybay-dev:feat/1206-event-schema-quarantine
Aug 30, 2026
Merged

mikewheeleer merged 5 commits into
Talenttrust:mainfrom
maybay-dev:feat/1206-event-schema-quarantine

Conversation

@maybay-dev

@maybay-dev maybay-dev commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What this fixes

An event from a newer contract version cannot safely enter projections that assume an older payload shape — applying it would silently corrupt contract state. Previously the ingestion boundary had no schema-version concept at all: a future-versioned event would either fail generic validation (losing the data) or, worse, be projected under the wrong shape.

This PR validates the schema version at the ingestion boundary, retains events this backend cannot yet process in a redacted quarantine store (never silently dropped), and makes operator reprocessing explicit and authenticated once support ships.

Root cause

validateContractEventPayload and the ingestion routes understood type, sequence, network, ledger, etc., but had no notion of a contract schema version. The pipeline was therefore unable to distinguish "valid event from a newer contract" (must be retained and reprocessed later) from "invalid event" (must be rejected) — it had exactly one rejection path, and no durable record of what it turned away.

The fix and why

  • Optional schemaVersion on the event envelope (src/contracts/types.ts, validated in src/contracts/validation.ts): absent = legacy (treated as version 1); present-but-invalid (0, negative, non-integer, non-number) is rejected fail-closed — an ambiguous version is never guessed.
  • Boundary classifier (src/events/schemaVersion.ts): known / unknown / malformed / absent. The known set is an explicit, append-only constant (KNOWN_SCHEMA_VERSIONS = [1]).
  • Redacted quarantine store (src/events/eventQuarantine.ts, SQLite event_quarantine table): an unknown-version event is persisted with redactPayload applied and the reason run through the safe-error sanitizer — no secrets or stack details stored. Capacity is bounded (oldest-pending eviction, mirroring the existing job-quarantine and webhook-DLQ policies) and replay attempts are capped so re-processing never loops forever or deletes silently.
  • Per-item isolation (POST /api/v1/events/batch): one RPC page is ingested with per-item outcomes, so a malformed or unknown-version event never blocks the rest of the page (the issue's "mixed versions in one page" case).
  • Authenticated, audited reprocessing: GET /api/v1/events/quarantine and POST /api/v1/events/quarantine/replay are admin-only (requireAuth + requireRole('admin')) and write an audit entry. Replay re-runs the stored (redacted) event through the boundary: once support ships it processes; while the version is still unknown it re-quarantines (the operator sees re-quarantined with a new quarantine id).

How it was tested

  • Unit — classifier (src/events/schemaVersion.test.ts): known / unknown / malformed / absent; overridden known set (simulated contract upgrade).
  • Unit — store (src/events/eventQuarantine.test.ts): persistence with identity fields; redaction of sensitive payload fields; filtered + paginated listing; replay-attempt accounting and max-exceeded; replayed/pending stats; capacity eviction; getPayload for missing id.
  • Route/integration (src/routes/events.quarantine.routes.test.ts): known version processed; legacy event (no version) processed; unknown version quarantined (202, status: quarantined, not projected — history stays empty); malformed version rejected with 400; mixed versions in one batch page isolated per item; oversized/empty batch rejected; quarantine inspection requires admin (401/403); replay succeeds once the version becomes known (event appears in history); replay re-quarantines while still unknown; replay validation (missing id/reason → 400, unknown id → 404).
  • Full CI run: npm run lint (0 errors), npm run build (exit 0), full Jest suite — no new failures beyond the pre-existing suites already red on main (verified identical failure set against the base branch).

Follow-ups worth filing separately

  • Config-driven known versions: KNOWN_SCHEMA_VERSIONS is a code constant; an env/config-driven allowlist would let operators roll out support for a new version without a deploy.
  • Quarantine GC: entries are only removed by explicit replay or capacity eviction; a retention policy for never-replayed entries (e.g. alert after N days) would prevent unbounded growth on permanently-unsupported versions.

Note on the three leading commits in this branch (chore(ci) baseline repairs + fix(errors) word-bound pattern): upstream main was merged with a red CI pipeline — PR #1237 referenced a src/context.ts that was never added, and several files carried syntax corruption (AppExrror, db::, typof, ??., an invalid /r regex flag, a MALFORMEDD_ typo) that broke ESLint parsing and Jest module loading. This branch includes those minimal repairs because CI cannot pass without them; they are isolated at the base of the branch.
closes #1207

maybay-dev and others added 4 commits August 29, 2026 15:57
Main was left with a red CI pipeline: PR Talenttrust#1237 referenced a
src/context.ts module that was never added, and three files carried
syntax corruption (AppExrror typo, db:: double-colon, typof, stray
paren, ??-operator) that broke both ESLint parsing and Jest module
loading. This restores the baseline so feature branches start from a
state where `npm run lint`, `npm test`, and `npm run build` pass.

- Add src/context.ts (AsyncLocalStorage request context) that app.ts,
  auth/middleware.ts, and eventIngestionService.ts already import.
- Fix parse errors in src/errors/appError.ts, src/dlqStore.ts, and
  src/dependencies/contractsClient.test.ts; restore the intended
  ECONNREFUSED constant in transport-error classification.

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
safeErrors.ts carried the same corruption family as appError.ts: a
regex with an invalid trailing flag (/r), a typo'd constant reference
(MALFORMEDD_RESPONSE_PATTERNS), and mangled syscall/timeout constants
(ETEMEOUTT/ESOCKETIMEOUTT/EHOSTUREACH/ECONNREFUNED) that broke Jest
module parsing for every module importing sanitizeErrorMessage.

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
The SQL-fragment pattern matched the word "updates" inside legitimate
safe messages (e.g. 'version field is required for updates'), so
containsUnsafeContent rejected them. Word-bound the alternation so
only real SQL keywords trigger sanitization.

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
An event from a newer contract cannot safely enter projections that
assume an older payload shape, but it must not be silently dropped
either. Previously the ingestion boundary had no schema-version
concept, so a future-versioned event would either fail validation or
be projected under the wrong shape.

Add schema-version handling at the ingestion boundary:
- optional `schemaVersion` on the contract event envelope; absent =
  legacy (version 1), present-and-invalid is rejected fail-closed;
- valid-but-unknown versions are quarantined: the redacted event is
  persisted to a SQLite event_quarantine store and the endpoint
  returns 202 status:quarantined — it never reaches projections;
- POST /events/batch ingests one RPC page with per-item isolation so
  one bad event never blocks the rest;
- GET /events/quarantine and POST /events/quarantine/replay (admin,
  audited) make reprocessing explicit once support ships; replays
  re-quarantine while the version is still unknown and replay
  attempts are bounded (no silent deletion).

New surface: schemaVersion classifier, EventQuarantineStorage,
event_quarantine table, quarantine admin routes.

Closes Talenttrust#1206

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
@drips-wave

drips-wave Bot commented Aug 29, 2026

Copy link
Copy Markdown

@maybay-dev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@mikewheeleer
mikewheeleer merged commit 9be0fd0 into Talenttrust:main Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make event-ingestion backpressure visible before queue loss

2 participants