feat(events): quarantine events with unknown contract schema versions - #1250
Merged
mikewheeleer merged 5 commits intoAug 30, 2026
Merged
Conversation
Main was left with a red CI pipeline: PR Talenttrust#1237 referenced a src/context.ts module that was never added, and three files carried syntax corruption (AppExrror typo, db:: double-colon, typof, stray paren, ??-operator) that broke both ESLint parsing and Jest module loading. This restores the baseline so feature branches start from a state where `npm run lint`, `npm test`, and `npm run build` pass. - Add src/context.ts (AsyncLocalStorage request context) that app.ts, auth/middleware.ts, and eventIngestionService.ts already import. - Fix parse errors in src/errors/appError.ts, src/dlqStore.ts, and src/dependencies/contractsClient.test.ts; restore the intended ECONNREFUSED constant in transport-error classification. Generated with Codebuff 🤖 Co-Authored-By: Codebuff <noreply@codebuff.com>
safeErrors.ts carried the same corruption family as appError.ts: a regex with an invalid trailing flag (/r), a typo'd constant reference (MALFORMEDD_RESPONSE_PATTERNS), and mangled syscall/timeout constants (ETEMEOUTT/ESOCKETIMEOUTT/EHOSTUREACH/ECONNREFUNED) that broke Jest module parsing for every module importing sanitizeErrorMessage. Generated with Codebuff 🤖 Co-Authored-By: Codebuff <noreply@codebuff.com>
The SQL-fragment pattern matched the word "updates" inside legitimate safe messages (e.g. 'version field is required for updates'), so containsUnsafeContent rejected them. Word-bound the alternation so only real SQL keywords trigger sanitization. Generated with Codebuff 🤖 Co-Authored-By: Codebuff <noreply@codebuff.com>
An event from a newer contract cannot safely enter projections that assume an older payload shape, but it must not be silently dropped either. Previously the ingestion boundary had no schema-version concept, so a future-versioned event would either fail validation or be projected under the wrong shape. Add schema-version handling at the ingestion boundary: - optional `schemaVersion` on the contract event envelope; absent = legacy (version 1), present-and-invalid is rejected fail-closed; - valid-but-unknown versions are quarantined: the redacted event is persisted to a SQLite event_quarantine store and the endpoint returns 202 status:quarantined — it never reaches projections; - POST /events/batch ingests one RPC page with per-item isolation so one bad event never blocks the rest; - GET /events/quarantine and POST /events/quarantine/replay (admin, audited) make reprocessing explicit once support ships; replays re-quarantine while the version is still unknown and replay attempts are bounded (no silent deletion). New surface: schemaVersion classifier, EventQuarantineStorage, event_quarantine table, quarantine admin routes. Closes Talenttrust#1206 Generated with Codebuff 🤖 Co-Authored-By: Codebuff <noreply@codebuff.com>
|
@maybay-dev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
An event from a newer contract version cannot safely enter projections that assume an older payload shape — applying it would silently corrupt contract state. Previously the ingestion boundary had no schema-version concept at all: a future-versioned event would either fail generic validation (losing the data) or, worse, be projected under the wrong shape.
This PR validates the schema version at the ingestion boundary, retains events this backend cannot yet process in a redacted quarantine store (never silently dropped), and makes operator reprocessing explicit and authenticated once support ships.
Root cause
validateContractEventPayloadand the ingestion routes understoodtype,sequence,network,ledger, etc., but had no notion of a contract schema version. The pipeline was therefore unable to distinguish "valid event from a newer contract" (must be retained and reprocessed later) from "invalid event" (must be rejected) — it had exactly one rejection path, and no durable record of what it turned away.The fix and why
schemaVersionon the event envelope (src/contracts/types.ts, validated insrc/contracts/validation.ts): absent = legacy (treated as version 1); present-but-invalid (0, negative, non-integer, non-number) is rejected fail-closed — an ambiguous version is never guessed.src/events/schemaVersion.ts):known/unknown/malformed/absent. The known set is an explicit, append-only constant (KNOWN_SCHEMA_VERSIONS = [1]).src/events/eventQuarantine.ts, SQLiteevent_quarantinetable): an unknown-version event is persisted withredactPayloadapplied and the reason run through the safe-error sanitizer — no secrets or stack details stored. Capacity is bounded (oldest-pending eviction, mirroring the existing job-quarantine and webhook-DLQ policies) and replay attempts are capped so re-processing never loops forever or deletes silently.POST /api/v1/events/batch): one RPC page is ingested with per-item outcomes, so a malformed or unknown-version event never blocks the rest of the page (the issue's "mixed versions in one page" case).GET /api/v1/events/quarantineandPOST /api/v1/events/quarantine/replayare admin-only (requireAuth+requireRole('admin')) and write an audit entry. Replay re-runs the stored (redacted) event through the boundary: once support ships it processes; while the version is still unknown it re-quarantines (the operator seesre-quarantinedwith a new quarantine id).How it was tested
src/events/schemaVersion.test.ts): known / unknown / malformed / absent; overridden known set (simulated contract upgrade).src/events/eventQuarantine.test.ts): persistence with identity fields; redaction of sensitive payload fields; filtered + paginated listing; replay-attempt accounting and max-exceeded; replayed/pending stats; capacity eviction;getPayloadfor missing id.src/routes/events.quarantine.routes.test.ts): known version processed; legacy event (no version) processed; unknown version quarantined (202,status: quarantined, not projected — history stays empty); malformed version rejected with 400; mixed versions in one batch page isolated per item; oversized/empty batch rejected; quarantine inspection requires admin (401/403); replay succeeds once the version becomes known (event appears in history); replay re-quarantines while still unknown; replay validation (missing id/reason → 400, unknown id → 404).npm run lint(0 errors),npm run build(exit 0), full Jest suite — no new failures beyond the pre-existing suites already red onmain(verified identical failure set against the base branch).Follow-ups worth filing separately
KNOWN_SCHEMA_VERSIONSis a code constant; an env/config-driven allowlist would let operators roll out support for a new version without a deploy.Note on the three leading commits in this branch (
chore(ci)baseline repairs +fix(errors)word-bound pattern): upstreammainwas merged with a red CI pipeline — PR #1237 referenced asrc/context.tsthat was never added, and several files carried syntax corruption (AppExrror,db::,typof,??., an invalid/rregex flag, aMALFORMEDD_typo) that broke ESLint parsing and Jest module loading. This branch includes those minimal repairs because CI cannot pass without them; they are isolated at the base of the branch.closes #1207