postgresql cdc v2 by serprex · Pull Request #4300 · PeerDB-io/peerdb

serprex · 2026-05-13T07:17:56Z

No description provided.

github-actions · 2026-05-13T07:41:58Z

❌ Test Failure

Analysis: Test_PartitionBy fails deterministically across all 4 test suites with a consistent assertion mismatch (expected partition key "num" but got "(num)"), indicating a real behavioral change — likely a newer ClickHouse version now wraps PARTITION BY expressions in parentheses.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-05-13T15:13:28Z

❌ Test Failure

Analysis: Test_CDC_V2_Protocol deterministically times out (200s) across all matrix jobs with other.NumRecords stuck at 1 instead of 2, indicating a real replication bug likely related to the ClickHouse ORDER BY/PARTITION BY change only partially fixed in the previous commit.
Confidence: 0.72

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-05-13T15:34:38Z

🔄 Flaky Test Detected

Analysis: Test_CDC_V2_Protocol times out at ~200s across all three matrix variants (199.33s–200.26s), indicating it's hitting an internal deadline rather than a real assertion failure, causing the overall e2e suite to exhaust the 900s timeout.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-05-13T15:56:58Z

🔄 Flaky Test Detected

Analysis: All failures are timeout errors ("UNEXPECTED TIMEOUT waiting on initial", "UNEXPECTED TIMEOUT v2 single insert", "context deadline exceeded" on Temporal) across all three CI matrix jobs, indicating overloaded CI runners rather than a code regression.
Confidence: 0.92

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-05-13T16:18:59Z

❌ Test Failure

Analysis: Test_CDC_V2_Protocol fails deterministically across all 3 matrix configurations (~200s each), consistent with a real regression likely introduced by the recent ClickHouse PARTITION BY/ORDER BY change rather than a flaky environment issue.
Confidence: 0.82

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-05-13T16:57:11Z

🔄 Flaky Test Detected

Analysis: The e2e test suite for the maria/8.0 matrix job hit the hard 900-second timeout and was killed mid-run, with no assertion failures, indicating infrastructure slowness or a stuck test rather than a code regression.
Confidence: 0.88

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-05-13T17:18:48Z

❌ Test Failure

Analysis: Test_NullEngine deterministically fails in all 4 test suites (MySQL_CH, MySQL_CH_Cluster, PG_CH, PG_CH_Cluster) because the ClickHouse NullEngine table never accumulates records (other.NumRecords stays at 0 across every polling cycle), which is likely a real regression from the recent ClickHouse version behavior change (the prior commit already patched PARTITION BY/ORDER BY issues from the same upgrade).
Confidence: 0.78

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

claude · 2026-05-21T16:30:18Z

Code review

Bug: v2 state not propagated through Lua stream adapter

File: flow/connectors/postgres/postgres_source.go line 274

SetV2Active and SetCommittedXIDs (line 222) are called on req.RecordStream, which is the pull-side CDCStream. When a Lua transform script is configured, AttachToCdcStream creates a separate CDCStream (outstream) for the sync goroutine. This outstream never receives v2Active or committedXIDs from the input stream.

The result: req.Records.V2Active() returns false in the sync goroutine (cdc.go line 281), so records are routed to the v1 raw table instead of the v2 WAL sink. Meanwhile, normalize reads from the WAL sink (based on the dynamic config flag), finds nothing, and committed data is silently lost.

The stream adapter already propagates SchemaDeltas and checkpoints (lines 78-81). The fix is to also propagate v2 state there -- v2Active should be propagated before SignalAsNotEmpty (so the sync goroutine sees it), and committedXIDs can be propagated at the end alongside SchemaDeltas.

codecov · 2026-05-21T18:05:33Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
2228	1	2227	207

View the full list of 1 ❄️ flaky test(s)

github.com/PeerDB-io/peerdb/flow/connectors/postgres::TestRunPipeline_DstExitsNonZero

Flake rate in main: 16.30% (Passed 77 times, Failed 15 times)

Stack Traces | 0s run time

=== RUN   TestRunPipeline_DstExitsNonZero
    pgdump_schema_test.go:109: error "src failed: signal: broken pipe\nstderr:\n" does not mention dst failure
--- FAIL: TestRunPipeline_DstExitsNonZero (0.00s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-05-21T18:09:00Z

🔄 Flaky Test Detected

Analysis: TestRunPipeline_DstExitsNonZero is racy: src got SIGPIPE because dst exited before reading the pipe, and goroutine scheduling determined whether "src failed" or "dst failed" was reported — the test assumes dst's Wait() always completes first, which isn't guaranteed.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

ilidemi

Fun stuff! First pass, haven't looked into details of CDC pull or schema changes yet

ilidemi · 2026-06-02T01:43:09Z

+			// possible. Without this guard a small streamed txn whose Commit arrives
+			// shortly after StreamStop could be split across two batches, leaving the
+			// records orphaned in the WAL sink with no committedXID to normalize.
+			inProgressStreams := p.inStream || len(p.activeStreams) > 0


If PG has events

start stream tx1 insert stop stream begin tx2 insert commit start stream tx1 (resume) insert

then no WAL activity for some time, do we know for sure the stop stream will come right away to unblock tx2?

ilidemi · 2026-06-02T01:48:06Z

+	// WHERE clause: WAL sink mode filters by committed XIDs, v1 mode filters by batch_id range
+	if t.walSinkMode {
+		fmt.Fprintf(&selectQuery,
+			" FROM %s WHERE _peerdb_txid IN (%s) AND _peerdb_destination_table_name = %s",


If the mirror is significantly behind, txids wrapping around wouldn't be unheard of (11 hours at 100K writes per second). Is there a way to deal with that (handle duplicates or stop sync until prev txid was cleaned up)?

github-actions · 2026-06-03T00:26:34Z

🔄 Flaky Test Detected

Analysis: TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Types_CH failed on a 1-microsecond QValueTime precision mismatch (17m38.020664s vs 17m38.020663s) between Postgres and ClickHouse, a value-dependent rounding flake rather than a logic bug.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

serprex mentioned this pull request May 13, 2026

v2 #1712

Closed

serprex force-pushed the v2-take-2 branch from 276d495 to 26d53b7 Compare May 13, 2026 14:47

serprex force-pushed the v2-take-2 branch from b43fd00 to f5a2668 Compare May 13, 2026 16:35

serprex force-pushed the v2-take-2 branch from f5a2668 to bd0a5a8 Compare May 14, 2026 04:46

serprex force-pushed the v2-take-2 branch from bd0a5a8 to a01a330 Compare May 21, 2026 16:08

serprex marked this pull request as ready for review May 21, 2026 16:12

serprex requested a review from a team as a code owner May 21, 2026 16:12

ilidemi reviewed Jun 2, 2026

View reviewed changes

serprex and others added 6 commits June 2, 2026 23:47

postgresql cdc v2

19dc94a

test streamed transactions work too

70abdee

TTL

0460bbe

lift streaming state across batches

e2909e4

pua

b0ecb9b

xid

68aeaa1

serprex force-pushed the v2-take-2 branch from a7b7538 to 68aeaa1 Compare June 2, 2026 23:49

small review items

48eb9b5

Uh oh!

Conversation

serprex commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented May 13, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented May 13, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented May 13, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented May 13, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented May 13, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented May 13, 2026

❌ Test Failure

Uh oh!

claude Bot commented May 21, 2026

Code review

Bug: v2 state not propagated through Lua stream adapter

Uh oh!

codecov Bot commented May 21, 2026

❌ 1 Tests Failed:

Uh oh!

github-actions Bot commented May 21, 2026

🔄 Flaky Test Detected

Uh oh!

ilidemi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilidemi Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

ilidemi Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 3, 2026

🔄 Flaky Test Detected

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants