Make catalog decoding consistent across DDL & restarts by serprex · Pull Request #73 · ClickHouse/walshadow

serprex · 2026-07-22T23:00:39Z

Replace live catalog cache & invalidation tracking with durable relation descriptor log

Log checkpoints at commit boundaries. This allows looking up relation id at a given LSN

Track next_lsn as EndRecPtr. Synchronize on catalog updating commits

This will be basis for keeping some time series tracking of catalog updates,
in meantime this allows exact comparisons with pg_last_wal_replay_lsn

Also enforce shadow catalog is using streaming replication, avoids deadlocks

Plan transactions before emitting rows

Build and validate each committed transaction before applying schema, config, or ClickHouse side
effects. Freeze descriptors and routes in a checksummed plan, spill large plans to disk, then replay
only after planning succeeds

Defer records from catalog-dirty transactions until commit, decode them against committed
descriptors, and fail ambiguous layouts or unsupported WAL shapes instead of silently dropping rows

This will be basis for keeping some time series tracking of catalog updates, in meantime this allows exact comparisons with pg_last_wal_replay_lsn Also enforce shadow catalog is using streaming replication, avoids deadlocks

Replace live catalog cache & invalidation tracking with durable relation descriptor log Log checkpoints at commit boundaries. This allows looking up relation id at a given LSN

serprex · 2026-07-23T19:45:56Z

PLAN_NO_SKIP_LOOP.md
PLAN_NO_SKIP.md
LOOP_JOURNAL.md

Last commit was built over the course of a 12 hour 35 iteration /loop

Build and validate each committed transaction before applying schema, config, or ClickHouse side effects. Freeze descriptors and routes in a checksummed plan, spill large plans to disk, then replay only after planning succeeds Defer records from catalog-dirty transactions until commit, decode them against committed descriptors, and fail ambiguous layouts or unsupported WAL shapes instead of silently dropping rows

…dable Capture marks span it can't prove decodable when a transaction changes a table in place. Stash resolution asks log at the commit's next_lsn, past that span's end, so it always got a clean Present back and rows written inside the span decoded under the final descriptor anyway Only mark a span when bytes really change. typmod and attstorage moves are declared shape only. Tuple walk reads attlen/attalign/attbyval, so keeps early valid_from and publish nothing. Publish span per key verdict can name, filenode and oid, so decode and truncate see same fence Close the neighbouring holes the same way. A filenode landing inside a span with no XLOG_SMGR_CREATE marker fails closed instead of being tracked without payload and silently dropped at commit. Draining stashed records with no verdict installed fails closed instead of sending every one through the discard arm. An aborted tree's verdict is removed so the next transaction reusing that xid can't fold rows under it Stashed records drop block images on blocks that also carry data, without this a dirty COPY spills its whole WAL volume Run descriptor log compaction on its own task, fed each persisted floor over a watch channel. Inline on the pump it rewrote the checkpoint while no keepalive was answered, long enough to hit the source's wal_sender_timeout. Tail-bytes gauge reads a mirror so it no longer dips to zero for the length of a compaction Split plan-failure counter by cause so an unmodelled WAL op is distinguishable from spill error

harshil-goel · 2026-07-29T16:08:00Z

+    /// Load `desc_log.ckpt` + `desc_log.tail` under `dir`, repairing a torn
+    /// tail. Creates an empty tail (header only) when absent. Never creates
+    /// the ckpt — [`Self::seed`] and GC own it.
+    pub async fn open(dir: &Path, identity: DescLogIdentity) -> Result<Self> {


on the restart, the log is already durable but clickhouse isn't updated yet. shadow pg is also already ahead. i have two concerns here:

Clickhouse will not receive the relevant alter.

What happens if before writng descriptor log we crash and shadow pg is alreayd updated?

on restart we go by manifest, not log. manifest only advances when CH durable. In 2nd scenario we shouldn't hit that since shadow shouldn't be written due to boundary_hold.rs, but catalog_capture.rs will raise "descriptor log lost coverage" if 2nd scenario did somehow happen

can add some tests:

Crash after shadow reaches boundary, before descriptor append

Crash after descriptor fsync, before commit enters worker queue

Crash after commit queues, before ClickHouse DDL begins

Crash after first ClickHouse ALTER statement, before remaining statements

Crash after complete ClickHouse DDL, before emitter acknowledgment

Crash after in-memory acknowledgment, before manifest fsync

Crash after manifest fsync, before descriptor GC

harshil-goel · 2026-07-29T16:17:07Z

+        self.stats.clone()
+    }
+
+    pub async fn capture_boundary(


For higher scale, we will need to think about making this hold per table or something? So that alter in one table doesn't affect other tables.

yes, https://github.com/ClickHouse/walshadow/blob/be-modular/plans/future/custom_rmgr.md seeks to improve overall efficiency, but we'd have to loosen our LSN ordering to be per table. I've been relying on these being slow path where we keep simpler design

harshil-goel · 2026-07-29T16:18:29Z

+    /// tail. Creates an empty tail (header only) when absent. Never creates
+    /// the ckpt — [`Self::seed`] and GC own it.
+    pub async fn open(dir: &Path, identity: DescLogIdentity) -> Result<Self> {
+        let mut index = Index::default();


Also what happens if this file gets corrupted during a restart? We will have to force resync?

we have some recovery mechanisms: detecting corrupt tail replays beforehand, duplicate detection between checkpoint & tail (checkpoint written before tail, crash in between could cause this), but malformed checkpoint yeah resync

harshil-goel · 2026-07-29T16:25:09Z

+                    "decoder worker terminated during catalog boundary hold at {commit_lsn:#X}"
+                ));
+            }
+            if start.elapsed() >= self.config.hold_timeout {


wouldn't walsender timeout first?

try to avoid it by having hold_timeout below wal_sender_timeout (30s vs default 60s), but we could do better about handling primary keep alives in meantime. but if this fails then we just reconnect & pull wal from s3 maybe

harshil-goel · 2026-07-29T16:29:19Z

+        if !is_commit {
+            return Ok(None);
+        }
+        // Local relcache invals: second oid source + capture-all trigger.


A missed alter can cause issues. (Due to restart or a new change by pg), should we also do an audit to make sure that the table the consistent?

That capability would be useful, maybe create issue or plans/future

serprex requested a review from harshil-goel July 22, 2026 23:00

serprex added 2 commits July 22, 2026 23:05

Make catalog decoding consistent across DDL & restarts

3bbc01e

Replace live catalog cache & invalidation tracking with durable relation descriptor log Log checkpoints at commit boundaries. This allows looking up relation id at a given LSN

serprex force-pushed the xact-r1 branch 2 times, most recently from 41526a5 to 7e0938b Compare July 23, 2026 19:43

serprex added 2 commits July 24, 2026 18:50

serprex force-pushed the xact-r1 branch from 7e0938b to b01e976 Compare July 24, 2026 21:50

harshil-goel approved these changes Jul 29, 2026

View reviewed changes

serprex merged commit d21bd3f into main Jul 31, 2026
5 checks passed

serprex deleted the xact-r1 branch July 31, 2026 01:42

serprex mentioned this pull request Jul 31, 2026

When shadowing one database (--dbname postgres) of a cluster that has other databases with any write activity, the daemon fatally exits #76

Closed

Uh oh!

Conversation

serprex commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serprex commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

serprex commented Jul 22, 2026 •

edited

Loading