qrep: derive pull columns from source instead of catalog schema#4583
Conversation
The QRep pull path loads the table schema from table_schema_mapping when Exclude columns or child table ranges are set, but the row was only written when SetupWatermarkTableOnDestination was true. Standalone QRep mirrors to non-PG destinations with excluded columns hit "no rows in result set". Move setupTableSchema out from behind the SetupWatermarkTableOnDestination guard so the schema is always available to the pull path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only standalone mirrors (ParentMirrorName == FlowJobName) lack a CDC setup_flow to populate table_schema_mapping; CDC initial-load children already have it under ParentMirrorName. Guard the unconditional setupTableSchema so CDC children don't run a redundant GetTableSchema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
The QRep pull path built the SELECT column list (for Exclude / child table ranges) by loading the table schema from table_schema_mapping keyed by (ParentMirrorName, DestinationTableIdentifier). That row can be absent for CDC snapshots after resync/table-addition (the destination is renamed with a _resync suffix), producing "failed to load table schema: no rows in result set". Use GetSelectedColumns to read the column list directly from the source's pg_attribute, matching how getTableSchemaForTable already builds it. This removes the catalog dependency and fixes standalone, CDC, resync and table-addition paths alike. catalogPool is no longer needed by corePullQRepRecords. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Removes QRep pull-time dependence on table_schema_mapping by deriving the SELECT column list directly from the source table (via pg_attribute), preventing pulls from hard-failing when the catalog schema row is missing.
Changes:
- Stop loading table schema from the catalog for building the
SELECTcolumn list during QRep pulls. - Derive selected columns from the source table using
GetSelectedColumnswhen excludes or child-table range partitions are involved. - Drop
catalogPoolusage fromcorePullQRepRecords(callers keep the parameter but ignore it).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| AND c.relname = $2 | ||
| AND a.attnum > 0 | ||
| AND NOT a.attisdropped ` + excludedColumnsSQL | ||
| AND NOT a.attisdropped` + excludedColumnsSQL + ` ORDER BY a.attnum` |
There was a problem hiding this comment.
optional, suggested by copilot to keep columns order consistent across calls
ilidemi
left a comment
There was a problem hiding this comment.
Would prefer to first understand what led to the error before making a change just in case. What if there's a consistency issue
I explained it in internal discussion, posting here as well: |
ilidemi
left a comment
There was a problem hiding this comment.
Ah right, I missed that thread. Conceptually having a single source of truth is nicer, and waiting on full Temporal cancellation before starting resync would be a cleaner fix (if possible), but this is also bringing us closer to not needing an overarching snapshot tx and still supporting schema changes so ok enough.
Problem
QRep pulls with excluded columns or child-table range partitions fail with:
The pull path (qrep_source.go) built the
SELECTcolumn list by loading the table schema fromtable_schema_mapping, keyed by(ParentMirrorName, DestinationTableIdentifier). When no row exists for that key,QueryRow().Scan()returnspgx.ErrNoRows("no rows in result set") and the pull hard-fails.This has been observed on a CDC mirror. The in-repo setup paths normally populate that row under a matching key, so the exact reason the row was absent for this mirror isn't pinned down (candidates: version skew / older mirror, or attaching to a pre-existing mirror). Regardless, the pull shouldn't depend on the catalog row being present when it can derive the columns directly from the source.
Fix
Derive the column list directly from the source via
GetSelectedColumns(which readspg_attribute) — the same primitivegetTableSchemaForTablealready uses to build the identical list, so column selection and ordering are unchanged.This removes the
table_schema_mappingdependency from the pull path entirely.catalogPoolis no longer needed bycorePullQRepRecords.🤖 Generated with Claude Code