Skip to content

qrep: derive pull columns from source instead of catalog schema#4583

Merged
dtunikov merged 5 commits into
mainfrom
qrep-load-schema-without-watermark-setup
Jul 16, 2026
Merged

qrep: derive pull columns from source instead of catalog schema#4583
dtunikov merged 5 commits into
mainfrom
qrep-load-schema-without-watermark-setup

Conversation

@dtunikov

@dtunikov dtunikov commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Problem

QRep pulls with excluded columns or child-table range partitions fail with:

[qrep] failed to pull records: failed to load table schema: no rows in result set

The pull path (qrep_source.go) built the SELECT column list by loading the table schema from table_schema_mapping, keyed by (ParentMirrorName, DestinationTableIdentifier). When no row exists for that key, QueryRow().Scan() returns pgx.ErrNoRows ("no rows in result set") and the pull hard-fails.

This has been observed on a CDC mirror. The in-repo setup paths normally populate that row under a matching key, so the exact reason the row was absent for this mirror isn't pinned down (candidates: version skew / older mirror, or attaching to a pre-existing mirror). Regardless, the pull shouldn't depend on the catalog row being present when it can derive the columns directly from the source.

Fix

Derive the column list directly from the source via GetSelectedColumns (which reads pg_attribute) — the same primitive getTableSchemaForTable already uses to build the identical list, so column selection and ordering are unchanged.

This removes the table_schema_mapping dependency from the pull path entirely. catalogPool is no longer needed by corePullQRepRecords.

🤖 Generated with Claude Code

The QRep pull path loads the table schema from table_schema_mapping when
Exclude columns or child table ranges are set, but the row was only written
when SetupWatermarkTableOnDestination was true. Standalone QRep mirrors to
non-PG destinations with excluded columns hit "no rows in result set".

Move setupTableSchema out from behind the SetupWatermarkTableOnDestination
guard so the schema is always available to the pull path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dtunikov
dtunikov requested a review from a team as a code owner July 14, 2026 14:06
Only standalone mirrors (ParentMirrorName == FlowJobName) lack a CDC
setup_flow to populate table_schema_mapping; CDC initial-load children
already have it under ParentMirrorName. Guard the unconditional
setupTableSchema so CDC children don't run a redundant GetTableSchema.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 14, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

The QRep pull path built the SELECT column list (for Exclude / child table
ranges) by loading the table schema from table_schema_mapping keyed by
(ParentMirrorName, DestinationTableIdentifier). That row can be absent for
CDC snapshots after resync/table-addition (the destination is renamed with a
_resync suffix), producing "failed to load table schema: no rows in result
set".

Use GetSelectedColumns to read the column list directly from the source's
pg_attribute, matching how getTableSchemaForTable already builds it. This
removes the catalog dependency and fixes standalone, CDC, resync and
table-addition paths alike. catalogPool is no longer needed by
corePullQRepRecords.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dtunikov dtunikov changed the title qrep: always populate table_schema_mapping for standalone mirrors qrep: derive pull columns from source instead of catalog schema Jul 14, 2026
@dtunikov
dtunikov requested a review from Copilot July 14, 2026 14:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Removes QRep pull-time dependence on table_schema_mapping by deriving the SELECT column list directly from the source table (via pg_attribute), preventing pulls from hard-failing when the catalog schema row is missing.

Changes:

  • Stop loading table schema from the catalog for building the SELECT column list during QRep pulls.
  • Derive selected columns from the source table using GetSelectedColumns when excludes or child-table range partitions are involved.
  • Drop catalogPool usage from corePullQRepRecords (callers keep the parameter but ignore it).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread flow/connectors/postgres/qrep_source.go
Comment thread flow/connectors/postgres/qrep_source.go
AND c.relname = $2
AND a.attnum > 0
AND NOT a.attisdropped ` + excludedColumnsSQL
AND NOT a.attisdropped` + excludedColumnsSQL + ` ORDER BY a.attnum`

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional, suggested by copilot to keep columns order consistent across calls

@ilidemi ilidemi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to first understand what led to the error before making a change just in case. What if there's a consistency issue

@dtunikov

Copy link
Copy Markdown
Collaborator Author

@ilidemi

Would prefer to first understand what led to the error before making a change just in case. What if there's a consistency issue.

I explained it in internal discussion, posting here as well:
this is what has happened most probably:

  • Resync cancels the active snapshot/QRep children.
  • DropFlowWorkflow deletes all table_schema_mapping rows, including during resync.
  • An already-running QRep pull gets far enough to perform this new schema lookup before its cancellation is observed.
  • It sees the deleted row and emits this exact error.

@dtunikov
dtunikov requested a review from ilidemi July 15, 2026 09:34

@ilidemi ilidemi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I missed that thread. Conceptually having a single source of truth is nicer, and waiting on full Temporal cancellation before starting resync would be a cleaner fix (if possible), but this is also bringing us closer to not needing an overarching snapshot tx and still supporting schema changes so ok enough.

@dtunikov
dtunikov merged commit b8357f1 into main Jul 16, 2026
22 checks passed
@dtunikov
dtunikov deleted the qrep-load-schema-without-watermark-setup branch July 16, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants