Skip to content

feat(snowflake): use maximum VARCHAR length (128 MB) for string columns#4401

Open
JGustavo0 wants to merge 1 commit into
PeerDB-io:mainfrom
JGustavo0:snowflake-max-varchar
Open

feat(snowflake): use maximum VARCHAR length (128 MB) for string columns#4401
JGustavo0 wants to merge 1 commit into
PeerDB-io:mainfrom
JGustavo0:snowflake-max-varchar

Conversation

@JGustavo0

Copy link
Copy Markdown

Snowflake's bare STRING/VARCHAR declaration defaults to 16 MB even on accounts where the larger-size behavior change bundles are enabled. This silently caps PeerDB-managed string columns (Postgres text/varchar/bpchar) at 16 MB instead of Snowflake's documented maximum of 128 MB (VARCHAR(134217728)).

This PR adds a shared SnowflakeMaxVarchar constant and uses it for the generic string kind (QValueKindString) and the unknown-kind fallback. The change propagates automatically through qvalue.ToDWHColumnType to:

  • generateCreateTableSQLForNormalizedTableCREATE TABLE
  • ReplayTableSchemaDeltasALTER TABLE ADD COLUMN
  • generateMergeStmtCAST(... AS <type>) in MERGE

QValueKindEnum (tiny labels) and QValueKindUUID (36 chars) intentionally stay as bare STRING — they can't exceed 16 MB anyway, so widening them would be noise.

Security & performance

  • Storage: None. Snowflake bills based on actual data stored; declared length has no storage impact.

  • Performance: None. Per the Snowflake docs on string data types:

    When choosing the maximum length for a VARCHAR column, consider that storing a smaller value (e.g. 'sample text') in a column with a larger declared length (e.g. VARCHAR(16777216)) has no impact on performance or storage. There is no performance difference between using the full-length VARCHAR(16777216) declaration and a smaller length.

    The same guarantee extends to VARCHAR(134217728) — declared length is purely a logical ceiling and does not affect compression, micro-partition layout, or query performance.

  • Security: Purely a column-width ceiling. Snowflake still enforces the 128 MB hard cap.

Compatibility

The 128 MB ceiling shipped via Snowflake behavior change bundles BCR-1779 (2024_08) and BCR-1942 (2025_03). On accounts where those bundles are disabled, VARCHAR(134217728) DDL will be rejected. Both bundles are enabled by default on current accounts, so this is safe in practice.

Test plan

  • flow/shared/types/... unit tests pass (used my account).
  • Snowflake schema-delta e2e suite (TestSnowflakeSchemaDeltaTestSuite) runs against a real Snowflake account and passes. Logs confirm string columns are now emitted as VARCHAR(134217728) via the ALTER TABLE ADD COLUMN path.
  • Non-string types (BOOLEAN, VARIANT, DATE, TIMESTAMP_*, NUMERIC, FLOAT, BINARY, TIME) emit unchanged.
  • BCR-1779/BCR-1942 confirmed enabled on the test account (validated with a manual CREATE TABLE ... VARCHAR(134217728) smoke test).

@JGustavo0
JGustavo0 requested a review from a team as a code owner June 9, 2026 11:39
@JGustavo0
JGustavo0 had a problem deploying to external-contributor June 9, 2026 11:39 — with GitHub Actions Failure
@JGustavo0
JGustavo0 had a problem deploying to external-contributor June 9, 2026 11:39 — with GitHub Actions Failure
@JGustavo0
JGustavo0 had a problem deploying to external-contributor June 9, 2026 11:39 — with GitHub Actions Failure
@JGustavo0
JGustavo0 had a problem deploying to external-contributor June 9, 2026 11:39 — with GitHub Actions Failure
@JGustavo0
JGustavo0 had a problem deploying to external-contributor June 9, 2026 11:39 — with GitHub Actions Failure
@JGustavo0

Copy link
Copy Markdown
Author

Anyone could help unlock this?

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Deprecated destination connector

This issue or pull request relates to a deprecated destination connector (Snowflake, BigQuery, ElasticSearch, Kafka, Redpanda, Confluent, Azure Event Hubs, Google Pub/Sub, or S3).

These destinations are no longer actively maintained, but remain functional. We are unlikely to prioritize new work here.

Note: BigQuery is deprecated only as a destination — it remains a supported source.

If you depend on one of these connectors, we recommend:

  • Pin to a known-good PeerDB version so behavior stays stable.
  • Fork the repository if you need to carry your own changes.

See the deprecated connectors documentation for details and migration guidance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants