fix: dbt build --empty compatibility via sv_ref/sv_source macros - #22
Open
zaramzamzam wants to merge 1 commit into
Open
fix: dbt build --empty compatibility via sv_ref/sv_source macros#22zaramzamzam wants to merge 1 commit into
zaramzamzam wants to merge 1 commit into
Conversation
dbt's --empty flag (used in CI/CD for schema validation) rewrites ref() and source() calls into subqueries: (select * from TABLE where false limit 0). This breaks Snowflake semantic views in two contexts: 1. CREATE SEMANTIC VIEW DDL — the TABLES clause requires bare table identifiers, not subqueries 2. semantic_view() query function — same requirement for table arguments 3. get_ddl() calls in tests — string arguments cannot contain subqueries The --empty rewriting is hardcoded in Python (BaseRelation.render_limited() in dbt-core) with no dispatchable macro to override it. A regex-based cleanup was considered but rejected: it would be tied to dbt's internal subquery format (an implementation detail that could change between versions), and the failure mode — silent regex mismatch — is harder to debug than a clear syntax error. Instead, this adds two utility macros: sv_ref() and sv_source(). They call dbt's native ref()/source() internally (preserving DAG lineage and execution ordering) but render the relation as DATABASE.SCHEMA.IDENTIFIER using the Relation object's component properties, which are stable public API and unaffected by --empty. This approach was chosen over a graph.nodes lookup (proposed in Snowflake-Labs#11) which bypasses ref() entirely, breaking DAG dependency tracking, and has correctness issues with custom schemas, cross-project refs, and aliases. Usage — anywhere a bare identifier is needed in semantic view contexts: TABLES(t1 AS {{ dbt_semantic_view.sv_ref('my_table') }}) select * from semantic_view({{ dbt_semantic_view.sv_ref('my_view') }} ...) get_ddl('SEMANTIC_VIEW', '{{ dbt_semantic_view.sv_ref('my_view') }}') Standard ref()/source() should still be used in normal SELECT contexts where subquery wrapping is harmless. Also disables the base_table existence test under --empty, since it checks for seed data presence which --empty empties by design. Closes Snowflake-Labs#11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dbt build --empty(used in CI/CD for schema validation) fails for semantic view models. The--emptyflag rewritesref()/source()into subqueries like(select * from TABLE where false limit 0), which breaks three contexts that require bare table identifiers:CREATE SEMANTIC VIEWDDL —TABLES()clausesemantic_view()query function — table argumentget_ddl()calls in tests — string argumentRoot cause
The
--emptyrewriting is hardcoded in Python (BaseRelation.render_limited()in dbt-core). There is no dispatchable macro to override it at the package level.Solution
Two new utility macros:
sv_ref()andsv_source().They call dbt's native
ref()/source()internally (preserving DAG lineage and execution ordering) but render the relation asDATABASE.SCHEMA.IDENTIFIERusing the Relation object's component properties — which are stable public API and unaffected by--empty.Usage
Standard
ref()/source()should still be used in normal SELECT contexts where subquery wrapping is harmless.Alternatives considered
Regex cleanup in materialization — Strip the
--emptysubquery pattern from compiled SQL before executing DDL. Rejected: tied to dbt's internal subquery format (an implementation detail), and silent regex mismatch is harder to debug than a clear error.graph.nodes lookup (proposed in #11) — Manually resolve relation names from the dbt graph without calling
ref(). Rejected: breaks DAG dependency tracking (dbt won't know about the relationship), and has correctness issues with custom schemas, cross-project refs, and aliases.Tag-based exclusion — Exclude semantic view models from
--emptyruns. Rejected: fragile CI/CD configuration, provides zero validation value, and silently breaks when new models are added without the tag.Verification
Both pass with 19/19:
Closes #11