Skip to content

fix: dbt build --empty compatibility via sv_ref/sv_source macros - #22

Open
zaramzamzam wants to merge 1 commit into
Snowflake-Labs:mainfrom
zaramzamzam:dbt-empty-failure
Open

fix: dbt build --empty compatibility via sv_ref/sv_source macros#22
zaramzamzam wants to merge 1 commit into
Snowflake-Labs:mainfrom
zaramzamzam:dbt-empty-failure

Conversation

@zaramzamzam

Copy link
Copy Markdown

Problem

dbt build --empty (used in CI/CD for schema validation) fails for semantic view models. The --empty flag rewrites ref()/source() into subqueries like (select * from TABLE where false limit 0), which breaks three contexts that require bare table identifiers:

  1. CREATE SEMANTIC VIEW DDLTABLES() clause
  2. semantic_view() query function — table argument
  3. get_ddl() calls in tests — string argument

Root cause

The --empty rewriting is hardcoded in Python (BaseRelation.render_limited() in dbt-core). There is no dispatchable macro to override it at the package level.

Solution

Two new utility macros: sv_ref() and sv_source().

They call dbt's native ref()/source() internally (preserving DAG lineage and execution ordering) but render the relation as DATABASE.SCHEMA.IDENTIFIER using the Relation object's component properties — which are stable public API and unaffected by --empty.

Usage

-- Semantic view definition (TABLES clause):
TABLES(t1 AS {{ dbt_semantic_view.sv_ref('my_table') }})

-- Querying a semantic view:
select * from semantic_view({{ dbt_semantic_view.sv_ref('my_view') }} metrics total_rows)

-- For sources:
TABLES(t2 AS {{ dbt_semantic_view.sv_source('my_source', 'my_table') }})

Standard ref()/source() should still be used in normal SELECT contexts where subquery wrapping is harmless.

Alternatives considered

Regex cleanup in materialization — Strip the --empty subquery pattern from compiled SQL before executing DDL. Rejected: tied to dbt's internal subquery format (an implementation detail), and silent regex mismatch is harder to debug than a clear error.

graph.nodes lookup (proposed in #11) — Manually resolve relation names from the dbt graph without calling ref(). Rejected: breaks DAG dependency tracking (dbt won't know about the relationship), and has correctness issues with custom schemas, cross-project refs, and aliases.

Tag-based exclusion — Exclude semantic view models from --empty runs. Rejected: fragile CI/CD configuration, provides zero validation value, and silently breaks when new models are added without the tag.

Verification

Both pass with 19/19:

dbt build --target snowflake --empty   # PASS=19 ERROR=0
dbt build --target snowflake           # PASS=19 ERROR=0

Closes #11

dbt's --empty flag (used in CI/CD for schema validation) rewrites ref()
and source() calls into subqueries: (select * from TABLE where false limit 0).
This breaks Snowflake semantic views in two contexts:

1. CREATE SEMANTIC VIEW DDL — the TABLES clause requires bare table
   identifiers, not subqueries
2. semantic_view() query function — same requirement for table arguments
3. get_ddl() calls in tests — string arguments cannot contain subqueries

The --empty rewriting is hardcoded in Python (BaseRelation.render_limited()
in dbt-core) with no dispatchable macro to override it. A regex-based
cleanup was considered but rejected: it would be tied to dbt's internal
subquery format (an implementation detail that could change between
versions), and the failure mode — silent regex mismatch — is harder to
debug than a clear syntax error.

Instead, this adds two utility macros: sv_ref() and sv_source(). They
call dbt's native ref()/source() internally (preserving DAG lineage and
execution ordering) but render the relation as DATABASE.SCHEMA.IDENTIFIER
using the Relation object's component properties, which are stable public
API and unaffected by --empty. This approach was chosen over a graph.nodes
lookup (proposed in Snowflake-Labs#11) which bypasses ref() entirely, breaking DAG
dependency tracking, and has correctness issues with custom schemas,
cross-project refs, and aliases.

Usage — anywhere a bare identifier is needed in semantic view contexts:

  TABLES(t1 AS {{ dbt_semantic_view.sv_ref('my_table') }})
  select * from semantic_view({{ dbt_semantic_view.sv_ref('my_view') }} ...)
  get_ddl('SEMANTIC_VIEW', '{{ dbt_semantic_view.sv_ref('my_view') }}')

Standard ref()/source() should still be used in normal SELECT contexts
where subquery wrapping is harmless.

Also disables the base_table existence test under --empty, since it checks
for seed data presence which --empty empties by design.

Closes Snowflake-Labs#11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--empty flag breaks semantic view compilation due to LIMIT 0 in ref()

1 participant