fix: recreate materialized views on query schema drift - #1621
Merged
Conversation
sd-db
added a commit
that referenced
this pull request
Jul 30, 2026
Coverage report
This PR does not seem to contain any modification to coverable code. |
jprakash-db
approved these changes
Aug 6, 2026
When CREATE embeds a user-specified column list, Databricks rejects REFRESH if the query-inferred schema changes (e.g. upstream select * gains a column). Detect drift and replace instead of refresh.
Replace the bespoke temp-view + `LIMIT 10` schema inference with the base adapter's `get_columns_in_query`, which emits `select * from (<sql>) as __dbt_sbq where false limit 0` -- no DDL and no data scan. Subquery wrapping also makes the check safe for model SQL ending in its own `LIMIT`, which the string-concatenated variant was not. The comparison collapses into the macro, so the `column_schemas_differ` adapter method is no longer needed. `get_columns_in_query` is part of the shared adapter contract (already used by this adapter in relations/view/create.sql and snapshot_helpers.sql), so no Databricks-specific method needs porting to Fusion. Also log the drifted column names, since a schema-driven replace was otherwise silent, and record why get_configuration_changes cannot catch this case: the model's query text is unchanged, only the upstream schema moved. Unit coverage moves from tests/unit/test_column_schemas_differ.py to a macro test exercising the same cases through the macro, plus empty-list cases. Co-authored-by: Isaac
The list is only populated via `returned.append` handed to the Jinja context as a callback, so mypy has no assignment to infer the element type from.
Schema drift short-circuited to a replace ahead of the on_configuration_change dispatch, so `fail` and `continue` were silently overridden: a drifted materialized view was rebuilt regardless of the setting. Drift is a configuration change the config components cannot observe -- the model's query text is unchanged, only the upstream schema moved -- so compute it alongside get_configuration_changes and let the existing dispatch handle it. `apply` replaces (REFRESH cannot reconcile a drifted schema), while `fail` and `continue` need no new branches; they fall out of the dispatch already there. Functional coverage grows from one test to four: the issue's own sequence of adding the column to the properties YAML after the upstream model gains it, plus the fail and continue paths. Both new dispatch tests were confirmed to fail against the previous ordering.
Drift previously called get_replace_sql directly, bypassing the alter macro's special handling for partition_by, which needs DROP + CREATE because CREATE OR REPLACE cannot change partitioning. Instead mark the changeset requires_full_refresh and let get_alter_materialized_view_as_sql pick the right rebuild. Drift with no component changes still replaces directly, since there is no changeset to carry the flag. model_copy keeps this a new object rather than mutating a shared one, and works under dbt's Jinja sandbox where __setattr__ is blocked.
sd-db
force-pushed
the
sd-db/fix/issue-1359-mv-schema-drift
branch
from
August 6, 2026 21:36
8f1a2bd to
9f26f22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
select *gains a column) and recreate via replace instead of issuingREFRESH, which Databricks rejects with an incompatible user-specified schema error.Resolves #1359
Test plan
tests/unit/test_column_schemas_differ.pytests/functional/adapter/materialized_view_tests/test_mv_schema_evolution.pytest_mv_alter_no_rebuildstill passes