branch-4.2: [fix](cast) Declare Variant casts to non-Variant targets nullable by eldenmoon · Pull Request #70359 · apache/doris

eldenmoon · 2026-11-06T16:45:32Z

What problem does this PR solve?

Cherry-pick of #70365 (BE scan fix) to branch-4.2, together with the FE cast-nullability fix.

Related PR: #70365

This PR fixes two related problems for Variant casts on branch-4.2 (the default legacy Variant
representation):

FE type mismatch (declared=String, actual=Nullable(String)): Cast.castNullable only
declared a Variant -> non-Variant cast nullable when the Variant type used the V2 execution
representation (isExecutionV2()). For the legacy representation it returned false, so the
cast was serialized as non-nullable while the legacy kernel legitimately returns NULL for
Variant nulls or non-convertible values. Direct casts and CTAS then failed with
Expr (...) return column type mismatch.
Wrong results for null tests over Variant casts: after fixing (1), CAST(v AS INT) IS NULL
survives to the scan, where _normalize_is_null_predicate rewrites it into v IS NULL and
drops rows whose Variant value is not convertible to INT ({"a":1}, [1,2], ...). This is the
same bug fixed on master in [fix](be) Keep null tests over Variant casts in the scan expression #70365 and included here as a backport.

Fix

FE: reuse the existing Cast.castNullable rule to declare Variant -> non-Variant casts nullable
for both representations. Identity casts keep the input nullability.
BE: keep the null test in the expression tree when the predicate's argument is a cast over a
Variant source (backport of [fix](be) Keep null tests over Variant casts in the scan expression #70365).

Tests

CastTest#testCastFromVariant updated (fails before, passes after).
New regression variant_p0/test_variant_cast_isnull_pushdown with a golden generated by the
regression runner: CAST(v AS INT) values plus IS NULL, IS NOT NULL and <=> NULL
filters over a VARIANT NOT NULL column.
Red/green verified end to end on a 4.2 FE+BE build:
- before the BE fix: WHERE CAST(v AS INT) IS NULL returns no rows while the projection
  returns 0,1,1,1;
- after: the filter and <=> NULL return 2,3,4, IS NOT NULL returns 1;
- CTAS create table ... as select ... parse_to_variant(cast(v as string)) ... succeeds after
  the FE fix (previously failed with the type mismatch).

Release note

Fix return column type mismatch when casting a NOT NULL Variant column, and fix wrong (empty)
results of CAST(variant_col AS T) IS NULL / <=> NULL filters.

Check List (For Author)

Test: Unit Test + Regression test + Manual end-to-end (4.2 FE+BE)
Behavior changed: Yes (casts are declared nullable and null filters return the correct rows)
Does this need documentation: No

hello-stephen · 2026-11-06T16:45:38Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2026-11-06T17:20:40Z

run buildall

eldenmoon · 2026-11-06T17:20:44Z

/review

github-actions

Reviewed apache/doris PR #70359 at the exact pinned head 566c2c0 against base 6612220. No new blocking findings are warranted. The Cast.castNullable change correctly declares every direct Variant-to-non-Variant cast nullable for both legacy and Variant V2 execution: legacy cast_from_variant_impl can return ColumnNullable for Variant nulls, incompatible shapes, and conversion failures, and the V2 scalar, array, string, and JSON paths have the same nullable-result contract. VExpr accepts a physically non-nullable fast result under a declared Nullable(T), so the relaxed declaration does not reject successful batches. Variant-to-Variant identity and mixed legacy/V2 rejection remain unchanged. FE translation, nested struct-field nullability propagation, and the updated CastTest coverage were inspected and remain conservative. Critical review checkpoints are satisfied: the two-file change is focused; no new concurrency, lifecycle, configuration, protocol, persistence, or data-write mechanism is introduced; the live PR has no existing inline review context; and the review-focus file contains no extra user concerns. All required delegated scopes completed with no new valuable findings, and risks R1-R3 were independently dismissed with evidence. Validation was static inspection only; no builds or tests were executed under the review contract. The review converged after round 1 with no unresolved coverage limits.

Review converged after 1 round(s).

hello-stephen · 2026-11-06T20:31:50Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

eldenmoon · 2026-11-07T02:07:16Z

/run buildall

…acks ### What problem does this PR solve? Related PR: apache#70359 Problem Summary: cover legacy Variant-to-non-Variant CAST nullable inference, optimizer preservation of IS NULL / <=> NULL and COALESCE / NVL, and scan filtering on CAST results. Include nullable inputs, incompatible values, identity casts, aliases, and all-Variant-null roots. Limit the representation- specific SQL golden to legacy Variant execution. The baseline FE tests fail on nullable inference and simplification; the baseline SQL fails with a nullable mismatch, and the FE-only fix still loses rows in scan filters. ### Release note None ### Check List (For Author) - Test: 59 FE tests passed; test_variant_cast_nullable and test_variant_cast passed with ASAN BE (2 suites); golden generated by run-regression-test.sh - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-11-07T05:12:08Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Related PR: apache#70359 Problem Summary: heavy schema change builds CastExpr directly without the nullable metadata normally supplied by Nereids. Changing a NOT NULL DATETIMEV2 column's scale therefore sends a non-nullable CAST to BE even though the existing converter returns a nullable column, and the alter task is cancelled with a declared/actual result type mismatch. At the schema-change construction site, reuse Cast.castNullable with the already resolved source/target types and source slot nullability, and set the existing nullableFromNereids metadata before task serialization. Keep legacy CastExpr fallback, runtime filters, BE conversion and storage unchanged. ### Release note Fix heavy schema change failures when modifying DATETIMEV2 scale. ### Check List (For Author) - Test: 56 FE unit tests passed; FE build/Checkstyle and unchanged BE ASAN build passed; expanded SQL golden generation passed with the scoped fix - Behavior changed: Yes. Schema change tasks declare the existing converter's nullable result correctly; conversion semantics are unchanged. - Does this need documentation: No

### What problem does this PR solve? Related PR: apache#70359 Problem Summary: check the CAST expression actually serialized by a heavy schema change task when DATETIMEV2 scale increases or decreases, including a nullable source column. Protect the legacy Variant-to-date fallback from unrelated changes. SQL regression checks existing rows, SQL NULL, the maximum datetime, and new microsecond writes after a scale change. ### Release note None ### Check List (For Author) - Test: 56 FE unit tests passed; FE build/Checkstyle and BE ASAN build passed; test_schema_change_cast_nullable and test_alter_table_column_nullable passed (2 suites); golden generated by run-regression-test.sh - Behavior changed: No - Does this need documentation: No

eldenmoon · 2026-11-07T06:53:03Z

Local validation on a 4.2 FE+BE build:

BE red/green: before this commit, WHERE CAST(v AS INT) IS NULL and <=> NULL return no rows while the projection returns 0,1,1; after it, both return the rows whose cast result is NULL and IS NOT NULL returns the other row.
Regression locally: test_variant_cast_isnull_pushdown (new), test_variant_cast (cast.groovy) and test_variant_cast_strict_mode all pass.
The 4.1-only test_variant_cast_nullable from branch-4.1: [fix](cast) Declare Variant casts to non-Variant targets nullable #70363 is intentionally not used here: CAST('{"a":1}' AS INT) is 0 on 4.1 and NULL on 4.2, so that golden is branch-specific. The new case only uses string/array values, whose cast is NULL on both branches.

eldenmoon · 2026-11-07T07:00:41Z

run buildall

hello-stephen · 2026-11-07T10:05:09Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

eldenmoon · 2026-11-07T12:23:40Z

run buildall

hello-stephen · 2026-11-07T14:25:13Z

BE UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	64.99% (30504/46937)
Line Coverage	49.97% (323939/648323)
Region Coverage	45.74% (261124/570949)
Branch Coverage	47.44% (123682/260699)

hello-stephen · 2026-11-07T15:22:31Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

eldenmoon · 2026-11-07T19:35:06Z

run cloud_p0

eldenmoon · 2026-11-07T19:35:10Z

run external

eldenmoon · 2026-11-07T21:15:40Z

run cloud_p0

eldenmoon · 2026-11-07T21:15:42Z

run external

eldenmoon · 2026-11-07T21:15:44Z

run feut

eldenmoon · 2026-11-07T21:15:46Z

run nonConcurrent

eldenmoon · 2026-11-08T02:15:24Z

run cloud_p0

eldenmoon · 2026-11-08T02:15:27Z

run external

eldenmoon · 2026-11-12T02:05:50Z

run feut

eldenmoon · 2026-11-12T02:05:56Z

run nonConcurrent

### What problem does this PR solve? Related PR: apache#70359 Problem Summary: Legacy Variant casts may return SQL NULL for NOT NULL inputs, but their non-nullable FE declaration allowed null-safe predicates to fold to FALSE and removed COALESCE/NVL fallbacks. Apply the shared nullable rule to both Variant execution representations, as in apache#70359. ### Release note Fix NULL predicate and conditional-function results, and nullable column type mismatches, for casts from NOT NULL Variant values. ### Check List (For Author) - Test: Unit Test - 59 focused FE unit tests passed; pre-fix CAST and optimizer tests failed as expected. SQL regression validation is recorded in the test commit and PR. - Behavior changed: Yes. Correct nullable declarations prevent invalid rewrites or result type mismatches; BE conversion semantics are unchanged. - Does this need documentation: No

…lization ### What problem does this PR solve? Related PR: apache#70359 Problem Summary: after fixing Variant CAST nullable metadata, a filter such as CAST(v AS INT) IS NULL can still lose rows. Scan normalization strips the Variant cast and substitutes a NullPredicate on the source column, although the cast can introduce SQL NULL for a non-NULL stored value. Keep null tests on casts as expressions using the existing pushdown rejection path. IS NOT NULL uses the same path. CAST conversion semantics and storage are unchanged. ### Release note Fix missing or extra rows from NULL filters over Variant casts. ### Check List (For Author) - Test: ASAN build.sh --be; focused SQL regression generation passed - Behavior changed: Yes. Evaluate the cast's NULL result before filtering. - Does this need documentation: No

…acks ### What problem does this PR solve? Related PR: apache#70359 Problem Summary: cover legacy Variant-to-non-Variant CAST nullable inference, optimizer preservation of IS NULL / <=> NULL and COALESCE / NVL, and scan filtering on CAST results. Include nullable inputs, incompatible values, identity casts, aliases, and all-Variant-null roots. Limit the representation- specific SQL golden to legacy Variant execution. The baseline FE tests fail on nullable inference and simplification; the baseline SQL fails with a nullable mismatch, and the FE-only fix still loses rows in scan filters. ### Release note None ### Check List (For Author) - Test: 59 FE tests passed; test_variant_cast_nullable and test_variant_cast passed with ASAN BE (2 suites); golden generated by run-regression-test.sh - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Related PR: apache#70359 Problem Summary: heavy schema change builds CastExpr directly without the nullable metadata normally supplied by Nereids. Changing a NOT NULL DATETIMEV2 column's scale therefore sends a non-nullable CAST to BE even though the existing converter returns a nullable column, and the alter task is cancelled with a declared/actual result type mismatch. At the schema-change construction site, reuse Cast.castNullable with the already resolved source/target types and source slot nullability, and set the existing nullableFromNereids metadata before task serialization. Keep legacy CastExpr fallback, runtime filters, BE conversion and storage unchanged. ### Release note Fix heavy schema change failures when modifying DATETIMEV2 scale. ### Check List (For Author) - Test: 56 FE unit tests passed; FE build/Checkstyle and unchanged BE ASAN build passed; expanded SQL golden generation passed with the scoped fix - Behavior changed: Yes. Schema change tasks declare the existing converter's nullable result correctly; conversion semantics are unchanged. - Does this need documentation: No

### What problem does this PR solve? Related PR: apache#70359 Problem Summary: check the CAST expression actually serialized by a heavy schema change task when DATETIMEV2 scale increases or decreases, including a nullable source column. Protect the legacy Variant-to-date fallback from unrelated changes. SQL regression checks existing rows, SQL NULL, the maximum datetime, and new microsecond writes after a scale change. ### Release note None ### Check List (For Author) - Test: 56 FE unit tests passed; FE build/Checkstyle and BE ASAN build passed; test_schema_change_cast_nullable and test_alter_table_column_nullable passed (2 suites); golden generated by run-regression-test.sh - Behavior changed: No - Does this need documentation: No

eldenmoon · 2026-11-12T02:12:12Z

run buildall

eldenmoon · 2026-11-12T07:20:31Z

run external

eldenmoon · 2026-11-12T14:46:21Z

run p0

eldenmoon · 2026-11-12T14:46:27Z

run external

eldenmoon · 2026-11-12T21:27:13Z

run external

eldenmoon · 2026-11-12T21:27:21Z

run p0

eldenmoon · 2026-11-13T02:20:45Z

run compile

eldenmoon · 2026-11-13T02:20:50Z

run feut

eldenmoon · 2026-11-13T07:25:46Z

run external

eldenmoon · 2026-11-13T07:26:43Z

run nonConcurrent

eldenmoon · 2026-11-13T12:20:05Z

run external

eldenmoon · 2026-11-13T12:20:41Z

run nonConcurrent

### What problem does this PR solve? Problem Summary: with the default `enable_variant_v2 = false`, a CAST from a NOT NULL Variant column to a non-Variant type fails when the legacy Variant kernel emits NULL for a non-NULL Variant input, for example: SELECT CAST(v AS STRING) FROM t; -- v VARIANT NOT NULL, some rows hold Variant nulls errCode = 2, detailMessage = Expr (CAST v(Variant(...)) TO String) return column type mismatch: declared=String, actual=Nullable(String) `Cast.castNullable` only declared a Variant -> non-Variant cast nullable when the Variant type used the V2 execution representation (`isExecutionV2()`). For the legacy representation it returned false, so the CastExpr was serialized with `is_nullable = false` and the BE slot type became a non-nullable String. The legacy kernel (`cast_from_variant_impl` in `be/src/exprs/function/cast/cast_to_variant.h`) serializes a scalar Variant through the root/WRAP path, and `wrap_in_nullable` merges the root null map; the trailing "drop the nullable wrapper" step only runs when the result has no NULLs. A NOT NULL Variant column that contains Variant nulls or values outside the root type therefore yields ColumnNullable with NULLs, and `VExpr::execute_column` rejects it against the non-nullable slot. Fix: declare Variant -> non-Variant casts nullable for both execution representations, which is what master already does (master no longer has the legacy representation). The values produced by the cast are unchanged; only the declared nullability of the output slot is relaxed, and `VExpr` already wraps a non-nullable column when the declaration is nullable. ### Release note Fix `return column type mismatch: declared=String, actual=Nullable(String)` when casting a NOT NULL Variant column to STRING with the legacy Variant representation. ### Check List (For Author) - Test: Unit Test - `CastTest#testCastFromVariant` (updated) - fails before, passes after - Behavior changed: No (cast results are unchanged; only the declared nullability of the output slot changes) - Does this need documentation: No

Backport of apache#70365 to branch-4.2. A null test over a Variant cast silently drops rows whose Variant value is not convertible to the target type: `_normalize_is_null_predicate` rewrites `CAST(v AS INT) IS NULL` into a `NullPredicate` on the source column (`v IS NULL`), but the cast returns NULL for non-NULL object, string or array values. Keep the null test in the expression tree when the predicate's argument is a cast over a Variant source. This is required together with the FE nullability fix in this branch: before it, the FE folded the null test to FALSE; after it, the test survives and would be pushed down with wrong semantics. The regression `variant_p0/test_variant_cast_isnull_pushdown` fails before this commit (the `IS NULL` / `<=> NULL` filters return no rows) and passes after it.

yiguolei · 2026-11-13T14:49:13Z

run buildall

eldenmoon · 2026-11-14T07:15:04Z

run external

eldenmoon · 2026-11-14T12:19:31Z

run external

eldenmoon · 2026-11-14T12:20:03Z

run external

eldenmoon · 2026-11-14T16:14:28Z

run external

eldenmoon · 2026-11-15T16:13:27Z

run external

eldenmoon · 2026-11-15T16:13:50Z

run nonConcurrent

eldenmoon requested a review from yiguolei as a code owner November 6, 2026 16:45

github-actions Bot reviewed Nov 6, 2026

View reviewed changes

This was referenced Nov 7, 2026

[fix](fe) Infer CAST nullability when building schema change tasks #70362

Open

branch-4.1: [fix](cast) Declare Variant casts to non-Variant targets nullable #70363

Open

eldenmoon force-pushed the fix-doris-30101-variant-cast-4.2 branch 2 times, most recently from 88be67f to be7c802 Compare November 7, 2026 06:52

eldenmoon changed the title ~~[fix](cast) Declare Variant casts to non-Variant targets nullable~~ branch-4.2: [fix](cast) Declare Variant casts to non-Variant targets nullable Nov 7, 2026

eldenmoon force-pushed the fix-doris-30101-variant-cast-4.2 branch from be7c802 to b20d007 Compare November 12, 2026 02:12

eldenmoon added 2 commits November 13, 2026 22:42

yiguolei force-pushed the fix-doris-30101-variant-cast-4.2 branch from b20d007 to 9b307fb Compare November 13, 2026 14:42

Conversation

eldenmoon commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Fix

Tests

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Nov 6, 2026

Uh oh!

eldenmoon commented Nov 6, 2026

Uh oh!

eldenmoon commented Nov 6, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Nov 6, 2026

FE UT Coverage Report

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

hello-stephen commented Nov 7, 2026

FE UT Coverage Report

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

hello-stephen commented Nov 7, 2026

FE UT Coverage Report

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

hello-stephen commented Nov 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Nov 7, 2026

FE UT Coverage Report

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 8, 2026

Uh oh!

eldenmoon commented Nov 8, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

eldenmoon commented Nov 6, 2026 •

edited

Loading