Skip to content

[fix](fe) Infer CAST nullability when building schema change tasks - #70362

Open
eldenmoon wants to merge 2 commits into
apache:branch-4.2from
eldenmoon:branch-variant-cast-nullable-42-20261107
Open

eldenmoon wants to merge 2 commits into
apache:branch-4.2from
eldenmoon:branch-variant-cast-nullable-42-20261107

Conversation

@eldenmoon

@eldenmoon eldenmoon commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related PR: #70359

Problem Summary: a heavy schema change that changes a NOT NULL DATETIMEV2 column's scale can be cancelled by BE with a result type mismatch, for example:

CAST required_dt(DateTimeV2(3)) TO DateTimeV2(6) return column type mismatch:
declared=DateTimeV2(6), actual=Nullable(DATETIMEV2)

SchemaChangeJobV2 directly creates a legacy CastExpr without the nullable metadata normally supplied by Nereids translation. The legacy fallback does not recognize DATETIMEV2 scale conversion, while the existing BE converter returns a nullable physical column.

Fix

At the schema-change CAST construction site, compute nullability using the existing Cast.castNullable function, the already resolved source/target types, and the source slot nullability. Store it through the existing setNullableFromNereids metadata, which takes precedence when serializing the task expression.

This changes one production file. It adds no per-type special case and leaves CastExpr.isNullable, normal query translation, and runtime-filter construction unchanged. The skipped string-to-string and DECIMALV2-to-DECIMALV2 paths remain unchanged.

Validation

  • Before the fix: the new task-serialization UT fails because the CAST node is non-nullable. The baseline SQL schema change was cancelled with the declared/actual nullable mismatch above, using the same ASAN BE.
  • After the fix: 56 FE unit tests passed (CastExprTest, CastTest, TryCastTest, CastExprThriftSimplificationTest, SchemaChangeJobV2Test including three new task cases, and CloudSchemaChangeJobV2Test).
  • build.sh --fe passed with zero Checkstyle violations. The unchanged BE was built with ASAN via build.sh --be.
  • SQL goldens were generated by run-regression-test.sh, then normal comparison passed: test_schema_change_cast_nullable and existing test_alter_table_column_nullable (2 suites). Both scale directions (3 -> 6 -> 3), NOT NULL and nullable columns, maximum datetime, historical NULL, and microsecond writes are covered. Downscale results match an ordinary CAST before the alter.

Compatibility and performance

BE CAST conversion semantics and storage formats are unchanged. No new Thrift fields, public APIs, or dependencies are introduced. The task declares the nullable result already returned by the converter; required-column constraints remain in effect.

The inference runs while constructing each changed-column expression, reusing types that are already resolved. There is no new per-row work. Other direct CastExpr entry points were inspected; their legacy fallback is preserved to avoid changing runtime-filter behavior. No performance benchmark is claimed.

Release note

Fix heavy schema change failures when modifying DATETIMEV2 scale.

Check List (For Author)

  • Test: 56 FE unit tests passed; FE Checkstyle and ASAN BE builds passed; 2 SQL regression suites passed
  • Behavior changed: Yes. DATETIMEV2 scale schema changes no longer fail because of an incorrect CAST result nullability declaration; conversion semantics are unchanged.
  • Does this need documentation: No

@eldenmoon
eldenmoon requested a review from yiguolei as a code owner November 7, 2026 04:41
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon
eldenmoon force-pushed the branch-variant-cast-nullable-42-20261107 branch from f9928b1 to d9dc16f Compare November 7, 2026 05:20
@eldenmoon eldenmoon changed the title [fix](fe) Reuse CAST nullability for directly constructed expressions [fix](fe) Infer CAST nullability when building schema change tasks Nov 7, 2026
@eldenmoon

Copy link
Copy Markdown
Member Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70362
commit: d9dc16fea95eb5a8fd51e9587ae78e87792f4c4a
base: 66122209cac78805a96a42f44d2a4a20b428f038
reviewed_at: 2026-11-07T13:33+08:00
reviewer: eldenmoon
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Static source review only; no builds or tests were run by the reviewers. Existing author logs were inspected: 56 FE tests, FE build/Checkstyle, generated goldens and two normal schema-change regression suites passed. CI status was not verified.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@eldenmoon

Copy link
Copy Markdown
Member Author

/run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@eldenmoon

Copy link
Copy Markdown
Member Author

/run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloud_p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloud_p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run feut

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloud_p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run feut

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloud_p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run feut

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run p0

### What problem does this PR solve?

Related PR: apache#70359

Problem Summary: heavy schema change builds CastExpr directly without the
nullable metadata normally supplied by Nereids. Changing a NOT NULL
DATETIMEV2 column's scale therefore sends a non-nullable CAST to BE even
though the existing converter returns a nullable column, and the alter task
is cancelled with a declared/actual result type mismatch.

At the schema-change construction site, reuse Cast.castNullable with the
already resolved source/target types and source slot nullability, and set
the existing nullableFromNereids metadata before task serialization. Keep
legacy CastExpr fallback, runtime filters, BE conversion and storage unchanged.

### Release note

Fix heavy schema change failures when modifying DATETIMEV2 scale.

### Check List (For Author)

- Test: 56 FE unit tests passed; FE build/Checkstyle and unchanged BE ASAN
  build passed; expanded SQL golden generation passed with the scoped fix
- Behavior changed: Yes. Schema change tasks declare the existing converter's
  nullable result correctly; conversion semantics are unchanged.
- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#70359

Problem Summary: check the CAST expression actually serialized by a heavy
schema change task when DATETIMEV2 scale increases or decreases, including
a nullable source column. Protect the legacy Variant-to-date fallback from
unrelated changes. SQL regression checks existing rows, SQL NULL, the maximum
datetime, and new microsecond writes after a scale change.

### Release note

None

### Check List (For Author)

- Test: 56 FE unit tests passed; FE build/Checkstyle and BE ASAN build
  passed; test_schema_change_cast_nullable and test_alter_table_column_nullable
  passed (2 suites); golden generated by run-regression-test.sh
- Behavior changed: No
- Does this need documentation: No
@eldenmoon
eldenmoon force-pushed the branch-variant-cast-nullable-42-20261107 branch from d9dc16f to 3631150 Compare November 12, 2026 02:11
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run p0

@eldenmoon

Copy link
Copy Markdown
Member Author

run compile

@eldenmoon

Copy link
Copy Markdown
Member Author

run feut

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

4 similar comments
@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run external

@eldenmoon

Copy link
Copy Markdown
Member Author

run nonConcurrent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants