[jacktengg]

Background

A follower FE forwards CTAS to the master FE. The request includes user variables as typed Thrift literals. The master restores these variables before it analyzes the statement.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

Forwarded TIMESTAMPTZ user variables can become DATE or DATETIME values. CTAS can then omit matching rows, create the wrong column type, or return an incorrect time difference. Restore both the declared type and its scale during decoding.

Details

• Frequency: When a follower forwards CTAS that reads a TIMESTAMPTZ user variable.
• Root cause: Literal decoding discards the TIMESTAMPTZ type. The shared type decoder also falls back to TIMESTAMPTZ with scale 0.
• Why existing tests missed it: Date forwarding tests cover TIMESTAMP_NS. TIMESTAMPTZ parser tests do not cover the complete Thrift round trip.
• Fix: Decode the TIMESTAMPTZ scale from the existing type descriptor. Pass that type to the existing literal parser. Add type and literal round-trip tests, plus a CTAS regression with one source row.
• Other paths and branches: The shared type decoder also serves Java UDF types and nested type descriptors. Its TIMESTAMPTZ branch now retains scale. A partition snapshot comment is updated; its conversion logic does not change. No release branch is modified.
• Behavior change: CTAS through a follower preserves the TIMESTAMPTZ type, scale, and UTC value used by a direct master connection.

Testing

• Local FE unit tests: 57 passed. Command: bash run-fe-ut.sh --run org.apache.doris.qe.ConnectProcessorTimestampTzTest,org.apache.doris.qe.ConnectProcessorTimestampNsTest,org.apache.doris.analysis.DateLiteralUtilsTest,org.apache.doris.catalog.TypeTest.
• Before the fix, all four forwarded TIMESTAMPTZ cases failed on source revision f7396d7. With only the literal fix, three cases still failed because scale became 0. Both fixes make all four cases pass.
• Final FE build and Checkstyle passed: FE_MAVEN_THREADS=8 DISABLE_BE_JAVA_EXTENSIONS=ON bash build.sh --fe.
• Ran bash run-regression-test.sh --run -d datatype_p0/timestamptz -s test_timestamptz_user_variable_ctas through both FE roles. Both passed with the final build in an isolated two-FE, one-BE cluster.
• The SQL test uses one source row and script-generated expected results. It verifies the matching row, TIMESTAMPTZ(6) column, +08:00 value, and zero hour difference.
• The earlier multi-FE reproduction used existing b9829c5 binaries: master passed; follower returned no filtered row, a DATE column, and an 8-hour difference. Final FEs use the current source plus this fix. The BE uses the existing worktree binary; BE code was not changed or rebuilt.
• git -c core.whitespace=-blank-at-eof diff --cached --check passed. The generated regression output retains its standard final blank line.

Release note

Fix CTAS results when a follower FE forwards TIMESTAMPTZ user variables. Matching rows, time differences, column types, and fractional precision now agree with direct master execution.

Check List (For Author)

• Behavior changed: Yes (see Details)
• Does this need documentation: No
• Pre-submission status: Ready

AI Disclosure

• Tool: Codex
• Generated: Code changes, tests, and this description.
• Human review: Pending.

Background

What problem does this PR solve?

Issue Number:
Related PR:
PR Type:

Problem Summary:

Details

Testing

Notes for Reviewers

Release note

Check List (For Author)

• Behavior changed:
• Does this need documentation:
• Pre-submission status:

AI Disclosure

• Tool:
• Generated:
• Human review:

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[jacktengg]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/42489753403

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[jacktengg]

/review

[jacktengg]

run buildall

[github-actions]

No substantiated new findings in the complete static review of PR #70380 at 41a78cc. Reviewed all 11 changed files and related producer/consumer paths. The two complete-scope reviewers and dedicated risk reviewer all returned NO_NEW_VALUABLE_FINDINGS; the main review independently verified their evidence. Converged after round 1. No existing inline findings or independently confirmed existing P0/P1 blockers.

Traced SET user variable -> legacy literal -> FEOpExecutor serialization -> forwarded request -> ConnectProcessor reconstruction -> Nereids literal conversion -> CTAS analysis and insertion. The declared temporal type and scale now survive decoding, and typed TIMESTAMPTZ parsing preserves the sender's UTC instant.

Critical checkpoint conclusions:

1. Goal and proof: The fix addresses TIMESTAMPTZ type/scale loss and midnight/trailing-zero inference for other dates. Type and decoder unit tests plus CTAS regressions target these behaviors; their code and expectations were inspected.
2. Scope and reuse: Two small production behavior changes reuse existing type factories and literal parsing. The other two production edits only correct comments.
3. Concurrency and locks: Reconstruction remains synchronous with per-request maps/literals; no new shared state, thread, lock acquisition, heavy work under metadata locks, or lock-order change.
4. Lifecycle and initialization: No new resource ownership, cleanup path, asynchronous lifetime, or static initialization dependency. Test configuration/thread-local changes have finally cleanup.
5. Configuration: No configuration items added. Date decoder tests cover both receiver enable_date_conversion settings.
6. Compatibility: Existing descriptor fields and wire formats are retained. Both HEAD and merge-base ScalarType serializers emit TIMESTAMPTZ precision and scale; reachable UDF wrappers preserve those descriptors. No concrete new missing-field failure found.
7. Thrift/protobuf definitions: No gensrc changes or field/enum ID changes.
8. Parallel paths: Checked DATE, DATEV2, DATETIME, DATETIMEV2, TIMESTAMPTZ, unchanged TIMESTAMP_NS, nested ARRAY/MAP/STRUCT decoding, Java UDF descriptors, and partition schema freezing. The common forwarding decoder covers redirected commands and forwarded queries.
9. Conditions and invariants: Valid sender strings satisfy the typed parser branches. TIMESTAMPTZ strings include +00:00 and select UTC during restoration, including before thread-local context installation. New field preconditions match actual producers and adjacent temporal decoding.
10. Test coverage: New tests cover temporal families, midnight/noon, scales 0/3/6, zero and trailing-zero fractions, +08 session interpretation, CTAS filtering and projection. Existing parser tests cover invalid inputs, offsets, boundaries and nanoseconds. No new malformed-input behavior requiring a separate negative test was identified.
11. Changed expected results: Both .out files agree with the SQL and inspected formatting paths, including datetime(6), timestamp(6), bigint(20), +08 rendering and zero-hour difference. Results are deterministically ordered; tables are dropped before use and retained afterward.
12. Observability: Existing parser/RPC errors remain adequate for this local deterministic conversion; no new operational path requires logs or metrics.
13. Transactions and persistence: No EditLog, replay, failover or transaction protocol changes.
14. Writes and atomicity: CTAS execution mechanics remain unchanged; the corrected literal feeds analysis and insertion. Storage visible-version, delete-bitmap and crash-cleanup logic are unchanged.
15. FE/BE variables: No new transmitted variable or serialization field; precision and scale use existing descriptors.
16. Performance and memory: The scalar decoder adds constant work, retaining existing recursive traversal and short-string parsing complexity. No new IO, retained resource or significant allocation mechanism.
17. Other issues: Final sweep and independent missed-case rechecks found no additional substantiated issue. Both initial risks were dismissed with concrete evidence; no unresolved candidate or uncovered static-review scope remains.

No additional user-provided focus was supplied. Validation limits: this invocation performed static inspection only, with no builds, tests, benchmarks or runtime validation. The SQL suites exercise forwarding when invoked through a follower FE; their configured connection does not itself guarantee that topology. Execution and generated-output provenance were not independently verified, and author-reported test runs are not claimed as reviewer-executed evidence.

Review converged after 1 round(s).

[jacktengg]

run buildall

[jacktengg]

/review

[github-actions]

Static review of all 14 changed paths at 8523e19 found no actionable new issues. No existing inline findings were present to carry forward.

The reviewed flow is: follower user variable -> FEOpExecutor/ExprToThriftVisitor (declared type and canonical text) -> ConnectProcessor -> DateLiteralUtils -> ConnectContext/Literal.fromLegacyLiteral -> master CTAS analysis. Preserving TIMESTAMPTZ type and scale keeps the canonical UTC fields intact and prevents midnight values from being inferred as DATE.

Critical checkpoint conclusions:

• Goal and proof: the two production changes preserve the sender's temporal type and fractional scale. New type/literal round-trip tests and CTAS suites cover the intended failures; these tests were inspected, not executed in this review.
• Scope and reuse: the fix reuses the existing parser and scalar-type factory. Two comment updates are accurate. The additional changelog and partition-cache test changes are separate stability fixes and were also reviewed.
• Concurrency and locks: no new production thread, shared state, metadata-lock acquisition, lock ordering, or IO under a lock is introduced. Test polling reuses the master TSO snapshot lock and existing scheduler behavior.
• Lifecycle and initialization: restored literals remain request-local; no new resource ownership, callback lifetime, reference cycle, or static initialization dependency is introduced. Tests restore the global date-conversion setting/scheduler interval and remove their thread-local context in finally blocks.
• Configuration: no new setting is added. Explicit sender types correctly avoid dependence on the receiver's enable_date_conversion default.
• Compatibility: no function symbol or storage/wire layout changes. The pre-fix FE serializer already writes both TIMESTAMPTZ precision and scale. Production Java UDF transport preserves FE descriptors; recursive array/map/struct decoding reaches the same corrected scalar branch. This does not add TIMESTAMPTZ Java UDF support.
• Thrift/Protobuf definitions: no gensrc files, field IDs, enum values, or field optionality change.
• Parallel paths: DATE, DATEV2, DATETIME, DATETIMEV2, TIMESTAMPTZ, and TIMESTAMP_NS producer/parser/consumer paths were checked. TIMESTAMP_NS already received its explicit type and remains unchanged. The partition snapshot still retains all scalar parameters.
• Conditions and error handling: canonical sender formats satisfy explicit parser shape and fractional limits. TIMESTAMPTZ text always carries +00:00, making restoration independent of when the receiver binds its context. Existing parse exceptions and descriptor invariant failures remain appropriate; zero-month/day rejection predates this PR.
• Test coverage: new cases cover midnight/noon, zero and nonzero fractions, trailing zeros, scales 0/3/6, both date-conversion defaults, a non-UTC receiver session, filtering, CTAS column types, and timestampdiff. Existing parser negatives, timezone/DST tests, and nanosecond forwarding coverage remain. The CTAS suites exercise actual forwarding when run through a follower; they do not create that topology themselves.
• Expected results: all three new output files match their SQL. UTC midnight renders as 08:00 at +08:00 with zero hour difference; date CTAS retains DATETIME(6); partition counts are 6 before recycling and 3 afterward. Changelog boundaries query the master's TSO, round strictly past prior writes, and wait before subsequent writes; inclusive start semantics preserve those later writes.
• Observability: existing parser/request errors and bounded polling failures provide adequate diagnostics for this correction; no new metrics or logging requirement identified.
• Transactions, persistence, and failover: production journal/replay and transaction state are unchanged. Partition-drop and replay cache invalidation were traced for the regression test.
• Data writes and atomicity: CTAS and regression DML use existing write paths; the fix only changes reconstructed literals. No new write atomicity, crash-cleanup, or lock-lifetime issue identified.
• FE/BE variable passing: no new transmitted variable; existing type descriptor fields are reused across forwarding and function metadata.
• Performance: descriptor reconstruction remains bounded per scalar, with existing recursion for compound types; no new hot-path IO or asymptotic work. Changelog waits are bounded. Changing retention from 6 to 3 preserves the warmed count cache until partition removal, so the cache regression still exercises its intended invalidation path.
• Additional risk review: both explicit-type parsing and shared descriptor field requirements were independently challenged and dismissed with producer/consumer evidence. No unresolved candidate or uncovered changed path remains.

User focus: no additional focus points were supplied. Validation limits: static inspection only; no builds, tests, benchmarks, or live mixed-version/two-FE validation were performed by this review.

Review converged after 2 round(s).

[jacktengg]

run external