Repository navigation
branch-4.2: [improvement](fe) Stream Hive directory listings with bounded caching - #70584
Open
CalvinKirs wants to merge 1 commit into
Open
CalvinKirs wants to merge 1 commit into
CalvinKirs wants to merge 1 commit into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#70078 Problem Summary: Cold Hive file-list cache misses materialize every file in a partition before producing its first split, delaying scans of large single partitions. ### Details Adapt the merged change to the branch-4.2 Hive metadata cache and native file system interfaces. Pull HDFS, S3 and Azure listings lazily, bound per-entry and catalog-wide pending cache retention, and publish only complete listings and byte summaries. Broker preserves its single-RPC listing boundary. Bind refresh loaders and streaming publication to the original physical cache entry so invalidation, retirement and concurrent writers cannot resurrect stale listings. Keep partition concurrency, count pushdown, sampling and transaction selection semantics, and avoid directory I/O for ordinary row-count estimates. Allow explicit ANALYZE to populate complete summaries, publish completed row estimates under table/catalog/database invalidation fences, and retain valid job/column statistics without making ordinary readers wait for pending analysis. Track streaming load outcomes once, including failures, without double-counting managed refreshes. ### Release note Hive scans can produce splits before a cold directory listing completes, with bounded optional file-list cache staging. ### Testing All 236 targeted FE tests across fourteen suites passed with run-fe-ut.sh, including cold ANALYZE, completed-only row publication, scope invalidation, nonblocking ordinary readers, load metrics, cache lifecycle, listing backends and the real single-partition split entry path. FE compilation and Checkstyle passed in that run. External regression fixtures were adapted from the original PR; they were not run locally. ### Check List (For Author) - Test - [x] Unit Test - [ ] Regression test - Behavior changed: - [x] Yes. Stream ordinary batch-mode Hive file listings and bound cache staging. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label (cherry picked from commit 89d4f28)
Member
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#70078