perf(core): skip IPC validation on the sort-shuffle read paths - #2365
Open
NoahKusaba wants to merge 1 commit into
Open
perf(core): skip IPC validation on the sort-shuffle read paths#2365NoahKusaba wants to merge 1 commit into
NoahKusaba wants to merge 1 commit into
Conversation
phillipleblanc
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change + Changes
Ballista already skips Arrow's per-batch IPC validation on some shuffle reads, behind the
default-on
arrow-ipc-optimizationsfeature (shuffle_reader.rs:1188,flight_service.rs:159).Two decode paths were missed, which between them cover every read of the default shuffle format:
MultiStreamPartitionStreamBlockDataStream→StreamDecoderMultiStreamPartitionStreamThe shuffle writer produced these bytes from arrays Arrow had already validated, so validating
again costs a UTF-8 / offset / view-index scan per batch and buys nothing. Same trade DataFusion
makes for spill files (
physical-plan/src/spill/mod.rs:96).AI Benchmark:
Decode only, not end-to-end. Release, best-of-15, variants interleaved per rep, LZ4, warm cache.
Tests:
Two tests in
ballista/client/tests/sort_shuffle.rs, each across the existingLocal/RemoteFlight/RemoteBlockIomatrix:..._group_by_binary_column— every pre-existing query groups on a fixed-width primitive, sono offsets buffer had ever crossed a shuffle here.
..._group_by_view_columns— twoUtf8Viewkeys, 8 bytes (inline) and 35 (data buffer), soboth halves of view validation are covered. No fixture column is a view type, hence
arrow_cast.Are there any user-facing changes?
No