Skip to content

perf(core): skip IPC validation on the sort-shuffle read paths - #2365

Open
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:perf/skip-sort-shuffle-ipc-validation
Open

perf(core): skip IPC validation on the sort-shuffle read paths#2365
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:perf/skip-sort-shuffle-ipc-validation

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change + Changes

Ballista already skips Arrow's per-batch IPC validation on some shuffle reads, behind the
default-on arrow-ipc-optimizations feature (shuffle_reader.rs:1188, flight_service.rs:159).
Two decode paths were missed, which between them cover every read of the default shuffle format:

read path decoder
local MultiStreamPartitionStream
remote, block transport (default) BlockDataStreamStreamDecoder
remote, Flight transport (opt-in) executor-side MultiStreamPartitionStream

The shuffle writer produced these bytes from arrays Arrow had already validated, so validating
again costs a UTF-8 / offset / view-index scan per batch and buys nothing. Same trade DataFusion
makes for spill files (physical-plan/src/spill/mod.rs:96).

AI Benchmark:
Decode only, not end-to-end. Release, best-of-15, variants interleaved per rep, LZ4, warm cache.

shape validate skip
lineitem-like, 24 batches / 197k rows 48.12 ms 28.18 ms 1.71x
lineitem-like, 16 batches / 131k rows 30.26 ms 20.90 ms 1.45x
lineitem-like, 16 sub-streams × 1 batch 28.62 ms 19.33 ms 1.48x
numeric only 6.3–10.9 ms 6.2–10.2 ms ~1.00x

Tests:

Two tests in ballista/client/tests/sort_shuffle.rs, each across the existing
Local/RemoteFlight/RemoteBlockIo matrix:

  • ..._group_by_binary_column — every pre-existing query groups on a fixed-width primitive, so
    no offsets buffer had ever crossed a shuffle here.
  • ..._group_by_view_columns — two Utf8View keys, 8 bytes (inline) and 35 (data buffer), so
    both halves of view validation are covered. No fixture column is a view type, hence arrow_cast.

Are there any user-facing changes?

No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants