Skip to content

fix: count SortExec TopK output_batches after emit split - #24496

Open
shinzoxD wants to merge 1 commit into
apache:mainfrom
shinzoxD:fix/sort-exec-topk-output-batches
Open

fix: count SortExec TopK output_batches after emit split#24496
shinzoxD wants to merge 1 commit into
apache:mainfrom
shinzoxD:fix/sort-exec-topk-output-batches

Conversation

@shinzoxD

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

When SortExec runs in TopK mode (ORDER BY ... LIMIT k on unsorted input) and the k result rows span more than one output batch, output_batches reported 1 regardless of how many batches the operator actually emitted.

For example, a top-25 query at batch_size = 10 emits [10, 10, 5] but reported output_batches=1.

This is the SortExec TopK counterpart of #24470 / #24495 (PartitionedTopKExec metrics). That sibling work is out of scope here.

What changes are included in this PR?

TopK::emit now records baseline output metrics (output_rows, output_batches, output_bytes) on each batch sent to the consumer after splitting to batch_size, instead of once on the pre-split heap batch.

Are these changes tested?

Added the issue's reproduction as topk_output_batches_metric_matches_emitted_batches: it asserts output_batches equals the number of batches actually emitted ([10, 10, 5] → 3).

Also ran:

  • cargo test -p datafusion-physical-plan --lib topk (82 passed)
  • cargo test -p datafusion --test core_integration topk_output_batches_metric_matches_emitted_batches (passed)
  • cargo fmt --all
  • cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings
  • cargo clippy -p datafusion --test core_integration -- -D warnings

Are there any user-facing changes?

EXPLAIN ANALYZE now reports the correct output_batches for SortExec in TopK mode when the result spans multiple batches.

Record SortExec TopK baseline metrics on each batch-size-split
batch sent to the consumer so output_batches matches the number
of emitted batches.

Closes apache#24468
Copilot AI lite review requested due to automatic review settings August 19, 2026 18:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Aug 19, 2026
@Jefffrey

Copy link
Copy Markdown
Contributor

how does this differ from

@Jefffrey

Copy link
Copy Markdown
Contributor

for anyone looking at this PR please see my comment here:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SortExec (TopK) output_batches metric does not match the number of emitted batches

3 participants