Skip to content

fix: expose PartitionedTopKExec metrics - #24495

Open
shinzoxD wants to merge 1 commit into
apache:mainfrom
shinzoxD:fix/partitioned-topk-metrics
Open

fix: expose PartitionedTopKExec metrics#24495
shinzoxD wants to merge 1 commit into
apache:mainfrom
shinzoxD:fix/partitioned-topk-metrics

Conversation

@shinzoxD

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

PartitionedTopKExec already collected ExecutionPlanMetricsSet internally, but ExecutionPlan::metrics() was unimplemented so it returned None. EXPLAIN ANALYZE therefore showed no metrics, and output_rows / output_batches could not be checked against what the operator actually emits.

What changes are included in this PR?

  • Implement PartitionedTopKExec::metrics() so it returns the existing metrics set, matching SortExec / window operators.
  • Record baseline output metrics (output_rows, output_batches, output_bytes) on the coalesced batches sent to the consumer, not on the pre-coalesce per-partition heap batches. elapsed_compute was already tracked via TopKMetrics.

This does not address #24468 (SortExec TopK output_batches).

Are these changes tested?

Added the issue's reproduction as partitioned_topk_output_batches_metric_matches_emitted_batches: it asserts metrics() is Some and that output_batches equals the number of batches actually emitted (5 coalesced batches of 10 rows).

Also ran:

  • cargo test -p datafusion-physical-plan --lib topk (82 passed)
  • cargo test -p datafusion --test core_integration partitioned_topk_output_batches_metric_matches_emitted_batches (passed)
  • cargo fmt --all
  • cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings
  • cargo clippy -p datafusion --test core_integration -- -D warnings

Are there any user-facing changes?

EXPLAIN ANALYZE now reports metrics for PartitionedTopKExec (output_rows, output_batches, elapsed_compute, etc.).

Copilot AI lite review requested due to automatic review settings August 19, 2026 17:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Jefffrey

Copy link
Copy Markdown
Contributor

how does this differ from

@Jefffrey

Copy link
Copy Markdown
Contributor

for anyone looking at this PR please see my comment here:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PartitionedTopKExec does not expose any metrics (metrics() returns None)

3 participants