Skip to content

Add aggregate-specific metrics to legacy grouped hash and TopK - #24523

Draft
kosiew wants to merge 3 commits into
apache:mainfrom
kosiew:aggmetrics-03-23570
Draft

Add aggregate-specific metrics to legacy grouped hash and TopK#24523
kosiew wants to merge 3 commits into
apache:mainfrom
kosiew:aggmetrics-03-23570

Conversation

@kosiew

@kosiew kosiew commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Aggregate-specific timing metrics should remain consistent across grouped aggregation implementations. Previously, queries using the legacy grouped hash or grouped TopK paths could expose less per-aggregate timing information than queries using the migrated grouped hash implementation, making metrics dependent on which internal aggregation path was selected.

This change extends aggregate-specific metrics to those remaining paths for the phases they actually perform.

What changes are included in this PR?

  • Adds per-aggregate accumulator timing metrics to GroupedHashAggregateStream for update, merge, state, evaluate, and convert-to-state phases.
  • Reuses the same aggregate labels and accumulator metric helpers used by the migrated grouped aggregation implementation.
  • Adds per-aggregate argument evaluation timing to GroupedTopKAggregateStream.
  • Documents that grouped TopK does not expose accumulator update, merge, state, or evaluate timing because it maintains MIN/MAX values directly in its priority map.
  • Extends final-mode metric coverage to exercise both values of datafusion.execution.enable_migration_aggregate.

Are these changes tested?

Yes.

The patch adds test_legacy_groupby_aggregate_accumulator_metrics, which disables datafusion.execution.enable_migration_aggregate and verifies aggregate-specific arguments_time, update_time, and state_time metrics for the legacy grouped hash path, including positive update and state timings.

test_groupby_metrics_final_mode now exercises final-mode metrics with datafusion.execution.enable_migration_aggregate set to both true and false.

The patch also adds test_topk_aggregate_argument_metrics, which exercises grouped TopK aggregation and verifies that the per-aggregate agg_expr_0_arguments_time metric is present, labeled with MIN(a), and non-zero.

Are there any user-facing changes?

Yes. Query execution metrics now expose more consistent aggregate-specific timing information when grouped aggregation uses the legacy grouped hash or grouped TopK implementations.

There are no public API changes.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

kosiew added 3 commits August 20, 2026 22:23
… TopK paths

- Legacy grouped hash: per-aggregate update/merge/state/convert/evaluate
   timers.
 - Reused canonical phase contract; spill-safe Single merge coverage.
 - Grouped TopK: per-aggregate argument timer; accumulator phases documented
   N/A.
 - Tests: legacy fallback labels/times; TopK metric label/time.
- Final metrics test now checks migrated + legacy paths.
- TopK test now asserts bounded MIN output + metric.
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 20, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.97814% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.31%. Comparing base (cd8cef9) to head (120b2c8).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...hysical-plan/src/aggregates/grouped_topk_stream.rs 87.67% 2 Missing and 7 partials ⚠️
...ysical-plan/src/aggregates/group_values/metrics.rs 85.45% 2 Missing and 6 partials ⚠️
...hysical-plan/src/aggregates/grouped_hash_stream.rs 90.90% 5 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24523    +/-   ##
========================================
  Coverage   81.30%   81.31%            
========================================
  Files        1117     1117            
  Lines      395798   396065   +267     
  Branches   395798   396065   +267     
========================================
+ Hits       321815   322049   +234     
- Misses      55164    55187    +23     
- Partials    18819    18829    +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants