Skip to content

fix(benchmark): compare PR-comment baseline at the flat-label level#4887

Open
tadelesh wants to merge 1 commit into
mainfrom
fix/benchmark-scoped-emitter-baseline
Open

fix(benchmark): compare PR-comment baseline at the flat-label level#4887
tadelesh wants to merge 1 commit into
mainfrom
fix/benchmark-scoped-emitter-baseline

Conversation

@tadelesh

@tadelesh tadelesh commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Problem

The benchmark PR comments show phantom emitters emit/@azure-tools and emit/@typespec that don't exist, and the real emitters' baseline is wrong.

Root cause: to build the rolling baseline, the PR-comment path reconstructed a RuntimeStats from the flat metrics stored on the benchmark-data branch (expandRuntimeMetrics). It split each emit/... label on every / and used the second segment as the emitter name — but a scoped package name contains a slash:

Flat history label Parsed as (before)
emit/@azure-tools/typespec-client-generator-core emitter @azure-tools, step typespec-client-generator-core
emit/@typespec/openapi3 emitter @typespec, step openapi3

So it produced a fake @azure-tools / @typespec emitter and left the real emitter with no baseline.

This is pre-existing (introduced with the rolling-baseline reconstruction), now also visible on the external-spec benchmark comment.

Fix

The flatten → reconstruct structure → flatten again round-trip was the only place that had to parse labels, and parsing a name that itself contains / is inherently ambiguous. Instead, compare the current run against the baseline directly at the flat-label level:

  • Labels are opaque keys (compareFlatMetrics) — scoped names like emit/@azure-tools/pkg are compared as-is, never split.
  • The current run's runtime is flattened with flattenRuntime (the same function that produces the stored history), so both sides use identical keys.
  • expandRuntimeMetrics is removed entirely.

The benchmark-data storage format (history.json / latest.json) is unchanged, so existing data and the website dashboard are unaffected. The compare CLI keeps its structural path (it compares two real runs and never had this bug).

Validation

Ran the fixed upload-pr-comment against the real benchmark-data history for both datasets:

  • external-results: comment shows only emit/@azure-tools/typespec-client-generator-core — no phantom @azure-tools.
  • results (main): comment shows all real emitters (@azure-tools/typespec-autorest, @azure-tools/typespec-python, @typespec/http-client-js, @typespec/openapi3 + steps) — no phantom @azure-tools/@typespec rows.

Tests

Added test/compare-flat.test.ts: scoped labels are compared as opaque keys (no phantom emitters), missing values default to 0, and flattenRuntime keeps scoped emitter names intact.

@azure-tools/typespec-benchmark is private, so no changeset is required.

@azure-sdk-automation

azure-sdk-automation Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

No changes needing a change description found.

@github-actions

github-actions Bot commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚡ External Spec Benchmark Results

⚠️ 21 metric(s) regressed above the +5% threshold:

Metric Baseline Current Change
total 🔴 6.97s 🔴 7.55s +8.3% 🔴
loader 🔴 2.04s 🔴 2.28s +11.7% 🔴
checker 🔴 1.55s 🔴 1.64s +5.7% 🔴
 ↳ validation/@typespec/http 🔴 158.3ms 🔴 171.1ms +8.1% 🔴
linter 🔴 2.83s 🔴 3.06s +8.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/byos 🔴 131.4ms 🔴 141.9ms +8.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-header-explode 🔴 282.9ms 🔴 313.8ms +10.9% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-private-usage 🔴 35.1ms 🔴 37.2ms +6.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-query-explode 🔴 290.7ms 🔴 318.1ms +9.5% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces 🟡 16.4ms 🟡 18.3ms +11.9% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch 🔴 128.6ms 🔴 139.1ms +8.1% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem 🔴 324.9ms 🔴 353.3ms +8.7% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-child-resources 🔴 66.9ms 🔴 70.5ms +5.3% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version 🔴 127.7ms 🔴 136.2ms +6.7% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts 🔴 146.4ms 🔴 156.8ms +7.1% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes 🟢 9.9ms 🟡 11.4ms +15.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response 🔴 66.4ms 🔴 70.5ms +6.2% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header 🔴 287.6ms 🔴 314.7ms +9.4% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body 🔴 473.8ms 🔴 520.1ms +9.8% 🔴
emit 🔴 11.50s 🔴 12.12s +5.3% 🔴
 ↳ emit/@azure-tools/typespec-client-generator-core 🔴 11.50s 🔴 12.12s +5.3% 🔴
Full details – comparing 14d12ca vs baseline rolling-baseline-3497e82-e161baf (rolling baseline (2 main runs))
Metric Baseline Current Change
total 🔴 6.97s 🔴 7.55s +8.3% 🔴
loader 🔴 2.04s 🔴 2.28s +11.7% 🔴
resolver 🟢 152.3ms 🟢 154.5ms +1.5%
checker 🔴 1.55s 🔴 1.64s +5.7% 🔴
validation 🟡 382.7ms 🟡 395.6ms +3.4%
 ↳ validation/@azure-tools/typespec-azure-core 🟢 0.2ms 🟢 0.2ms +1.0%
 ↳ validation/@azure-tools/typespec-client-generator-core 🔴 20.4ms 🔴 20.1ms -1.5%
 ↳ validation/@typespec/http 🔴 158.3ms 🔴 171.1ms +8.1% 🔴
 ↳ validation/@typespec/rest 🔴 23.4ms 🔴 22.1ms -5.6% 🟢
 ↳ validation/@typespec/versioning 🔴 154.3ms 🔴 154.8ms +0.3%
 ↳ validation/compiler 🔴 24.7ms 🔴 24.6ms -0.3%
linter 🔴 2.83s 🔴 3.06s +8.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/auth-required 🟢 0.1ms 🟢 0.1ms +8.9%
 ↳ linter/@azure-tools/typespec-azure-core/byos 🔴 131.4ms 🔴 141.9ms +8.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/casing-style 🟢 9.6ms 🟢 9.9ms +3.7%
 ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance 🟢 1.0ms 🟢 1.0ms +4.9%
 ↳ linter/@azure-tools/typespec-azure-core/documentation-required 🟡 14.1ms 🟡 14.2ms +1.0%
 ↳ linter/@azure-tools/typespec-azure-core/friendly-name 🟡 10.4ms 🟡 10.9ms +4.4%
 ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required 🟢 3.2ms 🟢 3.5ms +9.7%
 ↳ linter/@azure-tools/typespec-azure-core/known-encoding 🟢 2.9ms 🟢 3.0ms +3.5%
 ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required 🟢 5.5ms 🟢 5.9ms +6.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch 🟢 3.3ms 🟢 3.4ms +3.9%
 ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union 🟢 5.2ms 🟢 5.5ms +6.2%
 ↳ linter/@azure-tools/typespec-azure-core/no-enum 🟢 0.4ms 🟢 0.4ms +4.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes 🟢 2.1ms 🟢 2.1ms +3.3%
 ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops 🟢 1.5ms 🟢 1.6ms +4.0%
 ↳ linter/@azure-tools/typespec-azure-core/no-format 🟢 7.3ms 🟢 7.5ms +2.4%
 ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric 🟢 3.6ms 🟢 3.7ms +1.8%
 ↳ linter/@azure-tools/typespec-azure-core/no-header-explode 🔴 282.9ms 🔴 313.8ms +10.9% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage 🔴 22.5ms 🔴 22.4ms -0.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator 🟢 0.8ms 🟢 1.0ms +15.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-nullable 🟢 2.9ms 🟢 3.1ms +7.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime 🟡 15.2ms 🟡 15.9ms +4.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-openapi 🔴 36.5ms 🔴 37.1ms +1.7%
 ↳ linter/@azure-tools/typespec-azure-core/no-private-usage 🔴 35.1ms 🔴 37.2ms +6.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-query-explode 🔴 290.7ms 🔴 318.1ms +9.5% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces 🟡 16.4ms 🟡 18.3ms +11.9% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch 🔴 128.6ms 🔴 139.1ms +8.1% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params 🟢 1.9ms 🟢 1.9ms +0.8%
 ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator 🟢 0.6ms 🟢 0.7ms +5.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-unknown 🟢 2.2ms 🟢 2.4ms +7.3%
 ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union 🟢 5.4ms 🟢 5.5ms +2.1%
 ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version 🟢 3.3ms 🟢 3.3ms -0.7%
 ↳ linter/@azure-tools/typespec-azure-core/request-body-problem 🟢 5.1ms 🟢 5.1ms -0.9%
 ↳ linter/@azure-tools/typespec-azure-core/require-versioned 🟢 0.1ms 🟢 0.1ms +1.1%
 ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem 🔴 324.9ms 🔴 353.3ms +8.7% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body 🟢 0.7ms 🟢 0.7ms +7.2%
 ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model 🟢 3.4ms 🟢 3.5ms +0.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-child-resources 🔴 66.9ms 🔴 70.5ms +5.3% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-lifecycle-operations 🟢 0.2ms 🟢 0.2ms +3.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version 🔴 127.7ms 🔴 136.2ms +6.7% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key 🟢 0.9ms 🟢 0.9ms +7.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage 🟢 0.6ms 🟢 0.7ms +8.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes 🟢 3.7ms 🟢 4.0ms +8.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-feature-file-usage-discourage 🟢 0.0ms 🟢 0.1ms +11.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts 🔴 146.4ms 🔴 156.8ms +7.1% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record 🟢 3.4ms 🟢 3.5ms +3.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes 🟢 9.9ms 🟡 11.4ms +15.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes 🟢 0.3ms 🟢 0.3ms +9.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment 🟢 4.4ms 🟢 4.8ms +8.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property 🟢 1.0ms 🟢 1.1ms +6.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator 🟢 0.5ms 🟢 0.5ms +2.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb 🟢 1.5ms 🟢 1.5ms +1.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property 🟢 0.8ms 🟢 0.8ms +7.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format 🟢 0.3ms 🟢 0.3ms +6.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars 🟢 1.9ms 🟢 2.1ms +10.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern 🟢 0.2ms 🟢 0.3ms +8.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation 🟢 3.3ms 🟢 3.4ms +2.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response 🔴 66.4ms 🔴 70.5ms +6.2% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch 🟢 4.1ms 🟢 4.7ms +12.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars 🟢 1.4ms 🟢 1.4ms +5.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state 🟢 0.9ms 🟢 1.0ms +8.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels 🟢 0.8ms 🟢 0.8ms +3.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties 🟢 1.5ms 🟢 1.5ms +5.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation 🟢 0.1ms 🟢 0.2ms +17.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header 🔴 287.6ms 🔴 314.7ms +9.4% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint 🟢 0.2ms 🟢 0.2ms +15.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers 🟢 4.4ms 🟢 4.5ms +1.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model 🟢 0.7ms 🟢 0.8ms +11.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-override-props 🟢 0.7ms 🟢 0.8ms +7.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-reserved-resource-property 🟢 0.7ms 🟢 0.8ms +11.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation 🟢 2.2ms 🟢 2.5ms +11.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body 🔴 473.8ms 🔴 520.1ms +9.8% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope 🟢 1.5ms 🟢 1.6ms +4.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name 🟢 1.6ms 🟢 1.7ms +5.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop 🔴 82.4ms 🔴 82.7ms +0.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type 🟢 4.1ms 🟢 4.2ms +1.9%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression 🟢 0.3ms 🟢 0.3ms +10.2%
 ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict 🔴 29.7ms 🔴 30.4ms +2.5%
 ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix 🟢 6.6ms 🟢 7.0ms +5.1%
emit 🔴 11.50s 🔴 12.12s +5.3% 🔴
 ↳ emit/@azure-tools/typespec-client-generator-core 🔴 11.50s 🔴 12.12s +5.3% 🔴

Averaged across 3 specs (compute, network, web).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)

@azure-sdk-automation

azure-sdk-automation Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

You can try these changes here

🛝 Playground 🌐 Website

@github-actions

github-actions Bot commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚡ Benchmark Results

⚠️ 17 metric(s) regressed above the +5% threshold:

Metric Baseline Current Change
total 🔴 1.00s 🔴 1.07s +6.2% 🔴
loader 🟡 304.5ms 🟡 328.2ms +7.8% 🔴
checker 🟡 361.4ms 🟡 382.9ms +5.9% 🔴
linter 🟡 213.4ms 🟡 225.8ms +5.8% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-header-explode 🔴 26.5ms 🔴 28.5ms +7.6% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-query-explode 🔴 27.6ms 🔴 29.5ms +7.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-response-body 🔴 33.2ms 🔴 35.6ms +7.4% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem 🔴 32.3ms 🔴 34.1ms +5.6% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-child-resources 🔴 20.3ms 🔴 22.7ms +11.9% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts 🟡 16.9ms 🟡 18.0ms +6.2% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header 🟡 19.5ms 🔴 20.8ms +7.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body 🔴 29.5ms 🔴 31.4ms +6.3% 🔴
emit 🔴 5.98s 🔴 6.30s +5.4% 🔴
 ↳ emit/@azure-tools/typespec-python 🔴 4.46s 🔴 4.71s +5.7% 🔴
 ↳ emit/@typespec/openapi3 🟡 213.4ms 🟡 226.8ms +6.3% 🔴
 ↳ emit/@typespec/openapi3/compute 🟢 185.3ms 🟢 197.1ms +6.3% 🔴
 ↳ emit/@typespec/openapi3/write 🟢 27.5ms 🟢 29.1ms +6.0% 🔴
Full details – comparing 14d12ca vs baseline rolling-baseline-3b1e01f-e161baf (rolling baseline (20 main runs))
Metric Baseline Current Change
total 🔴 1.00s 🔴 1.07s +6.2% 🔴
loader 🟡 304.5ms 🟡 328.2ms +7.8% 🔴
resolver 🟢 40.1ms 🟢 41.8ms +4.2%
checker 🟡 361.4ms 🟡 382.9ms +5.9% 🔴
validation 🟢 84.2ms 🟢 87.7ms +4.2%
 ↳ validation/@azure-tools/typespec-azure-core 🟡 12.5ms 🟡 13.4ms +7.2%
 ↳ validation/@typespec/http 🟡 15.4ms 🟡 16.2ms +5.3%
 ↳ validation/@typespec/rest 🟢 1.6ms 🟢 1.7ms +5.1%
 ↳ validation/@typespec/versioning 🔴 50.8ms 🔴 52.4ms +3.1%
 ↳ validation/compiler 🟢 3.8ms 🟢 4.0ms +4.0%
linter 🟡 213.4ms 🟡 225.8ms +5.8% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/auth-required 🟢 0.1ms 🟢 0.1ms +11.8%
 ↳ linter/@azure-tools/typespec-azure-core/bad-record-type 🟢 0.6ms 🟢 0.6ms +3.2%
 ↳ linter/@azure-tools/typespec-azure-core/byos 🟢 8.3ms 🟢 8.7ms +5.0%
 ↳ linter/@azure-tools/typespec-azure-core/casing-style 🟢 1.4ms 🟢 1.5ms +3.5%
 ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance 🟢 0.2ms 🟢 0.2ms +3.4%
 ↳ linter/@azure-tools/typespec-azure-core/documentation-required 🟢 1.9ms 🟢 2.0ms +4.6%
 ↳ linter/@azure-tools/typespec-azure-core/friendly-name 🟢 1.4ms 🟢 1.5ms +4.9%
 ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required 🟢 0.4ms 🟢 0.4ms +4.1%
 ↳ linter/@azure-tools/typespec-azure-core/known-encoding 🟢 0.5ms 🟢 0.5ms +5.8%
 ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required 🟢 0.8ms 🟢 0.8ms +4.0%
 ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch 🟢 0.7ms 🟢 0.7ms +5.4%
 ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union 🟢 1.0ms 🟢 1.0ms +3.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-enum 🟢 0.2ms 🟢 0.3ms +4.0%
 ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes 🟢 0.3ms 🟢 0.3ms +4.3%
 ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops 🟢 0.2ms 🟢 0.2ms +4.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-format 🟢 0.8ms 🟢 0.9ms +4.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric 🟢 0.8ms 🟢 0.9ms +4.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-header-explode 🔴 26.5ms 🔴 28.5ms +7.6% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage 🟢 2.1ms 🟢 2.1ms +3.2%
 ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator 🟢 0.2ms 🟢 0.3ms +12.4%
 ↳ linter/@azure-tools/typespec-azure-core/no-nullable 🟢 0.4ms 🟢 0.4ms +4.9%
 ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime 🟢 2.0ms 🟢 2.1ms +5.7%
 ↳ linter/@azure-tools/typespec-azure-core/no-openapi 🟢 2.3ms 🟢 2.4ms +1.9%
 ↳ linter/@azure-tools/typespec-azure-core/no-private-usage 🟢 3.2ms 🟢 3.3ms +4.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-query-explode 🔴 27.6ms 🔴 29.5ms +7.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-response-body 🔴 33.2ms 🔴 35.6ms +7.4% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces 🟢 0.1ms 🟢 0.1ms +11.9%
 ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch 🟢 7.6ms 🟢 8.1ms +6.7%
 ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params 🟢 0.4ms 🟢 0.4ms +5.4%
 ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator 🟢 0.1ms 🟢 0.1ms +5.2%
 ↳ linter/@azure-tools/typespec-azure-core/no-unknown 🟢 0.3ms 🟢 0.4ms +5.7%
 ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union 🟢 0.8ms 🟢 0.8ms +4.0%
 ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version 🟢 0.4ms 🟢 0.4ms +3.1%
 ↳ linter/@azure-tools/typespec-azure-core/request-body-problem 🟢 0.5ms 🟢 0.5ms +3.4%
 ↳ linter/@azure-tools/typespec-azure-core/require-versioned 🟢 0.1ms 🟢 0.1ms +5.9%
 ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem 🔴 32.3ms 🔴 34.1ms +5.6% 🔴
 ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body 🟢 0.7ms 🟢 0.7ms +6.2%
 ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model 🟢 0.4ms 🟢 0.5ms +2.5%
 ↳ linter/@azure-tools/typespec-azure-core/use-standard-names 🟢 7.5ms 🟢 8.0ms +7.2%
 ↳ linter/@azure-tools/typespec-azure-core/use-standard-operations 🟢 0.3ms 🟢 0.3ms +2.9%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-child-resources 🔴 20.3ms 🔴 22.7ms +11.9% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-lifecycle-operations 🟢 0.2ms 🟢 0.2ms +0.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version 🟡 12.0ms 🟡 13.0ms +8.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key 🟢 0.2ms 🟢 0.2ms +4.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage 🟢 0.1ms 🟢 0.1ms +6.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes 🟢 5.2ms 🟢 4.5ms -13.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-feature-file-usage-discourage 🟢 0.1ms 🟢 0.1ms +3.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts 🟡 16.9ms 🟡 18.0ms +6.2% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record 🟢 0.6ms 🟢 0.7ms +2.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes 🟢 1.4ms 🟢 1.4ms -3.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes 🟢 0.2ms 🟢 0.2ms -2.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment 🟢 0.4ms 🟢 0.4ms +0.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property 🟢 0.4ms 🟢 0.4ms +4.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator 🟢 0.1ms 🟢 0.1ms +9.9%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb 🟢 0.2ms 🟢 0.2ms +1.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property 🟢 0.3ms 🟢 0.3ms +4.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format 🟢 0.2ms 🟢 0.2ms +1.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars 🟢 0.5ms 🟢 0.5ms +6.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern 🟢 0.1ms 🟢 0.1ms +3.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation 🟢 0.5ms 🟢 0.6ms +3.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response 🟢 8.9ms 🟢 9.2ms +3.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch 🟢 0.8ms 🟢 0.8ms +3.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars 🟢 0.4ms 🟢 0.4ms +4.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state 🟢 0.4ms 🟢 0.4ms +2.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels 🟢 0.2ms 🟢 0.2ms +4.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties 🟢 0.4ms 🟢 0.4ms +3.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation 🟢 0.1ms 🟢 0.1ms +8.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header 🟡 19.5ms 🔴 20.8ms +7.0% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint 🟢 0.1ms 🟢 0.1ms +3.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers 🟢 1.0ms 🟢 1.0ms +1.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model 🟢 0.3ms 🟢 0.3ms +7.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-override-props 🟢 0.3ms 🟢 0.3ms +3.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-reserved-resource-property 🟢 0.2ms 🟢 0.2ms +2.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation 🟢 0.5ms 🟢 0.5ms +5.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body 🔴 29.5ms 🔴 31.4ms +6.3% 🔴
 ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope 🟢 0.4ms 🟢 0.4ms +4.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name 🟢 0.4ms 🟢 0.4ms +1.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop 🟢 5.3ms 🟢 5.3ms -0.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type 🟢 0.7ms 🟢 0.7ms +2.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression 🟢 0.3ms 🟢 0.3ms +4.1%
 ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict 🟢 2.3ms 🟢 2.4ms +4.5%
 ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix 🟢 1.7ms 🟢 1.7ms +2.2%
emit 🔴 5.98s 🔴 6.30s +5.4% 🔴
 ↳ emit/@azure-tools/typespec-autorest 🟡 248.1ms 🟡 260.5ms +5.0%
 ↳ emit/@azure-tools/typespec-python 🔴 4.46s 🔴 4.71s +5.7% 🔴
 ↳ emit/@typespec/http-client-js 🔴 1.13s 🔴 1.18s +4.4%
 ↳ emit/@typespec/openapi3 🟡 213.4ms 🟡 226.8ms +6.3% 🔴
 ↳ emit/@typespec/openapi3/compute 🟢 185.3ms 🟢 197.1ms +6.3% 🔴
 ↳ emit/@typespec/openapi3/write 🟢 27.5ms 🟢 29.1ms +6.0% 🔴

Averaged across 3 specs (azure-arm-resource-manager, azure-core-dataplane, azure-full).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)

fix(benchmark): compare PR-comment baseline at the flat-label level
6fa20f4 
The PR comment reconstructed the baseline RuntimeStats from the flat history metrics via expandRuntimeMetrics, which split each emit/<name> label on every slash and mis-parsed scoped packages: emit/@azure-tools/typespec-client-generator-core became a fake @Azure-Tools emitter with the package name as a step. This produced phantom @Azure-Tools / @typespec emitter rows in the benchmark PR comments and a wrong baseline for the real emitters.

Compare current vs baseline directly at the flat-label level instead: labels are opaque keys (compareFlatMetrics), the current runtime is flattened with flattenRuntime, and no reconstruction/parsing of emitter names happens. Removes expandRuntimeMetrics entirely. The history/latest.json storage format is unchanged, so existing benchmark-data is unaffected; only the compare CLI keeps its structural path (which never had this bug).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tadelesh tadelesh force-pushed the fix/benchmark-scoped-emitter-baseline branch from 8555797 to 6fa20f4 Compare July 14, 2026 06:47
@tadelesh tadelesh changed the title fix(benchmark): keep scoped emitter names intact in PR comment baseline fix(benchmark): compare PR-comment baseline at the flat-label level Jul 14, 2026
@tadelesh tadelesh requested a review from Copilot July 14, 2026 06:59
Copilot AI reviewed Jul 14, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes incorrect baseline comparisons in benchmark PR comments for scoped emitter names (e.g. emit/@azure-tools/...) by switching the PR-comment workflow to compare runtime metrics at the flat label level instead of reconstructing/rehydrating structured runtime stats.

Changes:

  • Introduces compareFlatMetrics() to diff two flat label→ms maps without parsing labels (avoids scoped-name / ambiguity).
  • Updates upload-pr-comment to build/use flat baselines (rolling baseline from history.json, fallback from latest.json) and compare against the current run’s flattenRuntime() output.
  • Exports flattenRuntime() from generate-history and adds targeted tests covering scoped labels and missing-value defaults.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
packages/benchmark/test/compare-flat.test.ts Adds tests ensuring scoped labels are treated as opaque keys and missing metrics default to 0.
packages/benchmark/src/upload-pr-comment.ts Switches PR-comment baseline comparison from reconstructed RuntimeStats to flat-label comparisons.
packages/benchmark/src/generate-history.ts Exports flattenRuntime so it can be reused for PR-comment flattening.
packages/benchmark/src/compare.ts Adds compareFlatMetrics for flat-label comparisons (label treated as opaque).

Comment thread packages/benchmark/src/compare.ts
Comment thread packages/benchmark/src/upload-pr-comment.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@bterlson bterlson Awaiting requested review from bterlson bterlson is a code owner
@markcowl markcowl Awaiting requested review from markcowl markcowl is a code owner
@timotheeguerin timotheeguerin Awaiting requested review from timotheeguerin timotheeguerin is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tadelesh