perf-timeline: opt-in test262.total macro point

[mparrett]

Pairs with #15 (touches a different file, so it can land in either order). Wires the Test262 macro-benchmark (the cmd/bench-test262 tooling merged in #10) into the timeline, so an opted-in commit records a whole-suite point on the trend page.

Relationship to #10

#10 added the perf-test262 label, which runs an A/B on a PR (merge-base versus head) and comments the delta, a pre-merge "did this change move perf?" check. This is the complementary post-merge half: a single anchor-normalized test262.total point on the trend page. Same intent — run the test262 macro here — over two surfaces: a label on a PR, a marker in a commit.

Opt-in by design

The macro adds ~4 minutes to a snapshot, so it doesn't run on every push — only when the commit message contains [perf-test262], or via workflow_dispatch (with a boolean toggle so a manual backfill can skip the extra ~4 min). Without the marker the timeline is unchanged. The trade-off is a sparse, dotted trend rather than a continuous one: points appear where you ask for them.

How it works

On an opted-in commit, after the usual snapshot it runs the macro (built-ins and language, sharded so memory stays bounded), then injects a test262.total entry. The metric is the mean per-test execution time over the passing, non-timed-out tests (summed_ns / passing_count), anchor-normalized like every other series and tagged with the run's set_hash (from bench-test262, #24). The step is non-fatal, so a macro hiccup never drops the micro snapshot. The page (#15) renders it as its own self-relative, segmented series.

Metric: mean, not sum (updated after review)

The first version injected the summed per-test time. Summing conflates per-test speed with how many tests pass: a conformance win (more passing tests add their durations) reads as a slowdown on the trend (#26). The mean removes that count-inflation, and the set_hash tag carries the exact contributing set so a residual composition shift the mean can't cancel stays visible to the page. set_hash is soft — absent until #24 lands, in which case the entry simply omits it. A fixed-reference-set is the deeper follow-up tracked in #26; this is the near-term mitigation.

[mparrett]

[P1] Keep the measured Test262 set stable across snapshots

The workflow records test262.total by summing only the tests that pass in that particular run. That means the measured corpus changes whenever correctness changes: fixing tests adds their runtime and can look like a performance regression, while newly failing tests disappear from the sum and can look like a speedup. Recording the passing count provides useful provenance, but it does not make the points comparable.

Please use a fixed benchmark allowlist, or retain enough per-test timing data for the page to compare a common intersection across snapshots. At minimum, points with different contributing sets should not be presented as one continuous performance series.

The workflow otherwise passes actionlint; I did not run the full Test262 macro.

[mparrett]

Tracked in #26, which consolidates this into a standalone issue (survives this PR closing) and adds the set_hash relationship from #24 and the option analysis (fixed reference set vs. per-test mean vs. segment-and-caveat). The "different contributing sets should not be one continuous series" point is also captured as a rendering follow-on in the #23 tracker.

[nooga]

Overview

This adds one gated step-pair to .github/workflows/perf-timeline.yml: a test262 corpus cache and a step that runs scripts/macro-test262.sh, extracts the total record's ns_per_op/iterations, anchor-normalizes it, and splices it into the snapshot JSON as benchmarks["test262.total"] before publish.

Opt-in gating

Correctly implemented: both new steps share the same gating condition, so ungated pushes see zero behavior change, and continue-on-error: true correctly isolates a macro/jq failure from the primary snapshot.

One nit: workflow_dispatch unconditionally runs the macro, even though the file's existing comment describes workflow_dispatch as a backfill mechanism for historical micro snapshots. Every manual backfill dispatch now silently costs an extra ~4 minutes with no opt-out. Consider a workflow_dispatch boolean input (default false) instead.

test262.total comparability vs. #24

This is the substantive issue. #24 ("fingerprint the contributing test set so test262.total is comparable") explicitly states that #16 should gate its test262.total series on set_hash — #24 provides the tool, the page-side drop/flag is #16's job. Since #24 isn't merged yet and this PR doesn't reference it, merging #16 as-is emits unfingerprinted test262.total points onto the durable append-only timeline that will need special-casing (or discarding) once #24 lands.

Recommend either:

1. Sequencing #24 before #16 and threading set_hash through this step's jq splice, or
2. Landing #16 now but adding an explicit code comment/TODO referencing #24, so pre-fingerprint points are recognizably legacy/lower-confidence rather than silently indistinguishable later.

Minor

• jq -s '.[] | select(...) | .ns_per_op' slurps the whole file into an array only to re-stream it — jq 'select(.name=="total") | .ns_per_op' (no -s) is equivalent and simpler for JSONL input.
• No test coverage added (understandable for a CI-workflow change), but worth a manual workflow_dispatch dry run before merge to confirm the jq splice against a real snapshot shape.

[mparrett]

Addressed the review:

• test262.total is now the per-test mean (summed_ns / passing_count) rather than the sum, and carries a set_hash of the contributing set (from bench-test262: fingerprint the contributing test set so test262.total is comparable #24) — so a conformance change no longer reads as a perf move, and a residual composition shift stays visible.
• workflow_dispatch takes a test262 boolean (default false) so backfill dispatches opt out of the ~4-min macro.
• Dropped the unnecessary jq -s slurp on the JSONL extractions; added an empty-passing-set guard.

Validated end-to-end via a workflow_dispatch on my fork: the macro gated on correctly, and the published snapshot carried ns_per_op = mean with a real set_hash. The deeper fixed-reference-set metric stays tracked separately in #26; this is the near-term fix. Could you take another look when you get a chance? Thanks!

[nooga]

Thanks Matt — nice touch making this opt-in and continue-on-error, and switching to mean (vs sum) so the metric can't conflate conformance gains with perf regressions. Merging.