validate: mem-anchor contention normalization (no-op A/B) - #16
Closed
mparrett wants to merge 1 commit into
Closed
Conversation
Same VM code as perf/mem-anchor (base), so the repeat A/B is same-code — every gate is a false positive. Both sides carry BenchmarkRatchetMemAnchor, so the mem-normalized column is active. Comment-only change to trigger perf-pr-repeat.
Owner
Author
|
Closing — validation complete, negative result. The memory-aware anchor does not work: across 5 contended-CI samples the per-family β (family-swing regressed on anchor-swing, per cycle) is pure scatter (−0.58 to +2.66) with no co-variation, and full ÷mem normalization injects the anchor's own volatility (worst |median| 4.09% raw → 13.23% ÷mem on the run where the anchor swung ±14%). Root cause is structural: The N=7 median-of-N + ABBA gate holds 0 false positives across 10 runs without it. Branches kept as the shelved-prototype record. |
mparrett
added a commit
that referenced
this pull request
Jul 13, 2026
test262.total is a mean over the passing set (perf-timeline #16), whose membership drifts as conformance changes. Where a point's set_hash differs from its predecessor the mean isn't over the same workload, so the slope across that gap isn't a pure engine delta (nooga#26, the nooga#23 segmentation follow-on). Dash the connecting span, mark the changed point with a triangle, add a tooltip + count note — so a composition jump reads differently from a real engine move. Also fixes the ?mode= param regression from the tab rename: match case-insensitively so old ?mode=all / ?mode=overall links still resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mparrett
added a commit
that referenced
this pull request
Jul 18, 2026
Replace the "#16 should gate on set_hash" note (an exact-match gate degenerates on the timeline — the passing set drifts on nearly every point) with: segment the line at set_hash changes, keep the hash as a diagnostic. Add the sum-vs- conformance metric confound as follow-on nooga#26. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mparrett
added a commit
that referenced
this pull request
Jul 25, 2026
test262.total is a mean over the passing set (perf-timeline #16), whose membership drifts as conformance changes. Where a point's set_hash differs from its predecessor the mean isn't over the same workload, so the slope across that gap isn't a pure engine delta (nooga#26, the nooga#23 segmentation follow-on). Dash the connecting span, mark the changed point with a triangle, add a tooltip + count note — so a composition jump reads differently from a real engine move. Also fixes the ?mode= param regression from the tab rename: match case-insensitively so old ?mode=all / ?mode=overall links still resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork-only validation of the memory-aware anchor (stacked on the median-of-N gate #14).
Setup. Base branch
perf/mem-anchoralready carriesBenchmarkRatchetMemAnchor+ the gate driver's mem-normalization, so both base and head snapshots have the mem anchor to normalize against. The head is a comment-only no-op → same VM code both sides → every gate is a false positive (must be 0). Theperf-pr-repeatgate reports raw (÷cpu anchor) alongside mem-normalized (÷mem anchor) deltas.What to look for. Under real CI memory contention, the raw worst-family |median| should exceed the mem-normalized one — the phantom the CPU anchor can't normalize. Deterministic proof of the mechanism (a +20% raw phantom → 0% mem-normalized) is in the driver's synthetic differential; this run measures it on a contended runner.
Not for merge — validation only. Landing the benchmark on main is gated on generalizing bench-ratchet's anchor detection to a predicate (coordinated, sequenced on nooga#22).