test(rccl): gate build time against the PR's base commit

[alex-breslow-amd]

Stacked on #9889 — review that one first. This PR's diff is a single commit; the base will retarget to develop automatically once #9889 merges.

Why

The absolute 5-minute threshold added in #9889 only means something on hardware close to what it was tuned on. A clean single-arch build is about 50s on a 256-core host but would blow the same budget on a small runner, so the number needs re-tuning per machine class.

A ratio does not have that problem. --compare-base builds the PR's merge-base and its head on the same runner and fails when the head is more than --max-regression-pct slower.

Keeping it honest

Three things stop this from being a flaky test:

• Alternating rounds. Base and head builds interleave rather than running all of one then the other, so thermal drift or a noisy neighbour cannot land on just one side.
• Minimum across rounds, not the mean. Build-time noise is one-sided — interference only ever makes a build slower — so the minimum is the better estimator. This mattered: a single round read −1.5% on a change that touches no source, while min-of-2 tightened the same comparison to −0.1%.
• ccache forced off. A warm cache would let the second build finish almost instantly and silently invalidate the comparison. RCCL's own build path does not enable ccache, but TheRock CI does, so the script sets CCACHE_DISABLE=1 rather than trusting the surrounding configuration.

Folded into the existing gate

The comparison times the head build anyway, so it also enforces the absolute budget. That replaces the precheckin entry from #9889 instead of adding a second one, so there is still a single "RCCL Build Time" suite. The two checks are independent: with an artificially small budget a run failed with took 1:30.16, over the 0:10.00 budget while its delta was a healthy −0.7%.

Measurements

All on gfx950 at -j 64, through the precheckin harness:

scenario	result
Change touching only test files	−0.1%, PASS
Eight days of develop (1:19 → 1:30)	+13.8%, FAIL
Head over the absolute budget	FAIL, delta unaffected

The four raw readings in the last passing run spanned 0.1s on 90s, and two separate harness runs both landed on −0.1%. A 10% limit therefore sits well clear of the noise on this class of runner; it could go lower later if the data holds on the real CI hardware.

Cost

Four builds instead of one, 365s measured. Because it replaces the previous 92s entry, the net addition to precheckin is roughly 4.5 minutes.

Test plan

• mypy --strict clean
• Config validates against the harness JSON schema
• Passing case through test_runner.py (365s, exit 0)
• Failing case on a real regression (+13.8%, exit 1)
• Absolute-budget failure fires independently of the delta check
• Temporary git worktree removed on both success and failure
• Build logs cleaned up on success, retained on failure
• Confirm timings and pick a final threshold on the actual CI runner, which will be noisier than the dev host used here

Made with Cursor

Coverage is scoped to the toolchain it runs under

Worth stating plainly, since it bounds what this gate can promise: it only catches regressions that reproduce under the compiler the runner uses. Precheckin builds against the local /opt/rocm install, so a regression that only manifests on another toolchain will pass here.

This is not hypothetical. #9756 fixed a ce_reduce.cc translation unit that took ~48 minutes to compile on the srock toolchain, caused by LLVM's runtime loop-unroller auto-unrolling a rank loop by up to 64x. Measured against /opt/rocm 7.0.2.1 on gfx950, the same pre-fix file compiles in 2.63s (confirmed from ninja's per-edge log on a full clean build), and this gate scores the fix at -0.0% — 90.38s before, 90.37s after. The pathological unroll simply does not happen with that clang, so there was never anything here to catch.

Two consequences for reviewers:

• A green result means "no regression on this runner's toolchain", not "no regression". Covering the srock toolchain would need the gate wired into TheRock CI, where the ccache interaction described above has to be solved first.
• Wall-clock totals can hide large shifts in compile work. On /opt/rocm, perf(rccl): split ce_reduce into per-kernel TUs to fix ~48min build time #9756 takes ce_reduce from one 2.63s job to 42 jobs totaling 30.64s of CPU — roughly 12x the aggregate work — while wall clock is unchanged (83.80s vs 83.77s) because -j 64 has slack to absorb it. That is the right trade against a 48-minute serialized job, but on a core-starved runner the same split could cost wall clock instead of saving it.