From aeda5231d1e84f6dc8a7c48e63d5c675692552cf Mon Sep 17 00:00:00 2001 From: mmelnich Date: Wed, 12 Aug 2026 12:23:21 -0700 Subject: [PATCH 1/2] TEMPORARY: quarantine the macOS Accelerate gesdd test so core-macos can go green REVERT THIS ONCE MARK HAS REVIEWED THE BLAS++ / LAPACK++ ACCELERATE WORK. TestQB.Polynomial_Decay_general1 fails on Apple Silicon because Apple's old Accelerate LAPACK has a broken divide-and-conquer gesdd. It was left failing on purpose, as a canary for the planned migration to the new Accelerate interface. The problem with that arrangement is that core-macos has been permanently red, across build and build-asan, on main and on every branch. A lane that is always red carries no information: nobody can tell a new macOS regression from the known one, so in practice the job is ignored and the canary signals to no one. This quarantines the test rather than deleting the signal: - The main ctest run excludes it, so the job reflects real macOS health. - The test is then run separately and cannot fail the job. - Every run prints a warning that the suppression is active, naming the condition for removing it. - If the test ever PASSES, the job prints a louder warning saying to delete the quarantine. That is the migration signal the old always-red setup was meant to give, except it now arrives as a specific instruction instead of a colour change nobody was watching. Deliberately surgical: nothing else in these files is touched, so reverting is a clean revert of this one commit. --- .github/workflows/core-macos.yaml | 44 +++++++++++++++++++++++++++++-- docs/CI.md | 24 ++++++++++++----- 2 files changed, 60 insertions(+), 8 deletions(-) diff --git a/.github/workflows/core-macos.yaml b/.github/workflows/core-macos.yaml index 4092be8c..6ec0f90f 100644 --- a/.github/workflows/core-macos.yaml +++ b/.github/workflows/core-macos.yaml @@ -86,7 +86,27 @@ jobs: `pwd`/../RandLAPACK make -j$(sysctl -n hw.ncpu) make -j$(sysctl -n hw.ncpu) install - ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability" --output-on-failure + ##################################################################### + ## TEMPORARY -- REVERT THIS ENTIRE BLOCK ## + ## ## + ## TestQB.Polynomial_Decay_general1 fails on Apple Silicon because ## + ## Apple's OLD Accelerate LAPACK has a broken divide-and-conquer ## + ## gesdd. It is quarantined here ONLY to stop core-macos being ## + ## permanently red, which trained everyone to ignore this job. ## + ## ## + ## REVERT AS SOON AS Mark has reviewed the BLAS++ / LAPACK++ work ## + ## that migrates to the new Accelerate interface. The test is still ## + ## RUN below, so its result is not lost: if it starts PASSING, the ## + ## job prints a loud warning telling you to delete this block. ## + ##################################################################### + ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability|^TestQB\.Polynomial_Decay_general1" \ + --output-on-failure + echo "::warning title=TEMPORARY macOS suppression is ACTIVE::TestQB.Polynomial_Decay_general1 is quarantined (Apple Accelerate gesdd bug). REVERT once Mark has reviewed the BLAS++/LAPACK++ Accelerate migration. See docs/CI.md." + if ctest --tests-regex "^TestQB\.Polynomial_Decay_general1" --output-on-failure; then + echo "::warning title=DELETE THE macOS SUPPRESSION::TestQB.Polynomial_Decay_general1 now PASSES. The Accelerate gesdd bug appears fixed -- remove the quarantine block in .github/workflows/core-macos.yaml and the note in docs/CI.md." + else + echo "::notice title=Quarantined test still failing (expected)::TestQB.Polynomial_Decay_general1 still fails, as expected while Apple's old Accelerate gesdd is in use. The suppression is still needed." + fi - name: build and test extras run: | @@ -185,4 +205,24 @@ jobs: `pwd`/../RandLAPACK make -j$(sysctl -n hw.ncpu) make -j$(sysctl -n hw.ncpu) install - ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability" --output-on-failure + ##################################################################### + ## TEMPORARY -- REVERT THIS ENTIRE BLOCK ## + ## ## + ## TestQB.Polynomial_Decay_general1 fails on Apple Silicon because ## + ## Apple's OLD Accelerate LAPACK has a broken divide-and-conquer ## + ## gesdd. It is quarantined here ONLY to stop core-macos being ## + ## permanently red, which trained everyone to ignore this job. ## + ## ## + ## REVERT AS SOON AS Mark has reviewed the BLAS++ / LAPACK++ work ## + ## that migrates to the new Accelerate interface. The test is still ## + ## RUN below, so its result is not lost: if it starts PASSING, the ## + ## job prints a loud warning telling you to delete this block. ## + ##################################################################### + ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability|^TestQB\.Polynomial_Decay_general1" \ + --output-on-failure + echo "::warning title=TEMPORARY macOS suppression is ACTIVE::TestQB.Polynomial_Decay_general1 is quarantined (Apple Accelerate gesdd bug). REVERT once Mark has reviewed the BLAS++/LAPACK++ Accelerate migration. See docs/CI.md." + if ctest --tests-regex "^TestQB\.Polynomial_Decay_general1" --output-on-failure; then + echo "::warning title=DELETE THE macOS SUPPRESSION::TestQB.Polynomial_Decay_general1 now PASSES. The Accelerate gesdd bug appears fixed -- remove the quarantine block in .github/workflows/core-macos.yaml and the note in docs/CI.md." + else + echo "::notice title=Quarantined test still failing (expected)::TestQB.Polynomial_Decay_general1 still fails, as expected while Apple's old Accelerate gesdd is in use. The suppression is still needed." + fi diff --git a/docs/CI.md b/docs/CI.md index f69adca5..a6a56aba 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -24,12 +24,24 @@ candidates once they have a green track record. ## Things that are deliberate (do not "fix" without reading this) -- **core-macos is red on purpose.** `TestQB.Polynomial_Decay_general1` fails - on Apple Silicon because Apple's default (old) Accelerate LAPACK has a - broken divide-and-conquer `gesdd`. The failure is kept as a canary: when - the planned migration to the new Accelerate interface happens, this test - flipping green is the evidence the bug is gone. A local fix exists - (reference SVD via `gesvd`) but is intentionally unmerged. +- **`TestQB.Polynomial_Decay_general1` is QUARANTINED on macOS -- THIS IS + TEMPORARY AND MUST BE REVERTED.** The test fails on Apple Silicon because + Apple's default (old) Accelerate LAPACK has a broken divide-and-conquer + `gesdd`. It was previously left failing as a canary, which made core-macos + permanently red and trained everyone to ignore the job -- a red CI lane that + is always red reports nothing. + + **Revert as soon as Mark has reviewed the BLAS++ / LAPACK++ work migrating + to the new Accelerate interface.** Delete the marked block in + `.github/workflows/core-macos.yaml` (both `build` and `build-asan`) and this + entry. + + The canary is deliberately preserved: the test is still *run*, separately, + and cannot fail the job. Every macOS run prints a warning that the + suppression is active, and if the test ever **passes** the job prints a + louder one telling you to delete the quarantine -- which is exactly the + signal the old always-red arrangement was supposed to provide. A local fix + exists (reference SVD via `gesvd`) but is intentionally unmerged. - **The RandBLAS submodule's own tests do not run here** (`-DBUILD_TESTS=OFF` in the core recipes). The pinned commit is already tested by RandBLAS's CI; rebuilding its ~450 tests in every RandLAPACK job From 31310609310db41069fb41c817eda39139c3b032 Mon Sep 17 00:00:00 2001 From: mmelnich Date: Wed, 12 Aug 2026 12:27:46 -0700 Subject: [PATCH 2/2] docs/CI.md: name the upstream Accelerate PRs that gate reverting the macOS quarantine blaspp#134 and lapackpp#88 (in that order -- #88 needs #134's defines.h), plus the stalled blaspp#74 they continue. Without the PR numbers, 'revert once the Accelerate work lands' is not actionable by anyone who was not in the room. --- docs/CI.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/CI.md b/docs/CI.md index a6a56aba..cd3bd0ab 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -31,10 +31,20 @@ candidates once they have a green track record. permanently red and trained everyone to ignore the job -- a red CI lane that is always red reports nothing. - **Revert as soon as Mark has reviewed the BLAS++ / LAPACK++ work migrating - to the new Accelerate interface.** Delete the marked block in - `.github/workflows/core-macos.yaml` (both `build` and `build-asan`) and this - entry. + **Revert once Apple's new Accelerate interface is in use**, i.e. when both + of these land (order matters -- lapackpp#88 needs the `defines.h` from + blaspp#134): + - -- New Apple Accelerate + support, a rebased and completed continuation of the stalled + + - -- Support Apple's new + Accelerate interface + + Then RandLAPACK PR #155 (retire legacy-Accelerate accommodations) becomes + mergeable, `TestQB.Polynomial_Decay_general1` should pass on its own, and + the "DELETE THE macOS SUPPRESSION" warning below will fire. Delete the + marked block in `.github/workflows/core-macos.yaml` (both `build` and + `build-asan`) and this entry. The canary is deliberately preserved: the test is still *run*, separately, and cannot fail the job. Every macOS run prints a warning that the