Skip to content

TEMPORARY: quarantine the macOS Accelerate gesdd test (revert after Mark's BLAS++/LAPACK++ review) #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mmelnich merged 2 commits into from
Aug 12, 2026
Merged

TEMPORARY: quarantine the macOS Accelerate gesdd test (revert after Mark's BLAS++/LAPACK++ review) #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aeda523
TEMPORARY: quarantine the macOS Accelerate gesdd test so core-macos c…
mmelnich Aug 12, 2026
3131060
docs/CI.md: name the upstream Accelerate PRs that gate reverting the …
mmelnich Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 42 additions & 2 deletions .github/workflows/core-macos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,27 @@ jobs:
`pwd`/../RandLAPACK
make -j$(sysctl -n hw.ncpu)
make -j$(sysctl -n hw.ncpu) install
ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability" --output-on-failure
#####################################################################
## TEMPORARY -- REVERT THIS ENTIRE BLOCK ##
## ##
## TestQB.Polynomial_Decay_general1 fails on Apple Silicon because ##
## Apple's OLD Accelerate LAPACK has a broken divide-and-conquer ##
## gesdd. It is quarantined here ONLY to stop core-macos being ##
## permanently red, which trained everyone to ignore this job. ##
## ##
## REVERT AS SOON AS Mark has reviewed the BLAS++ / LAPACK++ work ##
## that migrates to the new Accelerate interface. The test is still ##
## RUN below, so its result is not lost: if it starts PASSING, the ##
## job prints a loud warning telling you to delete this block. ##
#####################################################################
ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability|^TestQB\.Polynomial_Decay_general1" \
--output-on-failure
echo "::warning title=TEMPORARY macOS suppression is ACTIVE::TestQB.Polynomial_Decay_general1 is quarantined (Apple Accelerate gesdd bug). REVERT once Mark has reviewed the BLAS++/LAPACK++ Accelerate migration. See docs/CI.md."
if ctest --tests-regex "^TestQB\.Polynomial_Decay_general1" --output-on-failure; then
echo "::warning title=DELETE THE macOS SUPPRESSION::TestQB.Polynomial_Decay_general1 now PASSES. The Accelerate gesdd bug appears fixed -- remove the quarantine block in .github/workflows/core-macos.yaml and the note in docs/CI.md."
else
echo "::notice title=Quarantined test still failing (expected)::TestQB.Polynomial_Decay_general1 still fails, as expected while Apple's old Accelerate gesdd is in use. The suppression is still needed."
fi

- name: build and test extras
run: |
Expand Down Expand Up @@ -185,4 +205,24 @@ jobs:
`pwd`/../RandLAPACK
make -j$(sysctl -n hw.ncpu)
make -j$(sysctl -n hw.ncpu) install
ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability" --output-on-failure
#####################################################################
## TEMPORARY -- REVERT THIS ENTIRE BLOCK ##
## ##
## TestQB.Polynomial_Decay_general1 fails on Apple Silicon because ##
## Apple's OLD Accelerate LAPACK has a broken divide-and-conquer ##
## gesdd. It is quarantined here ONLY to stop core-macos being ##
## permanently red, which trained everyone to ignore this job. ##
## ##
## REVERT AS SOON AS Mark has reviewed the BLAS++ / LAPACK++ work ##
## that migrates to the new Accelerate interface. The test is still ##
## RUN below, so its result is not lost: if it starts PASSING, the ##
## job prints a loud warning telling you to delete this block. ##
#####################################################################
ctest --exclude-regex "^TestABRIK\.ABRIK_catch_instability|^TestQB\.Polynomial_Decay_general1" \
--output-on-failure
echo "::warning title=TEMPORARY macOS suppression is ACTIVE::TestQB.Polynomial_Decay_general1 is quarantined (Apple Accelerate gesdd bug). REVERT once Mark has reviewed the BLAS++/LAPACK++ Accelerate migration. See docs/CI.md."
if ctest --tests-regex "^TestQB\.Polynomial_Decay_general1" --output-on-failure; then
echo "::warning title=DELETE THE macOS SUPPRESSION::TestQB.Polynomial_Decay_general1 now PASSES. The Accelerate gesdd bug appears fixed -- remove the quarantine block in .github/workflows/core-macos.yaml and the note in docs/CI.md."
else
echo "::notice title=Quarantined test still failing (expected)::TestQB.Polynomial_Decay_general1 still fails, as expected while Apple's old Accelerate gesdd is in use. The suppression is still needed."
fi
34 changes: 28 additions & 6 deletions docs/CI.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,34 @@ candidates once they have a green track record.

## Things that are deliberate (do not "fix" without reading this)

- **core-macos is red on purpose.** `TestQB.Polynomial_Decay_general1` fails
on Apple Silicon because Apple's default (old) Accelerate LAPACK has a
broken divide-and-conquer `gesdd`. The failure is kept as a canary: when
the planned migration to the new Accelerate interface happens, this test
flipping green is the evidence the bug is gone. A local fix exists
(reference SVD via `gesvd`) but is intentionally unmerged.
- **`TestQB.Polynomial_Decay_general1` is QUARANTINED on macOS -- THIS IS
TEMPORARY AND MUST BE REVERTED.** The test fails on Apple Silicon because
Apple's default (old) Accelerate LAPACK has a broken divide-and-conquer
`gesdd`. It was previously left failing as a canary, which made core-macos
permanently red and trained everyone to ignore the job -- a red CI lane that
is always red reports nothing.

**Revert once Apple's new Accelerate interface is in use**, i.e. when both
of these land (order matters -- lapackpp#88 needs the `defines.h` from
blaspp#134):
- <https://github.com/icl-utk-edu/blaspp/pull/134> -- New Apple Accelerate
support, a rebased and completed continuation of the stalled
<https://github.com/icl-utk-edu/blaspp/pull/74>
- <https://github.com/icl-utk-edu/lapackpp/pull/88> -- Support Apple's new
Accelerate interface

Then RandLAPACK PR #155 (retire legacy-Accelerate accommodations) becomes
mergeable, `TestQB.Polynomial_Decay_general1` should pass on its own, and
the "DELETE THE macOS SUPPRESSION" warning below will fire. Delete the
marked block in `.github/workflows/core-macos.yaml` (both `build` and
`build-asan`) and this entry.

The canary is deliberately preserved: the test is still *run*, separately,
and cannot fail the job. Every macOS run prints a warning that the
suppression is active, and if the test ever **passes** the job prints a
louder one telling you to delete the quarantine -- which is exactly the
signal the old always-red arrangement was supposed to provide. A local fix
exists (reference SVD via `gesvd`) but is intentionally unmerged.
- **The RandBLAS submodule's own tests do not run here**
(`-DBUILD_TESTS=OFF` in the core recipes). The pinned commit is already
tested by RandBLAS's CI; rebuilding its ~450 tests in every RandLAPACK job
Expand Down
Loading