Add multi-architecture CUDA container build - #866
Open
Sbozzolo wants to merge 21 commits into
Open
Conversation
Sbozzolo
added
ci
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
from
August 5, 2026 08:10
253c899 to
39a50ea
Compare
Sbozzolo
added
the
no-long-tests
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
2 times, most recently
from
August 5, 2026 10:00
b25dfa2 to
756a5e9
Compare
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
2 times, most recently
from
August 5, 2026 13:02
79a0279 to
8064aa7
Compare
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
from
August 5, 2026 15:47
10474f4 to
de5f86c
Compare
Sbozzolo
marked this pull request as ready for review
August 6, 2026 16:12
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
from
August 6, 2026 16:13
92f3086 to
dde4558
Compare
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
2 times, most recently
from
August 10, 2026 07:42
16bfb13 to
1013b53
Compare
Re-apply the review fixes not already present on the branch: - setup-runner: use ln -sf so a dirty runner does not abort on an existing spack_repo/local/packages link. - PalaceDockerfile: prune dangling Spack view symlinks on the CPU path too (was CUDA-only), and guard best-effort CUDA cleanup with || true so a successful dependency install is not turned into a hard failure. - ExternalSUNDIALS: guard the CUDA architecture forward with the same empty-list check the other CUDA dependencies already use. - palace package.py: normalize the mfem 4.9 patch guards from @:4.9 to @4.9.0 so a future 4.9.1 is not silently mishandled. - Factor the duplicated buildcache index-refresh job into a refresh-buildcache-index composite action used by spack.yml and lookahead.yml.
The escape-and-append of CMAKE_CUDA_ARCHITECTURES to each external project's options was copy-pasted across six External*.cmake files. Add palace_append_cuda_architectures() to ArchitectureHelpers and route HYPRE, MAGMA, MFEM, Palace, SUNDIALS, and Umpire through it. The helper is a no-op when no architectures are set, so callers invoke it unconditionally inside their CUDA branch. Covered by the MultiArchitecture cmake test.
The docs build synthesizes its own Spack environment and only linked the builtin mfem package into spack_repo/local, but package.py now also references libceed there, so 'Configure External Packages' failed with 'Package spack_repo.local.packages.libceed.package not found'. Link libceed too, mirroring setup-runner, and use ln -sf so a dirty runner does not abort on an existing link.
update-index intermittently fails with a 403 on a manifest GET from GHCR. Retry up to four times with exponential backoff (15s/30s/60s) before failing. Because both spack.yml and lookahead.yml now share this composite action, the retry covers both.
The index refresh is non-critical: it self-heals on the next successful run, and binaries stay reachable via Spack's direct OCI-tag fallback in the meantime. After exhausting the retries for a transient GHCR 403, warn and exit 0 instead of failing the job, so this step never blocks CI.
Make spack.yaml the single source of truth for the CUDA fat-binary architectures (cuda_arch=80,86,89,90 in cuda_constraints) instead of templating them in from the action. Remove the now-redundant Configure CUDA architectures sed step and the cuda.architectures OCI label. The cuda_arch action input is retained purely as the CPU-vs-CUDA profile toggle (empty builds CPU-only, non-empty selects the CUDA profile via PALACE_CUDA_ARCH); its value no longer carries the architecture list.
Two review items were lost when the branch was reset onto the remote tip; restore them: - palace_parse_cuda_architecture: accept the arch-conditional feature suffixes nvcc uses for Hopper/Blackwell (90a, 100f) and document why native/all/all-major are rejected. Covered by a new 90a parse test. - package.py: note that mfem CUDA multi-arch support comes from the builtin-recipe patch pr5895_mfem_multi_cuda_arch.diff, not from the source patches listed here.
The container CUDA architectures are fixed in spack.yaml, so the action no longer needs a cuda_arch input to carry them. Replace it with a Detect CUDA build step that runs nvidia-smi: runners with an NVIDIA GPU build the CUDA profile, everything else builds CPU-only. The result is exported as PALACE_CUDA via $GITHUB_ENV and threaded through the image name, the podman build-arg, the Dockerfile ARG/ENV, and spack.yaml's profile selection (renamed from PALACE_CUDA_ARCH). Drop the now-unused cuda_arch column from the containers.yml build matrices.
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
from
August 10, 2026 08:00
1013b53 to
1bbcdc3
Compare
Sbozzolo
force-pushed
the
gbozzola/container-cuda
branch
from
August 10, 2026 08:23
1bbcdc3 to
cb27547
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a workflow to compile CUDA-enabled containers.
Containers are built for x86_64_v3 and various CUDA architectures in a single executable. I could not build for targets newer than sm_90 because the job would take forever (probably the same problem as mfem/mfem#5363).
CUDA containers are huge (> 4 GB), so I spent considerable effort slimming them down. This is also needed to make CI faster by reducing upload time.
I upstreamed all the PRs: