PC-023 — GPU vendor lock-in (CUDA-only build)

[itigges22]

Status (at mirror time): Open (tracking) • Severity: High

One-liner: Build hard-codes NVIDIA/CUDA — AMD/Intel/CPU-only users can't install. Match llama.cpp's full backend matrix

---

• Status: Open (tracking — deferred to roadmap; not blocking Phase 1)
• Severity: High (blocks every non-NVIDIA user from installing)
• Location: inference/Dockerfile.v31, docker-compose.yml,
  scripts/install.sh

The current build hard-codes NVIDIA/CUDA in three places:

1. Dockerfile.v31:1 — base image is
   nvidia/cuda:12.9.0-devel-rockylinux9.
2. Dockerfile.v31:25 — cmake is invoked with -DGGML_CUDA=ON and
   no other backend.
3. docker-compose.yml:28 — GPU resource reservation specifies
   driver: nvidia.

llama.cpp itself supports a wide backend matrix — CUDA, ROCm/HIP
(AMD), Vulkan (universal), SYCL (Intel), Metal (Apple), and a CPU-only
fallback. ATLAS only exposes one. AMD users currently can't install
the stack at all.

geometric-lens runs PyTorch on CPU
(geometric_lens/service.py:60), so only the llama-server side needs
the abstraction. The Confidence Router, V3 pipeline, Pattern Cache,
proxy, and sandbox are all vendor-agnostic.

Fix

Two reasonable directions; pragmatic combo recommended:

• (a) Vulkan as the universal path. One Dockerfile, works on
  NVIDIA + AMD + Intel + CPU. Slower than native CUDA/ROCm (typically
  20–40% perf hit) but the operational simplicity is huge — no
  per-vendor matrix, no driver detection logic. Best fit for ATLAS's
  "easy to install" goal.
• (b) Per-vendor Dockerfiles. Dockerfile.v31 (CUDA),
  Dockerfile.v31.rocm (AMD), Dockerfile.v31.vulkan (universal),
  Dockerfile.v31.cpu (no GPU). Compose picks one via
  ATLAS_GPU_VENDOR=nvidia|amd|vulkan|cpu. Best raw performance per
  vendor; more surface area to maintain.

The pragmatic combo: ship (a) first to unbreak non-NVIDIA users,
then add (b) for users who want native-vendor speed. Vulkan stays
as the universal fallback.

Cross-references:

• Phase 0 hardware-tier presets roadmap item — naturally
  encompasses per-vendor presets once the Dockerfile matrix exists.
• Phase 0 atlas-bootstrap.sh installer — needs to detect GPU
  vendor and pick the right image variant; currently assumes NVIDIA.
• Phase 0 prebuilt GHCR images — multi-arch + multi-vendor tag
  matrix once vendor abstraction lands (e.g.,
  ghcr.io/.../atlas-llama-server:v3.1-cuda,
  :v3.1-rocm, :v3.1-vulkan).
• GitHub issues this resolves (close as covered by PC-023):
  • feat: ROCm support for AMD GPUs #26 — ROCm support for AMD GPUs (direct match for path (b)
    Dockerfile.v31.rocm)
  • feat: Intel oneAPI / SYCL backend support #27 — Intel oneAPI / SYCL backend (one of the universal-(a)
    backends + a per-vendor (b) image option)
  • feat: Apple Silicon (Metal) deployment guide #32 — Apple Silicon (Metal) deployment guide. Note: Apple
    Silicon doesn't run our docker stack natively (no nvidia/cuda
    base, no Linux containers with GPU passthrough on Mac). Either
    a separate Mac-native install path (brew + raw llama.cpp Metal
    binary) or wait until vLLM/MLX backend lands via PC-024. Track
    here so the GPU-vendor work is aware of it.
  • DirectMLSupport #7 — DirectML (Windows). Vulkan path (a) covers this for
    AMD-on-Windows and Intel-on-Windows. NVIDIA-on-Windows users
    just use the CUDA path. Pure-DirectML support would be its own
    backend; defer until there's demand.

Docs to update in the same change:

• docs/SETUP.md — vendor selection step in the install flow,
  per-vendor prerequisite blocks (NVIDIA Container Toolkit vs ROCm
  vs Vulkan ICD vs CPU-only).
• docs/ARCHITECTURE.md — the llama-server box becomes "llama-server
  (vendor-pluggable backend)" with a list of supported backends.
• docs/CONFIGURATION.md — document ATLAS_GPU_VENDOR and any
  per-vendor env vars (e.g., AMD's HSA_OVERRIDE_GFX_VERSION).
• docs/TROUBLESHOOTING.md — vendor-specific failure-mode entries
  (AMD ROCm version mismatches, Vulkan ICD missing, etc.).
• README.md — system requirements section.

---

---

Mirrored from local ISSUES.md. ISSUES.md is gitignored; this issue is the canonical public record. PC-### numbering preserved for cross-references.

[itigges22]

Status — V3.1.1 starts unwinding this. AMD ROCm via the llama.cpp ROCm backend and Apple Metal (once macOS lands) are explicitly on the V3.1.1 roadmap. Each specific backend has its own ticket: #26 ROCm, #32 Apple Metal, #27 Intel oneAPI / SYCL, #7 DirectML.

Architecturally V3.1.0 has not yet untangled CUDA from the stack — inference/Dockerfile.v31, docker-compose.yml, and the bootstrap installer still all assume NVIDIA. The non-inference services (proxy, v3-service, geometric-lens, sandbox) are already CPU-only, so the V3.1.1 work scope is the inference container plus the install path. Leaving this open as the umbrella; close once #26 / #32 land.

[itigges22]

partial close-out — ROCm shipped in V3.1.1 (commit 052f18c). that's option (b) from the issue body, just for AMD instead of via Vulkan first.

what we have now:

• Dockerfile.rocm (gfx1100/1101/1102/1030/90a — covers RDNA3, RDNA2, and CDNA2)
• entrypoint-rocm.sh with the equivalent flags
• docker-compose.rocm.yml overlay that swaps the GPU resource block
• atlas init wizard detects AMD GPUs and writes ATLAS_BACKEND=rocm automatically
• atlas doctor has a ROCm check path

so the NVIDIA-only blocker is gone for AMD users. still on the matrix:

• Intel SYCL — tracked separately at feat: Intel oneAPI / SYCL backend support #27
• Apple Metal — tracked separately at feat: Apple Silicon (Metal) deployment guide #32
• Vulkan universal fallback — not yet built (option (a) from this issue)
• CPU-only — falls out of the Vulkan path if/when we ship it

keeping this open as the umbrella tracker until at least one more backend lands (Vulkan would be the highest-leverage next add since it covers Intel + the CPU-only case in one Dockerfile).

[itigges22]

progress checkpoint... substantial movement on this umbrella since v3.1.0 shipped NVIDIA-only:

shipped:

• AMD ROCm via feat: ROCm support for AMD GPUs #26... Dockerfile.rocm + docker-compose.rocm.yml + entrypoint dispatch + atlas init AMD detection + atlas doctor rocm check. fat build covers gfx1100/1101/1102/1030/90a. @bscubed is testing on a 7900 XTX, waiting on hardware validation before feat: ROCm support for AMD GPUs #26 closes.
• Vulkan universal via feat: Vulkan universal backend (one image, covers NVIDIA + AMD + Intel + Apple + CPU) #114 (f88a717 + acae180)... covers NVIDIA + AMD + Intel + Apple + CPU through one image, using Mesa drivers (RADV/ANV/lavapipe) on linux and MoltenVK on macos. slower than vendor-native but works as the "atlas runs on your GPU regardless of vendor" fallback. open until non-NVIDIA hardware coverage gets validated.
• multi-backend plumbing: ATLAS_BACKEND env var (cuda/rocm/vulkan), atlas init --backend [flag], tier.vulkan_available() + tier.rocm_available(), vendor-specific compose overlays, doctor checks per backend, docs in SETUP.md + CONFIGURATION.md.

still split off as child tickets:

• feat: Intel oneAPI / SYCL backend support #27 — Intel SYCL native (vulkan covers Intel today via ANV, but native sycl unlocks the AMX/XMX matrix engines on Arc/Lunar Lake/Battlemage for the perf-sensitive path)
• feat: Apple Silicon (Metal) deployment guide #32 — Apple Silicon native install (MoltenVK-in-docker is the slow fallback... fast path needs brew + uv + metal-cpp build that skips docker entirely)
• feat: ARM64 multi-arch builds (DGX Spark, Snapdragon X Elite, Apple Silicon, Jetson, Pi 5) #115 — ARM64 multi-arch builds (foundation that unblocks DGX Spark, Snapdragon X Elite, Apple Silicon containers, Jetson, Pi 5)

keeping this open as the tracking ticket until the three children land, but the "atlas only runs on NVIDIA" framing is no longer accurate as of f88a717. anyone reading this issue: pick your platform from the child tickets.

[itigges22]

third child closed... #32 (Apple Silicon Metal) shipped + validated on a 32GB M2 Pro today.

umbrella status:

closed:

• feat: ROCm support for AMD GPUs #26 ROCm (V3.1.1, @bscubed validated on RX 7900 XTX)
• feat: Apple Silicon (Metal) deployment guide #32 Apple Silicon native install via Metal hybrid (V3.1.2, validated on M2 Pro)
• feat: Vulkan universal backend (one image, covers NVIDIA + AMD + Intel + Apple + CPU) #114 Vulkan universal backend (V3.1.2, NVIDIA-only validated so far... still tracks non-NVIDIA hardware coverage)

still split off:

• feat: Intel oneAPI / SYCL backend support #27 Intel SYCL native (vulkan covers Intel today via Mesa ANV, native sycl unlocks Arc/Lunar Lake/Battlemage perf)
• feat: ARM64 multi-arch builds (DGX Spark, Snapdragon X Elite, Apple Silicon, Jetson, Pi 5) #115 ARM64 multi-arch (foundation shipped, validated on M2 Pro for the Apple Silicon path... still waiting on DGX Spark / Jetson / Snapdragon / Pi 5 hardware reports for the linux arm64 paths)

the original "atlas only runs on NVIDIA x86_64" framing from when this ticket was filed isn't true anymore... atlas init on any of (NVIDIA, AMD x86_64, Apple Silicon, Intel via Vulkan) writes a working .env. AMD on arm64 still has no native path because rocm has no arm64 release... vulkan via Mesa RADV is the fallback there.

keeping open until #27 + #115 close, but the heavy lifting is done.