Summary
On the latest available AKS Ubuntu 24.04 GPU node image, Standard_NV36ads_A10_v5 nodes come up with an inconsistent NVIDIA vGPU driver: the aks-gpu layer stages the 580.159.04 user-space libraries and writes a dkms-marker claiming 580.159.04, but the VHD only contains the 570.211.01 kernel module and DKMS source. nvidia-smi therefore fails with Driver/library version mismatch on a fresh node with no GPU Operator, device plugin, or driver DaemonSet present.
This is reproducible on fresh nodes and is not fixable by the customer, because this SKU requires the AKS-managed licensed vGPU (GRID) driver, installing our own driver via --gpu-driver none + GPU Operator is not a viable workaround for this VM family.
Tried with removing GPU Operator entirely and trying to bring node back up, doesn't work.
Node image / environment
- Node image:
AKSUbuntu-2404gen2containerd-202607.02.0
az aks nodepool get-upgrades --query latestNodeImageVersion returns the same
version → this is the newest image available; there is nothing to upgrade to.- VM size:
Standard_NV36ads_A10_v5
- OS SKU:
Ubuntu 24.04
- AKS version:
v1.35.5
- Kernel:
6.8.0-1059-azure
- GPU Operator / device plugin / driver DaemonSet: none (deleted entirely to
isolate; issue persists on stock nodes)
- Region: East US
What's on the node
uname -r
6.8.0-1059-azure
nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.159
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 570.211.01 Tue Nov 25 16:36:36 UTC 2025
GCC version: gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)
cat /opt/azure/aks-gpu/dkms-marker
kernel=6.8.0-1059-azure
driver_version=580.159.04
driver_kind=cuda
arch=x86_64
find /lib/modules/$(uname -r) -name 'nvidia.ko*' -exec modinfo -F version {} ;
570.211.01
dkms status
nvidia/570.211.01, 6.8.0-1059-azure, x86_64: installed
ls -la /usr/lib/x86_64-linux-gnu/libnvidia-ml.so*
# both versions present
libnvidia-ml.so.570.211.01
libnvidia-ml.so.580.159.04
libnvidia-ml.so.1 -> libnvidia-ml.so.580.159.04
dpkg -S /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.580.159.04
dpkg-query: no path found matching pattern ... # staged by aks-gpu, not apt
To Reproduce
- Create AKS cluster with version 1.35.5
- Create NV36adsA10_v5 node pool with GPU driver installed by Azure
- Launch a privileged debug pod on the node and chroot to host, see driver mismatch with the commands above
Expected behavior
Pre-installed driver should be consistent
Summary
On the latest available AKS Ubuntu 24.04 GPU node image,
Standard_NV36ads_A10_v5nodes come up with an inconsistent NVIDIA vGPU driver: the aks-gpu layer stages the 580.159.04 user-space libraries and writes a dkms-marker claiming 580.159.04, but the VHD only contains the 570.211.01 kernel module and DKMS source.nvidia-smitherefore fails withDriver/library version mismatchon a fresh node with no GPU Operator, device plugin, or driver DaemonSet present.This is reproducible on fresh nodes and is not fixable by the customer, because this SKU requires the AKS-managed licensed vGPU (GRID) driver, installing our own driver via
--gpu-driver none+ GPU Operator is not a viable workaround for this VM family.Tried with removing GPU Operator entirely and trying to bring node back up, doesn't work.
Node image / environment
AKSUbuntu-2404gen2containerd-202607.02.0az aks nodepool get-upgrades --query latestNodeImageVersionreturns the sameversion → this is the newest image available; there is nothing to upgrade to.
Standard_NV36ads_A10_v5Ubuntu 24.04v1.35.56.8.0-1059-azureisolate; issue persists on stock nodes)
What's on the node
To Reproduce
Expected behavior
Pre-installed driver should be consistent