Migrate to use libnvml to check nvlink status directly (BugFix) by hanhsuan · Pull Request #2634 · canonical/checkbox

hanhsuan · 2026-07-01T08:18:44Z

Description

With nvidia-driver-580, the output of nvidia-smi nvlink --status was empty. However, with nvidia-driver-595, nvidia-smi nvlink --status now returns something, which broke the prime_offload_test.py and we don't want to depend on the output from nvidia-smi nvlink -s anymore. Therefore, migrating to use ctype would be a easier way to skip to add more dependencies into checkbox.
At the same time, add __GLX_VENDOR_LIBRARY_NAME=mesa for all non-nvidia to make sure the default is set to the right one.

Resolved issues

#2560

Documentation

Tests

26.04 (I+N): https://certification.canonical.com/hardware/202001-27667/submission/498212/
24.04 (I+N): https://certification.canonical.com/hardware/202001-27667/submission/498219/
22.04 run 24.04 test plan (I+N) : https://certification.canonical.com/hardware/202001-27667/submission/498211/
22.04 with NVLINK: https://certification.canonical.com/hardware/202303-31329/submission/498220/
24.04 (A+N): https://certification.canonical.com/hardware/202603-38521/submission/498366/

codecov · 2026-07-01T08:21:39Z

Codecov Report

❌ Patch coverage is 94.33962% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.88%. Comparing base (24ae8db) to head (48908f0).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
providers/base/bin/prime_offload_tester.py	94.33%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2634      +/-   ##
==========================================
+ Coverage   59.84%   59.88%   +0.03%     
==========================================
  Files         485      485              
  Lines       48752    48801      +49     
  Branches     8732     8739       +7     
==========================================
+ Hits        29177    29223      +46     
- Misses      18658    18659       +1     
- Partials      917      919       +2

Flag	Coverage Δ
provider-base	`36.15% <94.33%> (+0.16%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

2. set __GLX_VENDOR_LIBRARY_NAME=mesa for non NV card

Copilot

Pull request overview

This PR updates the providers/base prime offload helper to stop relying on nvidia-smi nvlink --status output and instead query NVLink status directly via NVML (loaded with ctypes). It also adjusts the GLX vendor selection for non-NVIDIA offload runs to ensure Mesa is used by default, aligning with the reported offload failure scenario.

Changes:

Add PrimeOffloader.check_nv_link_status() using ctypes.CDLL("libnvidia-ml.so.1") to detect active NVLink via NVML APIs.
Update check_nv_offload_env() to use NVML-based NVLink detection rather than parsing nvidia-smi output.
Set __GLX_VENDOR_LIBRARY_NAME=mesa for non-NVIDIA offload environments and add unit tests covering NVML/NVLink detection and updated behavior.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

File	Description
providers/base/bin/prime_offload_tester.py	Adds NVML/ctypes NVLink detection, switches NVLink gating logic to use it, and sets Mesa GLX vendor for non-NVIDIA offload.
providers/base/tests/test_prime_offload_tester.py	Adds unit tests for NVML-based NVLink detection and updates NVLink-related environment checks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tomli380576

lgtm the test results look ok

tomli380576 · 2026-07-03T02:29:37Z

+            # Define NVML return codes
+            NVML_SUCCESS = 0
+            NVML_ERROR_NOT_SUPPORTED = 3


source for future reviewers: https://github.com/NVIDIA/nvidia-settings/blob/5b341a9e54f08ac324e148e1e7e102030e42b1c4/src/nvml.h#L1293

tomli380576 · 2026-07-03T02:30:43Z

+            ret = nvmlInit()
+            if ret != NVML_SUCCESS:
+                self.logger.info("NVML initialization failed")
+                return False


source for future reviewers: https://github.com/NVIDIA/nvidia-settings/blob/5b341a9e54f08ac324e148e1e7e102030e42b1c4/src/nvml.h#L3974-L4000

tomli380576 · 2026-07-03T02:44:46Z

+            nvmlDeviceGetHandleByIndex = nvml.nvmlDeviceGetHandleByIndex_v2
+            nvmlDeviceGetHandleByIndex.argtypes = [
+                ctypes.c_uint,
+                ctypes.POINTER(ctypes.c_void_p),
+            ]
+            nvmlDeviceGetHandleByIndex.restype = ctypes.c_int


source for future reviewers: https://github.com/NVIDIA/nvidia-settings/blob/5b341a9e54f08ac324e148e1e7e102030e42b1c4/src/nvml.h#L4593-L4639

tomli380576 · 2026-07-03T02:47:23Z

+            nvmlDeviceGetNvLinkState = nvml.nvmlDeviceGetNvLinkState
+            nvmlDeviceGetNvLinkState.argtypes = [
+                ctypes.c_void_p,
+                ctypes.c_uint,
+                ctypes.POINTER(ctypes.c_uint),
+            ]
+            nvmlDeviceGetNvLinkState.restype = ctypes.c_int


source for future reviewers: https://github.com/NVIDIA/nvidia-settings/blob/5b341a9e54f08ac324e148e1e7e102030e42b1c4/src/nvml.h#L9486-L9504

the 1st arg is a void pointer because otherwise we need to bring in the entire nvmlDevice_st struct

tomli380576 · 2026-07-03T02:51:46Z

the pseudocode is basically

check_nv_link_status():
    nvmlInit_v2()
    if init failed: 
        panic
    
    num_devices = nvmlDeviceGetCount()
    
    for device_i = 0...num_devices:
        device_handle = nvmlDeviceGetHandleByIndex(i)
        
        for link_i = 0...6:  
            has_nvlink = nvmlDeviceGetNvLinkState(device_handle, link_i)
            if has_nvlink:
                return True

    return False

hanhsuan force-pushed the fix_nv_link_check_issue branch from fb39fca to 91aae37 Compare July 1, 2026 12:29

1. change to use ctype to call libnvml to verify nvlink status directly

48908f0

2. set __GLX_VENDOR_LIBRARY_NAME=mesa for non NV card

hanhsuan force-pushed the fix_nv_link_check_issue branch from 91aae37 to 48908f0 Compare July 1, 2026 12:44

hanhsuan marked this pull request as ready for review July 3, 2026 02:16

Copilot AI review requested due to automatic review settings July 3, 2026 02:16

Copilot started reviewing on behalf of hanhsuan July 3, 2026 02:16 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

tomli380576 approved these changes Jul 3, 2026

View reviewed changes

tomli380576 reviewed Jul 3, 2026

View reviewed changes

Hook25 assigned pieqq Jul 3, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate to use libnvml to check nvlink status directly (BugFix)#2634

Migrate to use libnvml to check nvlink status directly (BugFix)#2634
hanhsuan wants to merge 1 commit into
mainfrom
fix_nv_link_check_issue

hanhsuan commented Jul 1, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

tomli380576 left a comment

Uh oh!

tomli380576 Jul 3, 2026

Uh oh!

tomli380576 Jul 3, 2026

Uh oh!

tomli380576 Jul 3, 2026 •

edited

Loading

Uh oh!

tomli380576 Jul 3, 2026 •

edited

Loading

Uh oh!

tomli380576 Jul 3, 2026 •

edited

Loading

Uh oh!

tomli380576 commented Jul 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

hanhsuan commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Resolved issues

Documentation

Tests

Uh oh!

codecov Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tomli380576 left a comment

Choose a reason for hiding this comment

Uh oh!

tomli380576 Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

tomli380576 Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

tomli380576 Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomli380576 Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomli380576 Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomli380576 commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hanhsuan commented Jul 1, 2026 •

edited

Loading

codecov Bot commented Jul 1, 2026 •

edited

Loading

tomli380576 Jul 3, 2026 •

edited

Loading

tomli380576 Jul 3, 2026 •

edited

Loading

tomli380576 Jul 3, 2026 •

edited

Loading

tomli380576 commented Jul 3, 2026 •

edited

Loading