feat: activate one NVLink fabric partition for multi-whole-card vGPU VMs - #195
Open
lexfrei wants to merge 15 commits into
Open
feat: activate one NVLink fabric partition for multi-whole-card vGPU VMs#195lexfrei wants to merge 15 commits into
lexfrei wants to merge 15 commits into
Conversation
lexfrei
marked this pull request as ready for review
July 12, 2026 13:12
lexfrei
force-pushed
the
feat/fabric-multigpu-partitions
branch
from
July 20, 2026 00:43
de75596 to
a477397
Compare
lexfrei
force-pushed
the
feat/fabric-multigpu-partitions
branch
from
September 1, 2026 10:12
a477397 to
9e4618b
Compare
…work Ada/Hopper+ GPUs (vGPU 17+) no longer expose vGPU instances through mdev; instead, SR-IOV Virtual Functions stay bound to the "nvidia" host driver and are configured directly on the PCI function via sysfs. The existing vGPU discovery only walks /sys/bus/mdev/devices, so it finds nothing on these GPUs and every vGPU VF ends up indistinguishable from a whole-card passthrough device, since both share the same PCI vendor:device id. Add a second discovery path that walks /sys/bus/pci/devices looking for Nvidia functions bound to the "nvidia" driver, skips Physical Functions (identified by sriov_totalvfs), and treats a Virtual Function as a configured vGPU when its current_vgpu_type is non-zero. The profile name is resolved by matching that type id against the creatable_vgpu_types catalog, merged per physical card (scoped by each VF's physfn parent) since a configured VF's own list is frequently empty or reduced to the active entry, and since vGPU type ids are only unique within a single physical card. Because these VFs are ordinary PCI/IOMMU-group devices, discovered functions are added to the same iommu/BDF maps used by GPU passthrough and dispatched through the existing GenericDevicePlugin, grouped by vGPU profile name instead of PCI device id. No changes were needed to the allocate path or device contract; the mdev-backed vGPU discovery is untouched. Document the new hardware path in the README alongside the existing mdev instructions, since it applies to a different GPU generation and neither replaces the other. Signed-off-by: Aleksei Sviridkin <f@lex.la>
A fully consumed card reduces the creatable_vgpu_types list of every one of its functions down to the header, so the per-card catalog cannot resolve the types of already-configured VFs and the plugin advertised nothing exactly when all capacity was allocated. Fall back to a host-wide catalog merged across all cards; numeric ids that different cards map to different names stay ambiguous and are still skipped rather than guessed. Signed-off-by: Aleksei Sviridkin <f@lex.la>
On a host where every card is fully consumed, no creatable_vgpu_types catalog lists anything at all — sysfs alone cannot resolve configured type ids to profile names no matter how catalogs are merged. Resolve through NVML as the last resort: the supported-vGPU list of a card does not shrink as capacity is allocated. Scoped per physical card like the sysfs catalogs, cached per scan, and requires the host driver library to be loadable in the container. Signed-off-by: Aleksei Sviridkin <f@lex.la>
…other cards When a configured VF's card catalog is reduced and cannot resolve its type id, resolution fell back to a host-wide catalog merged across all cards. Numeric vGPU type ids are only unique within one physical card, so if the target card contributed no entry for the id and another card mapped the same id to a different profile, that entry looked unambiguous and the VF could be advertised under the wrong extended resource. Drop the host-wide catalog entirely. Resolve strictly card-local sysfs, then the parent Physical Function's own NVML supported-type list, then skip the VF with a log. NVML is authoritative per card and its supported list does not shrink as capacity is allocated, so it covers the fully consumed card without ever borrowing a sibling card's mapping. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ments The NVML name-resolution fallback needs libnvidia-ml.so.1 and the NVIDIA device nodes. The default DaemonSet mounts only /dev/vfio and runs unprivileged, so on a fully consumed node a restart could resolve no profile names and advertise no VFs. Add a dedicated manifest that provides them via a single-file bind of libnvidia-ml.so.1 plus the NVIDIA device nodes under privileged: true, leaving the default manifest minimal and unprivileged. Document in the README both the minimally privileged runtimeClassName: nvidia path and the hostPath variant, why the library must be bound as a single file rather than the whole host lib directory, and which device nodes the plugin's NVML queries actually require. Also document that vGPU profiles must be configured before the plugin starts, since it scans the Virtual Functions once at startup. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ation On NVSwitch systems running SR-IOV vGPU with Fabric Manager in FABRIC_MODE=2, a whole-card guest cannot initialise CUDA until its VF's NVLink fabric partition is activated through the Fabric Manager SDK (cuInit fails with error 802 otherwise). Add a self-contained pkg/fabric wrapping libnvfm (the nv_fm_agent API): - cgo binding (linux && cgo) linking libnvfm.so.1, with a portable stub for other platforms and CGO_ENABLED=0 so the whole module still builds and unit-tests on developer machines. The FM ABI is declared in the binding; NVIDIA's proprietary headers are not vendored. - Connect / GetSupportedPartitions / ActivateWithVFs / Deactivate / Close with sizeof-based struct-version handling and a full fmReturn_t -> error mapping. The connection address type (Unix socket vs host:port TCP) is selected from the address form. - NVML helpers to resolve a physical GPU's FM physicalId (module id) and MIG mode, and a VF -> single-GPU partition resolver keyed strictly on physicalId, never on PCI enumeration order. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Activate a whole-card vGPU VF's NVLink fabric partition during GenericDevicePlugin.Allocate, before the device is handed to the virt-launcher pod, so the guest's CUDA init succeeds with per-VM NVLink isolation from the first init. For each allocated VF the plugin resolves its single-GPU fabric partition (VF -> parent PF -> FM physicalId -> partition) and activates it for that one VF; fmActivateFabricPartitionWithVFs takes one VF per GPU, so a single-GPU partition is always activated with exactly one VF. MIG-mode GPUs have NVLink disabled and are not in the fabric, so their VFs need no activation and are skipped (nvmlDeviceGetMigMode). The feature is a no-op on non-fabric systems and classic passthrough GPUs. Configuration via environment: FABRIC_PARTITION_ACTIVATION (auto/off), FABRIC_MANAGER_ADDRESS (default the FM TCP command interface), FABRIC_FAIL_MODE (closed: an FM error fails the allocation; open: log and allow), and FABRIC_RECONCILE_INTERVAL. A pod-resources reconciler deactivates a partition once its VF is gone, since the device plugin API has no release callback. Unit-tested with a fake Fabric Manager client. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fabric partition activation binding links libnvfm.so.1 (NEEDED in the binary), so both the builder and the runtime image need it. Fetch a pinned libnvfm from the CUDA repository package at build time (only the versioned shared object; the FM ABI is declared in-tree, headers are not required), link against it, and copy it into the distroless runtime image on the loader search path. The version is a build ARG so it can track the target host's Fabric Manager daemon. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Document the fabric partition activation design: the FM SDK binding, the VF -> single-GPU partition resolver, single-VF activation (one VF per GPU), MIG-skip, the pod-resources reconciler for deactivation, the configuration, the runtime requirements (hostNetwork for the FM TCP command interface, pod-resources mount, FABRIC_MODE=2), and libnvfm image bundling. Includes a draft upstream PR description. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the hand-written inline cgo Fabric Manager SDK binding with the official github.com/NVIDIA/go-nvfm library. The previous binding declared the nv_fm_agent struct layouts and entry points by hand in the cgo preamble (NVIDIA's headers cannot be redistributed) and linked libnvfm.so.1 at build time. go-nvfm ships the FM SDK headers and dlopen's libnvfm at runtime, so both the hand-maintained ABI transcription and the link-time dependency are gone. Behaviour is unchanged: same Client interface and call sequence, same single-VF partition activation, same physicalId-based resolution and MIG-mode skip, same 5s connect timeout, same TCP/Unix address handling, and the same return-code to sentinel-error mapping the allocation logic branches on. The runtime image still bundles the versioned libnvfm.so.1; go-nvfm is pointed at that SONAME because the unversioned -dev symlink is not shipped in the distroless image. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The libnvfm build steps were written for x86_64 only, so the arm64 image this repository also publishes was broken in two independent places. The builder stage fetched the Fabric Manager package from the CUDA repository under the directory reported by uname -m. That is correct for x86_64, but the repository names its 64-bit ARM server directory sbsa, so the arm64 fetch requested a path that does not exist. Under set -eux the 404 from wget aborted the stage and the arm64 image never built at all. The repository path and the Debian architecture suffix are now chosen separately, since only one of them follows uname. The runtime stage then copied libnvfm.so.1 into /usr/lib/x86_64-linux-gnu. The loader searches the multiarch directory belonging to the image's own architecture, so on arm64 that destination names a directory that is never searched and go-nvfm's dlopen of libnvfm.so.1 would fail, leaving fabric partition activation silently unavailable. The library now goes to /usr/lib, which this base image's loader carries in its built-in search path on both published architectures, so one destination serves both. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The design document carried a pull-request draft that restated the sections above it, and a paragraph recording which shape the Fabric Manager binding passed through before the current one. Neither constrains the code as it stands, and both date the document to the moment it was written rather than to what the plugin does. The heading covering the gaps this feature has to fill now names what the plugin does not provide, rather than what a fork does not supply. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
On NVSwitch systems the fabric partition layout is predefined: an arbitrary set of GPUs can be activated as a single partition only if the fabric offers that exact set. Add PartitionForModuleIDs, which resolves a requested set of physical GPU ids (physicalId == NVML module id) to the one partition spanning exactly those GPUs, matched strictly by physicalId, or reports ErrNoMatchingPartition. This is the prerequisite for binding a multi-whole-card VM to a single partition so its cards get NVLink peer-to-peer. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
A VM allocated several whole cards in one request should get NVLink peer-to-peer between them, which requires all the cards in a single fabric partition. Activate the whole request's VF set at once: resolve each card to its GPU, find the partition spanning exactly that set, and activate it with one VF per GPU in the partition's GPU order (as fmActivateFabricPartitionWithVFs requires). A single-VF request keeps the existing single-GPU behavior unchanged. When no defined partition spans the requested cards, FABRIC_FAIL_MODE decides: closed fails the allocation (naming the set and the partitions the fabric offers); open activates each card's single-GPU partition so CUDA still works without cross-card NVLink. MIG cards are excluded from a set, and the feature stays inert on non-NVSwitch systems. GetPreferredAllocation now prefers, for a multi-device request, a VF set whose GPUs form a defined partition, choosing sets that fragment the fewest larger partitions; single-device and non-fabric requests fall through to the existing NUMA preference. The pod-resources reconciler groups devices per VM so a multi-GPU partition is reconstructed as one partition owning N VFs, and deactivates it only once all its VFs are released - partitions are atomic and never partially torn down. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe the predefined-partition constraint, the multi-VF activation path and VF ordering, the FABRIC_FAIL_MODE fallback for an unmatched set, the partition-aligned GetPreferredAllocation, and the reconciler's deactivate-after-last-VF semantics. Note the multi-card path is not yet hardware-validated. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
force-pushed
the
feat/fabric-multigpu-partitions
branch
from
September 1, 2026 13:30
9e4618b to
2b852cc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #193.
What & why
#193 activates a single-GPU NVLink fabric partition per vGPU VF, giving a whole-card guest working CUDA on NVSwitch systems (Fabric Manager in
FABRIC_MODE=2) — one card per VM. A VM allocated several whole cards in one request also needs NVLink peer-to-peer between those cards, which requires all of them in a single fabric partition; activating each card's single-GPU partition independently gives working CUDA per card but no cross-card NVLink. This change activates the one fabric partition spanning exactly the requested cards.The predefined-partition constraint
On an NVSwitch system the partition layout is fixed by the fabric, not chosen per request:
fmGetSupportedFabricPartitionsenumerates the partitions the system offers (on an 8-GPU HGX: all-8, two 4-GPU, four 2-GPU, eight single-GPU). A set of GPUs can be one partition only if the fabric offers that exact set — e.g.{1,2}may be a partition while{2,3}is not.Allocateresolves each requested card to its GPU (physicalId== NVML module id, matched strictly by id, never PCI order), finds the partition whose GPU set equals the request, and activates it with one VF per GPU in the partition's GPU order (asfmActivateFabricPartitionWithVFsrequires). A single-VF request is unchanged from #193.Fail modes for an unmatched set (
FABRIC_FAIL_MODE)When the requested cards do not form any defined partition (or two VFs map to the same GPU):
closed(default): fail theAllocate, naming the requested GPU set and the multi-GPU partitions the fabric offers.open: activate each card's own single-GPU partition, so CUDA works in every card of the guest but without cross-card NVLink.GetPreferredAllocation
For a multi-device request the whole-card resource prefers a VF set whose GPUs form a defined partition, and among those prefers sets that fragment the fewest larger still-available partitions (so small requests don't chop up the large partitions). Single-device requests, non-fabric systems, and requests with no partition-aligned option fall through unchanged to the existing NUMA-affinity preference. The preference is advisory —
Allocatestill validates and activates whatever set it is handed.Reconciler
A multi-GPU partition maps to N VFs. The pod-resources reconciler groups allocated devices per container (one VM), reconstructing a multi-GPU partition as one partition owning N VFs — needed after a device-plugin restart, since kubelet does not re-
Allocaterunning pods and FM does not report a partition's VF list. A partition is deactivated once its VM departs (all its VFs released); kubelet reports each container's devices atomically, so there is no sub-container partial-release state.Testing
Unit tests cover: matching set → one partition activation with N VFs in GPU order; no-match → fail-closed error and fail-open per-GPU fallback; MIG cards excluded from a set; module-id resolution error and FM-unreachable in both fail modes; partition-aligned
GetPreferredAllocation(including must-include and less-fragmenting preference); and the reconciler deactivating only after a VM departs and adopting a multi-GPU partition after a restart.go build,go vet,gofmt,golangci-lint, andgo test ./pkg/...are green.Not yet hardware-validated
The single-VF path (#193) was validated live on an 8-GPU H200 NVSwitch host. This multi-card path has not been validated on hardware — there was no free whole-card pair on the target system to exercise the multi-VF Fabric Manager SDK call sequence end-to-end. It is covered by unit tests against a fake Fabric Manager client, but the
fmActivateFabricPartitionWithVFscall for a multi-VF partition should be confirmed on real hardware before relying on it in production. This is why the PR opens as a draft.Stacking
Stacked on #193 (
feat/fabric-partition-activation); review that first. The #192 → #193 stack is pending a rebase (the #192 branch was recently reworked); this PR will be rebased along with the stack.