Skip to content

fix: scope Allocate env map to each container request - #196

Open
lexfrei wants to merge 1 commit into
NVIDIA:masterfrom
lexfrei:fix/allocate-envlist-per-container
Open

fix: scope Allocate env map to each container request#196
lexfrei wants to merge 1 commit into
NVIDIA:masterfrom
lexfrei:fix/allocate-envlist-per-container

Conversation

@lexfrei

@lexfrei lexfrei commented Jul 20, 2026

Copy link
Copy Markdown

envList was created once per Allocate() call and reused for every entry in reqs.ContainerRequests, so each entry's PCI addresses were appended to those of the entries processed before it. For requests naming devices 11 and 22, the second response's PCI_RESOURCE_NVIDIA_COM_* is emitted as 11,22 — it names a device belonging to a different entry.

Moving the declaration inside the per-container loop scopes it correctly. GenericVGpuDevicePlugin.Allocate() already scopes its env map this way and is unaffected, so the generic plugin was the outlier. The declaration predates #191, which changed what enters devAddrs but not the scope of envList.

This is latent in practice: upstream kubelet sends a single ContainerAllocateRequest per call and reads only ContainerResponses[0], so no released kubelet hits this path. The repeated field is part of the public device plugin API and nothing in its contract forbids batching, so the plugin should handle it correctly rather than depending on the caller sending one entry.

Testing

New spec in generic_device_plugin_test.go sends two container requests and asserts each response carries only its own device address and only its own /dev/vfio/<group> node. Verified it fails without the fix (entry 1 gets 11,22) and passes with it; the device-node assertions additionally catch hoisting deviceSpecs/seenDeviceSpecs out of the loop. Full suite 43/43, no new go vet or golangci-lint findings against master.

@lexfrei
lexfrei marked this pull request as ready for review July 20, 2026 01:19
AllocateRequest.ContainerRequests is a repeated field, but the env map
holding the allocated PCI addresses was created once per Allocate call
and reused for every entry in it, so each entry's addresses were appended
to those of the entries processed before it. For requests naming devices
11 and 22, the second response's PCI_RESOURCE_NVIDIA_COM_* is emitted as
"11,22" — it names a device belonging to a different entry.

Upstream kubelet sends exactly one ContainerAllocateRequest per call and
reads only ContainerResponses[0], so no released kubelet reaches this
path. The repeated field is part of the public device plugin API and
nothing in its contract forbids batching, so handle it correctly instead
of depending on the caller sending a single entry.

Move the map inside the per-container loop so a response advertises only
the devices requested in its own entry. The vGPU plugin already scopes
its env map this way.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei
lexfrei force-pushed the fix/allocate-envlist-per-container branch from 4c65c51 to d66b8f3 Compare September 1, 2026 13:17
@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant