EKS: nvidia-setup-kernel should no-op before apt when the running kernel already matches

[yuanchen8911]

Note

This issue was substantially rewritten on 2026-09-19. The original report claimed AICR
installed kernel 7.0.0 and poisoned the nodes. That was wrong — see "Correction" at the
bottom. The valid finding is narrower and is stated below. Earlier comments are retained as
history; prefer this body.

Summary

nvidia-setup-kernel runs an apt transaction even when the node is already booted on the
exact kernel the package targets. With NVIDIA_SETUP_INSTALL_KERNEL: "true" the version
check is bypassed entirely, so there is no path that short-circuits before apt.

On a node carrying unrelated pre-existing dpkg damage, that unnecessary transaction inherits
the damage, fails, and blocks node tuning — and therefore aicr validate.

Bounded by post-repair evidence (2026-09-19): after both nodes were repaired, tuning was
re-run with stock settings and reached 2/2 complete. The install path still ran despite the
nodes already being on the target kernel — apt reached the repositories, reported
0 upgraded, 0 newly installed, 0 removed, and GRUB was regenerated — but it succeeded. So
this is a latent reliability hazard, not a healthy-node failure: it performs avoidable
network, package-manager and boot-configuration work, and makes otherwise-idempotent tuning
depend on apt, repository, dpkg and GRUB health.

What is wrong in AICR

recipes/components/nodewright-customizations/manifests/tuning.yaml sets
NVIDIA_SETUP_INSTALL_KERNEL: "true" on nvidia-setup-kernel. In
ghcr.io/nvidia/nodewright-packages/nvidia-setup@sha256:1551abdd54476f6bddce863d383903a0a76f8a91bb4daa53c4f3c9490b980dae
(the digest tuning.yaml pins as containerSHA), steps/ensure_kernel.sh:

if [ "${NVIDIA_SETUP_INSTALL_KERNEL:-false}" = "true" ]; then
  install_kernel
  exit 0
fi
# the kernel checks live below this branch and are never reached

The package's own target for this lane is already satisfied: defaults/eks-gb300.conf sets
KERNEL=6.17.0-1019-aws, and resolve_full_kernel appends -64k on aarch64 →
6.17.0-1019-aws-64k, which both GPU nodes run. There was nothing to install.

AICR's comment at tuning.yaml:102 ("Check for kernel being >= to 6.17.0-1019 and if not
install") describes neither real branch.

Why flipping the flag is not the fix

Two independent reasons:

1. With NVIDIA_SETUP_KERNEL_ALLOW_NEWER: "false" (what AICR sets alongside it), the false
   path runs check_kernel_exact and exit 1s otherwise. It is check-then-fail, not
   check-then-install.

   Worth noting the check is not what its name implies. It strips everything after the first
   hyphen and compares only the upstream version:

   check_kernel_exact() {
     local required="$1"
     local current
     current=$(uname -r)
     local required_upstream="${required%%-*}"   # 6.17.0-1019-aws-64k -> 6.17.0
     local current_upstream="${current%%-*}"     # 6.17.0-1019-aws-64k -> 6.17.0
     [ "${current_upstream}" = "${required_upstream}" ]
   }

   So it accepts any 6.17.0-* kernel regardless of ABI or flavour, while the error text says
   "exact match required". A fix that relies on this path should either describe those semantics
   accurately or introduce a full-version comparison.

2. false is not a no-op either. A successful kernel check continues into the full EKS
   setup path in apply.sh, including upgrade.sh and further apt activity. Verified on
   cluster: patching the live CR to false still failed against the same broken dpkg state.

Proposed fix

Short-circuit before the first apt operation when the running kernel already matches the
target. A check inside the INSTALL_KERNEL=true branch of ensure_kernel.sh, immediately
before install_kernel, is sufficient — apply.sh exits after ensure_kernel.sh returns. The
requirement is simply that nothing touches the package database before that check.

Scope

Both follow-ups from this run share the same exposure:

	#2865	#2870
Structurally exposed	EKS profiles where the kernel-build path runs	Same — eks-h100, eks-gb200, eks-gb300
AICR recipes affected	A100, H100, H200, GB200, GB300 EKS	Same
Reproduced on	GB300 only	GB300 only
Not exposed	AKS, GKE, BCM, generic-GB300, RKE2, OKE, OCP	Same

Detail:

• Pre-existing package damage verified on both GB300 nodes of this cluster.
• AICR tuning failure reproduced on one node (ip-10-0-227-27). The current NodeWright CR
  processed only that node; ip-10-0-187-96 remains waiting and is absent from nodeState.
• Same package configuration: eks-h100 / eks-gb200 / eks-gb300 profiles (identical
  KERNEL=6.17.0-1019-aws and EFA=1.48.0), reached by AICR's A100/H100/H200/GB200/GB300 EKS
  recipes.
• Not exposed by the current AKS profile: aks-h100.conf omits KERNEL, and ensure_kernel.sh
  tests for that before the INSTALL_KERNEL branch.
• Not exposed: GKE, BCM, generic GB300, RKE2, OKE, OCP.

This particular tuning failure requires both AICR's unnecessary apt call and pre-existing
damage on the node. That is not to say the damage is otherwise benign — broken iU/iF kernel
packages are independently harmful to node maintenance and reboot safety, regardless of AICR.

Correction — what this issue originally got wrong

The original report attributed the broken kernel packages to AICR. Node evidence disproves it:

unattended-upgrades installed kernel 7.0.0 on 2026-09-16, three days before the AICR run.

Start-Date: 2026-09-16  06:22:11
Commandline: /usr/bin/unattended-upgrade
INFO Packages that will be upgraded: … linux-aws linux-headers-aws linux-image-aws …
INFO Package linux-aws is kept back because a related package is kept back

The EFA DKMS build failed against 7.0.0 — EFA 3.0.0 is source-incompatible
(kcompat.h:274: error: 'struct ib_umem' has no member named 'nmap') — leaving the packages
iU/iF. Both GPU nodes show the same state and the same 09-16 origin.

Today's AICR run requested only 6.17.0-1019-aws-64k; its 7.0.0 dpkg entries are
configure retries of the pre-existing unpacked packages, not installs:

2026-09-19 22:12:18 configure linux-image-7.0.0-1012-aws:arm64
2026-09-19 22:12:18 status half-configured linux-image-7.0.0-1012-aws:arm64

Therefore, withdrawn: that AICR installed or transitively pulled kernel 7.0.0; that AICR
poisoned the nodes; that the linux-aws metapackages were pulled by AICR's transaction; and
the "persistent node damage caused by AICR" escalation. The node damage is a fleet/image
concern for the cluster owner, not an AICR defect.

Upstream

Reopen requested upstream at NVIDIA/nodewright-packages#85 (currently CLOSED) —
"nvidia-setup: skip-if-newer kernel mode + flavor-aware check".

That issue predates this one (opened 2026-07-03) and already describes both the unconditional
install and the flavour-stripping comparison. It was closed 2026-09-15 as "already implemented",
but neither behaviour is present at
b90ba46
or in the published nvidia-setup:0.8.0 (the newest release). A reopen request with the source
evidence and a minimal patch is posted here:

• [FEA]: nvidia-setup: skip-if-newer kernel mode + flavor-aware check nodewright-packages#85 (comment)

This issue stays open as the downstream tracker. It closes when the corrected package is
released and this repo advances nvidia-setup's version / containerSHA in tuning.yaml.

Note the upstream minimal fix is an exact-match short circuit, which is what #2870 needs; it
does not implement #85's broader "skip when newer" behaviour.

Components Affected

The fix itself belongs upstream, not in this repo:

• Upstream nvidia-setup package logic — steps/ensure_kernel.sh (and the surrounding
  apply.sh path) is where the short-circuit must happen. AICR only passes the env var.
• AICR recipe pin — recipes/components/nodewright-customizations/manifests/tuning.yaml
  needs its version / containerSHA advanced to consume the corrected package once released.

AICR could additionally reconsider whether NVIDIA_SETUP_INSTALL_KERNEL: "true" is the right
default for lanes whose AMI already ships the target kernel, but that is a mitigation rather
than the fix — see "Why flipping the flag is not the fix" above.

[yuanchen8911]

Scope correction: this is not GB300-specific — all three EKS lanes share the exposure.

The lane configs inside ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.8.0:

config	KERNEL	EFA
eks-h100.conf	6.17.0-1019-aws	1.48.0
eks-gb200.conf	6.17.0-1019-aws	1.48.0
eks-gb300.conf	6.17.0-1019-aws	1.48.0
aks-h100.conf	unset	none
bcm-*.conf	empty	—

All three EKS configs are identical in both fields, and all three lanes render the shared
tuning.yaml with NVIDIA_SETUP_INSTALL_KERNEL: "true". So eks-h100 and eks-gb200 take
the same unconditional-install path and the same apt transaction as eks-gb300. GB300 is
simply where this was reproduced.

AKS is structurally immune, not incidentally lucky. ensure_kernel.sh tests for an unset
KERNEL before the INSTALL_KERNEL branch:

if [ -z "${KERNEL:-}" ]; then
  echo "No KERNEL configured for combination ${COMBINATION}; skipping kernel handling"
  exit 0
fi
if [ "${NVIDIA_SETUP_INSTALL_KERNEL:-false}" = "true" ]; then
  install_kernel
  exit 0
fi

aks-h100.conf omits KERNEL, so the package exits early — matching the existing comment in
tuning.yaml ("On AKS this package is a designed no-op: the aks defaults omit KERNEL and
ensure_kernel exits early"). A concurrent AKS H100 run of the same workflow completed node
tuning normally, which is consistent with this.

The title would be better as "on EKS lanes" rather than implying GB300.

[yuanchen8911]

Escalation: the failure is self-poisoning. One run leaves the node permanently unable to
complete tuning until repaired by hand.

After the first apply.sh failure, the 7.0.0 kernel packages remain on the node in a broken
dpkg state:

$ dpkg -l | grep -E "linux-(image|headers|aws)"
iU  linux-aws                      7.0.0-1012.12~24.04.1  arm64
iF  linux-headers-7.0.0-1012-aws   7.0.0-1012.12~24.04.1  arm64
iU  linux-headers-aws              7.0.0-1012.12~24.04.1  arm64
iF  linux-image-7.0.0-1012-aws     7.0.0-1012.12~24.04.1  arm64

$ dpkg --audit
The following packages have been unpacked but not yet configured...
 linux-aws
 linux-headers-aws

iU = unpacked-not-configured, iF = half-configured. install_kernel.sh wraps its apt calls
in apt_with_dpkg_heal, so every subsequent run tries to configure these packages first,
hits the same EFA DKMS build failure, and exits 100 again.

Consequence: setting NVIDIA_SETUP_INSTALL_KERNEL=false afterwards does not recover the
node. Verified here — the CR was patched to false (and confirmed live), and the very next
attempt produced the identical dpkg: error processing package linux-image-7.0.0-1012-aws.
The trigger is no longer the install step; it is the poisoned dpkg state.

So the blast radius of a single exposure is larger than the original report implies:

• The fix in the recipe prevents new nodes from entering this state.
• It does not rescue any node that already ran the package once. Those need manual repair
  (purging the 7.0.0 linux-image / linux-headers / linux-aws packages, or fixing the EFA
  DKMS build) before tuning can ever reach complete.
• Any EKS lane that has already run this package on a long-lived node is likely already in this
  state and would fail on the next tuning run regardless of a recipe fix.

Worth considering whether the package should fail before unpacking when the DKMS build cannot
succeed, rather than leaving a half-configured kernel behind.

[yuanchen8911]

Retracting an overreach in my escalation comment above.

I wrote:

Any long-lived EKS node that has run this is likely already stuck.

That is not established and I should not have stated it. A node only enters the broken state if
it actually unpacked the newer kernel packages and the DKMS configure step failed. Nodes that
ran the package before those packages were available in the archive, that had different package
state, or that completed the transaction successfully are not necessarily damaged. What is
verified is the single node in this run.

Also worth qualifying the proposed fix. Setting NVIDIA_SETUP_INSTALL_KERNEL=false in AICR
is only safe if every supported EKS AMI is guaranteed to already boot the exact target kernel;
otherwise it converts a kernel install into a hard validation failure instead. The safer shape
is upstream package idempotency — when the running kernel already matches the target, do
nothing even when installation is enabled — with AICR consuming the corrected package and
retaining enough resources for the genuine install path.

The persistent-poisoning behaviour itself (packages left iU/iF, every later run re-entering
dpkg --configure -a and re-failing, including after flipping INSTALL_KERNEL=false) is
verified on this node and stands.

Separately: NodeWright leaving runtime-required taints behind after CR deletion is upstream
behaviour and should not be folded into this issue.

[yuanchen8911]

Update after node repair: confirmed and bounded. The unnecessary work is real; healthy nodes
completed successfully.

Both GB300 nodes were repaired and tuning re-run with stock settings
(NVIDIA_SETUP_INSTALL_KERNEL=true, unmodified resources).

On both nodes nvidia-setup-kernel still entered the install path despite the nodes already
being booted on 6.17.0-1019-aws-64k. It accessed the apt repositories, found every target
package already at the newest version, reported 0 upgraded, 0 newly installed, 0 removed,
regenerated GRUB, and completed successfully.

This bounds the impact and is worth stating plainly: the earlier failure was not a
general healthy-node failure. It happened because the unconditional apt path encountered
unrelated pre-existing dpkg damage.

It does not make the transaction harmless, though. Even on a healthy node it performs
network and package-manager operations and regenerates boot configuration, none of which is
needed when the running kernel already matches. Nor is broken dpkg the only way it can fail —
repository outages, apt locks and ordinary apt errors remain possible.

The accurate rationale for the fix: the missing exact-match short circuit introduces avoidable
side effects and makes otherwise-idempotent tuning depend on apt, repository, dpkg and GRUB
health. Severity should be read as a latent reliability hazard, not a demonstrated
healthy-node failure.

The proposed fix is unchanged.