Skip to content

fix(ascend): reject -core on hard-split vNPU and fix "nor"->"not" typo - #2026

Merged
hami-robot[bot] merged 4 commits into
Project-HAMi:masterfrom
Wangmin362:fix/ascend-hardsplit-core-reject
Jul 9, 2026
Merged

fix(ascend): reject -core on hard-split vNPU and fix "nor"->"not" typo#2026
hami-robot[bot] merged 4 commits into
Project-HAMi:masterfrom
Wangmin362:fix/ascend-hardsplit-core-reject

Conversation

@Wangmin362

@Wangmin362 Wangmin362 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Fixes #2025

What this changes

On Ascend hard split the compute of a vNPU is fixed by the driver template, so a per-container -core request cannot be honored. Today it is dropped by PatchAnnotations but still reserved by the scheduler (Fit), and when it exceeds the physical core count the pod stays Pending with CardInsufficientCore for no visible reason.

This PR rejects the invalid combination at admission:

  • MutateAdmission: when the pod is not in hami-core (soft split) mode and a <resource>-core limit/request > 0 is present, admission fails with <resource>-core is only supported in hami-core (soft split) mode.
  • Soft split (hami-core) with -core is unchanged.
  • Hard split without -core is unchanged.

Also folds in a one-word typo fix: the multi-device rejection message "vNPU nor supported for multiple devices""not".

Commits

  1. fix(ascend): reject -core request on hard-split vNPU — fix + unit test Test_MutateAdmission_HardSplitCoreRejected.
  2. fix(ascend): correct typo "nor" -> "not" in multi-device rejection message.

Testing

Unit (go test ./pkg/device/ascend/...): all pass, incl. the new table test covering hard-split core (rejected), hard-split core=0 / no-core (allowed), and hami-core soft split with core (allowed).

Hardware validation (CONTRIBUTING.md gate 2):

  • Device: Huawei Ascend 910B4 (32G SKU, physical AICore=20), 8 cards, single node
  • Driver 25.5.1; ascend-device-plugin v1.3.0; k8s v1.29.15 (arm64)
  • Scheduler/webhook built from this branch (arm64) and deployed on the node; commercial device-plugin left untouched (its hami.io/node-register-Ascend910B4 is read fine by the open-source scheduler).
case request before this PR with this PR (verified)
f6-1 mem 16384 + core 10 (≤20), hard split Running, core dropped but reserved rejected at admission: huawei.com/Ascend910B4-core is only supported in hami-core (soft split) mode
f6-2 mem 16384 + core 25 (>20), hard split Pending CardInsufficientCore rejected at admission (fail fast, no misleading Pending)
hn-quarter mem 8192, hard split, no core Running (vir05_1c_8g) Running, temp=vir05_1c_8g (unchanged)
soft-core mem + core, hami-core annotation admitted (soft split) admitted (unchanged; not rejected by the new check)
f7-multidev 2 devices, sub-card mem, hard split rejected: ...nor... rejected: vNPU not supported for multiple devices

Server-side webhook log:

E ... webhook.go:83] validating pod failed:huawei.com/Ascend910B4-core is only supported in hami-core (soft split) mode   # f6-1
E ... webhook.go:83] validating pod failed:huawei.com/Ascend910B4-core is only supported in hami-core (soft split) mode   # f6-2
E ... webhook.go:83] validating pod failed:vNPU not supported for multiple devices                                        # f7

AI assistance disclosure

Per CONTRIBUTING.md: this change was investigated and implemented with the assistance of Claude Code. The root-cause analysis is based on my own hardware testing on an 8×910B4 node (6 rounds). I understand the code and can explain and defend it. AI helped navigate the codebase, draft the fix, and write the tests; I reviewed and verified everything, including the hardware validation above.

Summary by CodeRabbit

Behavior change & compatibility note

  • Admission now rejects pods requesting a non-zero Ascend “core” resource on hard-split vNPU. Pods in hami-core mode, or pods that omit the core request or set it to 0, are unaffected.
  • Bug Fixes

    • Reject non-zero core requests on hard-split vNPU.
    • Refined admission error wording for multi-device requests when not using hami-core.
  • Tests

    • Added coverage for hard-split vs hami-core core-resource admission outcomes.

wangmin added 2 commits July 7, 2026 23:05
On Ascend hard split the compute is fixed by the driver template, so a
requested -core is reserved by the scheduler (Fit) but dropped by
PatchAnnotations and never delivered to the container. When the request
also exceeds the physical core count the pod stays Pending with
CardInsufficientCore for no visible reason.

Reject the -core request at admission when the pod is not in hami-core
(soft split) mode, so the invalid combination fails fast with a clear
message instead of being silently mis-accounted.

Signed-off-by: wangmin <wangmin@riseunion.io>
…ssage

Signed-off-by: wangmin <wangmin@riseunion.io>
@hami-robot
hami-robot Bot requested review from lengrongfu and wawa0210 July 7, 2026 16:12
@hami-robot hami-robot Bot added the size/M label Jul 7, 2026
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9f872d56-5485-47ee-aa00-4d9ef0f518f1

📥 Commits

Reviewing files that changed from the base of the PR and between 6e832c2 and 85806d3.

📒 Files selected for processing (2)
  • pkg/device/ascend/device.go
  • pkg/device/ascend/device_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • pkg/device/ascend/device_test.go
  • pkg/device/ascend/device.go

📝 Walkthrough

Walkthrough

Ascend admission now rejects non-zero core resource requests unless the pod uses hami-core mode, checks both requests and limits, updates one rejection message, and adds unit coverage for hard-split and hami-core outcomes.

Changes

Core resource admission validation

Layer / File(s) Summary
Admission validation and tests
pkg/device/ascend/device.go, pkg/device/ascend/device_test.go
MutateAdmission rejects non-zero core resource requests outside hami-core mode, updates the multi-device rejection message, and the new test covers hard-split and hami-core cases with empty, zero, positive, and over-physical core requests.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Pod
  participant MutateAdmission
  participant Container

  Pod->>MutateAdmission: Submit pod with -core request
  MutateAdmission->>Container: Inspect Limits and Requests for core resource
  alt not hami-core mode and core > 0
    MutateAdmission-->>Pod: Reject admission with error
  else hami-core mode or core == 0
    MutateAdmission-->>Pod: Allow admission
  end
Loading

Possibly related issues

Suggested labels: kind/bug, approved

Suggested reviewers: FouoF, archlitchi

Poem

A bunny checks the core request,
And sends hard-split to hami-core test.
No phantom cores may sneak inside,
The typo fell, the tests abide. 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main behavior change and the typo fix.
Linked Issues check ✅ Passed The changes reject non-zero -core requests on hard-split pods while preserving hami-core and no-core cases, matching issue #2025.
Out of Scope Changes check ✅ Passed The diff stays focused on Ascend admission validation, the related typo fix, and matching tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot added the kind/bug Something isn't working label Jul 7, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds validation to reject -core requests when not in hami-core (soft split) mode, as they are only supported in that mode. It also fixes a typo in an error message and adds a corresponding unit test. The feedback suggests removing redundant local constants in the new test since they are already defined at the package level.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread pkg/device/ascend/device_test.go
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
unittests 59.58% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/device/ascend/device.go 83.44% <100.00%> (+0.23%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Test_MutateAdmission_HardSplitCoreRejected redeclared VNPUModeAnnotation
and VNPUModeHamiCore locally; use the package-level constants from
device.go instead.

Signed-off-by: wangmin <wangmin@riseunion.io>

@DSFans2014 DSFans2014 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems duplicated with #2024

Signed-off-by: wangmin <wangmin@riseunion.io>

@archlitchi archlitchi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@archlitchi archlitchi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@hami-robot hami-robot Bot added the lgtm label Jul 8, 2026
@archlitchi

Copy link
Copy Markdown
Member

/retest

@wawa0210

wawa0210 commented Jul 9, 2026

Copy link
Copy Markdown
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


/retest

@hami-robot

hami-robot Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, FouoF, Wangmin362

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Ascend] hard-split vNPU: requested "-core" is accounted by the scheduler but never delivered to the container (+ typo in multi-device message)

5 participants