feat: init-container GPU resource accounting by mayooot · Pull Request #2116 · Project-HAMi/HAMi

mayooot · 2026-07-23T14:02:14Z

What type of PR is this?

What this PR does / why we need it:
Implements GPU resource accounting for init containers, following the design in
docs/develop/initContainer-design.md (#2064).

Init containers run sequentially to completion before app containers start, so their
GPU usage should never be summed with app usage. The effective per-device footprint is:

effective = max( sum(app requests), max(init request) )

This is applied consistently across the three accounting paths:

Admission (webhook.go): quota check uses the effective value; MutateAdmission
now also runs over init containers.
Scheduling (score.go): each init container is fit independently against a fresh
node copy with an isolated allocation context; app containers accumulate on the real
node. Init allocations no longer pollute app quota/filter checks.
Usage recording (scheduler.go, pods.go): stores the collapsed view, and shrinks
to app-only once init containers finish (one-time, with symmetric quota delta).

Context

Follow-up to my earlier attempt in #1278, which only touched the webhook and was
incomplete. This PR implements the full model per the merged design doc.

Summary by CodeRabbit

New Features
- Improved GPU resource accounting for pods with init containers using init-aware effective usage semantics.
- Added automatic transition from init-inclusive usage to app-only usage once init containers complete successfully.
- GPU admission, scheduling, and scoring/fit now consistently account for both init and app containers.
Bug Fixes
- Prevented init-container device allocations from polluting app container allocation/footprints.
- Fixed quota usage tracking to remain consistent across pod updates, shrink transitions, and deletion.
Tests
- Added coverage for init-container accounting, quota evaluation, and allocation separation.

hami-robot · 2026-07-23T14:02:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mayooot
Once this PR has been reviewed and has the lgtm label, please assign fouof for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-07-23T14:02:42Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2315d078-9216-415f-83c3-5e514994a595

📥 Commits

Reviewing files that changed from the base of the PR and between b01667e and 0762cf6.

📒 Files selected for processing (9)

pkg/device/initcontainer.go
pkg/device/initcontainer_test.go
pkg/device/initcontainer_usage_test.go
pkg/device/pods.go
pkg/scheduler/scheduler.go
pkg/scheduler/score.go
pkg/scheduler/score_test.go
pkg/scheduler/webhook.go
pkg/scheduler/webhook_test.go

🚧 Files skipped from review as they are similar to previous changes (8)

pkg/device/pods.go
pkg/device/initcontainer_test.go
pkg/scheduler/webhook_test.go
pkg/device/initcontainer_usage_test.go
pkg/scheduler/scheduler.go
pkg/scheduler/score_test.go
pkg/scheduler/score.go
pkg/device/initcontainer.go

📝 Walkthrough

Walkthrough

The change adds init-container-aware GPU accounting. Usage is collapsed using max(init, summed app) semantics, released after successful init completion, preserved in pod state, and applied consistently across scoring, binding, admission mutation, and quota validation.

Changes

Init-container GPU accounting

Layer / File(s)	Summary
Collapsed usage views and lifecycle predicates `pkg/device/initcontainer.go`, `pkg/device/initcontainer_test.go`	Adds collapsed device views, app-only views, and helpers for detecting terminated or successful init containers, with table-driven coverage.
Pod usage release state `pkg/device/pods.go`, `pkg/device/initcontainer_usage_test.go`	Adds idempotent app-only shrinking, preserves released state across pod updates and copies, and verifies quota and deletion accounting.
Scheduling and bind accounting `pkg/scheduler/score.go`, `pkg/scheduler/scheduler.go`, `pkg/scheduler/score_test.go`	Separates init-container scoring from app allocation and uses collapsed usage for pod cache, quota updates, and rollback paths.
Admission mutation and quota validation `pkg/scheduler/webhook.go`, `pkg/scheduler/webhook_test.go`	Applies GPU mutation and privilege checks to init containers and calculates quota footprints using sequential init-container semantics.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Pod
  participant Scheduler
  participant PodManager
  participant QuotaManager
  Pod->>Scheduler: report init-container status and device usage
  Scheduler->>PodManager: store collapsed effective footprint
  Scheduler->>QuotaManager: add effective usage
  Pod->>Scheduler: report successful init completion
  Scheduler->>PodManager: shrink usage to app-only
  Scheduler->>QuotaManager: replace previous usage with app-only usage

Possibly related PRs

Project-HAMi/HAMi#2001: Also modifies GPU request aggregation in fitResourceQuota.
Project-HAMi/HAMi#2055: Also changes PodInfo deep-copy behavior.
Project-HAMi/HAMi#2064: Describes related init-container GPU resource accounting and release behavior.

Suggested labels: enhancement

Suggested reviewers: wawa0210

Poem

I’m a rabbit with GPUs in tow,
Where init peaks rise, then app sums grow.
Quotas shrink when setup is done,
Scores stay clean for everyone.
Hop, hop—no double-counting snow!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 69.23% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding GPU resource accounting for init containers.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/device/initcontainer_test.go`:
- Around line 59-68: Update the loops in podWithInit to use the modern
range-over-int form required by the rangeint linter, while preserving the
existing container counts and append behavior for both InitContainers and
Containers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e5e92b0a-8d3e-43ed-8531-8ca19a7b6883

📥 Commits

Reviewing files that changed from the base of the PR and between 9f2c88d and 56a2a30.

📒 Files selected for processing (9)

pkg/device/initcontainer.go
pkg/device/initcontainer_test.go
pkg/device/initcontainer_usage_test.go
pkg/device/pods.go
pkg/scheduler/scheduler.go
pkg/scheduler/score.go
pkg/scheduler/score_test.go
pkg/scheduler/webhook.go
pkg/scheduler/webhook_test.go

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/scheduler/webhook.go`:
- Around line 75-104: Update the privileged-container handling in the mutate
closure and its init/application container callers so detection produces an
explicit admission denial rather than only a warning and nil result. Propagate
that denial immediately for both pod.Spec.InitContainers and
pod.Spec.Containers, while preserving existing mutation errors and normal
admission behavior for non-privileged containers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 923852a8-9e68-4869-b248-44b13a5cea37

📥 Commits

Reviewing files that changed from the base of the PR and between 56a2a30 and b01667e.

📒 Files selected for processing (9)

pkg/device/initcontainer.go
pkg/device/initcontainer_test.go
pkg/device/initcontainer_usage_test.go
pkg/device/pods.go
pkg/scheduler/scheduler.go
pkg/scheduler/score.go
pkg/scheduler/score_test.go
pkg/scheduler/webhook.go
pkg/scheduler/webhook_test.go

🚧 Files skipped from review as they are similar to previous changes (7)

pkg/scheduler/webhook_test.go
pkg/scheduler/score.go
pkg/device/initcontainer.go
pkg/device/initcontainer_usage_test.go
pkg/scheduler/score_test.go
pkg/device/pods.go
pkg/scheduler/scheduler.go

Signed-off-by: Harry Li <harrymingh@gmail.com>

codecov · 2026-07-23T14:21:08Z

Codecov Report

❌ Patch coverage is 83.66337% with 33 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/scheduler/webhook.go	68.29%	10 Missing and 3 partials ⚠️
pkg/device/initcontainer.go	92.45%	4 Missing and 4 partials ⚠️
pkg/device/pods.go	76.00%	5 Missing and 1 partial ⚠️
pkg/scheduler/scheduler.go	50.00%	5 Missing and 1 partial ⚠️

Flag	Coverage Δ
unittests	`62.38% <83.66%> (+0.42%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/scheduler/score.go	`93.66% <100.00%> (+0.86%)`	⬆️
pkg/device/pods.go	`75.94% <76.00%> (+2.89%)`	⬆️
pkg/scheduler/scheduler.go	`68.33% <50.00%> (-0.41%)`	⬇️
pkg/device/initcontainer.go	`92.45% <92.45%> (ø)`
pkg/scheduler/webhook.go	`74.07% <68.29%> (-2.12%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

DSFans2014

hi @mayooot Thank you very much for your contribution. However, this overlaps with #1773 which @maishivamhoo123 is currently working on. And the documentation #2064 was also designed by him

mesutoezdil · 2026-07-27T07:09:48Z

/hold

hami-robot Bot added the dco-signoff: no label Jul 23, 2026

hami-robot Bot requested review from ouyangluwei163 and wawa0210 July 23, 2026 14:02

hami-robot Bot added the size/XL label Jul 23, 2026

coderabbitai Bot requested a review from Shouren July 23, 2026 14:03

coderabbitai Bot added the enhancement label Jul 23, 2026

mayooot force-pushed the feat/initcontainer-accounting branch from 56a2a30 to b01667e Compare July 23, 2026 14:04

hami-robot Bot added dco-signoff: yes and removed dco-signoff: no labels Jul 23, 2026

github-actions Bot added the kind/feature new function label Jul 23, 2026

coderabbitai Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread pkg/device/initcontainer_test.go

coderabbitai Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread pkg/scheduler/webhook.go

feat: init-container GPU resource accounting

0762cf6

Signed-off-by: Harry Li <harrymingh@gmail.com>

mayooot force-pushed the feat/initcontainer-accounting branch from b01667e to 0762cf6 Compare July 23, 2026 14:16

mayooot temporarily deployed to nvidia July 23, 2026 14:27 — with GitHub Actions Inactive

mesutoezdil reviewed Jul 23, 2026

View reviewed changes

Comment thread pkg/scheduler/score.go

Comment thread pkg/scheduler/webhook.go

Comment thread pkg/scheduler/score_test.go

DSFans2014 reviewed Jul 24, 2026

View reviewed changes

coderabbitai Bot mentioned this pull request Jul 27, 2026

[Bug] Mutating webhook allows pods with privileged containers when other containers request GPU resources #2138

Open

hami-robot Bot added the do-not-merge/hold label Jul 27, 2026

coderabbitai Bot mentioned this pull request Jul 27, 2026

fix webhook to deny privileged containers #2139

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: init-container GPU resource accounting#2116

feat: init-container GPU resource accounting#2116
mayooot wants to merge 1 commit into
Project-HAMi:masterfrom
mayooot:feat/initcontainer-accounting

mayooot commented Jul 23, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

hami-robot Bot commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

codecov Bot commented Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DSFans2014 left a comment

Uh oh!

mesutoezdil commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mayooot commented Jul 23, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Summary by CodeRabbit

Uh oh!

hami-robot Bot commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov Bot commented Jul 23, 2026

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DSFans2014 left a comment

Choose a reason for hiding this comment

Uh oh!

mesutoezdil commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mayooot commented Jul 23, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading