Skip to content

ci: scaffold unified nightly and stable release workflow#623

Merged
crookedstorm merged 23 commits into
mainfrom
AIREINF-254-unified-release-workflow/smuley
Jul 13, 2026
Merged

ci: scaffold unified nightly and stable release workflow#623
crookedstorm merged 23 commits into
mainfrom
AIREINF-254-unified-release-workflow/smuley

Conversation

@svvarom

@svvarom svvarom commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This replaces the separate nightly, RC, stable, and release-bundle workflows with one workflow for nightly and stable releases. Each run resolves an exact source commit and selected artifact set, optionally synchronizes NGC metadata, starts only the selected staging work, waits for the artifacts to reach their final locations, and reports the result. Stable releases also dispatch artifact registration, and full all releases create a GitHub Release. The legacy workflows, release-bundle metadata helper, and its tests are removed.

Linear: AIREINF-254 — Adapt Release Workflow by Newer Design

Stable and nightly behavior

Step Stable release Nightly pre-release State
Plan and validate Manual only. Requires an exact 40-character source SHA and MAJOR.MINOR.PATCH version. Runs on the weekday schedule or manually. Uses an optional pinned SHA and generates nightly-YYYYMMDDHHMMSS. Implemented
Select artifacts Operator selects all, wheels, containers, helm, or a custom set of IDs. Manual runs use the same selection model; scheduled runs select all. Implemented
Synchronize NGC metadata Optional manual input invokes the reusable metadata workflow. Available for manual runs; scheduled runs skip it. Implemented
Stage wheels Dispatches the selected wheels with release cadence and version. Dispatches the selected wheels with nightly cadence and timestamp. Implemented
Stage containers Dispatches the selected container IDs, source SHA, and release version. Dispatches the selected IDs and source SHA using the generated nightly label. Implemented
Stage Helm chart Packages k8s/helm using the release version and publishes it to NGC. Packages the same chart with a timestamped nightly version and publishes it to GHCR. Implemented
Register release artifacts Dispatches the selected wheel, container, and Helm IDs immediately after planning. Skipped completely. Implemented
Poll final locations Waits for all selected artifacts to reach their public release locations. Waits for all selected artifacts to reach their nightly locations. Implemented
Create GitHub Release Runs after polling succeeds for a full all release. Skipped. Implemented
Create deployment Dispatches deployment creation after polling succeeds when Helm is selected. Same behavior. Implemented
Signal completed deployment Separate downstream completion signal. Same behavior. Placeholder
Notify Sends a start notification, a delayed-readiness alert after two hours, and a final success or failure notification. Same behavior. Implemented

Dry runs resolve and validate the release plan and package Helm when selected, but skip publishing, downstream release dispatches, final-location polling, GitHub Release creation, and deployment creation.

Artifact catalog

Type IDs
Wheels nemo-platform, nemo-platform-plugin
Containers nmp-api, nmp-core, nmp-cpu-tasks, nmp-automodel-tasks, nmp-automodel-training, nmp-unsloth-training, auditor-tasks, safe-synthesizer-tasks
Helm nemo-platform from k8s/helm

Dispatch contracts

Handoff Runs for Client payload
stage-wheels Stable and nightly when wheels are selected ref, cadence, release_label, nightly_timestamp, wheels
release Stable and nightly when containers are selected ref, version, containers
register-release-artifacts Stable only version, source_sha, source_run_url, wheel_ids, container_ids, helm_id
create-release-deployment Stable and nightly after polling when Helm is selected ref, cadence, release_label, helm_version

Helm staging runs directly in this workflow and does not use a repository dispatch. For custom stable releases, unselected ID arrays are empty and helm_id is null.

Before merge

Configure the SLACK_ALERTS_WEBHOOK and SLACK_RELEASE_WEBHOOK Actions secrets used by the notification jobs.

Summary by CodeRabbit

  • New Features
    • Added a unified release.yaml workflow for nightly and stable releases, including artifact selection, validation, optional NGC metadata sync, Helm packaging/publishing, polling/verification, and Slack notifications (with dry-run).
    • Enforced an explicit required secret for NGC metadata synchronization.
  • Removed
    • Removed nightly/RC/stable and release-bundle automation workflows, plus release bundle metadata generation and its unit tests.
    • Removed release asset manifest entries and NGC container description content for nmp-core.
  • Documentation
    • Updated RELEASING.md to reflect the new release flow.
  • Chores
    • Updated .gitignore for local Nektos Act files; improved kind pre-pulling and adjusted docker build targets.

@github-actions github-actions Bot added the ci label Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 23845/31009 76.9% 61.7%
Integration Tests 13816/29658 46.6% 19.6%

@svvarom svvarom self-assigned this Jul 9, 2026
Comment thread .github/workflows/release.yaml Outdated
@crookedstorm
crookedstorm force-pushed the AIREINF-254-unified-release-workflow/smuley branch from 177e61f to d0127e6 Compare July 9, 2026 18:43
Comment thread .github/workflows/release.yaml Outdated
svvarom and others added 19 commits July 10, 2026 18:45
Signed-off-by: Swarom Muley <smuley@nvidia.com>
Signed-off-by: Swarom Muley <smuley@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Swarom Muley <smuley@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
@crookedstorm
crookedstorm force-pushed the AIREINF-254-unified-release-workflow/smuley branch from 2fe7357 to 399e17d Compare July 11, 2026 01:45
@crookedstorm
crookedstorm marked this pull request as ready for review July 11, 2026 01:45
@crookedstorm
crookedstorm requested review from a team as code owners July 11, 2026 01:45
@crookedstorm
crookedstorm requested a review from a team as a code owner July 11, 2026 01:45
@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9d9d1e20-94aa-4b2f-89ce-5411cea028b0

📥 Commits

Reviewing files that changed from the base of the PR and between 6e3d135 and d611a5d.

📒 Files selected for processing (1)
  • docker/base/Dockerfile.nmp-workspace
💤 Files with no reviewable changes (1)
  • docker/base/Dockerfile.nmp-workspace

📝 Walkthrough

Walkthrough

The release process is consolidated into release.yaml, covering artifact selection, validation, staging, publication, polling, GitHub Releases, and Slack notifications. Legacy release automation is removed, while release documentation and local workflow support are updated.

Changes

Release pipeline

Layer / File(s) Summary
Release planning and validation
.github/workflows/release.yaml
Adds release inputs, artifact catalogs, selection checks, artifact validation, and wheel version resolution.
Artifact staging and publication
.github/workflows/release.yaml, .github/workflows/ngc-metadata.yaml
Adds notifications, required NGC workflow secrets, optional metadata synchronization, artifact dispatches, and Helm publication.
Publication verification and completion
.github/workflows/release.yaml
Adds registry polling, delay alerts, stable GitHub Release creation, deployment signaling, and final Slack notifications.
Legacy release automation removal
.github/workflows/release-stable.yaml
Removes the previous stable release workflow.
Release procedure documentation
RELEASING.md
Documents the unified workflow, inputs, catalog, destinations, notifications, secrets, and verification steps.

Build and local workflow support

Layer / File(s) Summary
CPU build target selection
docker-bake.hcl
Removes nmp-core-docker from the CPU build group.
Kind image preloading
.github/actions/setup-kind-cluster/action.yaml
Adds the NMP API image to the images pre-pulled into Kind.
Nektos act file ignores
.gitignore
Ignores .act-variables and .act-secrets.

Suggested reviewers: a2bondar, crookedstorm, maxdubrinsky, tylersbray

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: replacing fragmented release paths with a unified nightly and stable release workflow.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch AIREINF-254-unified-release-workflow/smuley

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/release.yaml:
- Around line 702-705: Update the “Report packaged Helm chart” step to pass the
package output through the step’s env mapping, then reference that environment
variable in the shell command instead of directly expanding
steps.package.outputs.chart_package in run.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f033b4c6-ba65-40c8-80f1-29d0efe65bf9

📥 Commits

Reviewing files that changed from the base of the PR and between 275c8cd and 399e17d.

📒 Files selected for processing (9)
  • .github/scripts/write_release_bundle_metadata.py
  • .github/workflows/ngc-metadata.yaml
  • .github/workflows/release-bundle.yaml
  • .github/workflows/release-nightly.yaml
  • .github/workflows/release-rc.yaml
  • .github/workflows/release-stable.yaml
  • .github/workflows/release.yaml
  • .gitignore
  • tests/unit/release/test_write_release_bundle_metadata.py
💤 Files with no reviewable changes (6)
  • .github/workflows/release-rc.yaml
  • .github/workflows/release-nightly.yaml
  • tests/unit/release/test_write_release_bundle_metadata.py
  • .github/workflows/release-bundle.yaml
  • .github/scripts/write_release_bundle_metadata.py
  • .github/workflows/release-stable.yaml

Comment thread .github/workflows/release.yaml Outdated
@svvarom

svvarom commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@crookedstorm can we also delete https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/release/assets.yaml and delete/update https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/RELEASING.md?

Signed-off-by: Brooke Storm <brookes@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
RELEASING.md (1)

141-153: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing "Next Steps" section.

Doc ends without cross-links to related docs (e.g. release.yaml, CONTRIBUTING). As per coding guidelines, "Include 'Next Steps' section at the end with cross-links to related documentation content."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@RELEASING.md` around lines 141 - 153, Add a “Next Steps” section at the end
of RELEASING.md after the completed-release verification content, including
links to the related release workflow documentation (release.yaml) and
CONTRIBUTING guidance.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@RELEASING.md`:
- Around line 141-153: Add a “Next Steps” section at the end of RELEASING.md
after the completed-release verification content, including links to the related
release workflow documentation (release.yaml) and CONTRIBUTING guidance.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 38d174d9-1689-482f-8192-ec2dafd9f684

📥 Commits

Reviewing files that changed from the base of the PR and between 399e17d and 5c186f8.

📒 Files selected for processing (3)
  • .github/workflows/release.yaml
  • RELEASING.md
  • release/assets.yaml
💤 Files with no reviewable changes (1)
  • release/assets.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/release.yaml

Comment thread .github/workflows/release.yaml
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
Signed-off-by: Brooke Storm <brookes@nvidia.com>
@crookedstorm
crookedstorm enabled auto-merge July 13, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants