docs: add container deployment guides - #695
Conversation
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
WalkthroughThe PR documents published GPU container usage, Docker workflows, Kubernetes Jobs, private workload images, configurable multi-architecture builds, and navigation updates. ChangesRuntime documentation
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Greptile SummaryThe PR adds public-container deployment documentation for Docker, Kubernetes Jobs, and private derived workload images.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Reviews (3): Last reviewed commit: "docs: fix derived workload runtime paths" | Re-trigger Greptile |
|
|
||
| ```bash | ||
| docker run --rm --gpus all --shm-size=1g \ | ||
| --user "$(id -u):$(id -g)" \ |
There was a problem hiding this comment.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
docs/user-guide/docker.md (1)
8-8: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove decorative bold from Markdown body text.
**not**is decorative emphasis in a paragraph. Replace it with plain text.Proposed fix
-installed runtime; it does **not** contain input data or a workload +installed runtime; it does not contain input data or a workloadAs per coding guidelines, Markdown body text must avoid decorative bold and use backticks for code identifiers.
Sources: Coding guidelines, Path instructions
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 90c623e4-46ee-42c9-920e-ed96e5696956
📒 Files selected for processing (8)
README.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/docker.mddocs/user-guide/getting-started.mddocs/user-guide/kubernetes.mddocs/user-guide/private-workload-images.mdmkdocs.yml
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Greptile Review
🧰 Additional context used
📓 Path-based instructions (13)
mkdocs.yml
📄 CodeRabbit inference engine (.cursor/rules/repo-navigation.mdc)
Configure documentation site using MkDocs Material in
mkdocs.ymlKeep documentation navigation synchronized by adding new documentation pages to the
nav:section.
Files:
mkdocs.yml
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: IfAGENTS.local.mdexists, read it and give its instructions top priority.
Read and follow repository-specific skills in.agents/skills/when a task matches their scope.
Follow the detailed coding conventions inSTYLE_GUIDE.md.
Useuvfor project operations, neverpipor rawpython; useuv runfor Python execution.
Usemisetasks or wrapper scripts intools/instead of invokingruffortydirectly.
For a full GPU/development environment, useuv sync --frozen --extra cu129 --extra engine --group dev; bareuv sync --frozenis incomplete.
Do not commit unless the user asks for a commit or PR work. When committing, require DCO sign-off and GPG signing viagit commit --signoff --gpg-sign(or-s -S); never manually addSigned-off-byor use--no-gpg-sign.
Use feature branches based onmain; branch names commonly include an issue-number prefix such as<author>/123-short-name.
Use the documentedmisetasks and matching skills for testing, building, syncing, bootstrapping, worktrees, GitHub, and recurring workflows.End files with a newline, avoid trailing whitespace, use one space between sentences, and keep code, comments, and docstrings within 120 characters.
**/*: Do not move a published release tag; if release code changes, create and validate the next release candidate instead.
The stable release tag must point to the same tested commit SHA as the validated release candidate.
Files:
mkdocs.ymlREADME.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
⚙️ CodeRabbit configuration file
**/*: Review as a senior maintainer for NeMo Safe Synthesizer. Prioritize issues that can change behavior, break user workflows, weaken privacy guarantees, hide failures, make tests unreliable, or create maintenance risk. Avoid generic style commentary unless it points to a concrete project convention that automated tools will not catch.
Comment only when the finding is actionable and tied to changed code. For each finding, state the impact, the condition that triggers it, and the smallest practical fix. Prefer one precise comment over broad advice. Do not ask for refactors outside the PR scope unless the changed code creates the problem.
Review type guidance: - Potential issue: use for correctness bugs, data loss, privacy leaks,
security risks, broken public APIs, invalid config behavior, missing
validation, hidden failures, nondeterministic tests, or CI breakage.
- Refactor suggestion: use for local maintainability problems introduced
by the diff when they have clear future cost, such as duplicated setup,
unclear boundaries, over-mocking, avoidable complexity, or opaque test
helpers.- Nitpick: avoid in chill mode. Do not emit formatting, import-order,
wording, or style-only comments unless automated tools cannot catch the
issue and it affects maintainability.Severity guidance: - Critical: security/privacy leaks, data loss, training/test/holdout
contamination, or broken release/package/core pipeline execution.
- Major: incorrect generation/training/evaluation behavior, broken
CLI/SDK public API, invalid config defaults or validators, or GPU/vLLM
cleanup and process-isolation bugs likely to fail CI or production
runs.- Minor: localized bugs, missing focused tests for changed behavior, or
bad test patterns that weaken regression coverage.- Trivial: small cleanup with no behavior impact. Usually suppress in
chill mode.- Info: context only. Avoid unless it helps reviewers understand risk.
Safe-Synthesizer-specific review focus: - Data ...
Files:
mkdocs.ymlREADME.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
**/*.{yaml,yml}
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Use two-space indentation,
:for mappings, unquoted values unless required, a final newline, and SPDX headers. GitHub Actions section dividers use#followed by dashes.
Files:
mkdocs.yml
**/*.{py,sh,yaml,yml,md}
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Every source file requires the appropriate SPDX copyright and license header; Markdown files with YAML frontmatter place hash-comment headers inside the frontmatter.
**/*.{py,sh,yaml,yml,md}: All source files with.py,.sh,.yaml,.yml, or.mdextensions require SPDX copyright headers, except files listed in.copyrightignore.
Runmise run formatandmise run checkbefore contributing so formatting, linting, type checking, and copyright checks pass.
Files:
mkdocs.ymlREADME.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
**/*.yml
⚙️ CodeRabbit configuration file
Review YAML for 2-space indentation, SPDX headers when required, unquoted values unless needed, and newline at EOF.
Files:
mkdocs.yml
**/*.{md,markdown,py}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown,py}: Avoid decorative bold (**text**) in list items, body text, and docstrings; use structural cues (headers, list markers, colons, backticks) for emphasis instead
Use backticks for code identifiers, paths, and CLI commands in markdown and docstrings
Files:
README.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
**/*.{md,markdown}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown}: Bold is acceptable only in markdown tables where it's the conventional way to mark header-like cells in the body
Use##headers to segment markdown sections instead of bold text
Use--(em-dash) instead of-(hyphen) for asides in markdown
Files:
README.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
**/*.{py,md,sh,Dockerfile}
📄 CodeRabbit inference engine (AGENTS.md)
Keep durable implementation guidance in public function/class docstrings or local source comments; keep test-suite guidance in
tests/TESTING.md.
Files:
README.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
**/*.md
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Use single backticks for code, paths, and CLI commands; do not use decorative bold in body text; use
--for asides; and include the required SPDX HTML-comment header unless YAML frontmatter requires hash comments inside the frontmatter.Use the repository's documented Markdown conventions and MkDocs Material extensions for documentation content.
Files:
README.mddocs/user-guide/private-workload-images.mdcontainers/README.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
README.md
⚙️ CodeRabbit configuration file
Treat README.md as the project overview. Check that setup, usage, and links stay consistent with CONTRIBUTING.md, Makefile, and docs/.
Files:
README.md
docs/**/*.md
📄 CodeRabbit inference engine (.cursor/rules/writing-docs.mdc)
docs/**/*.md: Use MkDocs Material admonition syntax (!!! note, !!! warning, ??? tip) for highlighting important information and collapsible sections in documentation
Use MkDocs Material tabs syntax (=== "Label") to present alternative views or language-specific examples in documentation
Use code block syntax with title and highlight line parameters (title="filename", hl_lines="2 3") for code examples in documentation
Use Mermaid diagram syntax (```mermaid flowchart, etc.) for visualizations in documentationClassify documentation using the Diátaxis categories and use MkDocs Material syntax for admonitions, tabs, titled code blocks, and highlights.
Documentation pages must live under
docs/in the appropriate Diataxis subdirectory, and new or edited pages must be added to thenav:section ofmkdocs.yml.
Files:
docs/user-guide/private-workload-images.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
docs/**
⚙️ CodeRabbit configuration file
Review documentation as MkDocs Material content. Check Diataxis fit, accurate commands, internal links, code fences, and markdown style from STYLE_GUIDE.md.
Files:
docs/user-guide/private-workload-images.mddocs/developer-guide/docker.mddocs/user-guide/kubernetes.mddocs/user-guide/getting-started.mddocs/user-guide/docker.md
containers/**
⚙️ CodeRabbit configuration file
Review container changes against STYLE_GUIDE.md and containers/README.md. Check cache mounts, uv usage, non-root runtime behavior, CUDA settings, and copy order.
Files:
containers/README.md
🧠 Learnings (1)
📚 Learning: 2026-08-03T18:08:15.626Z
Learnt from: zywind
Repo: NVIDIA-NeMo/Safe-Synthesizer PR: 687
File: README.md:7-15
Timestamp: 2026-08-03T18:08:15.626Z
Learning: Do not flag missing SPDX copyright or license headers in README.md, CONTRIBUTING.md, or SECURITY.md. These community-facing Markdown files are explicitly exempt through .copyrightignore; tools/codestyle/copyright_fixer.py, run via `mise run format`, is the authoritative check for this requirement.
Applied to files:
README.md
🔇 Additional comments (8)
README.md (1)
43-47: LGTM!docs/user-guide/docker.md (1)
6-7: LGTM!Also applies to: 9-9, 11-35, 37-104, 114-118, 119-192
docs/user-guide/getting-started.md (1)
103-119: LGTM!docs/user-guide/kubernetes.md (1)
1-201: LGTM!docs/user-guide/private-workload-images.md (1)
1-23: LGTM!Also applies to: 32-74, 90-100
containers/README.md (1)
116-118: LGTM!docs/developer-guide/docker.md (1)
280-290: LGTM!mkdocs.yml (1)
184-185: LGTM!
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
zywind
left a comment
There was a problem hiding this comment.
Review performed by an AI agent (Claude Code) on my behalf; I've checked the findings against the repo before posting.
Docs read well and the technical claims check out — I verified ghcr.io/nvidia-nemo/safe-synthesizer:latest-cu129 exists in GHCR and matches the tag scheme in container-build.yml, that --artifact-path/--run-path are real options (and --run-path is valid on run generate, since common_run_options is applied to it), and that the uid/gid 1000 appuser, WORKDIR /workspace, HF_HOME, tini entrypoint, and 256 MiB /dev/shm warning all match containers/Dockerfile.cuda / containers/entrypoint.sh. All cross-page anchors used by the new pages resolve.
The 0440 + --user "$(id -u):$(id -g)" conflict in private-workload-images.md is already flagged by CodeRabbit and Greptile, and CodeRabbit's note about the derived-image Kubernetes args still pointing at /workspace/config and /workspace/input instead of /workspace/workload is correct — no need for me to repeat those. Two additional items that only show up when reading the unchanged pages this PR now delegates to:
1. HF_TOKEN guidance is lost (docs/user-guide/docker.md)
The old page's secrets table documented HF_TOKEN: what it's for, where to get one, and that a token already present in the mounted HF cache is sufficient. The new Secrets section replaces it with "The complete variables and their purposes are in Environment Variables" — but environment.md's master reference table has no HF_TOKEN row (only NSS_INFERENCE_KEY and WANDB_API_KEY). After this PR, the only remaining mentions in the user guide are the hf_token_missing preflight code in troubleshooting.md and the secretKeyRef in the new Kubernetes template, so a user hitting a gated model has nowhere to land.
Suggest adding HF_TOKEN / HUGGING_FACE_HUB_TOKEN rows to environment.md, or keeping a short secrets table on the Docker page.
2. Two stale pointers into the rewritten page (docs/user-guide/environment.md)
Both still resolve as links, so no link checker will catch them:
- Line 271: "See Docker for mount paths, secrets, GPU flags, and mise container tasks." The Mise Container Tasks section was removed from that page — it now lives only in
docs/developer-guide/docker.mdandcontainers/README.md, so the phrase should be retargeted or dropped. - Line 63:
NVIDIA_VISIBLE_DEVICES→ "Limit GPUs inside a container" links todocker.md#gpu-access, but the condensed GPU Access section no longer shows the--gpus '"device=0,1"'/-e NVIDIA_VISIBLE_DEVICES=0,1examples that answered it.
Minor/optional: the Job template has no ttlSecondsAfterFinished, so completed Jobs linger — worth a line given the page is explicitly a portable starting point. (defaultMode: 0444 parses as octal correctly, so that one is fine as written.)
| pages define configuration and credential requirements without binding them to | ||
| a particular secret-management system. | ||
|
|
||
| ## Portable Job template |
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/user-guide/private-workload-images.md (1)
20-26: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd titles and highlights to the changed code blocks.
The Dockerfile, shell, and Kubernetes examples use language-only fences. Add
title="..."andhl_lines="..."to each changed fence. Apply this to the Dockerfile, build/push, Docker run, and Kubernetes YAML examples.As per coding guidelines,
docs/**/*.mdcode examples must use titled code blocks with highlight parameters.Also applies to: 38-41, 75-83, 94-103
Sources: Coding guidelines, Path instructions
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 669386d5-43c7-4c9c-9c3d-445b27a67073
📒 Files selected for processing (1)
docs/user-guide/private-workload-images.md
📜 Review details
⏰ Context from checks skipped due to timeout. (6)
- GitHub Check: Unit Tests (3.12)
- GitHub Check: Unit Tests (3.11)
- GitHub Check: Unit Tests (3.13)
- GitHub Check: Unit Tests (3.14)
- GitHub Check: Greptile Review
- GitHub Check: Analyze (Python)
🧰 Additional context used
📓 Path-based instructions (8)
**/*.{md,markdown,py}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown,py}: Avoid decorative bold (**text**) in list items, body text, and docstrings; use structural cues (headers, list markers, colons, backticks) for emphasis instead
Use backticks for code identifiers, paths, and CLI commands in markdown and docstrings
Files:
docs/user-guide/private-workload-images.md
**/*.{md,markdown}
📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)
**/*.{md,markdown}: Bold is acceptable only in markdown tables where it's the conventional way to mark header-like cells in the body
Use##headers to segment markdown sections instead of bold text
Use--(em-dash) instead of-(hyphen) for asides in markdown
Files:
docs/user-guide/private-workload-images.md
docs/**/*.md
📄 CodeRabbit inference engine (.cursor/rules/writing-docs.mdc)
docs/**/*.md: Use MkDocs Material admonition syntax (!!! note, !!! warning, ??? tip) for highlighting important information and collapsible sections in documentation
Use MkDocs Material tabs syntax (=== "Label") to present alternative views or language-specific examples in documentation
Use code block syntax with title and highlight line parameters (title="filename", hl_lines="2 3") for code examples in documentation
Use Mermaid diagram syntax (```mermaid flowchart, etc.) for visualizations in documentationClassify documentation using the Diátaxis categories and use MkDocs Material syntax for admonitions, tabs, titled code blocks, and highlights.
Documentation pages must live under
docs/in the appropriate Diataxis subdirectory, and new or edited pages must be added to thenav:section ofmkdocs.yml.
Files:
docs/user-guide/private-workload-images.md
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: IfAGENTS.local.mdexists, read it and give its instructions top priority.
Read and follow repository-specific skills in.agents/skills/when a task matches their scope.
Follow the detailed coding conventions inSTYLE_GUIDE.md.
Useuvfor project operations, neverpipor rawpython; useuv runfor Python execution.
Usemisetasks or wrapper scripts intools/instead of invokingruffortydirectly.
For a full GPU/development environment, useuv sync --frozen --extra cu129 --extra engine --group dev; bareuv sync --frozenis incomplete.
Do not commit unless the user asks for a commit or PR work. When committing, require DCO sign-off and GPG signing viagit commit --signoff --gpg-sign(or-s -S); never manually addSigned-off-byor use--no-gpg-sign.
Use feature branches based onmain; branch names commonly include an issue-number prefix such as<author>/123-short-name.
Use the documentedmisetasks and matching skills for testing, building, syncing, bootstrapping, worktrees, GitHub, and recurring workflows.End files with a newline, avoid trailing whitespace, use one space between sentences, and keep code, comments, and docstrings within 120 characters.
**/*: Do not move a published release tag; if release code changes, create and validate the next release candidate instead.
The stable release tag must point to the same tested commit SHA as the validated release candidate.
Files:
docs/user-guide/private-workload-images.md
⚙️ CodeRabbit configuration file
**/*: Review as a senior maintainer for NeMo Safe Synthesizer. Prioritize issues that can change behavior, break user workflows, weaken privacy guarantees, hide failures, make tests unreliable, or create maintenance risk. Avoid generic style commentary unless it points to a concrete project convention that automated tools will not catch.
Comment only when the finding is actionable and tied to changed code. For each finding, state the impact, the condition that triggers it, and the smallest practical fix. Prefer one precise comment over broad advice. Do not ask for refactors outside the PR scope unless the changed code creates the problem.
Review type guidance: - Potential issue: use for correctness bugs, data loss, privacy leaks,
security risks, broken public APIs, invalid config behavior, missing
validation, hidden failures, nondeterministic tests, or CI breakage.
- Refactor suggestion: use for local maintainability problems introduced
by the diff when they have clear future cost, such as duplicated setup,
unclear boundaries, over-mocking, avoidable complexity, or opaque test
helpers.- Nitpick: avoid in chill mode. Do not emit formatting, import-order,
wording, or style-only comments unless automated tools cannot catch the
issue and it affects maintainability.Severity guidance: - Critical: security/privacy leaks, data loss, training/test/holdout
contamination, or broken release/package/core pipeline execution.
- Major: incorrect generation/training/evaluation behavior, broken
CLI/SDK public API, invalid config defaults or validators, or GPU/vLLM
cleanup and process-isolation bugs likely to fail CI or production
runs.- Minor: localized bugs, missing focused tests for changed behavior, or
bad test patterns that weaken regression coverage.- Trivial: small cleanup with no behavior impact. Usually suppress in
chill mode.- Info: context only. Avoid unless it helps reviewers understand risk.
Safe-Synthesizer-specific review focus: - Data ...
Files:
docs/user-guide/private-workload-images.md
**/*.{py,md,sh,Dockerfile}
📄 CodeRabbit inference engine (AGENTS.md)
Keep durable implementation guidance in public function/class docstrings or local source comments; keep test-suite guidance in
tests/TESTING.md.
Files:
docs/user-guide/private-workload-images.md
**/*.md
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Use single backticks for code, paths, and CLI commands; do not use decorative bold in body text; use
--for asides; and include the required SPDX HTML-comment header unless YAML frontmatter requires hash comments inside the frontmatter.Use the repository's documented Markdown conventions and MkDocs Material extensions for documentation content.
Files:
docs/user-guide/private-workload-images.md
**/*.{py,sh,yaml,yml,md}
📄 CodeRabbit inference engine (STYLE_GUIDE.md)
Every source file requires the appropriate SPDX copyright and license header; Markdown files with YAML frontmatter place hash-comment headers inside the frontmatter.
**/*.{py,sh,yaml,yml,md}: All source files with.py,.sh,.yaml,.yml, or.mdextensions require SPDX copyright headers, except files listed in.copyrightignore.
Runmise run formatandmise run checkbefore contributing so formatting, linting, type checking, and copyright checks pass.
Files:
docs/user-guide/private-workload-images.md
docs/**
⚙️ CodeRabbit configuration file
Review documentation as MkDocs Material content. Check Diataxis fit, accurate commands, internal links, code fences, and markdown style from STYLE_GUIDE.md.
Files:
docs/user-guide/private-workload-images.md
🔇 Additional comments (1)
docs/user-guide/private-workload-images.md (1)
28-36: LGTM!Also applies to: 43-44, 46-68, 70-73, 85-88, 90-92, 105-106, 108-116
Summary
Came up during a customer call that we needed to update our public docs to reflect using the new container image. This pr does the following:
Validation
Summary by CodeRabbit
Documentation