feat(agents): add deployments runner backend for container agents#590
Open
arpitsardhana wants to merge 5 commits into
Open
feat(agents): add deployments runner backend for container agents#590arpitsardhana wants to merge 5 commits into
arpitsardhana wants to merge 5 commits into
Conversation
Wire nemo-agents to the nemo-deployments plugin so agents can run as containers instead of local subprocesses. The new DeploymentsRunnerBackend translates the RunnerBackend interface into Deployment/DeploymentConfig entity operations, which the deployments controller reconciles onto a configured executor. Because nemo-deployments already targets both Docker and Kubernetes, substrate selection is config-only (deployments.executor). - New DeploymentsRunnerBackend (create/status/delete/list/health/logs), status mapping, and loopback->host.docker.internal gateway URL rewrite so containerized agents can reach the host Inference Gateway. - Add "deployments" branch to the runner registry (lazy import) plus a DeploymentsRunnerConfig (executor, default_image, container_port, gateway_url_override). - Thread an image field end-to-end (entity, schema, API) and add the image param to the RunnerBackend ABC / in-memory backend. - Controller adopts the async-assigned endpoint on health check. - Optional [deployments] extra and workspace source for the plugin. - Unit tests for status mapping, gateway URL rewrite, config shape, and the entity lifecycle via a mocked entity client. Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
Contributor
|
Complete the deployments (container) runner backend end to end: - CLI: add `--image`/`-i` and repeatable `--env`/`-e` (KEY=VALUE or bare KEY forwarded from the shell) to `nemo agents deploy`; thread image + env through schema, API, entity, controller, and backend. - deployments_backend: delete the DeploymentConfig only after the Deployment entity is gone (bounded 30s poll), so the deployments controller's _load_configs no longer 404s every cycle during teardown. - in_memory: accept (and ignore) the new env param for interface parity. - examples/Dockerfile: patch nvidia-nat's NIM LangChain builder to exclude `verify_ssl` so it stops leaking into the request body under `nat serve`. - README: document container-backend setup gotchas (controllers, Docker socket, config, env vars, verify_ssl). Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
The image/env additions to the agent deployment API left the plugin OpenAPI spec and the Orval-generated web SDK out of sync, failing the "Generated files are out of sync" lint check. Regenerate both. Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
The prior regen used a stale local orval (8.5.3) that stripped the `.ts` import extension, diverging from the lockfile-pinned orval (8.18.0) CI uses. Regenerate with frozen deps so the web SDK matches CI output. Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
Drop the verbose deployments-backend gotchas section; keep a concise reference for the --image/--env flags on `nemo agents deploy`. Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
Contributor
📝 WalkthroughWalkthroughAdds a container-based "deployments" runner backend for nemo-agents: new config model, entity/schema/OpenAPI fields for image/env, CLI flags, a full DeploymentsRunnerBackend implementation using nemo-deployments entities, registry/controller wiring, unit tests, docs, and a Dockerfile patch. ChangesDeployments Backend Feature
Sequence Diagram(s)sequenceDiagram
participant CLI as nemo agents deploy
participant API as deployments.py (API)
participant Controller as AgentDeploymentController
participant Backend as DeploymentsRunnerBackend
participant Deployments as nemo-deployments entities
CLI->>API: POST create_deployment(image, env)
API->>API: create AgentDeployment(image, env)
Controller->>Backend: create_deployment(workspace, name, config, port, image, env)
Backend->>Deployments: create DeploymentConfig
Backend->>Deployments: create Deployment (PENDING)
Backend-->>Controller: DeploymentInfo(status=starting)
Controller->>Backend: health_check / get_deployment_status
Backend->>Deployments: fetch Deployment status/endpoint
Deployments-->>Backend: status, endpoint
Backend-->>Controller: DeploymentInfo(status, endpoint)
Controller->>Controller: update dep.endpoint
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Trivy (0.69.3)Trivy execution failed: 2026-07-08T00:50:52Z FATAL Fatal error run error: fs scan error: scan error: scan failed: failed analysis: post analysis error: post analysis error: cloudformation scan error: fs filter error: fs filter error: walk error range error: stat .coderabbit-oasdiff.yaml: no such file or directory: range error: stat .coderabbit-oasdiff.yaml: no such file or directory Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py (1)
180-188: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winCaller-supplied
--envcan silently override platform-critical vars.
container_env.update(env)(line 188) applies afterNMP_GATEWAY_BASE_URL/NMP_WORKSPACE/NMP_AGENT_NAMEare set, so a user passing--env NMP_GATEWAY_BASE_URL=...overrides the platform-computed gateway URL. Given the module docstring's stated assumption that "Gateway routing is supplied to the container through env vars," an accidental collision breaks the agent's ability to reach the gateway with no validation/warning.Consider rejecting or warning on caller-supplied keys with the
NMP_prefix before merging.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py` around lines 180 - 188, Caller-supplied env merging in deployments_backend.py can overwrite platform-critical NMP_* settings like NMP_GATEWAY_BASE_URL without warning. Update the deployment env assembly around container_env and the env merge in the deployment backend to reject or warn on any caller-supplied keys with the NMP_ prefix before applying container_env.update(env), while still allowing non-conflicting user env vars to pass through.plugins/nemo-agents/tests/unit/test_runner_deployments.py (1)
97-136: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMissing coverage for
health_checkandshutdown.Good coverage of status mapping, create/delete/list, but
health_check(HTTP GET path) andshutdown(client cleanup) aren't exercised.health_checkgates the starting→running transition, so it's worth a direct test (e.g. mockedhttpx.AsyncClient.getfor success/failure/exception paths).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/tests/unit/test_runner_deployments.py` around lines 97 - 136, Add unit coverage for the DeploymentBackend health and shutdown paths that are currently untested. In test_runner_deployments.py, extend the existing backend tests (or add a focused test near test_status_delete_and_list) to exercise DeploymentBackend.health_check with mocked httpx.AsyncClient.get for success, non-200, and exception cases so the starting-to-running transition is covered. Also add a test for DeploymentBackend.shutdown that verifies the client cleanup/close behavior is invoked when shutting down.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-agents/src/nemo_agents_plugin/entities.py`:
- Around line 119-122: The AgentDeployment.env field is currently persisted and
returned as raw values, which can leak secrets from create/list/get responses.
Update the AgentDeployment model and the create/read serialization path in
entities.py and the related deployment handling so env is stored as secret
references or redacted before persistence and exposure, and make sure any
read/list/get helpers no longer emit the original values.
In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py`:
- Around line 189-208: The create_deployment flow leaves an orphaned
DeploymentConfig when the subsequent Deployment create fails. Update the logic
in deployments_backend.py around build_deployment_config, entities.create, and
Deployment creation so the two writes are handled atomically from a cleanup
perspective: if creating the Deployment entity fails after the DeploymentConfig
is persisted, delete the previously created config before re-raising. Keep the
existing logging and return path unchanged, but make sure retries for the same
dep_name do not hit a stale config.
- Around line 222-273: The bounded wait in delete_deployment() blocks the
sequential reconcile loop, so a stuck Deployment can stall later deployments for
up to _DELETE_CONFIG_WAIT_S. Update delete_deployment() and/or
_wait_for_deployment_gone() to avoid awaiting the polling wait on the main
reconcile path, such as by making the config cleanup non-blocking or deferring
the wait so reconcile can continue processing other deployments.
---
Nitpick comments:
In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py`:
- Around line 180-188: Caller-supplied env merging in deployments_backend.py can
overwrite platform-critical NMP_* settings like NMP_GATEWAY_BASE_URL without
warning. Update the deployment env assembly around container_env and the env
merge in the deployment backend to reject or warn on any caller-supplied keys
with the NMP_ prefix before applying container_env.update(env), while still
allowing non-conflicting user env vars to pass through.
In `@plugins/nemo-agents/tests/unit/test_runner_deployments.py`:
- Around line 97-136: Add unit coverage for the DeploymentBackend health and
shutdown paths that are currently untested. In test_runner_deployments.py,
extend the existing backend tests (or add a focused test near
test_status_delete_and_list) to exercise DeploymentBackend.health_check with
mocked httpx.AsyncClient.get for success, non-200, and exception cases so the
starting-to-running transition is covered. Also add a test for
DeploymentBackend.shutdown that verifies the client cleanup/close behavior is
invoked when shutting down.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: edb19f0a-3f52-4023-b095-90da36ca0cf0
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (15)
plugins/nemo-agents/README.mdplugins/nemo-agents/examples/Dockerfileplugins/nemo-agents/openapi/openapi.yamlplugins/nemo-agents/pyproject.tomlplugins/nemo-agents/src/nemo_agents_plugin/api/v2/deployments.pyplugins/nemo-agents/src/nemo_agents_plugin/cli.pyplugins/nemo-agents/src/nemo_agents_plugin/config.pyplugins/nemo-agents/src/nemo_agents_plugin/entities.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/backend.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/controller.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/in_memory.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/registry.pyplugins/nemo-agents/src/nemo_agents_plugin/schema.pyplugins/nemo-agents/tests/unit/test_runner_deployments.py
Comment on lines
+119
to
+122
| env: dict[str, str] = Field( | ||
| default_factory=dict, | ||
| description="Environment variables injected into the agent container. Used by the 'deployments' backend.", | ||
| ) |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map relevant files and inspect the mentioned entity/API/openapi references.
git ls-files \
plugins/nemo-agents/src/nemo_agents_plugin/entities.py \
plugins/nemo-agents/src/nemo_agents_plugin/api/v2/deployments.py \
openapi.yaml \
| sed -n '1,120p'
printf '\n--- entities.py outline ---\n'
ast-grep outline plugins/nemo-agents/src/nemo_agents_plugin/entities.py --view expanded || true
printf '\n--- deployments.py outline ---\n'
ast-grep outline plugins/nemo-agents/src/nemo_agents_plugin/api/v2/deployments.py --view expanded || true
printf '\n--- search for env / secret refs ---\n'
rg -n "env: dict\\[str, str\\]|anthropic_api_key_secret|secret|deployments|saved entity|NVIDIA_API_KEY|Field\\(" \
plugins/nemo-agents/src/nemo_agents_plugin \
openapi.yamlRepository: NVIDIA-NeMo/nemo-platform
Length of output: 37902
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the relevant slices with line numbers.
printf '\n--- entities.py around lines 90-140 ---\n'
sed -n '90,140p' plugins/nemo-agents/src/nemo_agents_plugin/entities.py | cat -n
printf '\n--- deployments.py around likely response construction ---\n'
sed -n '1,260p' plugins/nemo-agents/src/nemo_agents_plugin/api/v2/deployments.py | cat -n
printf '\n--- openapi.yaml secret-related snippets ---\n'
rg -n -A4 -B4 "anthropic_api_key_secret|AnalyzeBatchConfig|EvaluateSuiteConfig|env" openapi.yamlRepository: NVIDIA-NeMo/nemo-platform
Length of output: 11807
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Narrowly inspect the exact lines and surrounding API behavior.
sed -n '110,130p' plugins/nemo-agents/src/nemo_agents_plugin/entities.py | cat -n
printf '\n---\n'
sed -n '120,220p' plugins/nemo-agents/src/nemo_agents_plugin/api/v2/deployments.py | cat -n
printf '\n---\n'
sed -n '1,220p' plugins/nemo-agents/src/nemo_agents_plugin/api/v2/deployments.py | cat -n | sed -n '1,220p'Repository: NVIDIA-NeMo/nemo-platform
Length of output: 14176
Do not persist raw env values
AgentDeployment.env is saved and returned verbatim by create/list/get, so CLI-passed secrets like NVIDIA_API_KEY can leak to anyone with deployment read access. Use secret refs or redact this field before persistence/serialization.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-agents/src/nemo_agents_plugin/entities.py` around lines 119 -
122, The AgentDeployment.env field is currently persisted and returned as raw
values, which can leak secrets from create/list/get responses. Update the
AgentDeployment model and the create/read serialization path in entities.py and
the related deployment handling so env is stored as secret references or
redacted before persistence and exposure, and make sure any read/list/get
helpers no longer emit the original values.
Comment on lines
+189
to
+208
| deployment_config = build_deployment_config( | ||
| name=dep_name, | ||
| workspace=workspace, | ||
| image=resolved_image, | ||
| port=self._config.container_port, | ||
| env=container_env, | ||
| ) | ||
| await entities.create(deployment_config) | ||
| deployment = Deployment( | ||
| name=dep_name, | ||
| workspace=workspace, | ||
| deployment_config=dep_name, | ||
| executor=self._config.executor, | ||
| status="PENDING", | ||
| ) | ||
| await entities.create(deployment) | ||
| logger.info("Created deployment entities '%s/%s' (image=%s)", workspace, dep_name, resolved_image) | ||
| # Endpoint is assigned asynchronously once the deployments controller schedules | ||
| # the container; the agents controller adopts it on a later reconcile cycle. | ||
| return DeploymentInfo(name=name, status="starting", endpoint="") |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Orphaned DeploymentConfig if second create fails.
entities.create(deployment_config) (line 196) and entities.create(deployment) (line 204) are two independent writes with no compensation. If the second call fails (network blip, conflict, validation error), the DeploymentConfig entity is left behind under dep_name. A retry of create_deployment for the same agent name will then attempt to create a DeploymentConfig that already exists, likely failing again — leaving the deployment permanently stuck at failed until someone manually deletes the orphaned config.
🔧 Proposed fix: clean up config on failed deployment creation
await entities.create(deployment_config)
deployment = Deployment(
name=dep_name,
workspace=workspace,
deployment_config=dep_name,
executor=self._config.executor,
status="PENDING",
)
- await entities.create(deployment)
+ try:
+ await entities.create(deployment)
+ except Exception:
+ await entities.delete(DeploymentConfig, name=dep_name, workspace=workspace)
+ raise📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| deployment_config = build_deployment_config( | |
| name=dep_name, | |
| workspace=workspace, | |
| image=resolved_image, | |
| port=self._config.container_port, | |
| env=container_env, | |
| ) | |
| await entities.create(deployment_config) | |
| deployment = Deployment( | |
| name=dep_name, | |
| workspace=workspace, | |
| deployment_config=dep_name, | |
| executor=self._config.executor, | |
| status="PENDING", | |
| ) | |
| await entities.create(deployment) | |
| logger.info("Created deployment entities '%s/%s' (image=%s)", workspace, dep_name, resolved_image) | |
| # Endpoint is assigned asynchronously once the deployments controller schedules | |
| # the container; the agents controller adopts it on a later reconcile cycle. | |
| return DeploymentInfo(name=name, status="starting", endpoint="") | |
| deployment_config = build_deployment_config( | |
| name=dep_name, | |
| workspace=workspace, | |
| image=resolved_image, | |
| port=self._config.container_port, | |
| env=container_env, | |
| ) | |
| await entities.create(deployment_config) | |
| deployment = Deployment( | |
| name=dep_name, | |
| workspace=workspace, | |
| deployment_config=dep_name, | |
| executor=self._config.executor, | |
| status="PENDING", | |
| ) | |
| try: | |
| await entities.create(deployment) | |
| except Exception: | |
| await entities.delete(DeploymentConfig, name=dep_name, workspace=workspace) | |
| raise | |
| logger.info("Created deployment entities '%s/%s' (image=%s)", workspace, dep_name, resolved_image) | |
| # Endpoint is assigned asynchronously once the deployments controller schedules | |
| # the container; the agents controller adopts it on a later reconcile cycle. | |
| return DeploymentInfo(name=name, status="starting", endpoint="") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py`
around lines 189 - 208, The create_deployment flow leaves an orphaned
DeploymentConfig when the subsequent Deployment create fails. Update the logic
in deployments_backend.py around build_deployment_config, entities.create, and
Deployment creation so the two writes are handled atomically from a cleanup
perspective: if creating the Deployment entity fails after the DeploymentConfig
is persisted, delete the previously created config before re-raising. Keep the
existing logging and return path unchanged, but make sure retries for the same
dep_name do not hit a stale config.
Comment on lines
+222
to
+273
| async def delete_deployment(self, workspace: str, name: str) -> bool: | ||
| entities = self._entity_client() | ||
| dep_name = self._dep_name(name) | ||
| found = False | ||
| try: | ||
| deployment = await entities.get(Deployment, name=dep_name, workspace=workspace) | ||
| if deployment.status != "DELETING": | ||
| deployment.status = "DELETING" | ||
| await entities.update(deployment) | ||
| found = True | ||
| except NemoEntityNotFoundError: | ||
| pass | ||
| # The deployments controller removes the container and deletes the Deployment entity | ||
| # on DELETING; the DeploymentConfig is ours to clean up. Wait for the Deployment to | ||
| # be gone first — deleting the config while the controller still holds a DELETING | ||
| # Deployment makes its _load_configs 404 every cycle (harmless but noisy log spam). | ||
| if found: | ||
| await self._wait_for_deployment_gone(workspace, dep_name) | ||
| try: | ||
| await entities.delete(DeploymentConfig, name=dep_name, workspace=workspace) | ||
| found = True | ||
| except NemoEntityNotFoundError: | ||
| pass | ||
| return found | ||
|
|
||
| async def _wait_for_deployment_gone(self, workspace: str, dep_name: str) -> None: | ||
| """Block until the Deployment entity is deleted or the wait budget elapses. | ||
|
|
||
| Args: | ||
| workspace: Workspace the Deployment belongs to. | ||
| dep_name: Prefixed Deployment entity name. | ||
|
|
||
| """ | ||
| # ponytail: bounded polling blocks this reconcile iteration for up to | ||
| # _DELETE_CONFIG_WAIT_S; on timeout we delete the config anyway and accept a | ||
| # single 404 warning rather than leak the config or hang the loop. Upgrade path | ||
| # is a multi-cycle, reconcile-driven delete if this ever needs to be non-blocking. | ||
| entities = self._entity_client() | ||
| deadline = time.monotonic() + _DELETE_CONFIG_WAIT_S | ||
| while time.monotonic() < deadline: | ||
| try: | ||
| await entities.get(Deployment, name=dep_name, workspace=workspace) | ||
| except NemoEntityNotFoundError: | ||
| return | ||
| await asyncio.sleep(_DELETE_CONFIG_POLL_S) | ||
| logger.warning( | ||
| "Deployment '%s/%s' still present after %.0fs; deleting its config anyway.", | ||
| workspace, | ||
| dep_name, | ||
| _DELETE_CONFIG_WAIT_S, | ||
| ) | ||
|
|
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check how NemoController base class drives reconcile_one across objects (sequential vs concurrent).
fd -e py | xargs rg -n -B2 -A15 'def reconcile\b|reconcile_one' --type=py -g '!*/tests/*'Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50384
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,260p' plugins/nemo-agents/src/nemo_agents_plugin/runner/controller.py
printf '\n---\n'
sed -n '1,220p' packages/nemo_platform_plugin/src/nemo_platform_plugin/controller.py
printf '\n---\n'
rg -n "_wait_for_deployment_gone|delete_deployment\(" plugins/nemo-agents/src/nemo_agents_plugin/runner -nRepository: NVIDIA-NeMo/nemo-platform
Length of output: 18883
30s delete wait blocks the whole reconcile loop. delete_deployment() is awaited inside the sequential reconcile path, so one stuck DELETING deployment can delay every later deployment in the cycle for up to _DELETE_CONFIG_WAIT_S. If that tradeoff is intentional, keep the cap; otherwise move the wait off the main reconcile path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py`
around lines 222 - 273, The bounded wait in delete_deployment() blocks the
sequential reconcile loop, so a stuck Deployment can stall later deployments for
up to _DELETE_CONFIG_WAIT_S. Update delete_deployment() and/or
_wait_for_deployment_gone() to avoid awaiting the polling wait on the main
reconcile path, such as by making the config cleanup non-blocking or deferring
the wait so reconcile can continue processing other deployments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires
nemo-agentsto thenemo-deploymentsplugin so agents can run as containers in addition to localnatsubprocesses. A newDeploymentsRunnerBackendtranslates the existingRunnerBackendinterface intoDeployment/DeploymentConfigentity operations, which the deployments controller reconciles onto a configured executor.Because
nemo-deploymentsalready targets both Docker and Kubernetes, substrate selection is config-only (deployments.executor) — no agents-side code change is needed to move from Docker to K8s later.Assumes the agent image is self-contained (NAT config baked in); gateway routing is passed via env vars rather than config rewriting.
Changes
runner/deployments_backend.py—DeploymentsRunnerBackend:create_deploymentwrites a prefixed (agent-<name>)DeploymentConfig+Deployment;get_deployment_status,delete_deployment,list_deployments,health_check(GET /health),get_log_location,shutdown.delete_deploymentsetsDELETING, then waits (bounded 30s poll) for the deployments controller to remove theDeploymententity before deleting theDeploymentConfig— so the controller's_load_configsnever 404s on a config we deleted out from under it.map_status,container_gateway_url(rewrites loopback host →host.docker.internalso the container reaches the host Inference Gateway),build_deployment_config.runner/registry.py— lazy-import"deployments"branch.config.py—DeploymentsRunnerConfig(executor,default_image,container_port,gateway_url_override).runner/backend.py+in_memory.py—imageandenvparams oncreate_deployment(in-memory ignores them).runner/controller.py— passesdep.image/dep.env; adopts the async-assigned endpoint on health check.entities.py/schema.py/api/v2/deployments.py—image+envfields end-to-end.cli.py—nemo agents deploy --image/-iand repeatable--env/-e(KEY=VALUEor bareKEYforwarded from the shell).examples/Dockerfile— patch nvidia-nat's NIM LangChain builder to excludeverify_ssl(see Issues encountered).pyproject.toml— optional[deployments]extra + workspace source.README.md— container-backend setup gotchas.Issues encountered
Documented in the plugin README ("Container (
deployments) backend — gotchas"). Environment/setup issues unless noted as a bug:| 1 |
TypeError: kwargs_from_env() got an unexpected keyword argument 'base_url'| Bug in nemo-deployments docker backend:docker_hostconfig is passed asbase_urltofrom_env()| UseDOCKER_HOSTenv var;docker_hostexecutor config is unusable until fixed upstream (follow-up) || 2 | No way to pass the image / secrets via CLI |
deploycommand didn't expose image/env | Added--image/-iand--env/-e(this PR) || 3 | Invoke fails:
Validation: Unsupported parameter(s): verify_ssl| Bug in nvidia-nat: undernat serve, the NIM LangChain builder does not excludeverify_sslfromChatNVIDIAkwargs (the OpenAI builder does), so it leaks into the request body and NVIDIA endpoints reject it |examples/Dockerfilepatchesnat/plugins/langchain/llm.pyto mirror the OpenAI exclude set; remove once fixed upstream || 4 |
Failed to load DeploymentConfig ...404 traceback spammed every cycle on delete | Race:delete_deploymentdeleted theDeploymentConfigwhile the deployments controller still held aDELETINGDeploymentreferencing it |delete_deploymentnow waits for theDeploymententity to be gone before deleting the config (this PR) |Test plan
uv run pytest plugins/nemo-agents/tests/unit/test_runner_deployments.py— unit tests: status mapping, gateway URL rewrite, config shape, entity lifecycle (create/status/delete/list) via mocked client,envthreaded throughcreate_deployment, and delete gated on theDeploymentdisappearing before config deletion.uv run pytest plugins/nemo-agents/tests/unit/test_runner_in_memory.py— regression from ABC signature (image/env) change. 20 tests pass.uv run ruff check,ruff format --check,ty check— clean on all changed files.imagePullSecretsfor private registries).End-to-end verification (real run, Docker executor via Colima)
Deployed a packaged agent image through the new
deploymentsbackend and drove it torunning, passingNVIDIA_API_KEYfrom the shell via the new--envflag:Invoked the running container through the agents gateway and got a valid model response (the
verify_sslDockerfile fix is what lets the NIM call through):{"input_message": "Who are you?"}.--envsecret injection verified (-e NVIDIA_API_KEYforwarded from the shell into the container).nemo agents undeploy, theDeploymentConfigis deleted only after the deployments controller removed theDeploymententity — noFailed to load DeploymentConfigwarning in platform logs during teardown.Follow-ups (out of scope)
docker_hostbug (issue test(agentic-use): add evaluator agent benchmark matrix #3 above):_create_clientpassesbase_urltofrom_env(); branch toDockerClient(base_url=...)vsfrom_env(). Separate PR.verify_sslfix (issue feat(evaluator): port metric output protocol #6): drop the Dockerfile patch once the NIM builder excludesverify_ssl.secretKeyRefyet.get_log_locationreturns a hint only; no remote log fetch for containerized deployments.Summary by CodeRabbit
New Features
nemo agents deploynow accepts a container image and environment variables.Bug Fixes