test(e2e): run agent container-deploy e2e on kubernetes (kind) too by benmccown · Pull Request #738 · NVIDIA-NeMo/nemo-platform

benmccown · 2026-07-16T20:11:08Z

Extends the docker-mode agent deployment e2e to also run on the kind (Kubernetes) CI job, proving the container-mode chain (gateway → agent container → Inference Gateway → mock provider → response) on the k8s deployments backend.

What

Shared core: extracted the backend-agnostic deploy/invoke/assert flow into e2e/agents_deploy_helpers.py (mock-backed agent config, deploy → wait running → assert container-mode endpoint shape → invoke → assert mocked round-trip → cleanup). A reap_backend_resources hook lets a backend reap leaked resources (docker container) during teardown.
Docker module (test_nemo_agents_docker.py) refactored to call the shared core. Behavior and markers (subprocess_only, needs_nmp_api_image, Linux-only) unchanged.
New k8s module (test_nemo_agents_k8s.py): deployment_mode="k8s", marked container_only (runs only against an external cluster — the kind e2e job) + needs_nmp_api_image.
Kind Helm values (e2e/k8s/values/kind.yaml): wire a nemo-deployments k8s executor. The nmp-core controller Role already grants the needed deployments/services/configmaps/pods permissions; default_namespace is the release namespace so the agent lands beside the platform and the injected in-cluster IGW URL is reachable.
CI (kind-cpu-e2e): export NMP_E2E_IMAGE_REGISTRY/NMP_E2E_IMAGE_TAG so the k8s test composes the (node-pre-pulled) nmp-api image ref.

Why not one dual-marked test function

The docker and k8s variants have mutually-exclusive markers (subprocess_only vs container_only) and different platform sources (self-driven subprocess platform + docker executor vs external Helm cluster) and networking (docker-bridge base_url juggling vs in-cluster DNS). So they are two thin wrappers over one shared core rather than one function with two markers.

Validation

Local: both modules collect; k8s skips without NMP_BASE_URL ("requires NMP_BASE_URL"), docker skips on non-Linux; ruff/ty/pre-commit clean.
helm template with kind.yaml renders the expected agents.deployments + deployments.executors[k8s] config into the platform ConfigMap.
Full k8s path runs in the Kind CPU e2e CI job on this PR.

Summary by CodeRabbit

New Features
- Added Kubernetes mode agent end-to-end test deploying a Deployment + Service and invoking via the gateway.
- Introduced shared container-mode agent deploy/invoke/teardown helpers for e2e runs.
- Wired CPU-only e2e Kubernetes executor configuration and CI-provided agent image registry/tag.
Tests
- Docker and Kubernetes agent tests now validate mocked gateway round-trips using the shared flow.
- Added unit coverage for effective Kubernetes namespace resolution.
Bug Fixes
- Improved Kubernetes deployment/job operations to consistently use the effective target namespace.

The docker-mode agent deployment e2e (test_nemo_agents_docker.py) proved the container-mode chain (gateway -> agent container -> IGW -> mock provider) only for the Docker backend. Add the Kubernetes equivalent so the same chain is exercised on the kind CPU e2e CI job. - Extract the backend-agnostic core (mock-backed agent config, deploy -> wait running -> assert container-mode endpoint shape -> invoke -> assert -> cleanup) into e2e/agents_deploy_helpers.py, shared by both modules. A reap_backend_resources hook lets a backend reap leaked resources (docker container) during teardown. - Refactor test_nemo_agents_docker.py to call the shared core; behavior and markers (subprocess_only, needs_nmp_api_image, Linux-only) are unchanged. - Add test_nemo_agents_k8s.py: deployment_mode=k8s, marked container_only (runs only against an external cluster, i.e. the kind e2e job) + needs_nmp_api_image. A single dual-marked function isn't viable — the markers are mutually exclusive and the platform source / networking differ — so it's a thin wrapper over the shared core. - Wire a nemo-deployments k8s executor into the kind Helm values (e2e/k8s/values/kind.yaml); the nmp-core controller Role already grants the needed deployments/services/configmaps/pods permissions. default_namespace is the release namespace so the agent lands beside the platform and the injected in-cluster IGW URL is reachable. - Export NMP_E2E_IMAGE_REGISTRY/NMP_E2E_IMAGE_TAG in the kind-cpu-e2e job so the k8s test can compose the (node-pre-pulled) nmp-api image ref. Signed-off-by: Ben McCown <bmccown@nvidia.com>

github-actions · 2026-07-16T20:22:45Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	25531/32732	78.0%	62.6%
Integration Tests	14729/31381	46.9%	19.2%

coderabbitai · 2026-07-16T21:16:19Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 05a56f4d-42b6-4e65-94dc-c0fd13c62352

📥 Commits

Reviewing files that changed from the base of the PR and between 050911f and af72451.

📒 Files selected for processing (1)

e2e/k8s/values/kind.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

e2e/k8s/values/kind.yaml

📝 Walkthrough

Walkthrough

Changes

Container Agent E2E

Layer / File(s)	Summary
Shared deployment and invocation flow `e2e/agents_deploy_helpers.py`	Adds shared mock-provider setup, container deployment, readiness checks, endpoint validation, gateway invocation, response assertions, and cleanup utilities.
Docker test integration `e2e/test_nemo_agents_docker.py`	Refactors the Docker test to use the shared flow and retain Docker-specific resource cleanup.
Kubernetes test and CI wiring `e2e/k8s/values/kind.yaml`, `.github/workflows/ci.yaml`, `e2e/test_nemo_agents_k8s.py`	Configures the Kubernetes executor, passes the agent image through CI, and adds the gated Kubernetes deployment test.
Kubernetes namespace resolution `plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/...`, `plugins/nemo-deployments/tests/unit/backends/k8s/test_backend.py`	Resolves namespaces from explicit configuration, `POD_NAMESPACE`, or `default`, and applies the resolved value across backend operations with unit coverage.

Sequence Diagram(s)

sequenceDiagram
  participant E2ETest
  participant SharedHelper
  participant NeMoPlatform
  participant AgentDeployment
  participant AgentsGateway
  participant MockProvider
  E2ETest->>SharedHelper: deploy and invoke with mode and image
  SharedHelper->>NeMoPlatform: create mock provider and agent
  SharedHelper->>AgentDeployment: create container deployment
  AgentDeployment-->>SharedHelper: running deployment and endpoint
  SharedHelper->>AgentsGateway: invoke deployed agent
  AgentsGateway->>MockProvider: request mocked completion
  MockProvider-->>AgentsGateway: return TEST_AGENT_RESPONSE
  AgentsGateway-->>SharedHelper: return completion response

Possibly related PRs

NVIDIA-NeMo/nemo-platform#644: Changes gateway routing for container-mode deployment endpoints exercised by these tests.
NVIDIA-NeMo/nemo-platform#682: Adds deployment-mode and image API support used by the shared helper.
NVIDIA-NeMo/nemo-platform#715: Introduces the Docker deployment E2E flow refactored into the shared helper.

Suggested labels: ci

Suggested reviewers: mckornfield

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: extending the agent container-deploy e2e test to Kubernetes on kind.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch agent-deployment-k8s-e2e

_{Comment @coderabbitai help to get the list of available commands.}

…pace The k8s deployments executor hardcoded default_namespace to "default" and had no way to self-locate, so the e2e kind values had to hardcode the release namespace (nemo-platform) — brittle and wrong if the namespace differs. Make default_namespace optional and resolve an effective namespace with precedence: explicit config value > POD_NAMESPACE (the controller pod's own namespace, injected via the downward API in the core controller deployment) > "default". A deployed platform can now omit the setting and agents land in its own namespace; an operator can still pin one, and per-deployment backend_config.k8s.namespace continues to override. Drop the hardcoded default_namespace from e2e/k8s/values/kind.yaml accordingly. Add unit tests covering the precedence (explicit wins, POD_NAMESPACE fallback, default fallback, blank POD_NAMESPACE treated as unset). Signed-off-by: Ben McCown <bmccown@nvidia.com>

mckornfield

minor wording stuff

Signed-off-by: Ben McCown <bmccown@nvidia.com>

github-actions Bot added the test conventional-commit type label Jul 16, 2026

benmccown self-assigned this Jul 16, 2026

benmccown marked this pull request as ready for review July 16, 2026 21:11

benmccown requested review from a team as code owners July 16, 2026 21:11

benmccown requested a review from mckornfield July 16, 2026 21:14

mckornfield approved these changes Jul 16, 2026

View reviewed changes

Comment thread e2e/agents_deploy_helpers.py

Comment thread e2e/k8s/values/kind.yaml Outdated

Comment thread e2e/test_nemo_agents_k8s.py

Comment thread plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/config.py

CR feedback

af72451

Signed-off-by: Ben McCown <bmccown@nvidia.com>

benmccown enabled auto-merge July 17, 2026 16:38

svvarom approved these changes Jul 17, 2026

View reviewed changes

benmccown added this pull request to the merge queue Jul 17, 2026

Merged via the queue into main with commit 27c9a8d Jul 17, 2026
58 checks passed

benmccown deleted the agent-deployment-k8s-e2e branch July 17, 2026 16:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(e2e): run agent container-deploy e2e on kubernetes (kind) too#738

test(e2e): run agent container-deploy e2e on kubernetes (kind) too#738
benmccown merged 3 commits into
mainfrom
agent-deployment-k8s-e2e

benmccown commented Jul 16, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 16, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

mckornfield left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

benmccown commented Jul 16, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why not one dual-marked test function

Validation

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

mckornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

benmccown commented Jul 16, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

coderabbitai Bot commented Jul 16, 2026 •

edited

Loading