feat(agents): authenticate deployed agents via service-principal sidecar - #914
Merged
Conversation
When platform auth is enabled, a deployed agent's Inference Gateway calls were rejected with 401 because the agent (a NAT runtime whose OpenAI HTTP client we do not control) carries no platform credential — its LLM api_key is the placeholder "not-used". This injects a loopback auth-proxy sidecar (the nmp-api image running `nemo services run --sidecars auth-proxy`) into k8s/docker agent deployments when auth is enabled. The agent targets the sidecar on localhost; the sidecar forwards to the platform stamping X-NMP-Principal-Id: service:agents, which the OPA policy authorizes via the ServiceSystem role. This is the same static service identity the platform's own SDK clients use (get_platform_sdk as_service=...); the proxy exists only for workloads that cannot set the header themselves. The LoRA adapters sidecar already self-injects service:models via the SDK and is unaffected. - New workload_proxy sidecar: loopback FastAPI forwarder that strips inbound auth/principal headers and stamps the service principal, streaming responses. Registered as "auth-proxy" in AVAILABLE_SIDECARS. - Agents backend injects the sidecar (as a native init-container sidecar with restartPolicy=Always) and points the agent's llms.*.base_url at it when auth is enabled; unchanged behavior when auth is off. - Fix k8s compiler exec-probe: V1Probe needs the `_exec` kwarg (`exec` is a Python keyword); the sidecar uses an exec readiness probe because it binds loopback only and a pod-IP httpGet probe would be refused. - Unit tests for the sidecar forwarder, agents injection/base_url wiring, and the exec-probe compiler path. Verified end-to-end on kind with auth enabled: agent deployment reaches 2/2 Running (sidecar ready via exec probe) and the agent's IGW call authenticates (reaches model resolution instead of 401). Signed-off-by: Ben McCown <bmccown@nvidia.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a FastAPI auth-proxy sidecar that stamps service-principal headers, registers and configures the sidecar, routes authenticated agent traffic through loopback, injects it into Kubernetes and Docker workloads, and adds probe and forwarding tests. ChangesAuth proxy deployment
Sequence Diagram(s)sequenceDiagram
participant Agent
participant AuthProxy
participant Platform
Agent->>AuthProxy: Send inference request to loopback
AuthProxy->>Platform: Forward request with stamped principal header
Platform-->>AuthProxy: Return response
AuthProxy-->>Agent: Stream response
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.py`:
- Around line 66-74: Update _platform_auth_enabled so failures from
get_auth_config are not converted to False; propagate the exception or
explicitly fail deployment instead. Preserve the boolean enabled result for
successfully resolved configuration, and ensure deployment cannot proceed as
unauthenticated when auth-config resolution fails.
- Around line 364-370: Update the deployment-building flow around
_build_auth_proxy_sidecar so Docker deployments do not append the proxy to
init_containers, which Docker rejects. Add a Docker-compatible sidecar and
network wiring for auth_proxy, while preserving the existing native sidecar
behavior for non-Docker modes; add coverage for an auth-enabled Docker
deployment verifying the proxy is configured and startup remains valid.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: df617f3b-67d2-46c4-9b27-8935a13853b4
📒 Files selected for processing (8)
packages/nmp_common/src/nmp/common/auth/workload_proxy/main.pypackages/nmp_common/tests/auth/test_workload_proxy.pypackages/nmp_platform_runner/src/nmp/platform_runner/registry.pyplugins/nemo-agents/src/nemo_agents_plugin/config.pyplugins/nemo-agents/src/nemo_agents_plugin/runner/deployments_backend.pyplugins/nemo-agents/tests/unit/test_runner_deployments.pyplugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.pyplugins/nemo-deployments/tests/unit/backends/k8s/test_compiler.py
Contributor
|
…flags Addresses PR feedback: the auth-proxy sidecar belongs to nemo-deployments (which compiles containers into deployments), not nemo-agents. Also removes a direct nmp_common reference from the agents plugin. - DeploymentConfig gains `auth_proxy_sidecar: bool` and `auth_proxy_sidecar_identity: str | None`. The deployments plugin compiles and injects the sidecar (nemo_deployments_plugin.auth_proxy) from these flags, interpolating the identity into the service-principal header. Injection is a no-op when platform auth is disabled. - nemo-agents now just sets the two flags (identity="agents") and points the agent's llms.*.base_url at the proxy port when auth is enabled; it no longer builds the sidecar container or reads deployments image/port config. - Plugins no longer import nmp_common: added nemo_platform_plugin.auth .platform_auth_enabled() wrapper; both agents and deployments consult it. - Sidecar image/port config moved from AgentsConfig to DeploymentsConfig. - Tests: deployments-side build_auth_proxy_container (requested/auth-off no-op/ identity default) and agents-side flag-setting + base_url wiring. Signed-off-by: Ben McCown <bmccown@nvidia.com>
…h fallback Addresses PR feedback. - Docker backend now injects the auth-proxy sidecar too (build_docker_plan), reusing build_auth_proxy_container. The sidecar shares the primary container's network namespace, so the agent reaches it on localhost:8090 — the same loopback address used in k8s. Previously the sidecar was injected only in the k8s compiler, so auth-on docker agent deployments got no identity and would 401. - The docker sidecar's upstream (NMP_BASE_URL) is rewritten to a docker-reachable host via determine_loopback_override() (e.g. host.docker.internal on macOS) when the platform base URL is a loopback, mirroring the agent-side rewrite from #899. K8s uses the Service DNS verbatim. - Documented platform_auth_enabled()'s fail-to-False behavior: the realistic failure is ImportError (package used outside the platform image); other failures are effectively unreachable in the controller (missing config -> defaults; malformed config would have crashed startup; the read is cached). - Tests: docker auth-proxy injection, auth-off no-op, and loopback upstream rewrite. Signed-off-by: Ben McCown <bmccown@nvidia.com>
…er stripping Addresses PR feedback. - Remove the default service-principal identity. A DeploymentConfig with auth_proxy_sidecar=True now requires auth_proxy_sidecar_identity: a model_validator rejects the invalid combination, surfaced as a 4xx at the create endpoint. The sidecar's run() likewise requires NMP_AUTH_PROXY_PRINCIPAL rather than defaulting. No silent "agents" fallback anywhere. - Trim the auth-proxy's header sanitization to only what would be actively wrong. Request: host + content-length (httpx recomputes), authorization + x-nmp-principal-id (we stamp the identity; must not be spoofed/conflict). Response: content-length + transfer-encoding (responses are streamed). Dropped the hop-by-hop policing that a trusted loopback sidecar doesn't need. - Tests: replace the identity-default test with a validation-rejection test. Signed-off-by: Ben McCown <bmccown@nvidia.com>
…nt cleanup Addresses review feedback on the auth-proxy sidecar. - Add a FastAPI lifespan that closes the shared httpx AsyncClient on shutdown so its connection pool is released gracefully instead of relying on process exit. - Remove the redundant BackgroundTask(response.aclose): the finally in the streaming generator already runs on normal completion, exception, and client disconnect, so the response is always closed there. httpx's aclose() is idempotent, so the duplicate was harmless but implied cleanup was missing. - Test that the lifespan actually closes the upstream client. Not changed: the request body is still buffered before forwarding. Requests through this proxy are a workload's outbound platform API calls (bounded in size), and streaming them would force chunked transfer-encoding on every request since we strip content-length. Revisit if large payloads are routed through the proxy. Signed-off-by: Ben McCown <bmccown@nvidia.com>
benmccown
enabled auto-merge
July 27, 2026 19:41
benmccown
disabled auto-merge
July 27, 2026 19:49
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
When platform auth is enabled, a deployed agent's Inference Gateway calls were rejected with 401 — the agent (a NAT runtime whose OpenAI HTTP client we don't control) carries no platform credential (
api_key: "not-used"). This injects a loopback auth-proxy sidecar that stamps a service-principal identity so those calls authenticate. Works for both k8s and docker agent deployments.The problem
Follows #899 (which fixed connectivity — the agent reaching the internal API Service). This fixes authorization: with auth on, the agent's IGW call had no principal and got a 401 → OpenAI-client retry loop.
Per @ironcommit, static
service:*identity is the intended fallback while workload token exchange is unavailable, and it's what the docker monolith already depends on. So this uses that mechanism rather than the (out-of-scope) RFC 8693 token exchange.The fix
nemo-deployments owns the sidecar; nemo-agents just requests it.
nmp_common/auth/workload_proxy): a loopback FastAPI forwarder run vianemo services run --sidecars auth-proxy(registered inAVAILABLE_SIDECARS). It strips inbound auth/principal headers and stampsX-NMP-Principal-Id: service:<identity>, which OPA authorizes via the ServiceSystem role. Streams responses.auth_proxy_sidecar: bool+auth_proxy_sidecar_identity: str | None. The deployments plugin compiles and injects the sidecar from these flags (nemo_deployments_plugin.auth_proxy), for both backends:restartPolicy: Always); upstream is the in-cluster API Service DNS.host.docker.internal/ bridge addr) viadetermine_loopback_override(), mirroring fix(agents): make deployed agent inference URL container-reachable #899.agents) and points the agent'sllms.*.base_urlat the proxy (http://127.0.0.1:8090) when auth is on. Unchanged when auth is off (direct Service DNS per fix(agents): make deployed agent inference URL container-reachable #899). It does not build the sidecar or know its mechanics.nmp_common: addednemo_platform_plugin.auth.platform_auth_enabled()as the wrapper both plugins consult.V1Probeneeds the_execkwarg (execis a Python keyword). The sidecar uses an exec readiness probe since it binds loopback only (a pod-IP httpGet probe is refused).The NIM/vLLM model images are never touched. The LoRA adapters sidecar already self-injects
service:modelsviaget_platform_sdk(as_service=...)and is unaffected — the proxy is only for workloads that can't set the header themselves.Verified end-to-end (dev pod, auth enabled)
2/2 Running(auth-proxy: ready=truevia the exec probe); invoke returns HTTP 200{"value":"...96..."}through the sidecar + a real nvidia-build model.nemo services run): deploy--mode dockeryields the agent +auth-proxycontainers (Up (healthy)), sidecarnetwork=container:<agent>(shared netns), envNMP_AUTH_PROXY_PRINCIPAL=agents/NMP_BASE_URL=http://172.17.0.1:8080. Agent httpx log shows it calling127.0.0.1:8090; the IGW logsstatus_code=404(model resolution) — not 401, confirming theservice:agentsprincipal authenticated.Tests
/healthz.build_auth_proxy_container: requested / auth-off no-op / identity default / docker loopback-upstream rewrite; dockerbuild_docker_planinjection.base_urlwiring (auth on/off)._exec).Summary by CodeRabbit
/healthzfor health/readiness checks.