[Epic] Integrate Switchyard routing into NemoClaw-managed Hermes through native NeMo Relay

[cjagwani]

Scope decision — accepted

Canonical NemoClaw scope was accepted in the 2026-08-13 maintainer decision recorded in this issue. The Hermes-first integration may proceed as dependency-aware implementation work. General availability remains gated on stable released upstream artifacts and all compatibility, credential, policy, lifecycle, and E2E criteria below.

Problem

NemoClaw can run Hermes, and the upstream native integration is converging, but NemoClaw does not yet provide an approved, released, secure, and end-to-end validated way to enable Switchyard model routing for a managed Hermes sandbox.

Without that integration, users must choose compatible upstream revisions, build or install experimental plugin code, write Relay and Switchyard configuration, attach multiple provider endpoints and credentials, widen network policy, diagnose silent activation failures, and decide retry/fallback behavior themselves.

V5 goal

A user explicitly enables an approved Switchyard routing configuration for a NemoClaw-managed Hermes agent. Normal Hermes turns then use the released native path:

• no Relay CLI wrapper;
• no switchyard-server sidecar;
• no upstream source checkout or ad hoc build;
• at least two real target models behind OpenShell-secured provider boundaries;
• persistent, fail-closed activation;
• canonical non-secret routing evidence; and
• restart, rebuild, cleanup, and negative-security qualification.

The first milestone qualifies seeded weighted-random and LLM-classifier routing. Stage routing, response-judging escalation, and other routers require separate qualification before being called supported.

Exact architecture and ownership

NemoClaw-managed OpenShell sandbox
└── normal supervised Hermes process
    └── native NeMo Relay runtime
        └── released nvidia.switchyard plugin
            └── OpenShell-secured providers
                ├── judge/classifier, when required
                ├── weak/efficient target
                └── strong/capable target

Relay intercepts and replaces Hermes downstream LLM execution with the Switchyard plugin async generator. Switchyard produces the RoutingOutcome, rewrites the request, and executes the selected target. Relay owns plugin discovery, activation, correlation, and lifecycle. Switchyard owns the routing algorithm and selected-target execution. Everything runs in the supervised Hermes process with zero Relay or Switchyard sidecars.

Within this accepted scope, NemoClaw owns:

1. selecting and verifying a released compatible Hermes, Relay, Switchyard, plugin, and OpenShell set;
2. installing verified artifacts for supported Linux architectures without source builds;
3. generating the released two-file activation/deployment contract and failing readiness when requested routing is inactive;
4. provisioning judge, weak, and strong provider identities through OpenShell without exposing raw credentials;
5. enforcing network, redirect, timeout, retry, fallback, cancellation, and stream-failure policy;
6. exposing bounded non-secret active-state and per-turn routing evidence; and
7. managed E2E, security, upgrade, rollback, operations, and support documentation.

NemoClaw does not own Switchyard algorithms, provider HTTP transport, a fork of the Relay plugin, benchmark claims, or a second harness-specific routing implementation.

Current status — 2026-09-14

The architecture is proven experimentally, but this is not yet a supported NemoClaw integration.

• Hermes and Relay are ready as a released pair: Hermes Agent 0.21.3 includes the native Relay loader and locks Relay 0.8.3.
• Switchyard is close at source level: the native plugin, routing events, semantic model groups, credential-header boundary, fallback behavior, and important classifier fixes have merged.
• The production bundle is still missing: the official NeMo Relay Plugins producer now builds and installs Switchyard on Linux amd64/arm64 and other supported platforms, but it has not published a production release. Its current bundles are checksum-only and require a weaker trust override.
• Qualification remains: Hermes interruption recovery, remaining provider transport bounds, OpenShell credential/endpoint attestation, fail-closed activation/readiness, two real targets, and managed E2E/security tests must still pass.

NemoClaw can prepare the disabled configuration, provider binding, readiness, and E2E work now. The production selector must wait for the released, immutable plugin tuple and completed qualification. Detailed upstream monitoring evidence is kept in the private readiness ledger rather than added as issue comments.

V0–V5 evidence ledger

Iteration	Architecture	Result	Meaning
V0	Relay 0.6 sidecar plus separate switchyard-server	Superseded	Historical compatibility path only
V1	Relay CLI wrapper launches one-off Hermes; Switchyard is statically linked	Passed; frozen at d58276a	Proved the components could classify, choose semantic targets, and stream inside a managed sandbox
V2	Supervised Hermes loads Relay and the external Switchyard plugin in-process	Passed; frozen at 3c6932b	Proved the intended zero-sidecar process topology and repeated turns
V3	V2 plus final calls through inference.local	Passed; frozen at 28749b8; repaired at 34d1b06	Proved gateway-owned credential substitution while exposing the single-host-model limitation
V4 demo	Native in-process path plus three attached OpenShell providers, two real cloud targets, route watcher, and read-only UI label	Passed as experimental evidence; frozen to Switchyard #270 commit c69a8b68f7c85e4b610c077690f90db6de9053ed	Proved the desired experience and security shape, not a released support contract
V5	Stable released artifacts, accepted scope, canonical configuration, secure providers, failure policy, E2E, and docs	Dependency-aware design and draft implementation are underway; the production selector remains blocked	Final supported integration after the immutable release tuple and qualification gates pass

Preserve V4 on its qualified immutable commit. Do not replace it with a moving PR head. Remove prototype overlays only after the released V5 replacements pass equivalent and stronger evidence.

Delivery tracker

Work package	Deliverable	Issue	Current state
Product decision	Canonical NemoClaw vs Community Solutions; lifecycle, compatibility, security, E2E, and support owners	This issue	Accepted 2026-08-13
Released compatibility and packaging	Stable tuple, exact locks, two architectures, immutable hashes, producer provenance, no source builds	#8885	Relay 0.8.2 and Hermes #96633 now align at source level; blocked on a released Hermes lock and signed Switchyard bundles
Activation and readiness	Released manifest-backed Relay activation plus Switchyard deployment; exact route identity; fail-closed readiness; zero sidecars	#8886	Two-file contract prepared; runtime selector remains blocked on #8885
Secure provider boundary	Separately attached judge/weak/strong providers; gateway-authored evidence; no raw credentials	#8887	Fresh-only design prepared; reuse/mutation/rotation remain blocked on OpenShell #2722 or an accepted equivalent
Network and failure behavior	Destination policy, redirect/body/SSE bounds, deadlines, retries, fallback, cancellation, redaction	#8888	Contract prepared; upstream transport gaps remain
Managed qualification	Released topology, two real targets, canonical decision events, restart/rebuild, negative security, exact evidence and teardown	#8889	Contract prepared; waits on prior gates
Documentation and operations	Compatibility matrix, enable/disable, upgrade/rollback, limitations, ownership, troubleshooting	Follow-up after #8889	Not started
Additional routers	StageRouter and response-judging escalation behavior/evaluation	Separate follow-ups	Out of first milestone
Other harnesses	Reusable Relay-host contract and gaps for OpenClaw/Deep Agents	Separate design after Hermes	Out of Hermes-first scope

Configuration and provider decisions

The released path uses two explicit files:

1. a manifest-backed Relay activation file containing plugin identity, native API/ABI compatibility, immutable entrypoint/bundle identity, and deployment path; and
2. a Switchyard v1 deployment document containing llm_clients, targets, and routes.

NemoClaw must reject the older plugin_id-only form, unknown route IDs, incompatible API/ABI/ranges, missing assets, and requested-but-inactive routing before any provider request.

For the first V5 provider design, use three separately attached OpenShell providers: judge, weak, and strong. Keep inference.local as Hermes's ordinary bootstrap/default route; do not use it for Switchyard targets because it represents one gateway-wide host-selected model.

Each Switchyard llm_client receives an endpoint-scoped raw-key placeholder through api_key_env; Switchyard constructs the provider Authorization header. forward_auth remains false. Caller authorization, cross-provider placeholder use, direct-provider egress, and silent fallback to inference.local must be denied.

Fresh creation of uniquely named providers is the initial boundary. Existing-provider reuse, update, mutation, and credential rotation remain blocked until OpenShell provides identity-bound conditional mutation and secret-free endpoint/profile attestation or an accepted equivalent.

Supported acceptance criteria

• Maintainers accepted canonical product scope and the documented ownership boundary on 2026-08-13.
• A fresh sandbox consumes only stable, compatible, immutable, provenance-verified artifacts for every supported architecture.
• Normal supervised Hermes loads Relay and nvidia.switchyard persistently with zero routing sidecars.
• Requested routing becomes active and ready before the first provider request.
• Repeated Hermes turns select at least two distinct real target models.
• Provider credentials remain outside agent-visible files, environment, argv, logs, errors, telemetry, and artifacts and resolve only at intended endpoints.
• Unauthorized egress, caller-auth forwarding, cross-endpoint use, and routing bypass fail closed.
• Invalid activation, missing/corrupt plugin, provider denial, redirect, timeout, retry exhaustion, fallback, cancellation, and post-first-stream failure are qualified.
• Buffered, streaming, tool, session, restart, rebuild, cleanup, and redaction behavior remain correct.
• Every routing assertion comes from the canonical switchyard.routing.decision event correlated to the Hermes turn—not reconstructed provider logs.
• Machine-readable evidence binds expected cases, observed pass status, exact artifacts/digests, architecture, topology, provider identities, route correlation, redaction scan, and teardown.
• Documentation names supported versions, routers, ownership, enable/disable, limitations, upgrade, rollback, and troubleshooting.

Constraints and non-goals

• Do not rewrite or vendor Switchyard routing algorithms in NemoClaw.
• Do not harden the Relay CLI/static-plugin or Decision API sidecar prototypes into the supported default.
• Do not treat merged source, a prerelease, a green partial CI run, or a working demo as a released compatibility set.
• Do not put raw provider credentials in the sandbox.
• Do not broaden the first milestone to every Relay feature, Switchyard router, model recipe, or agent harness.
• [Epic] Deliver the Hermes Harness Optimization Blueprint with NeMo Platform Insights #8501 owns benchmark-backed Hermes recipes; this Epic owns the routing/security/platform contract.

Other agent harnesses

The platform design can extend beyond Hermes only when another harness provides equivalent Relay lifecycle, downstream model-call interception, configuration, routing metadata/feedback, and credential-bound provider execution. NemoClaw should reuse the secure provider and policy layers where possible, but each harness needs separately accepted adapter/lifecycle scope and E2E evidence. Hermes is first because the native loader exists upstream.

Remaining decisions

• Who owns compatibility, security review, live E2E, release qualification, and support lifecycle?
• Which future stable Hermes/Relay/Switchyard/plugin/OpenShell tuple is the first supported record?
• What released producer publishes the native bundle for both required architectures with immutable provenance?
• Is fresh-only provider provisioning acceptable for the first milestone, or must OpenShell docs: add Brev troubleshooting guide #2722 land first?
• Which seeded random/classifier recipe is the first supported preset, and who owns its benchmark evidence?
• What exact upstream fallback behavior is accepted before and after the first streamed event?

Discovery and coordination

• Will Curran: confirmed the Hermes-first goal and requested assessment for other harnesses.
• Sean Lopp: provided Hermes integration and setup/runtime UX context and directed contract questions to Michael Demoret.
• Michael Demoret: identified Relay as the intended Hermes integration surface.
• Bryan Bednarski: confirmed the in-process direction and release sequencing.
• Ryan Lempka: provided Switchyard configuration, benchmarking, metadata, and library-mode context.
• Apurv Kumaria: Security Observer's Deep Agents weak/strong prototype informed demo clarity and secure provider handling.
• Aaron Erickson: requested this Epic and the concrete issues/PR plan.
• Supporting material from Alex Fournier, the NeMo Relay team, the Switchyard team, and the Hermes team was also reviewed.

Related work

• [Epic] Deliver the Hermes Harness Optimization Blueprint with NeMo Platform Insights #8501 owns benchmark-backed Hermes optimization recipes that may consume this platform capability.
• Closed feat(hermes): define native Switchyard routing contract #9012 remains historical lifecycle/prototype evidence but is not the V5 implementation path or final released configuration and packaging contract.

Checklist

• Prototype evidence and limitations are preserved.
• Current upstream release state and exact remaining gates are documented.
• NemoClaw, Relay, Switchyard, Hermes, OpenShell, and benchmark ownership are separated.
• Compatibility, activation, provider, failure-policy, and E2E child issues are concrete.
• The frozen V4 demo is separated from the released V5 path.
• Canonical NemoClaw scope accepted on 2026-08-13.