[Substrate] Dispatch integration — route runs to a Substrate Actor vs Job

[Sam123ben]

Part of the Agent Substrate epic ⬆️. Depends on #D.

Goal

Route a DevAI pipeline run/stage to a Substrate Actor when the agent targets Substrate, and to an ephemeral Job otherwise — on demand, with clean fallback.

Today (grounded)

JobRunnerStage.execute checks _maybe_dispatch_kagent (A2A over {kagent_url}/api/a2a/{ns}/{agent}) BEFORE the Job path; unlabelled agents fall through to a K8s Job. There's an operator kill-switch (DEVAI_KAGENT_ENABLED, default off) and per-user passthrough (forwards the user's key as the A2A bearer). We also just landed a NATS WorkQueue run-dispatch backend (separate, off by default).

Tasks

• Extend _maybe_dispatch_kagent to address a Substrate Actor (A2A endpoint may differ; confirm the SandboxAgent A2A surface).
• Keep degrade-to-Job on any Substrate error (never a SPOF) + the kill-switch gate.
• On-demand semantics: confirm cold-start is acceptable per-run (#A); if the Actor is scaled-to-zero, the first dispatch wakes it.
• Carry per-user key passthrough into the SandboxAgent ModelConfig (see #G).
• Decide interplay with the NATS WorkQueue run-dispatch (run-level queue) vs Actor (agent-level exec) — they compose, document it.

Acceptance

A labelled agent run dispatches to its Substrate Actor and returns; killing the switch or a Substrate error transparently falls back to a Job.

Refs: src/devai/pipeline/stages/job_runner.py, src/devai/agentic/kagent_client.py; memory: Agent Harness + kagent, Model Policy + Boardroom, NATS WorkQueue Dispatch.

— Part of #69

[Sam123ben]

Implemented and released the dispatch integration slice.

Delivered

• Confirmed kagent 0.9.12 serves SandboxAgent invocations at /api/a2a-sandboxes/{namespace}/{agent}.
• Added an explicit dispatch target while retaining classic /api/a2a/... compatibility for direct Agent callers.
• Runtime-labelled pipeline stages now select SandboxAgent / Substrate and return additive execution_target: substrate metadata.
• Preserved DEVAI_KAGENT_ENABLED and transparent Job fallback for transport, HTTP, JSON-RPC, non-object, and LLM failures.
• Preserved personal-connector-only key passthrough; tenant, organization, team, global, and platform keys remain excluded.
• Documented that NATS owns run-level work assignment, then JobRunnerStage selects Actor vs Job at agent-execution level.

Source and CI

• PR: feat(runtime): dispatch labelled agents to Substrate #270
• Main merge: 1f45df44a7e95809e179d6349ff58a2cb7717995
• PR CI: https://github.com/tesserix/devai/actions/runs/32277145532
• Main ALM build: https://github.com/tesserix/devai/actions/runs/32277735807
• Main Runner build: https://github.com/tesserix/devai/actions/runs/32277735776
• Image/tag guard: https://github.com/tesserix/devai/actions/runs/32277735779

Validation:

• focused routing/isolation suite: 22 passed
• full suite: 1606 passed, 1 skipped
• changed-file Ruff: passed
• strict MyPy on both changed source modules: passed
• wheel and sdist build: passed
• PR and main CI, image builds, and Trivy gates: passed

GitOps production verification

• Image tag: main-1f45df4
• Kargo freight: d4c534bf98f32cea8e5b79b6b06b91b0b7fbbd33
• Kargo production stage: Healthy and Verified for commit 1f45df44
• Seven promoted DevAI applications: Synced/Healthy
• Seven main-1f45df4 pods: Ready, zero restarts
• API /healthz: HTTP 200
• DEVAI_AGENTGATEWAY_URL: private in-cluster MCP gateway
• DEVAI_KAGENT_URL: private in-cluster controller
• DEVAI_KAGENT_ENABLED=false: retained pending readiness acceptance

Delivery note: the first PR (#269) targeted the deploy marker before matching images existed and briefly produced ImagePullBackOff replacement pods. Existing replicas remained ready. The standard main build/tag workflow then advanced deploy to the verified SHA; Kargo reconciled naturally. Final verification found no pods using the bad tag.

This issue remains open for the runtime acceptance criterion: the production canary is still Accepted=True, Ready=False (ActorTemplateNotReady) under #70, so a truthful scaled-to-zero wake-and-return test cannot yet be completed. No kill switch was enabled and no load test or imperative production repair was performed.

[Sam123ben]

The dispatch implementation is shipped and tested, including the SandboxAgent A2A route, personal-connector-only passthrough, kill switch, error-to-Job fallback, and NATS composition. A live Actor wake-and-return rollout is not planned under #70 current NO-GO. Production keeps DEVAI_KAGENT_ENABLED=false, so all runs use the supported Job path without a Substrate SPOF. Reopen when a compatible hardened runtime passes the documented canary and load gates.

[Sam123ben]

2026-08-20 acceptance audit: routing, personal-connector-only passthrough, kill switch, and transparent Job fallback remain merged and deployed. The required live Actor wake-and-return proof cannot be run safely because #76 fails the WorkerPool isolation gate and no compatible hardened runtime artifact is published. Production correctly remains on Jobs with DEVAI_KAGENT_ENABLED=false.

[Sam123ben]

Local Actor dispatch acceptance update (2026-08-20):

• Controller dispatch through /api/a2a-sandboxes/kagent-system/devai-substrate-mock selected a session Actor through atenet-router rather than the Job path.
• A cold Actor restored in 218.6 ms, executed a raw A2A SendMessage, returned The answer is 4., then checkpointed in 39.3 ms. End-to-end HTTP time was 411.7 ms.
• The earlier callback timeout was resolved without weakening default deny: a local additive policy allowed only WorkerPool -> kagent-controller TCP 8083/8084. The synthetic provider allowance was separately pinned to one host /32 on TCP 8090.
• A2A Go v2.5.0 is required so application errors with single-object error.data are decoded instead of hidden by a transport decode failure.

Upstream implementation is in kagent-dev/kagent#2503 at f13d370df494826e532596c51aadd281bc343201. Production remains fail-closed with DEVAI_KAGENT_ENABLED=false; this issue stays open until upstream merge and production security/capacity acceptance.