[Epic] Adopt kagent Agent Substrate — DevAI-authored agents, sandboxed full lifecycle

[Sam123ben]

Epic — Adopt kagent Agent Substrate as DevAI's sandboxed agent runtime

One-line: let DevAI users author their own agents in the dashboard (adhering to the registry design), then have kagent's Agent Substrate run them as isolated, on-demand, sandboxed Actors — through a full dev → test → deploy → run lifecycle.

Why now

Earlier we decommissioned kagent on the prod cluster because the classic kagent model is one always-warm Deployment per agent (× model variant) — a standing-pod explosion that does not fit our 3-node cluster. The user's instinct was right: "I thought these would be like jobs — pop up when called, not always running." So today every agent runs as an on-demand K8s Job (default JobRunnerStage), kagent is dormant behind the operator kill-switch (DEVAI_KAGENT_ENABLED=false).

The Agent Substrate (solo.io + kubernetes-sigs/agent-sandbox + GKE Agent Sandbox) changes the calculus. Its WorkerPool + Actor model gives "fast cold starts, low resource overhead, and secure gVisor/Firecracker isolation per agent" — agents are lightweight Actors multiplexed inside a shared WorkerPool, not a pod each. That directly answers both reasons we backed off kagent (standing-pod cost → low overhead; "should pop up when called" → fast cold start), and it adds strong per-agent sandboxing — exactly what we need to let users run their own agent code safely.

The end-to-end we're building

USER authors agent in DevAI dashboard (ArtifactEditor, registry schema/layout)
   → validate (schema + lint + handover types)            [Authoring]
   → BUILD (resolve refs) → TEST (eval) → SECURITY gate    [Lifecycle harness]
   → PUBLISH to agentic-registry                            [Registry]
   → agent-sync renders SandboxAgent + ActorTemplate        [Registry→Substrate]
   → kagent Substrate runs it as a gVisor-sandboxed Actor   [Runtime]
      in a shared WorkerPool, on demand, on the user's key  [Dispatch + per-user keys]
   → DevAI dispatches a run to the Actor over A2A           [Dispatch]
   → live status + cost surfaced in the dashboard           [Observability]

Sub-issues (decide + sequence here)

• [Substrate] GO/NO-GO: WorkerPool+Actor resource model on 3-node GKE #70 — GO/NO-GO: Substrate resource model on 3-node GKE (the gate; reconcile with the decommission)
• [Substrate] Install & configure Agent Substrate (tesserix-k8s / ArgoCD) #71 — Install & configure Agent Substrate (tesserix-k8s / ArgoCD; CRDs, WorkerPool, gVisor)
• [Substrate] User-authored agents in DevAI (dashboard → registry schema → publish) #72 — User-authored agents in DevAI adhering to the registry design/layouts (dashboard → schema → publish)
• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 — Registry → Substrate render (agentic-registry emits SandboxAgent + ActorTemplate/AgentHarnessSubstrateSpec; agent-sync reconciles)
• [Substrate] Dispatch integration — route runs to a Substrate Actor vs Job #74 — Dispatch integration (route a run to a Substrate Actor vs an ephemeral Job; kill-switch; on-demand semantics)
• [Substrate] Full agent lifecycle gate — dev → test → deploy → run (harness) #75 — Full lifecycle gate: dev → test → deploy → run (the agent-harness: build/test/security before publish+deploy)
• [Substrate] Security & multi-tenant isolation (gVisor, agentgateway, per-user keys) #76 — Security & multi-tenant isolation (gVisor/Firecracker, agentgateway egress/ingress, per-user/tenant + passthrough keys)
• [Substrate] Observability & UI — Actor/lifecycle status in the dashboard #77 — Observability & UI (substrate Actor/lifecycle status in the dashboard; cold-start, per-agent health)
• [Substrate] Cost & autoscaling — KEDA on the WorkerPool, 3-node budget #78 — Cost & autoscaling (KEDA on the WorkerPool, scale-to-zero, 3-node budget)

Guardrails / constraints

• No prod standing-pod explosion. Adoption is conditional on [Substrate] GO/NO-GO: WorkerPool+Actor resource model on 3-node GKE #70 proving the WorkerPool model fits 3 nodes. If it doesn't, we stay on on-demand Jobs.
• GitOps only (tesserix-k8s + ArgoCD); no manual kubectl apply. connect-local first for any cluster op.
• Additive + reversible. Substrate is opt-in behind the existing DEVAI_KAGENT_ENABLED operator switch; Jobs remain the default until Substrate is proven.

References

• Article: https://www.cloudnativedeepdive.com/kagent-agent-substrate-configuration-setup/
• Agent Substrate (solo.io): https://www.solo.io/blog/agent-substrate-powers-kubernetes-agents-with-kagent
• kubernetes-sigs/agent-sandbox: https://github.com/kubernetes-sigs/agent-sandbox · docs https://agent-sandbox.sigs.k8s.io/docs/
• GKE Agent Sandbox: https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate
• DevAI context: docs/agentic/KAGENT-INTEGRATION.md §0/§0a, docs/plans/{agent-harness,kagent-backend,kagent-dynamic-provisioning}/

Sequencing (dependencies)

#70 GO/NO-GO ──gate──> everything below
   └─> #71 Install ──> #73 Registry→Substrate render ──> #74 Dispatch ──> #77 Observability
        #72 Authoring ──┘ (feeds #73)                       │
        #75 Lifecycle gate (build/test/security) ───────────┤ (blocks "deploy to Substrate")
        #76 Security/isolation ──underpins──> #71,#74,#75
        #78 Cost/autoscaling ──depends──> #70,#71

Decision order: start with #70 (does it fit 3 nodes?). If GO → #71 + #72 in parallel → #73 → #74 → #75/#76/#77/#78. If NO-GO → stay on on-demand Jobs and revisit when node budget grows.

[Sam123ben]

Progress — the SOFTWARE chain is done; the LIVE runtime is gated on #70

Shipped + tested this pass (additive, dormant until kagent is re-enabled):

• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 promptRef→systemMessage resolution (the real schema mismatch) — agentic-registry.
• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 cross-validation API GET /v0/agents/{name}/export/kagent?validate=true → {ok, issues} + RegistryClient.kagent_validate() (DevAI). [[Substrate] User-authored agents in DevAI (dashboard → registry schema → publish) #72/[Substrate] Full agent lifecycle gate — dev → test → deploy → run (harness) #75]
• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 SandboxAgent rendering (?workerPool=<pool> → spec.substrate.workerPoolRef).

So DevAI-agent → validated → rendered-as-SandboxAgent → dispatchable works end to end in code, all unit-tested.

The one thing not done — and deliberately so: the LIVE Substrate runtime (install Substrate + stand a WorkerPool + gVisor) on the cluster. That is #70's GO/NO-GO and it must run on the sandbox (connect-local) first — we will NOT push a WorkerPool + gVisor onto the 3-node prod we just protected from pod-explosion without proving it fits. SandboxAgent's premise (Actors multiplexed in a shared pool, not a pod each) is what makes it plausibly fit, but that's a measurement, not an assumption.

Next concrete step: run #70 on the sandbox — stand up Substrate + a 1-replica WorkerPool, apply a rendered SandboxAgent, measure per-Actor overhead + cold-start, decide GO/NO-GO.

[Sam123ben]

#70 verdict: NO-GO on the current 3-node cluster — no GKE-Sandbox node pool, so the gVisor WorkerPool can't schedule. The software chain is done + validated against the live prod CRD (server-side dry-run of a rendered SandboxAgent passes, after fixing workerPoolRef to an object). Substrate just needs a small dedicated gVisor node pool (a gcloud cluster change that adds a node — conflicts with the 3-node limit, so it's an operator decision). See #70 for the options.

[Sam123ben]

Epic decision complete: #70 is a documented NO-GO for enabling the current Substrate runtime on production. The reusable software slices are delivered—user authoring, registry export/reconcile, dispatch/fallback, lifecycle gates, per-user key controls, and honest dormant UI—but the Actor rollout is not planned until a published kagent/Substrate pair satisfies the hardened runtime, readiness, migration, isolation, and measured capacity gates.

Production remains on the supported on-demand Job model with DEVAI_KAGENT_ENABLED=false: zero idle agent footprint, per-user connector isolation, and gateway-only AI routing. The issue closures preserve the exact reopen criteria; they do not claim the current privileged/unready WorkerPool is production-safe.

[Sam123ben]

Substrate epic gate update (2026-08-20)

The pinned upstream compatibility test for #76 is complete. kagent commit 059c01b68584dea113ccdf80f2e356c2d051e02a fixes the original valueFrom / pauseImage API mismatch, but it still needs three local compatibility fixes before a SandboxAgent reaches Ready. With those fixes, the local gVisor canary did reach Accepted=True, Ready=True, and produced a golden snapshot.

The epic remains blocked from production because the Substrate worker still runs with root, AppArmor unconfined, a hostPath, the default ServiceAccount and automounted API token, plus broad capabilities including SYS_ADMIN. That does not meet DevAI's per-user multi-tenant isolation requirement.

Full reproduction, test evidence, and gate decision: #76 (comment)

Production remains correctly fail-closed on the Jobs path with DEVAI_KAGENT_ENABLED=false. Do not close the epic or roll out Substrate until the compatible artifact, hardened worker identity/security context, cross-tenant tests, and named rollout approval are all present.

[Sam123ben]

2026-08-20 issue-by-issue audit complete. Closed as completed with current production evidence: #72 (user authoring on supported Job runtime), #75 (lifecycle gate), and #77 (honest owner-scoped runtime/lifecycle UI). Remaining Actor-dependent issues #73, #74, #76, and #78 stay open because the current WorkerPool fails the untrusted multi-tenant security boundary and no compatible hardened artifact is published. The epic therefore remains open and production remains fail-closed on Jobs.

[Sam123ben]

Upstream work is now reviewable rather than local-only: WorkerPool tokenless identity/default-deny egress is in kagent-dev/substrate#22, and kagent SandboxAgent compatibility is in kagent-dev/kagent#2503. The live sandbox-gvisor pool has target size 0 and no autoscaling configuration, so the production epic remains fail-closed on Jobs pending upstream review, trusted-worker hardening, capacity, and named rollout approval.