[Epic] Agent Sandbox & Evaluation — make the ADK a test bench

[Sam123ben]

Epic — Agent Sandbox & Evaluation: make the ADK a test bench

One line: DevAI can already author agents with the ADK and run them in
production, but there is nowhere in between to run an agent safely and measure whether
it is any good. This epic builds that middle.

BUILD (ADK) → SANDBOX (isolated run) → EVALUATE (scored run) → COMPARE (vs baseline)
  → GATE (thresholds) → PUBLISH / PROMOTE → OBSERVE

The two concepts

A sandbox is the same agent runtime as production with different boundaries — not
a separate "test runtime", or you get works in sandbox, fails in production. It pins
an immutable configuration (agent version, model, prompt version, tool set, dataset
version, token/cost limits, TTL) and, crucially, controls what the agent's actions
actually reach. That last part is what makes an agent sandbox different from an
application sandbox: a normal sandbox isolates a process, an agent sandbox has to
isolate side effects, because the agent decides at runtime to call
refund_customer().

An eval is a test suite for non-deterministic software. You cannot assert
f(x) == y, so you fix a dataset of cases, run the agent over all of them, and score
along several dimensions: deterministic (expected JSON / tool call / schema / task
completed), trajectory (right tools, sensible order, correct arguments, recovered
from failure, no forbidden action), LLM-as-judge (groundedness, helpfulness, reasoning),
and operational (P95 latency, tokens, cost, safety). The output is never just a number —
Evaluation score: 62% tells an engineer nothing; the trace of the failing case tells
them everything.

What we already have (grounded — this epic reuses, it does not rebuild)

Have	Where
ADK authoring + publish	src/devai/adk/, devai adk new-*/validate/publish
40 agent artifacts + 26 personas	architecture/registry-seeds/agents/, specializations/
The agent runtime — on-demand K8s Jobs, resolved tools/skills/prompts	pipeline/stages/job_runner.py + runtime/job_spec.py
Ephemeral env with a 4h idle TTL and a reaper	preview/service.py
A tool allowlist chokepoint	tools/dispatch.py, tools/registry.py
One run-quality scorer + analytics view	pipeline/stages/evals.py, /api/analytics/evals
Telemetry adapter, MCP hub, object store	adapters/telemetry/, mcphub/, adapters/object_store/

The five real gaps

1. Sandbox is not a first-class object → [Sandbox] SandboxSpec contract, sandboxes table and REST API #179, [Sandbox] Provision sandboxes through the existing K8s Job runtime (isolation + TTL) #180
2. The tool layer is binary allow/deny — no mock, replay or block, and no record of what
   was called → [Gateway] Tool gateway modes: real / mock / replay / block #181
3. No per-invocation trace to look at when something fails → [Trace] Per-invocation trace spine — prompt → LLM → tool → response #182
4. Evaluation is one hardcoded scorer over a whole pipeline run — no datasets, no
   per-agent eval, no trajectory, no judge → [Evals] Datasets and eval suites — versioned, published, stored independently #184, [Evals] Eval runner and scorer registry #185, [Evals] Trajectory scorer — tool choice, order, arguments, recovery, forbidden actions #186, [Evals] LLM-as-judge scorer, provider-agnostic and version-pinned #187
5. No comparison and no gate, so quality never blocks a promotion → [Compare] Baseline vs candidate comparison on one dataset version #189, [Gate] Evaluation gates before publish and promotion #190

Issues

Phase 1 — Foundation

• [Sandbox] SandboxSpec contract, sandboxes table and REST API #179 SandboxSpec contract, sandboxes table, REST API
• [Sandbox] Provision sandboxes through the existing K8s Job runtime (isolation + TTL) #180 Provision sandboxes through the existing Job runtime (isolation + TTL)

Phase 2 — Developer sandbox (MVP ends here, plus a slice of Phase 3)

• [Gateway] Tool gateway modes: real / mock / replay / block #181 Tool gateway modes: real / mock / replay / block ← build before evals
• [Trace] Per-invocation trace spine — prompt → LLM → tool → response #182 Per-invocation trace spine
• [ADK] Sandbox and test surface in the ADK (devai adk sandbox / devai adk test) #183 Sandbox + test surface in the ADK

Phase 3 — Evaluation

• [Evals] Datasets and eval suites — versioned, published, stored independently #184 Datasets and eval suites, versioned and stored independently
• [Evals] Eval runner and scorer registry #185 Eval runner and scorer registry
• [Evals] Trajectory scorer — tool choice, order, arguments, recovery, forbidden actions #186 Trajectory scorer
• [Evals] LLM-as-judge scorer, provider-agnostic and version-pinned #187 LLM-as-judge scorer
• [Evals] Golden eval suites for DevAI's own built-in agents #188 Golden eval suites for DevAI's own built-in agents

Phase 4 — Comparison and gates

• [Compare] Baseline vs candidate comparison on one dataset version #189 Baseline vs candidate comparison
• [Gate] Evaluation gates before publish and promotion #190 Evaluation gates before publish and promotion

UI (after the API)

• [UI] Agent page: Playground, Traces, Evaluations, Compare #191 Agent page: Playground, Traces, Evaluations, Compare

Cross-cutting

• [Sandbox] Sandbox-scoped credentials — no production secrets in a sandbox #194 Sandbox-scoped credentials — no production secrets in a sandbox
• [Evals] CI integration — run agent evals on PRs that touch agents #196 CI integration — run agent evals on PRs that touch agents
• [Docs] Concepts guide: sandboxes and evals, with in-context help #197 Concepts guide + in-context help

Phase 5 — Scale and governance

• [Sandbox] Budgets, quotas and cost accounting per sandbox and tenant #195 Budgets, quotas and cost accounting per sandbox and tenant
• Still to file: approval policy, shared dataset registry, stronger workload isolation.

Sequencing

#179 ──> #180 ──> #181 ──> #182 ──> #183
                    │        │
                    └────────┴──> #184 ──> #185 ──> #186 ──> #187 ──> #188
                                              │
                                              └──> #189 ──> #190 ──> #191

#194 underpins #180  ·  #195 depends on #182  ·  #196 depends on #185+#189  ·  #197 tracks #191

Guardrails

• Do not build a second agent runtime. The sandbox is a configuration of the K8s Job
  runtime we already run in production.
• Tool gateway before eval engine. An eval suite without mode control can issue 500
  real refunds.
• BLOCK/MOCK is the default for side-effecting tools; engineers opt into real.
• Eval results outlive sandboxes. Destroying a sandbox never destroys history.
• Everything versioned and immutable — agent, prompt, dataset, model config — or
  comparison is not trustworthy.
• Provider-agnostic judging through the LLM adapter + PrincipalLLMResolver.
• Security from day one: TTL, least-privilege identity, default-deny egress, no
  production secrets, explicit tool policy, immutable audit, token and cost budgets,
  rate limits, destructive tools blocked unless approved.
• GitOps only; schemas live in tesserix-k8s (repo rule 5).

Relationship to #69 (kagent Agent Substrate)

#69 asks where sandboxed agents execute (WorkerPool + Actor vs on-demand Jobs) and
is gated on #70. This epic is deliberately substrate-agnostic: the sandbox contract,
tool gateway, traces, datasets, scorers, comparison and gates all sit above whatever
executes the agent. If Substrate lands, #180 swaps its backend and nothing else moves.
#75 (the dev → test → deploy → run lifecycle harness) is the natural consumer — its
missing "test" step is an eval run in a sandbox (#185), and #190 is its gate.

Plan

docs/plans/agent-sandbox-evals/IMPLEMENTATION-PLAN.md

[Sam123ben]

Extended the epic with a workspace sandbox track, after reading agent-infra/sandbox, Browser Use's two ways to sandbox agents, sandboxed agent worktrees and a practical security guide for agentic AI.

Design: docs/plans/agent-sandbox-evals/WORKSPACE-SANDBOX.md (PR #193).

What was missing. #179–#181 make a sandbox a boundary around a run. They do not give it a place to work — a filesystem, a shell, a browser, a preview URL, an editor. Without that, an agent can be measured but cannot build, and a failing case cannot be opened and inspected.

The framing decision. Browser Use names two patterns: isolate the tool (agent stays put, risky capabilities move behind a service) and isolate the agent (agent runs inside the sandbox with no credentials, reaching the world only via a control plane). DevAI does both — they are the same picture at two scopes. The property to preserve is theirs: the agent should have nothing worth stealing and nothing worth preserving.

Where we deliberately differ from AIO Sandbox, whose defaults are wrong for a multi-tenant platform: auth is optional there (ours is closed by construction, no unauthenticated mode exists); it runs seccomp=unconfined (we keep the restricted context and pay for the browser with a narrow profile or gVisor); it shares one container across callers (ours is per-owner and TTL-bounded, which is what makes the eval numbers reproducible).

New issues:

• [Workspace] Sandbox workspace runtime — /workspace volume, shell and file services #201 Workspace runtime — /workspace volume, shell and file services
• [Workspace] Browser leg — CDP for agents, VNC for humans #202 Browser leg — CDP for agents, VNC for humans
• [Workspace] Sandbox MCP profile — workspace tools through our own hub #203 Sandbox MCP profile — workspace tools through our own hub, gated by [Gateway] Tool gateway modes: real / mock / replay / block #181
• [Sandbox] Egress allowlist proxy — the only route out of a sandbox #204 Egress allowlist proxy — the only route out; today's no-internet policy is safe and unusable
• [Sandbox] Credential broker — short-lived scoped credentials, never a standing secret #205 Credential broker — short-lived scoped credentials, extends [Sandbox] Sandbox-scoped credentials — no production secrets in a sandbox #194
• [Workspace] Preview proxy and human takeover — open the failing case and look #206 Preview proxy and human takeover — git worktree workspace, snapshot, open the failing case

Sequence: #201 → #203 → #204 → #202 → #205 → #206, interleaved with the eval track (#182 next).

[Sam123ben]

Current epic status after the 2026-08-20 release pass:

• The implementation issues for sandbox objects, provisioning, tool modes, traces, ADK surface, datasets/scorers, comparisons/gates, UI, workspace/browser/MCP/egress/broker/preview, CI, docs, quotas, and the nine golden suite definitions are delivered.
• [Evals] Golden eval suites for DevAI's own built-in agents #188 remains open only for nine durable owner-scoped baseline runs.
• [Sandbox] Sandbox-scoped credentials — no production secrets in a sandbox #194 remains open only for the signed-in owner acceptance proving the selected LLM grant, absent SCM/cloud credentials, and identifier-only audit record in production.
• No connected signed-in browser is available in this session, and there is no dedicated owner-scoped Actions connector. Running either acceptance with a platform/shared key or an impersonated identity would violate the epic isolation rules.

Required next input: connect a browser session signed in as the owner of the intended personal LLM connector (or provision a dedicated owner-scoped CI connector), then rerun #194 once and each of the nine #188 suites. Close this epic only after both evidence sets are retained.

The repository has been returned to PRIVATE after the verified release.

[Sam123ben]

2026-08-20 epic recheck: #188 and #194 remain the only owner-session acceptance blockers. Browser discovery returned no connected session, and no dedicated owner-scoped CI connector exists. The epic remains open; production is private and continues on the isolated, gateway-required Job path.