[Evals] Golden eval suites for DevAI's own built-in agents

[Sam123ben]

Part of the Agent Sandbox & Evaluation epic.

Goal

Dogfood. DevAI ships 40 agent artifacts
(architecture/registry-seeds/agents/) and 26 specialization personas, and we cannot
currently say whether any individual one got better or worse between releases — only
whether a whole pipeline run produced a PR.

Today (grounded)

src/devai/pipeline/stages/evals.py scores the run. A regression in, say,
db-engineer-agent shows up only as a vaguer end-to-end signal, if at all. The ALM
pipeline harnesses the code our agents write, never the agents themselves.

Tasks

• Golden datasets for the highest-leverage agents first — the ones on the critical
  path and the ones with real side effects:
  engineering-manager, code-remediator / senior-developer, db-engineer,
  security-expert, qa-tester, release-promoter, incident-responder,
  discovery, cost-optimizer.
• Each dataset covers: happy path, adversarial/prompt-injection, tool failure,
  and a case that should be refused.
• An eval suite per agent, published as a registry artifact next to the agent.
• Seed a baseline eval run per agent so later comparisons have something to diff.
• Then extend to the remaining agents; track coverage as a number on the dashboard.

Why this is the payoff issue

Once every built-in agent has a suite, "did this change make DevAI better?" becomes a
measurement rather than an opinion — and every user-authored agent inherits the same
machinery on day one.

Acceptance

Nine built-in agents have versioned golden datasets, published suites, and a stored
baseline eval run.

Plan: docs/plans/agent-sandbox-evals/IMPLEMENTATION-PLAN.md §8

[Sam123ben]

Current acceptance status (production read-only verification, 2026-08-20):

• Nine versioned public golden datasets exist for the required built-in agents.
• Nine matching published suite seeds exist.
• Every dataset has four cases covering happy path, adversarial/prompt injection, tool failure, and refusal; seed validation tests are green.
• Production eval_runs currently contains zero baseline rows for the nine built-in agents.
• The repository has no Actions secret for a dedicated owner-scoped evaluation identity/connector.
• Production connector metadata shows one personal OpenAI connector, but no authenticated browser session is connected to act as that owner. No connector secret or identity data was read or exposed.

I did not manufacture baseline rows, use a platform/shared key, or impersonate the connector owner. Remaining acceptance is to connect a signed-in owner session (or provision a dedicated owner-scoped CI connector), run each published suite once, and retain the nine durable baseline runs.

[Sam123ben]

2026-08-20 recheck: nine versioned datasets and suites remain delivered, but the required nine durable owner-scoped baseline runs cannot be created in this session because no signed-in browser session is connected and no dedicated owner-scoped CI connector exists. No shared/platform credential or impersonated identity was used.

[Sam123ben]

2026-08-20 owner-session acceptance recheck: the connected in-app browser runtime returned zero available sessions (agent.browsers.list() => []). The remaining nine durable owner-scoped baseline runs therefore cannot be truthfully executed. No shared/platform credential was substituted and no owner was impersonated; this issue remains open pending a real signed-in owner session.