Skip to content

spec-loop-runner.md

Latest commit

 

History

History
253 lines (229 loc) · 15.2 KB

spec-loop-runner.md

File metadata and controls

253 lines (229 loc) · 15.2 KB
Raw

title: Spec-loop runner status: experimental kind: feature mode: infra source: > tools/spec-loop/README.md, tools/spec-loop/loop.sh, tools/spec-loop/lib.sh, tools/spec-loop/tests/test_runner_fixtures.sh, tools/spec-loop/PROMPT_plan.md, tools/spec-loop/PROMPT_build.md, tools/spec-loop/PROMPT_update.md, tools/spec-loop/PROMPT_consolidate.md, and docs/spec-driven-development.md. acceptance:

  • The loop supports plan, build, update, and consolidate beats with one fresh agent context per iteration.
  • Build and update beats never commit to the integration base; they create reviewable local branches and stop before push or PR creation.
  • Open PRs and local work-item branches are passed into plan/build prompts as in-flight work so the loop does not duplicate already-built items.
  • The headless-agent harness contract is explicit for Claude Code, Codex, Cursor, Antigravity CLI, OpenCode, Kiro, Grok Build, Pi, and Goose.
  • The update marker tools/spec-loop/.last-sync is owned by the runner, not by the prompt.
  • At most one loop runs per checkout: a second start while a lock is held prints already_running and exits 0, so a scheduler records a skipped run rather than a failure.

Spec-loop runner

What it does

The spec-loop runner is the framework's local, review-first development loop. It runs a headless agent CLI against fixed prompts, one iteration at a time, to compare specs with the working tree, produce planned changes, back-fill specs after normal contributions, or consolidate a long implementation plan.

The loop is not an autonomous merge system. Its boundary is local branch creation: it may edit, validate, and commit locally, but it never pushes and never opens a pull request.

Where it lives

  • tools/spec-loop/loop.sh — Bash runner for the four beats.
  • tools/spec-loop/lib.sh — deterministic prompt assembly, harness command rendering, agent launch, the single-flight lock (spec_loop_lock_acquire / spec_loop_lock_release), and .last-sync marker helpers used by the runner and fixture tests.
  • tools/spec-loop/tests/test_runner_fixtures.sh — deterministic fixture tests for prompt assembly, harness command construction, and .last-sync marker helpers; executed by spec-validate.
  • tools/spec-loop/PROMPT_plan.md — gap-analysis prompt that rewrites IMPLEMENTATION_PLAN.md.
  • tools/spec-loop/PROMPT_build.md — implementation prompt for exactly one work item.
  • tools/spec-loop/PROMPT_update.md — spec back-fill prompt for functionality that landed outside the loop.
  • tools/spec-loop/PROMPT_consolidate.md — plan-size reduction prompt.
  • tools/spec-inventory/ — spec-inventory --brief supplies the compact repository inventory appended to every prompt, and spec-scope maps changed paths to the specs whose Where it lives sections name them.
  • tools/spec-loop/AGENTS.md — loop-specific operational context: repository map, validation commands, branch rules, hard limits, and commit rules.
  • tools/spec-loop/README.md — operator quickstart.
  • docs/spec-driven-development.md — full explanation of the loop's posture and lifecycle.

Behaviour & contract

  • Four beats, one mechanism. plan compares specs to code and updates the implementation plan without committing. build implements one uncovered work item. update compares code to specs and back-fills the spec directory. consolidate shrinks the implementation plan without dropping planned work.
  • One work item, one branch, one PR. Build work creates a bare <slug> branch off SPEC_LOOP_BASE and commits exactly one work item there. Update work creates a uniquely named sync-specs-<timestamp> branch. Consolidate commits only the plan file on the control branch.
  • No remote state changes. The runner and prompts forbid git push and gh pr create; each successful build/update beat prints the human-run push and gh pr create --web commands instead.
  • In-flight duplicate guard. Before each relevant iteration, the runner appends open PR context and local work-item branch context to the prompt. Plan/build treat both as in-flight work. This matters because a loop-built branch may be local-only and invisible to GitHub.
  • Control branch vs integration base. TOOLING_REF is captured before checkout. When the integration base does not carry the spec-loop files, the runner tells the agent to read prompts/specs/plan from the control branch with git show, while implementing product changes on the work branch.
  • Update marker ownership. tools/spec-loop/.last-sync records the base SHA last synced by the update beat. The runner reads it to append incremental-scope guidance, then amends or creates a marker commit after the agent finishes. Prompts must not instruct the agent to edit the marker.
  • Deterministic scope mapping. When .last-sync names a previous sync, the runner lists the paths changed since it (git diff --name-only <prev>..<BASE_HEAD> -- <SPEC_LOOP_UPDATE_PATHSPEC>, defined once in lib.sh: plugins skills .claude/skills tools docs projects organizations .github plus AGENTS.md and README.md), pipes them through spec-scope, and appends the specs it returns as likely relevant. spec-scope maps a skill edit, which git reports under plugins/<plugin>/skills/<alias>/, back to skills/<name>/ through the skills/ symlinks, so a spec's 'Skill: <name>' bullet matches it. The mapping is advisory: a spec-scope failure is silently skipped and the agent still reads the raw path list.
  • Single flight. Before its first iteration the runner creates the lock tools/spec-loop/.lock (gitignored; SPEC_LOOP_LOCK relocates it): a symbolic link whose target is <pid> <host> <start>, so ln -s creates the lock and its owner in one atomic step. A background heartbeat refreshes the link's mtime every SPEC_LOOP_LOCK_BEAT_SEC seconds (default 60). While the lock is held, a second start prints one already_running: … line naming the holder and exits 0, never launching an agent and never touching the holder's lock. A lock is stale when its pid is gone on the same host (a pid of another user counts as alive), or when its heartbeat is older than SPEC_LOOP_LOCK_STALE_MIN minutes (default 10) on any host. A stale lock is taken over under a second link, <lock>.takeover, which serialises takeovers: the taker re-reads the owner and removes the lock only if it still names the owner judged stale, so of two starters racing over one stale lock only one acquires. The lock is released on every exit path (normal end, STOP, error break, Ctrl+C, SIGTERM), but only once the backgrounded agent has stopped: on INT/TERM the runner signals the agent and waits up to SPEC_LOOP_STOP_TIMEOUT seconds (default 10). If the agent is still running, the lock is handed to the agent's pid (an atomic rename of a new link) and the heartbeat keeps it fresh until the agent exits. A stopped run exits 130 (INT) or 143 (TERM). An exit before acquisition (--help, a bad argument, a missing agent CLI) never releases it. SPEC_LOOP_LOCK=off disables the lock for a run the operator knows is safe to overlap.
  • Plan-size hysteresis. Build mode switches to one consolidate pass when IMPLEMENTATION_PLAN.md exceeds SPEC_LOOP_PLAN_MAX, then builds even if planned work alone keeps the file over the threshold. The latch resets once the plan drops below the threshold.

Headless harness contract

Every supported harness must provide the same loop-level behaviour: run one non-interactive agent iteration in the repository root, receive the assembled prompt, allow local edits/validation/commits under the external sandbox, and stop without pushing or opening a PR.

Harness Prompt transport Cwd contract Unattended flag Model override Output mode Extra denial
Claude Code stdin to claude -p launched from repo root --dangerously-skip-permissions --model --output-format --disallowedTools denies push and gh
Codex stdin to codex exec - --cd "$ROOT" --dangerously-bypass-approvals-and-sandbox --model --json for stream JSON external sandbox and exec policy
Cursor positional prompt to cursor agent --print or cursor-agent --print --workspace "$ROOT" --force --trust --model --output-format external sandbox and Cursor policy
Antigravity CLI -p "<prompt>" to agy launched from repo root none — never --dangerously-skip-permissions; headless keeps request-review --model --output-format user-scope terminal sandbox, permission rules and the agent-guard --antigravity hook; headless actions needing approval are soft-denied (exit 0, listed in denied_actions)
OpenCode positional prompt to opencode run launched from repo root --auto --model --format json for stream JSON external sandbox and OpenCode policy
Pi -p "<prompt>" to pi launched from repo root none — Pi has no approval flag; the user-scope Magpie extension refuses every asked command when there is no UI --model
Goose run --no-session -t "<prompt>" to goose launched from repo root none — a non-interactive run cannot answer approve-mode prompts, and outside approve mode the Magpie hook blocks what would need approval --model --output-format json the Magpie plugin hook (guard, deny rules); the whole process in sandbox-runtime when started through agent-iso goose
Kiro positional prompt to kiro-cli chat launched from repo root --no-interactive agent config (.kiro/agents) plain text only external sandbox and the agent-guard --kiro hook
Grok Build --prompt-file <assembled-prompt> --cwd "$ROOT" --always-approve --model plain; streaming-json for stream JSON --deny blocks git push and gh; external sandbox and agent guard remain active

SPEC_LOOP_EFFORT is the harness-neutral reasoning-effort band (low / medium / high). Unset means no effort flag is added. Invalid values fail at startup before the first iteration. Where a CLI offers more levels than the framework band, high maps to that CLI's top value.

Flag names and accepted values below were taken from each CLI's own --help (or the vendor docs when the binary was unavailable) at the time this mapping landed. They are version-dependent: an installed CLI that rejects a mapped value will fail the agent launch mid-loop. When bumping a harness CLI, re-check --help and update this table and spec_loop_launch_agent together.

Harness Effort flag Source low medium high
Claude Code --effort claude --help (low|medium|high|xhigh|max) low medium max
Codex -c model_reasoning_effort=… codex exec --help (-c/--config); key from Codex config low medium xhigh
OpenCode --variant opencode run --help (e.g. high, max, minimal) minimal medium max
Kiro --effort Kiro CLI effort docs (low|medium|high|xhigh|max) low medium max
Cursor (none — omitted silently) cursor agent --help (no effort/thinking-level flag) — — —
Antigravity CLI --effort agy --help 1.3.6 (low|medium|high|xhigh|max) low medium max
Pi --thinking pi --help 1.4.0 (off|minimal|low|medium|high|xhigh|max, clamped to the model) low medium max
Grok Build --reasoning-effort grok 1.0.50 --help (--effort alias also accepted) low medium high

SPEC_LOOP_AGENT chooses the CLI. SPEC_LOOP_HARNESS chooses the invocation convention and defaults from the agent basename. Adding a new harness means extending both matrices above (headless contract and effort mapping), documenting the safety boundary, and updating spec_loop_launch_agent in lib.sh (plus the harness case in loop.sh) in the same change.

Out of scope

  • Auto-pushing, auto-opening PRs, auto-merging, or changing remote state.
  • Replacing the project sandbox or adding new filesystem/network permissions.
  • Teaching individual skills how to do their domain work; those contracts live in the corresponding functional specs.
  • Editing docs/rfcs/; RFCs remain the separate governance layer.

Acceptance criteria

  1. loop.sh accepts plan, build, update, and consolidate, plus a bare numeric build count, and rejects unknown modes or non-numeric iteration counts.
  2. Build/update iterations check out SPEC_LOOP_BASE, create a non-base branch, and stop if a commit lands on the base.
  3. Plan/build prompts receive both open PR context and local work-item branch context.
  4. Update prompts receive incremental-scope guidance from .last-sync when present, but the runner remains the only writer of .last-sync.
  5. Claude Code, Codex, Cursor, Antigravity CLI, OpenCode, Kiro, Grok Build, Pi, and Goose are documented in the headless harness matrix and implemented by the runner.
  6. Operator docs and prompts agree on branch naming: build uses bare <slug> branches; update uses sync-specs-<timestamp> branches.
  7. Security docs describe unattended agent flags as harness-specific agent-level bypasses that must run under the external sandbox.
  8. With a live lock held, loop.sh <mode> exits 0, prints already_running, launches no agent, and leaves the lock in place; a stale lock (dead pid, same host) is taken over, and of two starters racing over it only one acquires; a lock from another host whose heartbeat is older than SPEC_LOOP_LOCK_STALE_MIN is taken over; a TERM while an agent ignores it exits 143 and leaves the lock naming the agent; SPEC_LOOP_LOCK=off skips the lock; a completed run leaves no lock behind.

Validation

bash -n tools/spec-loop/loop.sh
shellcheck tools/spec-loop/loop.sh
shellcheck tools/spec-loop/lib.sh tools/spec-loop/tests/test_runner_fixtures.sh
uv run --project tools/spec-validator --group dev spec-validate
bash tools/spec-loop/tests/test_runner_fixtures.sh

Known gaps

  • Cursor, Antigravity CLI, OpenCode, Kiro, Pi, and Goose rely on external policy/config plus the OS sandbox for push/PR denial. Claude Code and Grok Build additionally receive per-invocation hard-deny rules from the runner.
  • The single-flight stop signals only the agent's own pid. Processes the agent started and does not stop on SIGTERM keep running after the runner exits, and a SPEC_LOOP_AGENT wrapper that does not exec the agent leaves the agent running. Stopping the whole process group would need job control, which moves the agent off the terminal's foreground group.
  • A holder that is alive but has stopped its heartbeat for longer than SPEC_LOOP_LOCK_STALE_MIN (a host suspended for that long, for example) loses its lock to the next start, and the two runs then overlap. Its release at exit leaves the newer lock alone, because release only removes a lock that names its own pid.