fix(tests): make the Claude Code skill suite runnable and realign SDD assertions with v6.2.0 - #2071
Open
ericyen97903-lab wants to merge 2 commits into
Open
Conversation
…stic
Two defects in tests/claude-code/test-helpers.sh made this suite unusable
as a signal.
1. run_claude did not redirect stdin. The prompt is passed as an argument,
so these calls never want piped input, but an inherited non-TTY stdin
that never closes (CI, or the suite invoked from a script) makes
`claude -p` block waiting on it. The script then hits run-skill-tests.sh's
900s per-test cap and is reported as a timeout instead of reporting its
assertions. Locally this took test-subagent-driven-development.sh from a
900s timeout to a 110s complete run.
2. assert_contains matched line by line. Its patterns deliberately link two
prose keywords with `.*` ("implementer.*fix", "read.*plan"), but grep is
line-based, so a correct answer that puts those keywords in different
paragraphs failed. Sampling "what happens if a reviewer finds issues?"
three times: the answer contained "implementer" 3x and "fix" 7-9x every
time, yet the assertion passed only 1 of 3 runs purely on how the model
happened to wrap lines. Newlines are now flattened before matching.
Flattening is confined to assert_contains, and it can only turn a FAIL into
a PASS, so no currently-passing assertion changes meaning.
assert_not_contains and assert_order keep their line-based semantics;
assert_count is line-counting by definition and has no callers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two assertions in test-subagent-driven-development.sh still described pre-v6.2.0 behaviour. Because the script exits at the first failure, only the earlier one was ever visible. 1. "Read at beginning" looked for `Step 1|beginning|start|Load Plan`. The plan-scoped workspace work renamed that phase to Setup, so the model now answers "during Setup" and the assertion only passed when the wording happened to also contain "start". Sampling the prompt 5 times: "Setup" appeared 5/5, the assertion passed 2/5. Added Setup to the alternation. 2. "Provides text directly" looked for `provide.*directly|full.*text|paste| include.*prompt`, but SKILL.md now routes the task through `scripts/task-brief`: the dispatch carries the brief path, exact values "appear only in the brief", and "Never make a subagent read the whole plan file". The prompt asks for "<directly or by file>" and the answer is now "by file" (3/3 samples), so the assertion was testing for the behaviour the rewrite removed. It now asserts the brief-file handoff and is renamed accordingly. The file header's example keyword list is updated from "Step 1" to "Setup" for the same reason. With these and the harness fixes, the suite runs green end to end (15/15 assertions, exit 0) on macOS with GNU coreutils installed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests/claude-code/run-skill-tests.shdoes not currently produce a usable signal fortest-subagent-driven-development.sh. Two harness defects and two stale assertions compound: the script blocks on stdin, is reported as a 900s timeout, and when it does run it exits at the first of three failing assertions, hiding the rest.All four are fixed here. The changes are test-only; no skill content is touched.
1.
run_claudeblocks on inherited stdinThe prompt is passed as an argument, so these calls never want piped input. But when stdin is a non-TTY that never closes (CI, or the suite invoked from a script),
claude -pwaits on it. The script then hits the runner's 900s per-test cap and is reported as a timeout rather than reporting its assertions.Redirecting stdin from
/dev/nulltooktest-subagent-driven-development.shfrom a 900s timeout to a 110s complete run locally.2.
assert_containsmatched line by lineIts patterns deliberately link two prose keywords with
.*(implementer.*fix,read.*plan), butgrepis line-based, so a correct answer that puts those keywords in different paragraphs fails.Sampling "what happens if a reviewer finds issues? Is it a one-time review or a loop?" three times, the answer contained
implementer3x andfix7-9x every time, yetimplementer.*fix|fix.*issuespassed only 1 of 3 runs, purely on how the model happened to wrap lines. In a failing sample,implementersat on lines 7-8 andfixon lines 3, 5, 9, 19, 21.Newlines are now flattened before matching. This is confined to
assert_contains, and flattening can only turn a FAIL into a PASS, so no currently-passing assertion changes meaning.assert_not_containsandassert_orderkeep their line-based semantics, andassert_countis line-counting by definition and has no callers.3. "Read at beginning" still looked for the pre-rename vocabulary
The assertion searched
Step 1|beginning|start|Load Plan. The plan-scoped workspace work renamed that phase to Setup, so the model answers "during Setup" and the assertion only passed when the wording happened to also contain "start".Sampling that prompt 5 times:
Setupappeared 5/5, the assertion passed 2/5.Setupis added to the alternation.4. "Provides text directly" asserts behaviour v6.2.0 removed
The assertion searched
provide.*directly|full.*text|paste|include.*prompt.SKILL.mdnow routes the task throughscripts/task-brief: the dispatch carries the brief path, exact values "appear only in the brief", and "Never make a subagent read the whole plan file."The prompt asks for
<directly or by file>and the answer is now consistently "by file" (3/3 samples), so this assertion was testing for the behaviour the rewrite deliberately replaced. It now asserts the brief-file handoff and is renamed to match.Verification
Two consecutive clean runs of the full runner, no external stdin redirect, exit code 0:
15/15 assertions in
test-subagent-driven-development.sh. Environment: macOS 15 (arm64), GNU coreutils 9.11 fortimeout, Claude Code 2.1.156.Because assertion 3 and 4 are prose-matching against a model's free-form answer, each was sampled repeatedly rather than trusted on a single green run. Sample counts are given above.
🤖 Generated with Claude Code