fix(skills): the seven follow-ups from the #1226 review - #1227
Merged
Merged
Conversation
jswir
approved these changes
Sep 24, 2026
jswir
left a comment
Collaborator
There was a problem hiding this comment.
Approving. The seven follow-ups are all addressed, and I ran the branch end to end on the storefront-tour set to check them:
- The three changed test files run clean the way CI runs them (
python3 <file>): 186, 118 and 4 tests. - A 12-case arm scored 10 of 12, with 95.5% recall. 6 of the 12 cases had a tool result saved to a file, so the merged reader was exercised for real.
--rebuildreproduced the live run exactly, with no answerer spend.- I moved one saved file aside and rebuilt: that call was recorded as unmeasured (
rankedSummary: null), not zero, and the other cases' ledger lines were kept. --rejudge --only <qid>reused the saved answer byte for byte, ran only the judge, and left the other 11 cases' ledger lines identical.
Three things I found are inline. Each one changes a recorded result, so I'd fix them before merge if it's quick, or in a follow-up otherwise:
- Semicolon fix: a query with an escaped quote, or an apostrophe in a comment, no longer matches its printed form. The harness then grades the wrong query. This is a regression against the old code.
- Saved-to pattern: an ordinary result that mentions "written to " is replaced by that file's contents, and is then scored as zero entities.
--model-dirpointing at a folder that doesn't exist inside the repo still records a dirty model as clean.
Two smaller cleanups are inline too: the tautological assert, and the unused OFFLOADED alias with the leftover comment.
Two older problems, not from this PR and not blocking:
- The run viewer shows an unmeasured call as zero.
build_run_package.py:375-382turnsrankedSummary: nullinton_returned: 0with an emptyerror. I checked it on the rebuilt run above: the unmeasured call shows incalls.csvas0, the same as a search that found nothing, andempty_callscounts it. It now happens on every rebuild whose saved files are gone, which--rejudgenow triggers too. Anunmeasuredcolumn, or leavingn_returnedempty, would fix it. - A
get_contextcall that errors is stored as zero results. One answerer left outscopes, and the MCP error was recorded as{resultCount: 0}with the error attached, rather thanrankedSummary: null.
Four test classes added by #1226 (HeldGoldensAreNotReDerived, ANotQueryableKeyNamesItsRootSource, NarrowedRebuildKeepsTheLedger, PersistedStubIsTheResult) sat below the if __name__ guard. CI runs each test file as a script, so unittest.main() exited before Python defined them: 113 tests ran where 117 exist, 169 where 175. Both files now end with the guard, and each carries a test that reads its own source and fails if a class or function is ever defined below it again, which is how #1210 hit the same thing in the same file. Signed-off-by: Girish Jeswani <girish@credibledata.com>
…d splices the ledger --rejudge is documented as reusing the saved answers and scoring them again, but every gate in run_baseline reads a.rebuild, and --rejudge alone left it false: a new answerer ran per case ($0.33 each on a storefront run), and with --only the ledger took the full-write path and a 12-case run came back holding one case. --from already set both flags; imply_flags makes the bare --rejudge do the same, and the help text says so. Signed-off-by: Girish Jeswani <girish@credibledata.com>
PERSISTED_STUB and OFFLOADED both matched "saved to" and handled the colon and the trailing period differently, and two readers sat behind them. A rebuild after the CLI's temporary tool-results/ file is gone is the ordinary case, and there result_text matched the note, found no file, returned the note, and the fallback regex missed the colon in "saved to: /path", so the call scored zero entities with nothing in the ledger saying it was unmeasured. saved_result() is the one helper: (None, None) for no note, (path, None) for a named file that cannot be read, (path, body) otherwise; result_text, offloaded_json and the transcript parser all read through it, and the parser records no summary for the missing-file case. The OSError handler no longer repeats the read that just failed. Signed-off-by: Girish Jeswani <girish@credibledata.com>
…inning clean git_sha resolved the scope against the working directory and judged git status by stdout alone. A relative --model-dir given from anywhere but the repo root pointed outside the repo; git exited 128 with empty stdout, and a dirty model was recorded as clean. The exit code is now checked like HEAD's (no pin, which the schema calls honest), and a relative --model-dir is resolved against --model-repo. The worked example in running-a-run.md passes --model-dir, and the --model-repo help and the schema row no longer say the marker is decided over the whole repo. Signed-off-by: Girish Jeswani <girish@credibledata.com>
_norm turned every ; into a space, so two queries that both ran, one filtering on 'Books;Media' and one on 'Books Media', normalised to the same text, and the declared block could resolve to whichever ran later. Quoted text is now left alone; whitespace inside a literal is still collapsed, as main always did. Signed-off-by: Girish Jeswani <girish@credibledata.com>
… reader build_run_package.py kept its own copy of the raw tool-result reader, so a get_context call whose result the CLI had saved to a file showed the note as its detail and no result count in the run package. It now imports run_baseline.result_text, the one reader. Signed-off-by: Girish Jeswani <girish@credibledata.com>
The bundle carries the run reference and ledger schema changes. No version bump: main's release stamp already moves the skills and scaffolder packages one patch ahead of npm, so both gates pass on what main declares. Signed-off-by: Girish Jeswani <girish@credibledata.com>
…the lexer does Inside a string a backslash escapes the next character (MalloyLexer.g4 ESCAPED), and outside one -- or // runs to the end of the line (COMMENT_TO_EOL). The previous loop read an escaped quote as the end of the string, so the real closing quote opened one that never closed and a later ; stayed quoted; an apostrophe in a comment did the same. Either made the printed and executed forms of one query normalise differently and the harness graded the probe that ran last, a regression against the blanket replace. Tests for the escaped quote, both comment spellings, double quotes and backticks. Signed-off-by: Girish Jeswani <girish@credibledata.com>
girishjeswani
force-pushed
the
girish/eval-skills-review-followups
branch
from
September 25, 2026 21:14
2c4ab47 to
0e3dca6
Compare
…hen the text is not a result already Merging the two patterns loosened what result_text treated as a saved-result note: a bare "saved to|written to" ran over every tool result first, so an ordinary get_context JSON whose source doc said "written to /etc/hosts" was replaced by that file and scored as zero entities, the false zero this work is removing. The pattern is the CLI's marker again (<persisted-output> ... Full output saved to, or Output has been saved to) with a .json or .txt path, and result_text consults it only when the text does not already parse as a result. The unused OFFLOADED alias and the comment about the deleted PERSISTED_STUB are gone; the test that compared a path's name to itself now asserts the name the note carried. Signed-off-by: Girish Jeswani <girish@credibledata.com>
…f pinning clean git status on a pathspec that does not exist exits 0 with empty output, so a typo (pgk) or a doubled path (--model-repo repo/pkg --model-dir pkg, the shape the storefront README's diagnose command invites) recorded a dirty model as clean. git_sha now returns no pin when the resolved scope does not exist. Both --model-dir help texts say what the path is relative to: --model-repo here, the working directory in diagnose.py. Signed-off-by: Girish Jeswani <girish@credibledata.com>
…d, not as zero results An answerer that left out scopes got an MCP validation error back, and the transcript parser recorded it as a ranking with resultCount 0 and the error attached. The retrieval score then read a search that found nothing. An errored call now records no summary, the same shape as a spilled file that cannot be read. Signed-off-by: Girish Jeswani <girish@credibledata.com>
…ot as zero A get_context call with no rankedSummary (the CLI's saved file gone at rebuild time, or an errored call) was written to calls.csv as n_returned 0, the same row a search that found nothing writes, and empty_calls counted it. Every rebuild whose saved files are gone produces these, and --rejudge now takes that path too. n_returned stays empty for such a call, a new 'unmeasured' column says why, and the package gains unmeasured_calls beside empty_calls. Signed-off-by: Girish Jeswani <girish@credibledata.com>
girishjeswani
force-pushed
the
girish/eval-skills-review-followups
branch
from
September 25, 2026 21:15
0e3dca6 to
2bb496e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1226, addressing the seven review comments there. One commit per item, a test per code change, run both as CI runs test files (
python3 <file>) and under-m unittest discover.if __name__guard, and CI runs each test file as a script. Both files now end with the guard and carry a test that reads its own source and fails if anything is defined below it again (An eval set over the storefront example, with verified goldens and a worked run #1210 hit the same thing). 113 -> 118 and 169 -> 176 tests as CI runs them.--rejudge --onlyoverwrote the whole ledger, and--rejudgere-answered every case. Every gate readsa.rebuild; the bare flag left it false.imply_flagssets it, as--fromalready did, so a narrowed re-judge splices and the saved answers are reused. Help text updated.PERSISTED_STUBandOFFLOADEDwere two regexes and two readers for one note, disagreeing on the colon and the trailing period.saved_result()is the one helper: no note, a named file that cannot be read, or the body.result_text,offloaded_jsonand the transcript parser read through it; the missing-file case records no summary (unmeasured), not zero. TheOSErrorhandler no longer repeats the failed read.--model-diroutside the repo pinned a dirty model as clean.git_shanow checks the status exit code (128 for a pathspec outside the repo) and pins nothing; a relative--model-dirresolves against--model-repo. The worked example inrunning-a-run.mdpasses--model-dir; the--model-repohelp and the schema row no longer say the marker covers the whole repo._normtreated a;inside a string literal as a separator, so'Books;Media'and'Books Media'were one query. Only a;outside quotes separates clauses now.build_run_package.pykept its own copy of the old result reader; it importsrun_baseline.result_text.Versions:
@malloy-publisher/skills0.1.25 andcreate-malloy-package0.0.21 (skill content changed: the run reference and the ledger schema). Bundle regenerated.