Skip to content

fix(skills): the seven follow-ups from the #1226 review - #1227

Merged
girishjeswani merged 13 commits into
mainfrom
girish/eval-skills-review-followups
Sep 25, 2026
Merged

girishjeswani merged 13 commits into
mainfrom
girish/eval-skills-review-followups

Conversation

@girishjeswani

Copy link
Copy Markdown
Collaborator

Follow-up to #1226, addressing the seven review comments there. One commit per item, a test per code change, run both as CI runs test files (python3 <file>) and under -m unittest discover.

  • The four tests fix(skills): eight harness defects from a real-model eval pass #1226 added never ran in CI. They sat below the if __name__ guard, and CI runs each test file as a script. Both files now end with the guard and carry a test that reads its own source and fails if anything is defined below it again (An eval set over the storefront example, with verified goldens and a worked run #1210 hit the same thing). 113 -> 118 and 169 -> 176 tests as CI runs them.
  • --rejudge --only overwrote the whole ledger, and --rejudge re-answered every case. Every gate reads a.rebuild; the bare flag left it false. imply_flags sets it, as --from already did, so a narrowed re-judge splices and the saved answers are reused. Help text updated.
  • A rebuild after the spilled result file is gone scored zero entities. PERSISTED_STUB and OFFLOADED were two regexes and two readers for one note, disagreeing on the colon and the trailing period. saved_result() is the one helper: no note, a named file that cannot be read, or the body. result_text, offloaded_json and the transcript parser read through it; the missing-file case records no summary (unmeasured), not zero. The OSError handler no longer repeats the failed read.
  • A --model-dir outside the repo pinned a dirty model as clean. git_sha now checks the status exit code (128 for a pathspec outside the repo) and pins nothing; a relative --model-dir resolves against --model-repo. The worked example in running-a-run.md passes --model-dir; the --model-repo help and the schema row no longer say the marker covers the whole repo.
  • _norm treated a ; inside a string literal as a separator, so 'Books;Media' and 'Books Media' were one query. Only a ; outside quotes separates clauses now.
  • build_run_package.py kept its own copy of the old result reader; it imports run_baseline.result_text.

Versions: @malloy-publisher/skills 0.1.25 and create-malloy-package 0.0.21 (skill content changed: the run reference and the ledger schema). Bundle regenerated.

@jswir jswir left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The seven follow-ups are all addressed, and I ran the branch end to end on the storefront-tour set to check them:

  • The three changed test files run clean the way CI runs them (python3 <file>): 186, 118 and 4 tests.
  • A 12-case arm scored 10 of 12, with 95.5% recall. 6 of the 12 cases had a tool result saved to a file, so the merged reader was exercised for real.
  • --rebuild reproduced the live run exactly, with no answerer spend.
  • I moved one saved file aside and rebuilt: that call was recorded as unmeasured (rankedSummary: null), not zero, and the other cases' ledger lines were kept.
  • --rejudge --only <qid> reused the saved answer byte for byte, ran only the judge, and left the other 11 cases' ledger lines identical.

Three things I found are inline. Each one changes a recorded result, so I'd fix them before merge if it's quick, or in a follow-up otherwise:

  1. Semicolon fix: a query with an escaped quote, or an apostrophe in a comment, no longer matches its printed form. The harness then grades the wrong query. This is a regression against the old code.
  2. Saved-to pattern: an ordinary result that mentions "written to " is replaced by that file's contents, and is then scored as zero entities.
  3. --model-dir pointing at a folder that doesn't exist inside the repo still records a dirty model as clean.

Two smaller cleanups are inline too: the tautological assert, and the unused OFFLOADED alias with the leftover comment.

Two older problems, not from this PR and not blocking:

  • The run viewer shows an unmeasured call as zero. build_run_package.py:375-382 turns rankedSummary: null into n_returned: 0 with an empty error. I checked it on the rebuilt run above: the unmeasured call shows in calls.csv as 0, the same as a search that found nothing, and empty_calls counts it. It now happens on every rebuild whose saved files are gone, which --rejudge now triggers too. An unmeasured column, or leaving n_returned empty, would fix it.
  • A get_context call that errors is stored as zero results. One answerer left out scopes, and the MCP error was recorded as {resultCount: 0} with the error attached, rather than rankedSummary: null.

Comment thread skills/eval-loop/scripts/run_baseline.py
Comment thread skills/eval-loop/scripts/run_baseline.py Outdated
Comment thread skills/eval-loop/scripts/run_baseline.py
Comment thread skills/eval-loop/scripts/run_baseline_test.py Outdated
Comment thread skills/eval-loop/scripts/run_baseline.py Outdated
Comment thread skills/eval-loop/scripts/run_baseline.py Outdated
Four test classes added by #1226 (HeldGoldensAreNotReDerived,
ANotQueryableKeyNamesItsRootSource, NarrowedRebuildKeepsTheLedger,
PersistedStubIsTheResult) sat below the if __name__ guard. CI runs each test
file as a script, so unittest.main() exited before Python defined them:
113 tests ran where 117 exist, 169 where 175. Both files now end with the
guard, and each carries a test that reads its own source and fails if a
class or function is ever defined below it again, which is how #1210 hit
the same thing in the same file.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
…d splices the ledger

--rejudge is documented as reusing the saved answers and scoring them again,
but every gate in run_baseline reads a.rebuild, and --rejudge alone left it
false: a new answerer ran per case ($0.33 each on a storefront run), and with
--only the ledger took the full-write path and a 12-case run came back holding
one case. --from already set both flags; imply_flags makes the bare --rejudge
do the same, and the help text says so.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
PERSISTED_STUB and OFFLOADED both matched "saved to" and handled the colon
and the trailing period differently, and two readers sat behind them. A
rebuild after the CLI's temporary tool-results/ file is gone is the ordinary
case, and there result_text matched the note, found no file, returned the
note, and the fallback regex missed the colon in "saved to: /path", so the
call scored zero entities with nothing in the ledger saying it was unmeasured.
saved_result() is the one helper: (None, None) for no note, (path, None) for
a named file that cannot be read, (path, body) otherwise; result_text,
offloaded_json and the transcript parser all read through it, and the parser
records no summary for the missing-file case. The OSError handler no longer
repeats the read that just failed.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
…inning clean

git_sha resolved the scope against the working directory and judged git
status by stdout alone. A relative --model-dir given from anywhere but the
repo root pointed outside the repo; git exited 128 with empty stdout, and a
dirty model was recorded as clean. The exit code is now checked like HEAD's
(no pin, which the schema calls honest), and a relative --model-dir is
resolved against --model-repo. The worked example in running-a-run.md passes
--model-dir, and the --model-repo help and the schema row no longer say the
marker is decided over the whole repo.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
_norm turned every ; into a space, so two queries that both ran, one filtering
on 'Books;Media' and one on 'Books Media', normalised to the same text, and
the declared block could resolve to whichever ran later. Quoted text is now
left alone; whitespace inside a literal is still collapsed, as main always
did.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
… reader

build_run_package.py kept its own copy of the raw tool-result reader, so a
get_context call whose result the CLI had saved to a file showed the note as
its detail and no result count in the run package. It now imports
run_baseline.result_text, the one reader.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
The bundle carries the run reference and ledger schema changes. No version
bump: main's release stamp already moves the skills and scaffolder packages
one patch ahead of npm, so both gates pass on what main declares.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
…the lexer does

Inside a string a backslash escapes the next character (MalloyLexer.g4
ESCAPED), and outside one -- or // runs to the end of the line
(COMMENT_TO_EOL). The previous loop read an escaped quote as the end of the
string, so the real closing quote opened one that never closed and a later
; stayed quoted; an apostrophe in a comment did the same. Either made the
printed and executed forms of one query normalise differently and the
harness graded the probe that ran last, a regression against the blanket
replace. Tests for the escaped quote, both comment spellings, double quotes
and backticks.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
@girishjeswani
girishjeswani force-pushed the girish/eval-skills-review-followups branch from 2c4ab47 to 0e3dca6 Compare September 25, 2026 21:14
…hen the text is not a result already

Merging the two patterns loosened what result_text treated as a saved-result
note: a bare "saved to|written to" ran over every tool result first, so an
ordinary get_context JSON whose source doc said "written to /etc/hosts" was
replaced by that file and scored as zero entities, the false zero this work is
removing. The pattern is the CLI's marker again (<persisted-output> ... Full
output saved to, or Output has been saved to) with a .json or .txt path, and
result_text consults it only when the text does not already parse as a
result. The unused OFFLOADED alias and the comment about the deleted
PERSISTED_STUB are gone; the test that compared a path's name to itself now
asserts the name the note carried.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
…f pinning clean

git status on a pathspec that does not exist exits 0 with empty output, so a
typo (pgk) or a doubled path (--model-repo repo/pkg --model-dir pkg, the shape
the storefront README's diagnose command invites) recorded a dirty model as
clean. git_sha now returns no pin when the resolved scope does not exist.
Both --model-dir help texts say what the path is relative to: --model-repo
here, the working directory in diagnose.py.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
…d, not as zero results

An answerer that left out scopes got an MCP validation error back, and the
transcript parser recorded it as a ranking with resultCount 0 and the error
attached. The retrieval score then read a search that found nothing. An
errored call now records no summary, the same shape as a spilled file that
cannot be read.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
…ot as zero

A get_context call with no rankedSummary (the CLI's saved file gone at
rebuild time, or an errored call) was written to calls.csv as n_returned 0,
the same row a search that found nothing writes, and empty_calls counted it.
Every rebuild whose saved files are gone produces these, and --rejudge now
takes that path too. n_returned stays empty for such a call, a new
'unmeasured' column says why, and the package gains unmeasured_calls beside
empty_calls.

Signed-off-by: Girish Jeswani <girish@credibledata.com>
@girishjeswani
girishjeswani force-pushed the girish/eval-skills-review-followups branch from 0e3dca6 to 2bb496e Compare September 25, 2026 21:15
@girishjeswani
girishjeswani enabled auto-merge (squash) September 25, 2026 21:35
@girishjeswani
girishjeswani merged commit c3e52cc into main Sep 25, 2026
17 checks passed
@girishjeswani
girishjeswani deleted the girish/eval-skills-review-followups branch September 25, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants