Add zcode agent preset for checkpoint attribution - #2159
Conversation
ZCode emits Claude Code-compatible PreToolUse/PostToolUse hook payloads, but its transcript_path is a temp snapshot rather than a persistent transcript. The preset derives the live rollout (~/.zcode/cli/rollout/model-io-sess_<session_id>.jsonl) from the session id and extracts the model from its first model-io record (bounded read, missing/unparsable files fall back to "unknown"). Failed tool calls (PostToolUseFailure) and non-tool events are ignored since nothing landed on disk; ApplyPatch is classified as a file edit to match zcode's matcher aliases.
|
|
|
The workflow runs (Test / Lint & Format / End-to-End Tests) are pending approval since this is the first PR from this fork. Could a maintainer approve the workflow runs so CI can execute? Happy to iterate on any failures once they run. |
Model-io records are append-only per request, so reading only the first record mislabels every edit after a mid-session model switch. Extraction now scans the tail window's complete records newest-first (bounded at 4MB from EOF, since each record embeds the whole conversation) and falls back to the opening record when no parsable record fits the window.
c153c83 to
8c164c5
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Route the common cases through extract_model_from_jsonl_tail_with and extract_model_from_jsonl_head_with like the other agents, keeping the wider last-lines window and bounded first-line read only as fallbacks for model-io records that embed the whole conversation and exceed the shared scan windows.
|
Addressed the follow-up finding: |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The previous chain consulted the head scanner (oldest records) before the wide last-lines window, so a session whose early records fit the shared per-line cap but whose grown records did not was labelled with its starting model even after a mid-session switch. Reorder to tail scan, wide last-lines window, head scan, first line, and add a regression test for the switch-after-growth case.
|
Good catch — the fallback chain did consult the head scanner (oldest records) before the wide last-lines window, so sessions whose records outgrow the shared per-line cap were labelled with their starting model. Reordered to newest-first probes (tail scan, wide last-lines window) ahead of the oldest fallbacks (head scan, first line) and added a regression test for the switch-after-growth case (7486bba). All 9 extraction tests green. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Patch-style ApplyPatch payloads carry the edited paths inside the patch text rather than a file_path key, so those checkpoints recorded no files and the edits lost AI attribution. file_paths_for_event now falls back to parsing the patch text (shared collect_apply_patch_paths_from_text) and, for PostToolUse, the tool_response file_path, mirroring the droid preset. read_last_complete_lines_bounded also stopped discarding the bytes before the window's first newline when the window covers the whole file (window_start == 0): that prefix is a complete first line, not a truncated prior line, so single-record rollouts between the first-line cap and the wide window size no longer lose their model.
|
Addressed the new findings (9284f83):
Changes were reviewed internally before push; all 29 zcode unit tests and clippy are green. |
| for line in read_last_complete_lines_bounded(path, MAX_ZCODE_ROLLOUT_LAST_LINE_BYTES) | ||
| .iter() | ||
| .rev() | ||
| { | ||
| if let Some(model) = extract_model_from_zcode_rollout_line(line) { | ||
| return Ok(Some(model)); | ||
| } | ||
| } | ||
|
|
||
| if let Some(model) = | ||
| extract_model_from_jsonl_head_with(path, extract_model_from_zcode_rollout_line) | ||
| { | ||
| return Ok(Some(model)); | ||
| } |
There was a problem hiding this comment.
🟡 Long AI sessions can be labelled with the model used at session start instead of the model actually in use
When the newest entry in the transcript is bigger than the read window, the code falls back to the very first entry of the session (extract_model_from_jsonl_head_with at src/streams/model_extraction.rs:141-145) instead of the most recent readable one, so a session that switched models mid-way is reported under the old model.
Impact: Attribution for long sessions can name the wrong AI model, and the longer the session the more likely it is wrong.
Why the newest-record probes fail and the oldest-record fallback wins
Each zcode model-io record embeds the whole conversation, so the last line grows monotonically and, per the comment at src/streams/model_extraction.rs:57-61, "routinely" exceeds the windows.
extract_model_from_jsonl_tail_withreads only the last 50KB (MAX_JSONL_SCAN_BYTES), which is a truncated fragment of the giant last line → no parse.read_last_complete_lines_bounded(path, 4MB)seeks tosize - 4MB. If the last record is larger than 4MB, the entire window lies inside that single line,window.find('\n')returnsNone, and the function returns an empty vector (src/streams/model_extraction.rs:103-107) — it never reaches the second-to-last record, which is complete and readable just before the giant one.- Control therefore reaches the head scanner, which returns the first record of the session (oldest model), defeating the stated goal that "newest-record probes come first so mid-session model switches are attributed correctly".
A backwards scan for the last newline preceding the final record (or repeatedly widening/stepping the window back past the oversized line) would recover the newest parseable record instead of falling all the way back to the session's opening record.
Was this helpful? React with 👍 or 👎 to provide feedback.
Model-io records embed the whole conversation, so the previous wide window full-line read cost several megabytes and multiple copies per hook invocation, and when the newest record exceeded the window the extraction fell all the way back to the session's opening record. Extraction now locates the newest record's start with a bounded backward chunk scan and reads only that record's head for model.modelId (JSON string contents escape their quotes, so a raw "modelId":" match only lands on real key positions). Records past the 16MB backward budget still fall back to the opening records.
|
Addressed the two extraction findings (443321b):
Both changes were reviewed internally before push; 11 extraction tests green, clippy clean. |
|
@svarlamov apologies for the direct ping — could you take a look when you have a moment? Two things would unblock this stack:
All Devin Review rounds are resolved, local test suites (31 unit + 4 e2e) and clippy are green. Happy to iterate on anything that comes up. |
2 participants
Part of #2150 (closing keyword lives in the stacked PR #2161, which completes the feature).
Summary
Adds a
ZcodePresetso ZCode edits are attributed to AI sessions, wired through the samegit-ai checkpoint zcode --hook-input stdinflow as other agents.ZCode emits Claude Code-compatible
PreToolUse/PostToolUsehook payloads, with two differences that shape this PR:transcript_pathis a temp snapshot under the OS tmpdir, so the preset derives the live rollout (~/.zcode/cli/rollout/model-io-sess_<session_id>.jsonl) from the session id instead (constant-time path join, with path-traversal guards on the session id).model-ioJSONL format, so the model is extracted from the newest record'smodel.modelIdvia a new bounded tail-scanextract_model_from_zcode_rollout(falls back to the opening record when no parsable record fits the window; missing files fall back to"unknown").Design notes:
PostToolUseFailure,PermissionRequest,SessionStart,UserPromptSubmit, andStopproduce no events (a failed edit never landed on disk;PreToolUsealready captures pre-state).ApplyPatchis classified as a file edit to match zcode's matcher aliases (Write/Edit←ApplyPatch).~/.claudetranscript are rejected with a preset error, mirroring the guards in the claude preset.StreamSourcein v1: zcode's transcript format is not one of the streamed formats; session records (with model) are still created from the checkpoint'sAgentId(verified by the e2e tests).Changes
src/commands/checkpoint_agent/presets/zcode.rs— new preset + unit testssrc/commands/checkpoint_agent/presets/mod.rs— registry entrysrc/commands/checkpoint_agent/bash_tool.rs—Agent::Zcodeclassificationsrc/streams/model_extraction.rs—extract_model_from_zcode_rollout+ testssrc/commands/git_ai_handlers.rs— help texttests/fixtures/zcode-rollout-simple.jsonl— sanitized model-io fixturetests/integration/zcode.rs— parse tests + file-edit and bash e2e cycleszcodeadded tois_known_checkpoint_preset(otherwisegit_ai()rewrites checkpoint args as pathspec files) andAI_AUTHOR_NAMESTesting
cargo test zcode— unit + integration tests (incl. auto-generated worktree variants)cargo test preset/cargo test model_extractionregressions greencargo clippy --all-targets -- -D warningsclean,cargo fmtappliedA follow-up PR adds the
install-hooksinstaller for~/.zcode/cli/config.jsonand README support.