streams: cap JSONL line size and transcript batch bytes - #2245
Draft
scouredimage wants to merge 1 commit into
Draft
streams: cap JSONL line size and transcript batch bytes#2245scouredimage wants to merge 1 commit into
scouredimage wants to merge 1 commit into
Conversation
Transcript batches were bounded by event count only (1000), and read_jsonl_line buffered lines of unbounded length. A transcript whose events embed file contents (normal for agent tool results) could put hundreds of MB into a single batch, which downstream redaction and metrics conversion amplify several times over -- ballooning daemon RSS past the memory watchdog within seconds (git-ai-project#2244). - read_jsonl_line: cap a single line at MAX_JSONL_LINE_BYTES (8 MiB). The read is byte-based (read_until), NOT read_line: the cap can slice a multi-byte character, and read_line's UTF-8 validation would return InvalidData instead of classifying the line as Oversized -- wedging the stream at a fixed watermark. Oversized lines are skipped without being buffered (skip_until); callers advance their watermark past them via the new JsonlLineState::Oversized state. Non-UTF-8 content within the cap keeps read_line's InvalidData contract. - All JSONL byte-offset stream parsers (claude, codex, copilot, cursor, droid, gemini, pi, windsurf): stop a batch early once MAX_BATCH_BYTES (8 MiB) of raw JSON has been accepted; remaining events arrive in later batches. Not covered here: amp, continue_cli, and opencode parse whole files into a DOM (serde_json::from_reader) and need a separate treatment -- called out in git-ai-project#2244 as follow-up. Part 1/7 of the git-ai-project#2244 fix stack.
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1/7 of the #2244 fix stack (transcript-sweep memory balloons -> watchdog abort loop -> blocked traced git).
Transcript batches were bounded by event count only (1000), and
read_jsonl_linebuffered lines of unbounded length; a transcript whose events embed file contents can put hundreds of MB into one batch, which downstream redaction + metrics conversion amplify several times over.read_jsonl_line: 8 MiB per-line cap, byte-based (read_until) so a cap that slices a multi-byte character classifies as the newJsonlLineState::Oversizedinstead ofInvalidData(which would wedge the stream at a fixed watermark). Oversized lines are skipped without buffering; non-UTF-8 within the cap keeps theInvalidDatacontract. Unit tests cover the multi-byte boundary.Not covered: amp / continue_cli / opencode parse whole files into a DOM and need separate treatment (follow-up noted in #2244).
Stack
Each PR targets
main; the incremental diff of part N is its last commit. Root-cause analysis, production evidence, and the repro live in #2244.Verification
daemon memory limit): unpatched v1.6.24 aborts in ~1 s and re-aborts after respawn
(watermark 0); with the full stack the daemon survives, peak RSS 76 MB vs 589 MB,
zero memory emergencies, traced-git probes at 0.05-0.07 s throughout.
cargo fmt --checkclean;cargo test --lib: 2,401 passed. Three failures(
commands::upgrade::...pending_update,daemon::...conflict_resolution_note_read_errors...,git::authorship_traversal::...ai_touched_files...) reproduce identically on cleanmainin this environment - pre-existing, unrelated.