Skip to content

fix(speed): count a response copied into a forked subagent once - #820

Open
eric-engberg wants to merge 4 commits into
sirmalloc:mainfrom
eric-engberg:fix/speed-response-across-files
Open

eric-engberg wants to merge 4 commits into
sirmalloc:mainfrom
eric-engberg:fix/speed-response-across-files

Conversation

@eric-engberg

Copy link
Copy Markdown
Contributor

What

Input, Output and Total Speed no longer count a forked subagent's opening copy of the response that started it. That copy made a second request out of the same API response. Right as a fork starts, the windowed speeds read 2–3x the real rate, and the session average stayed too high afterwards.

Why (evidence)

A forked subagent's transcript opens with a copy of the assistant row that spawned it: same message.id, same usage. After #622, #725 and #735, the speed metrics count a response once per transcript file but never across files, so each copy adds the response's tokens again. Token totals don't have this problem, since they only read the main transcript.

I scored main against a standalone reference (each response counted once across the main transcript and its subagent files) on 19 transcripts: 14 of my own recent sessions (Claude Code 2.1.267–2.1.314), two of them cut mid-response, and 3 synthetic ones.

  • Token totals match the reference on all 19.
  • Speed request counts and token sums are off wherever a fork copied a response. In the session with 7 forked subagents (copies of 4 responses), main counts 3,417 requests instead of 3,410, and 33.9k output tokens too many. Another session with 6 copies of 3 responses counts 6 extra requests and 31.5k extra tokens.
  • The error is largest right as a fork starts, when the copy and the original are both inside the window. I rendered those two sessions at each fork's first row after its copy: 13 renders, at 7 distinct moments, since parallel forks start together. Their 30 s Output Speed readings:
Moment Forks starting Reference main This PR
1 3 149.4 t/s 448.3 t/s 149.4 t/s
2 1 114.6 t/s 216.1 t/s 114.6 t/s
3 1 105.1 t/s 105.1 t/s 105.1 t/s
4 2 282.4 t/s 564.8 t/s 282.4 t/s
5 1 130.3 t/s 130.3 t/s 130.3 t/s
6 2 196.0 t/s 588.1 t/s 196.0 t/s
7 3 184.6 t/s 553.7 t/s 184.6 t/s

At 5 of the 7 moments, main reads 1.9–3x the reference. A response that starts several forks at once is counted again for each copy.

How

  • mergeCollectedSpeedMetrics folds requests that share a message.id across files into one: the larger input and output counts (a copy carries the same usage), the later timestamp, and every interval. Intervals are merged when the speed is built, as before.
  • Each file still dedups its own rows first, because its interval handling depends on row order.
  • A copy has no user row before it in the fork's file, so it adds no interval. Durations don't change, only the tokens and request count do.
  • Requests without a message.id are kept as they are.
  • Upstream's test aggregates subagents independently with message id shared-id (from fix: count each API response once in speed and token metrics #622) expected three requests when three files share one id. In real transcripts, an id is only shared across files when a fork copies a response. That case now has its own test, built on the fork file's real shape (a fork-context-ref row, the copy, then the fork's prompt). The old test now gives each file's response its own id when ids are on, so it still checks that separate responses in separate files stay separate.

Demo

Three Output Speed widgets (session average, 30 s and 60 s windows) on a fake transcript where every response runs at 50 t/s. An 8 s response spawns a forked subagent, rendered right as the fork starts and again after it answered in 4 s.

Powerline

Before:

Speed with a forked subagent before the fix, Powerline

After:

Speed with a forked subagent after the fix, Powerline

Plain

Before:

Speed with a forked subagent before the fix, plain

After:

Speed with a forked subagent after the fix, plain

Testing

  • New test: a response copied into a forked subagent's transcript, checked at fork launch (30 s window) and after the fork's own response. It fails on main and passes here.
  • On main as of 2.2.34 (0c94df7): it merges cleanly. bun run lint is clean, and bun test on Linux CI gives 4404 pass, 0 fail (4403 on main). The built CLI under Bun 1.4.4 and Node 26.13.0 agrees across runtimes and matches main in plain and Powerline mode, and the TUI opens and exits cleanly under both. It adds no JavaScript or Node API newer than Node 14, the build's target.
  • Vitest under Node 26.10: the transcript-metrics and speed test files pass 68/68 (main: 67/67).
  • The 19 transcripts against the reference: every speed request count and token sum now matches, in the session average and in 1, 5, 30, 60, 119 and 120 s windows. Token totals are identical on main and this branch. The remaining differences are session durations, which this PR doesn't change: main merges a response's rows into one interval, so a parallel tool result logged between them counts as active time (+0.03% to +0.79%).
  • The 13 fork-launch renders: request counts and 30 s Output Speed match the reference at every render.
  • The built dist under Node 26.10 and Bun 1.4.2: the demo transcript reads 50.0 t/s everywhere on this branch, and 100.0 and 83.3 t/s on main, under both runtimes.

A forked subagent's transcript opens with a copy of the response that
spawned it: same message.id, same usage. Speed metrics dedup message ids
within each file only, so the copy counted as a second request. Right as
a fork starts, the windowed speeds read about twice the real rate (more
with several forks).

Merge requests that share a message.id across the main transcript and
its subagent files, keeping the larger token counts, the latest
timestamp and every interval. Each file still dedups its own rows first,
since the interval handling depends on row order.

The subagent aggregation test now gives each file's response its own id
when ids are on; a shared id across files is the copy case, covered by
its own test.
Keeps mergeCollectedSpeedMetrics to the loop: requests without an id
pass straight through, a repeated id folds into the first copy.
# Conflicts:
#	src/utils/jsonl-metrics.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant