Token widgets over-report ~1.84x: one JSONL entry per content block is counted per API call

Summary

The transcript-summing path in src/utils/jsonl-metrics.ts counts several JSONL entries per API call, so tokens-cached, tokens-input, tokens-output and tokens-total over-report. Measured across 148 transcripts / 11,007 real API calls on ccstatusline 2.2.27 + Claude Code 2.1.222–2.1.228:

widget	shown	actual	over-report
`tokens-cached`	6,062,470,358	3,292,144,479	1.84x
`tokens-output`	20,390,026	9,484,311	2.15x
`tokens-input`	2,770,164	1,240,935	2.23x

context-length and context-percentage-* are not affected — they read a single entry rather than summing.

Root cause

The dedup filter at jsonl-metrics.ts:207-212 assumes duplicate entries are streaming partials carrying stop_reason: null:

const entriesToCount = hasStopReasonField
    ? parsedEntries.filter((entry, index) => {
        const stopReason = entry.data.message?.stop_reason;
        return Boolean(stopReason) || (stopReason === null && index === parsedEntries.length - 1);
    })
    : parsedEntries;

That holds for one transcript shape but not the dominant one. Claude Code also writes one entry per content block — thinking, text, and each tool_use — and those entries all share a single message.id and the final non-null stop_reason. Boolean(stopReason) is true for every one of them, so none are filtered.

Grouping messages by shape:

share	shape	filter outcome
57.3%	multi-entry, all entries carry a non-null `stop_reason`	fails — every duplicate kept
0.6%	multi-entry, several non-null	fails
12.3%	multi-entry, exactly one non-null (the streaming-partial case)	works as intended
29.8%	single entry	n/a

So the filter is doing real work for the 12.3% — this isn't a "revert it" fix.

Why prompt-side fields are safe to dedup, and output needs care

Comparing entries that share a message.id, only output_tokens differs, and it is non-decreasing — the last entry carries the complete value. input_tokens, cache_read_input_tokens and cache_creation_input_tokens are byte-identical across an id's entries. Measured: 1,390 ids disagreed on usage, and in all 1,390 the differing field set was exactly {output}, always non-decreasing.

message …tlpxvdiq   (3 JSONL entries, one message.id)
  in=2  out=1     cread=3978  ccreate=12278  stop=None       blocks=['text']
  in=2  out=1     cread=3978  ccreate=12278  stop=None       blocks=['tool_use']
  in=2  out=287   cread=3978  ccreate=12278  stop=tool_use   blocks=['tool_use']

Suggested fix

Dedup by message.id, keeping the entry with the highest output_tokens. This subsumes the streaming-partial case (the partial always has a lower output_tokens than the finalized entry), so it handles both shapes and doesn't depend on stop_reason semantics staying put.

// Claude Code writes one JSONL entry per content block (thinking / text / each
// tool_use). All entries for one API call share a message.id and repeat identical
// prompt-side usage; only output_tokens grows, and is complete on the last one.
// Dedup by id, keeping the highest output_tokens, then restore transcript order
// so the compaction / most-recent-entry logic below is unchanged.
const byId = new Map<string, { data: TranscriptLine; lineIndex: number }>();
const withoutId: typeof parsedEntries = [];
for (const entry of parsedEntries) {
    const id = entry.data.message?.id;
    if (typeof id !== 'string' || id.length === 0) {
        withoutId.push(entry);
        continue;
    }
    const prev = byId.get(id);
    const out = entry.data.message?.usage?.output_tokens ?? 0;
    const prevOut = prev?.data.message?.usage?.output_tokens ?? 0;
    if (!prev || out >= prevOut) {
        byId.set(id, entry);
    }
}
const entriesToCount = [...byId.values(), ...withoutId]
    .sort((a, b) => a.lineIndex - b.lineIndex);

Notes:

message.id was present on all 23,168 usage-bearing entries scanned, but the withoutId passthrough keeps the old behaviour if that ever changes.
Live/in-flight updates are preserved: the current in-flight message still has an id, so it is still counted.
contextLength is unaffected, since it uses only prompt-side fields, which are identical across an id's entries.
hasStopReasonField becomes unused and can be dropped.

Repro

Run against your own transcripts — prints what ccstatusline counts vs. distinct message.id:

import json, glob, os, collections
root = os.path.expanduser("~/.claude/projects")
counted = ids = 0
for f in glob.glob(os.path.join(root, "**", "*.jsonl"), recursive=True):
    entries = []
    for line in open(f, errors="replace"):
        try: d = json.loads(line)
        except Exception: continue
        if (d.get("message") or {}).get("usage"): entries.append(d)
    if not entries: continue
    has_sr = any("stop_reason" in e["message"] for e in entries)
    kept = [e for i, e in enumerate(entries)
            if (e["message"].get("stop_reason")
                or (e["message"].get("stop_reason") is None and i == len(entries) - 1))] if has_sr else entries
    counted += len(kept)
    ids += len({e["message"].get("id") for e in entries})
print(f"counted by ccstatusline : {counted:,}")
print(f"real API calls          : {ids:,}")
print(f"inflation               : {counted/ids:.2f}x")

Observed: counted 20,307 / real 11,007 / 1.84x.

Possibly related

#439 covered a different mismatch in the same equation (statusline context_window fields vs. transcript sums) and is closed. This one is entirely within the transcript-summing path, so the four widgets can be internally consistent and still all be ~1.8x high together.

Happy to send a PR if the approach looks right.

Independent confirmation from a different environment (Claude Code 2.1.231).

One session showed 180 usage-bearing JSONL lines collapsing to 89 unique message.ids — about 2.02x inflation on the cumulative token widgets.

Same root cause: every line for a given id shares that id and a non-null stop_reason.

In this session's case, duplicate lines for a given id also carried byte-identical usage — not just the prompt-side fields, but output_tokens too. That's consistent with Claude Code splitting an already-completed multi-content-block message across lines, rather than writing genuine streaming partials.

The proposed fix ("dedup by message.id, keep max output_tokens") handles this case correctly as well, since all copies tie and any one gets kept.

feat(widgets): add Tokens Last Turn widget for per-message token usage (

fix(jsonl): dedup cumulative token counts by message.id (#8) (#9)

+1 — hit this independently and measured the same ~1.84× over-report (macOS, verified present on upstream/main @ 19c6665). So it's not just you.

Mechanism

A single assistant message with N content blocks produces N JSONL transcript entries that share the same message.id. Each entry carries its own usage object, and within one message output_tokens is a running (cumulative) value across those entries, not a per-block value. The cumulative aggregation path sums every entry's usage, so one multi-block message gets counted once per block — prompt side included.

Repro

Take a real transcript where one assistant reply has several content blocks (text + tool_use is enough — any tool-using turn).
Naive sum over the entries (what the cumulative path does) vs. dedup by message.id: count the prompt side once per id, keep the running max of output_tokens across the message's entries.
On a real 13 MB transcript the naive sum came out ~1.84× above the deduped value — matching the factor in this report.

Fix direction

Dedup consecutive counted entries by message.id as described above (which also matches the grouping discussed earlier in this thread). I've implemented exactly this with red→green tests on the #549 shape — see #725. Happy to adjust the approach if a different one is preferred.

Still present on ccstatusline 2.2.32 (latest on npm) + Claude Code 2.1.302, macOS. I'm adding three things that I don't think are in this thread yet: a check against Claude Code's own counters, data from many versions, and a summary of the open fixes.

1. Grouping rows by `message.id` gives exactly Claude Code's own numbers

Since 2.1.251, Claude Code sends a prompt_cache object in the status line JSON. Claude Code computes it from the API responses. I saved that JSON once during a session, then recomputed the same values from that session's transcript (main chain, rows up to the moment the JSON was saved):

Source	requests	cache-write tokens	cache-read tokens	hit ratio
Claude Code `prompt_cache`	36	129,841	—	95.24%
Transcript, rows grouped by `message.id`	36	129,841	2,596,896	95.24%
Transcript, current ccstatusline logic	76 rows	222,741 (1.72×)	5,699,520 (2.19×)	96.24%

Hit ratio is read ÷ (read + write + uncached input), the same formula Claude Code uses. Counting each message.id once gives the same numbers as Claude Code's own counters, down to the token.

One caveat: in a second session, the transcript had fewer responses than Claude Code counted (11 vs 14 requests). There, grouping by message.id gave fewer cache writes than Claude Code reported (378,681 vs 526,632), so it is only a lower bound for that session. The current logic still reported 1,395,504, which is 2.65× Claude Code's number.

2. All versions are affected

I scanned 13,202 local transcripts from 66 Claude Code versions, 2.1.202 → 2.1.302:

75% of API responses are written as several rows, one per content block.
In 70% of those, more than one row has a non-null stop_reason. The stop_reason filter in jsonl-metrics.ts assumes only the last row has one, so it counts the same response several times.
Cumulative token widgets over-report by 1.43×–3.15× depending on the version (median 2.12×).
In the 41,449 multi-row responses on the main chain, rows with the same message.id always had identical usage and were always next to each other. So keeping the first row, keeping the last row, or only merging consecutive rows all give the same result on this data. I could not reproduce the case from Token widgets overcount usage when transcript contains multiple usage snapshots for the same message.id #664 where rows of one response have different usage.

3. Open fixes (none reviewed yet)

PR	Token totals	Speed widgets	CI
#591	✅	✅	passing
#595	✅	✅	not run
#622	✅	✅	not run
#725	✅	—	passing
#735	—	✅	passing

Because the rows are identical and next to each other, all of these approaches should give correct numbers on real transcripts. The quickest way to close #549, #643 and #664 looks like #591 alone, or #725 + #735 together. All of these have passing CI.

4. Possible follow-up

For the cache widgets, Claude Code ≥ 2.1.251 already provides prompt_cache.requests, cache_write_tokens, hit_ratio, ttl and expires_at. Reading these when they are present would remove the need to parse the transcript for cache data. cache-timer could then use the real ttl and expires_at instead of the ttlSeconds the user configures. Right now it keeps counting down 5 minutes even when the session uses a 1-hour cache. These fields could also serve as expected values in the regression tests.

fix(metrics): count each API response once when its rows share a mess…

fix(jsonl): dedup cumulative token totals by message.id (sirmalloc#725)

Navigation Menu

Repository navigation

Token widgets over-report ~1.84x: one JSONL entry per content block is counted per API call #549

Summary

Root cause

Why prompt-side fields are safe to dedup, and output needs care

Suggested fix

Repro

Possibly related

1. Grouping rows by `message.id` gives exactly Claude Code's own numbers

2. All versions are affected

3. Open fixes (none reviewed yet)

4. Possible follow-up

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Token widgets over-report ~1.84x: one JSONL entry per content block is counted per API call #549

Description

Summary

Root cause

Why prompt-side fields are safe to dedup, and output needs care

Suggested fix

Repro

Possibly related

Activity

bmihaila-bd commented on Aug 31, 2026

axisrow commented on Oct 13, 2026

VSN2015 commented on Oct 17, 2026

1. Grouping rows by message.id gives exactly Claude Code's own numbers

2. All versions are affected

3. Open fixes (none reviewed yet)

4. Possible follow-up

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

1. Grouping rows by `message.id` gives exactly Claude Code's own numbers