Skip to content

avoid re-tokenizing large tool output multiple times - #44

Open
ben564885 wants to merge 1 commit into
Compresr-ai:mainfrom
ben564885:fix-redundant-tokenize-large-output
Open

avoid re-tokenizing large tool output multiple times#44
ben564885 wants to merge 1 commit into
Compresr-ai:mainfrom
ben564885:fix-redundant-tokenize-large-output

Conversation

@ben564885

Copy link
Copy Markdown

one large tool output slows the gateway down badly (reported as an outright hang for very large payloads). traced it to internal/pipes/tool_output/tool_output.go: the tool_output pipe re-tokenizes the same original content with tiktoken up to 4 times per request — once for the min/max threshold check, then again in each result-handling branch (fallback logging, fallback compression stats, and the compression-ratio check) — instead of reusing the count already computed.

tiktoken encoding is linear but not free, and it's not cached, so this multiplies real cpu time by the number of redundant passes. for a single large blob this adds up fast, and there's no size cap or timeout on this cpu-bound step, so on a big enough tool output it looks like the gateway hung.

fix: thread the already-computed token count through compressionTaskcompressionResult instead of recomputing it from the original content string in the result-handling loop.

verified with a controlled a/b (same config, same 20MB tool_result, before/after binary):

  • before: 11.3s
  • after: 7.9s

(~30% cut just from eliminating the one redundant tokenize call that fires on every request; the fallback/passthrough path eliminates 3 redundant calls instead of 1, so the win is larger there.)

no behavior change — output is identical, this only removes duplicate work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants