avoid re-tokenizing large tool output multiple times - #44
Open
ben564885 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
one large tool output slows the gateway down badly (reported as an outright hang for very large payloads). traced it to
internal/pipes/tool_output/tool_output.go: the tool_output pipe re-tokenizes the same original content with tiktoken up to 4 times per request — once for the min/max threshold check, then again in each result-handling branch (fallback logging, fallback compression stats, and the compression-ratio check) — instead of reusing the count already computed.tiktoken encoding is linear but not free, and it's not cached, so this multiplies real cpu time by the number of redundant passes. for a single large blob this adds up fast, and there's no size cap or timeout on this cpu-bound step, so on a big enough tool output it looks like the gateway hung.
fix: thread the already-computed token count through
compressionTask→compressionResultinstead of recomputing it from the original content string in the result-handling loop.verified with a controlled a/b (same config, same 20MB tool_result, before/after binary):
(~30% cut just from eliminating the one redundant tokenize call that fires on every request; the fallback/passthrough path eliminates 3 redundant calls instead of 1, so the win is larger there.)
no behavior change — output is identical, this only removes duplicate work.