v3 uses ~3× more tokens than v2 for the same input

[jehna]

Background

In #724, @huyz reported after first trying v3:

I'm trying v3.
Just an initial comment: warning, it eats up a lot more tokens than the JS version

To which @0xdevalias replied:

It would be interesting to side by side the old JS implementation with the new Rust one to see what's being sent as context / etc that would explain the extra token usage.

This issue is the side-by-side. To keep it deterministic and independent of model output, both v2 and v3 were instrumented with a dry-run path that walks every identifier the real pipeline would walk, builds the exact (system, user, schema) triple that would be sent, tokenises it with cl100k_base, and reports totals — no LLM calls, no API keys.

Reproduction

• v3: new humanify dry-run <file> subcommand on branch claude/vigilant-noether-awdmR (uses tiktoken-rs).
• v2: standalone dry-run.ts script in a worktree at commit 5b30583^ (last v2-shaped commit before the Rust merge), using tiktoken from npm.

Both report calls, system/user/schema/total tokens, and per-call averages.

Numbers

Fixture 1: fixtures/example.min.js (140 bytes, no shadowing)

metric	v2 (ctx=500)	v3 (ctx=500)	v3 / v2
calls	6	6	1.00×
system tokens	114	240	2.11×
user tokens	486	822	1.69×
schema tokens	342	234	0.68×
total tokens	942	1296	1.38×
avg total/call	157	216	1.38×

Fixture 2: 4 minified functions with shadowed e, t, n, r, i, ... (396 bytes)

metric	v2 (ctx=500)	v3 (ctx=500)	v3 / v2
calls	11	24	2.18×
system tokens	209	960	4.59×
user tokens	1013	3799	3.75×
schema tokens	627	936	1.49×
total tokens	1849	5695	3.08×
avg total/call	168	237	1.41×

Where the extra tokens go

Two independent multipliers stack:

1. Walker iterates per binding, not per name → ~2.2× more calls

• v2 src/plugins/local-llm-rename/visit-all-identifiers.ts: visited is a Set<string> keyed on the identifier name. Once e has been renamed once, every other e binding in any other scope is silently skipped.
• v3 src/rename/walker.rs: visited is a HashSet<SymbolId> keyed on the binding. Each shadowed e, t, n, r, ... is a distinct symbol and gets its own LLM call.

v3's behaviour is arguably more correct (different bindings can mean different things), but on real minified output — which leans hard on a, b, c, d, e, t, n, r, i, o, s, u reused in every closure — it multiplies the call count substantially.

2. Larger per-call prompt template → ~1.4× more tokens per call

v2 (src/plugins/openai/openai-rename.ts):

system: Rename Javascript variables/function `${name}` to have descriptive name based on their usage in the code."
user:   <surroundingCode>

v3 (src/llm/renamer.rs):

system: You are a senior software engineer reviewing minified or obfuscated JavaScript.
        Your job is to assign a single descriptive identifier name based on how the
        variable is used in the surrounding code. Return JSON only.
user:   Surrounding code:
        ```javascript
        <surroundingCode>
        ```

        The identifier currently named `<name>` appears in this code. Suggest a single
        descriptive replacement name. Rules:
        - camelCase for variables and functions, PascalCase for classes/constructors
        - ASCII letters, digits, underscores only; first character must be a letter or underscore
        - Avoid JavaScript reserved words
        - If the current name is already meaningful, return it unchanged

The fenced code block, the "Rules:" enumeration, and the longer system role each add fixed per-call overhead — ~70–90 extra tokens before the surrounding code is even included.

Note: the v3 default --context-size is 500 chars vs. v2's 1000, so v3 already sends half as much surrounding code, and the overhead above is on top of that.

Possible next steps

• Dedupe walker work by name where it's safe (e.g. cache rename decision for identical (name, surrounding_code) pairs).
• Slim the per-call prompt: most of the Rules: block can be enforced post-hoc by the existing safe-name pipeline (src/rename/safe_name.rs) which already handles case, reserved words, and identifier-character normalisation — the model doesn't need to be told.
• Make --context-size consistent with v2's old default, or document the change.

I'm happy to take a swing at the prompt slim-down once we agree on direction.

[0xdevalias]

Looking at the prompt; would it help with token caching if the code was moved to be the final part of the prompt; and the formatting rules were moved either above it in the user prompt; or possibly even into the system prompt?

[jehna]

Thanks for the suggestion — this is the correct general structure for prefix caching: static instructions first, variable code last. Moving the rules above the code would therefore be reasonable cleanup, but it would not materially affect Humanify’s current token usage.

The dry-run measurements show an average complete input of only 216–237 tokens per call, including the system prompt, user prompt, and schema. That is well below the providers’ cache thresholds:

Provider	Minimum cacheable prefix	Current Humanify behavior
OpenAI	1,024 tokens	Automatic caching, but these prompts are too short
Gemini	1,024–4,096, model-dependent	Implicit caching, but again too short
Anthropic Sonnet 4.6	1,024 tokens	Also requires cache_control, which we do not currently send
Ollama	No billing threshold	May reuse a longer local KV prefix and slightly reduce prefill latency
OpenRouter	Backend-dependent	Unlikely to help with prompts this short

Reordering also does not itself remove any tokens: the same instructions are still transmitted on every call.

So while I would be happy to move the static rules before the code as good hygiene and for possible Ollama/future benefits, it does not explain or solve the ~3× increase. The meaningful wins remain reducing the ~2.2× call multiplier and slimming instructions already enforced by safe_name.rs.