Skip to content
This repository was archived by the owner on Oct 28, 2026. It is now read-only.

perf(local-agent): deduplicate assembly token counts - #62

Closed
To3akaRin wants to merge 1 commit into
langbot-app:mainfrom
To3akaRin:codex/localagent-assembly-token-counts-2716
Closed

To3akaRin wants to merge 1 commit into
langbot-app:mainfrom
To3akaRin:codex/localagent-assembly-token-counts-2716

Conversation

@To3akaRin

Copy link
Copy Markdown

LocalAgent currently performs repeated Host token-count RPCs while fitting and reporting the same context. A simple under-budget assembly makes eight calls for only four distinct inputs. This change reuses successful counts within one compact_async() invocation, reducing that path to four calls while preserving the complete assembly result.

Refs langbot-app/LangBot#2716. This addresses the assembly-local duplicate-counting portion of that issue.

Implementation

  • Create an independent compactor worker and bounded counter wrapper for each assembly, including concurrent calls on the same original compactor.
  • Build immutable keys from the complete SDK Message.model_dump() values. Preserve types, order, nested fields and signed zero without repeatedly JSON-encoding large Base64 strings. Unsupported values fall back to the original counter.
  • Keep at most 128 successful results per assembly. Empty input retains the Host's actual count, including nonzero overhead. Usage-anchor estimates remain separate from real full-context counts.
  • Release cached snapshots with the assembly. Snapshots retain immutable string references during that invocation; they do not retain mutable messages or containers.

Validation

  • uv sync --locked --extra dev --python 3.11
  • uv run --no-sync pytest -q: 327 passed, with existing SDK Pydantic deprecation warnings.
  • 25 new regressions cover deduplication, independent/concurrent assemblies, zero/nonzero empty-input counts, usage anchors, compaction, multimodal content, tool arguments, mutations, failure/cancellation, and SDK value-encoding edge cases.
  • ruff check, ruff format --check, git diff --check, and lbp build passed. The built package's production-source SHA-256 matches the benchmarked source.
  • Independent review found no remaining blocking issues.

Reproducible comparison

The committed harness loads the original source from immutable baseline 7b532cfd6222aa79f05f113b4db6c876d4782434 and compares every message, frame, compaction result and diagnostic with the current implementation. All 24 configurations passed exact output comparisons (four scenarios, 0/2/10 ms simulated RPC waits, concurrency 1/8, 20 measured batches after two warmups).

Scenario Host calls per assembly Median assembly time, 10 ms simulated RPC, concurrency 1
Plain text 8 → 4 88.577 → 44.655 ms
History and RAG 8 → 6 87.778 → 66.755 ms
Compaction 42 → 30 471.181 → 339.684 ms
256 KiB synthetic image 8 → 4 90.547 → 45.991 ms

The full results, including zero-wait overhead and P95, are in the benchmark guide. Some zero-wait paths are slower due to snapshot work; the history/RAG 2 ms, concurrency-8 P95 also increased in this run (21.018 → 44.239 ms). This synthetic benchmark does not establish production throughput or universal latency improvement, and does not include real Core/Runtime RPC, models, database, Box or IM delivery.

No dependency, environment variable, configuration, Host authorization, cancellation polling, or SDK protocol changes are included.

@RockChinQ

Copy link
Copy Markdown
Member

Hi @To3akaRin , langbot-plugins have been migrated to https://github.com/langbot-app/LangBot/tree/master/plugins , please file a new PR there~

@RockChinQ RockChinQ closed this Oct 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants