Skip to content

fix(sweep): stop crediting declined turns, see the prefix bust, and name the fallback failure - #211

Open
OsherElhadad wants to merge 6 commits into
rossoctl:mainfrom
OsherElhadad:fix/extract-llm-waste-removal
Open

fix(sweep): stop crediting declined turns, see the prefix bust, and name the fallback failure#211
OsherElhadad wants to merge 6 commits into
rossoctl:mainfrom
OsherElhadad:fix/extract-llm-waste-removal

Conversation

@OsherElhadad

@OsherElhadad OsherElhadad commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

What this changes

Three defects found while measuring extract_llm and extract_llm_sweep over the 16-day
production corpus (220,933 requests, 892 model calls). Two are telemetry, one is economics.
Every figure below carries the query or the file:line behind it; full working in
cg-research/results/extractllm-findings.md and the page at
cg-research/artifacts/extract-llm-deep-dive.html.

Mechanism claims are read at the deployed commit 3a1b438, not at main — prod runs a
side branch that is 2 commits ahead of and 22 behind main. All three defects are live on
main as well, which is why they are fixed here rather than merely reported.

1. ExtractSweep.Offload reported mutated for turns it declined

Offload has two return paths and only the tail one set rep.Skipped. The inventory floor
returns early:

if len(cands) < e.minInventory {        // defaultMinInventory = 10
    rep.GateN("sweep_inventory_below_min", len(cands))
    return keys, nil                   // never reaches the changed == 0 guard
}

Outside the pre-expiry window phase 1 collects no candidates at all, so len(cands) = 0 < 10,
the early return fires, and dash/event.go's Mutated = !Reverted && !Skipped reads true.
GateN is a no-op at n <= 0, so those turns recorded neither a gate nor a skip.

29,321 of 29,372 production rows read mutated=1, acted=0, skipped=0:

select acted, mutated, skipped, count(*) n,
  sum(events like '%sweep_offered%') has_offered,
  sum(gates like '%sweep_inventory_below_min%') has_below_min,
  sum(gates like '%not_in_pre_expiry_window%') has_notwindow
from request_components where component='extract_llm_sweep' group by 1,2,3;

-- acted mutated skipped     n  offered  below_min  not_window
--   0      0       1        8      8        0          0        <- reached the ask; guard worked
--   0      1       0    29321      7        7      29310        <- the early return
--   1      1       0       43     11        0         32

sweep_offered is emitted before the floor check, so it partitions the two exits cleanly: all
8 correctly-skipped rows have it, only 7 of the 29,321 do, and those same 7 are the only ones
with a non-zero sweep_inventory_below_min. 29,310 + 4 + 7 = 29,321 exactly.

No dollar figure is affectedacted derives from saved_gross and is correct. What is
wrong is the mutated census: the component is credited with 29,364 mutations, of which 29,321
are a component that declined. Fixed by guarding the outcome once in a defer, so both exits
and any future one are covered; the duplicated tail check is what went wrong here.

2. sweep_prefix_cache_read_ZERO could not see the harm it exists to catch

The counter tested usage.CacheRead == 0, so a partial hit silenced it however large the
write. The prefix ask's whole justification is that it reads the agent's cached prefix; when
the entry has gone it writes it instead, at 1.25× fresh on the agent's own model rather than
a tenth of it — which is precisely what extract_econ.go cited when it rejected prefix reuse.

Three production asks did exactly that, for 704,925 opus cache_write tokens = $3.42 = 73 %
of this component's entire spend
. Two of the three had read 39,805 tokens of a stable
sub-prefix, so CacheRead == 0 was false and $2.26 of the $3.42 went uncounted:

req      idle before  ask read  ask write     cost  agent's own read / write
151760         1.3 s    39,805    260,604  $1.2685      48,040 / 572,018
175582       299.4 s         0    240,309  $1.1565      42,572 / 805,897   <- only this one fired
172127       276.1 s    39,805    204,012  $0.9920           0 / 687,058

Now counted separately as sweep_prefix_cache_write, because a zero read and a non-zero write
name different failures with different fixes. The existing counter keeps its exact meaning.

Worth recording alongside the fix, since it bears on whether to re-enable the component: on all
three turns the agent's own request also re-created its prefix, so the sweep walked into a
miss rather than causing one. But the trigger guarantees that — sweeping() fires only when
0 < CacheTTLMs − IdleMs <= 60 s, i.e. in the last minute of the entry's life, which is when a
prefix re-read is most likely to fail. No window width fixes that; the permission to act and the
need for a live prefix are mutually exclusive by construction.

3. block_fallback said the wrong thing about which failure it declined on

Widening the condition in change 2 also put the blockFallback branch on the partial-hit path, where
it still wrote "the prefix ask read nothing from cache" — false at CacheRead = 39,805 with a
260,604-token write. Branched, so the two cases stay distinguishable.

One thing worth stating plainly, and a correction to how this section originally described the
string's reach:

  • Correction: the string is NOT unreachable — it is live in production and is now tested. This
    section originally claimed rec.Component stays "" on this path, so Offload discards the row
    on its call.rec.Component != "" guard, and it shipped only a counters test on that premise. That
    premise held for the unit test's own rep := &components.Report{}, but not for production:
    components/pipeline.go's runOne sets Report{Component: comp.Name(), ...} before any component
    runs, and extract_sweep.go assigns r.rec = components.ModelCall{Component: rep.Component, ...}
    before the block_fallback branch is ever reached. So the row survives, reaches rep.Calls, and
    the branched Rejection string is exactly what an operator reads on extraction_calls.rejection
    the column this same PR calls out as already mis-analysed twice. Added
    TestBlockFallbackDecliningAWritePublishesTheCorrectRejection, which sets rep.Component the way
    the pipeline does and asserts the exact published string.
  • A policy change worth stating plainly: block_fallback: true previously declined only on a
    zero read and now also declines on any partial hit that wrote. A write on the agent's own prefix
    bills at 1.25× fresh on the agent's model rather than a tenth of it, so declining is the point of
    the switch — but this is a change to a config-gated brake, not just a counter split.

4. cg.db and cg-control.db sit untracked, and *.db is not ignored

The proxy writes its dashboard and control DBs with default names into whatever directory it is
started from, so a proxy run inside a checkout leaves production-shaped tenant data in the working
tree — untracked, but one git add -A away from a repo with three remotes. Both files were 0 bytes
with no tables
, so nothing leaked this time. Three lines of .gitignore.

5. The floor guard: the measurement, not the literal — and no literal changes

An earlier version of this branch raised housellm's min_tokens from 3,000 to 5,000. Dropped.
The guard that replaces it carries the per-band cost measurement and the per-session replay
multiplier and derives the floor from them, so the next edit has to argue with the data. It asserts
the shipped 3,000 as a lower bound, and it currently passes.

Two corrections are baked in because the first version of the guard had both:

  • It named a constant intervalTop and gave it $14.71, which is the MEDIAN token value, not the
    ceiling. The p90 is $31.25. With the median standing in for the top, the same rule selects 5,000
    instead of 2,000 — it reversed the direction of a config change. That number has now moved three
    times on the same 873 calls: an imported k=12 from another corpus, k=12 again through a mislabelled
    band, and the median through a constant's name.
  • It asserted equality, not a bound. A higher floor only ever declines, and the bands are
    non-monotonic in cost (2–3k at $14.27 is cheaper than 3–5k at $15.76), which puts the ordering
    inside the noise of 25–359 calls per band. Equality would have pinned the literal to a rule fitting
    that noise.

The open-ended top band is skipped explicitly: treating its sentinel as a token count would derive a
floor of that size and silently disable the component.

6. The deterministic-only arm, as a runnable test

The question "would the free character window have done as well as the paid model leg?" had never
been measured, and it is the one arm that answers it without spending anything: with model == nil,
strategyOrder collapses to ["deterministic"] and no call is issued. Skips without CG_H0_DIR, so
it costs CI nothing; the doc comment carries the export query and both internal-validity limits.

One gotcha recorded in the source because it cost real time: the first version of this file was
named ..._arm_test.go, which Go reads as the GOARCH filename build constraint.
It landed in
IgnoredGoFiles on amd64 and go test reported "no tests to run" and exited 0 — a green pass for
a test that never executed. Detect that class with
go list -f '{{.IgnoredGoFiles}}' ./..., never with an exit code.

Tests

Seven, and each guarding changed logic was confirmed to fail without its fix:

  • TestSweepDeclinedAtInventoryFloorReportsSkipped — both exits, table-driven. Reverted, it
    reproduces the production row shape exactly:
    gates: map[not_in_pre_expiry_window:1] with Skipped=false, which is request 141575.
  • TestSweepPrefixWriteTripsTheCounterEvenOnAPartialHit — asserts the write counter fires on a
    39,805-token read with a 260,604-token write, and that the zero-read counter does not, so
    the two are not collapsed.
  • TestBlockFallbackDistinguishesAZeroReadFromAWrite — table-driven over both cases, asserting the
    right counter fires, the wrong one does not, and a declined ask leaves the transcript untouched.
  • TestBlockFallbackDecliningAWritePublishesTheCorrectRejection — asserts rep.Calls itself, with
    rep.Component set the way pipeline.go sets it in production; see the correction to section 3
    above.
  • TestPanicClearsSkippedSetByTheComponentsOwnDefer — a component's own deferred Skipped-guard
    (the pattern this PR's fix feat: Go context-engineering proxy/library (tree-sitter, TOON, Starlark extractor, config, metrics, real integrations) #1 introduced) can run during the same panic unwind that reaches
    runOne's recover, which sets Reverted. Fixed runOne to clear Skipped there, so a
    Report never carries both — closing a half-state fix feat: Go context-engineering proxy/library (tree-sitter, TOON, Starlark extractor, config, metrics, real integrations) #1's own defer made newly reachable.
  • TestHousellmFloorClearsTheBandsThatLoseAtEveryK — passes at the shipped 3,000, and verified to
    fail at 1,000 with the derived bound of 2,000 named in the message.
  • TestDeterministicOnlyArmOverCapturedBodies — a measurement harness rather than a guard; skips
    without its corpus.

Also added sweep_prefix_cache_write to the counters list in
docs/components/extract_llm_sweep.md, which had gone stale for this PR's own change (no code
wiring needed — metrics/metrics.go and proxy/promexport.go already iterate gates generically).

go build ./..., go vet, and go test ./... are green.

Deliberately not in this PR

Five things this investigation measured and is not shipping. Each is in the write-up with its
query.

  • The per-output floor. An earlier draft of this branch raised the housellm literal from 3,000
    to 5,000 and I committed it. Withdrawn and dropped from the branch. The break-even band it
    rested on was mislabelled: solving $4.755 + (k−1)×$0.3814 for the three printed values gives
    k = 3.80 / 12.00 / 27.11, so the middle figure was not this corpus's median but the 12.0
    imported from docs/components/extract_llm.md and measured elsewhere. The measured band is
    $5.82 / $14.71 / $31.25, and at it the change is worth +$1.27 / +$0.13 / −$1.99 — $0.13 at
    the median, on a 7 % overshoot well inside the noise of a 70-session median. The rule the guard
    encoded now selects 2,000, which the shipped 3,000 already clears. The real exposure is six
    tenants overriding min_tokens to 500, which is a deployment fix.
  • Routing to the free deterministic window. A paired experiment over the 468 candidate bodies
    stored in full shows the free window removes 2.56× more tokens for $0.00, and class-gated
    routing would avoid $4.51 of $5.02 and 3,619 s. Declined: the window keeps a median 33.1 %
    of characters and drops 44.2 % of identifiers against the paid program's 30.8 %. And the
    paired corpus excludes the paid leg's only winning size band entirely — max candidate is 6,405
    tokens, so the 8–15k band at $4.05/MTok has zero representation — which biases the comparison
    toward the free arm. The next measurement is reward and steps, not tokens.
  • Widening the sweep's pre-expiry window. Struck, not deferred. Grouping its 19 firing requests
    on whether it removed anything: 7 removal turns wrote 3,363,850 agent cache_write tokens
    against 4 same-idle-band controls' 4,907 — 686× at comparable growth
    , all ten accepted rows at
    ≤0.196 cache warmth and all four no-removal warm rows at ≥0.997, and one request that grew by
    exactly zero tokens and wrote 592,279. extract_llm refuses depth, so the sweep is the only
    component editing inside the already-cached region, and a mid-history edit re-hashes everything
    after it. Revised net −$19.6 to −$27.2, not the ledger's −$3.60 (attributed, not proven; n=7
    vs 4). Its own premise fails too: it fired on a live cache on 13 of 19 requests and on a
    genuinely expired one once.
  • A pre-filter for the 364 calls that bought nothing. Scored on both sides of the ledger, no
    free content predictor pays across the band. looksLikeFileRead — the predicate the codebase
    already has, whose comment calls line-numbered dumps irreducible — is net negative at every
    point
    , because those are the most profitable class the component sees (83 calls, 71.1 %
    acceptance, 169,171 tokens saved).
  • The two Starlark-generation fixes. 96 of 356 dead calls ($1.3356, 20 minutes of model time)
    are programs the interpreter could not run, but they are ~six distinct bugs rather than one. The
    reply-budget theory is refuted — only 5 of 39 cut-off replies reach 4,000 output tokens
    against a 4,096 cap; the median is 859. The two small ones are the while-loop refusal (8 calls)
    and taking the first fenced block rather than only a leading one (5 calls; stripFences at
    internal/extract/extract.go:397 strips only a leading fence). Together $0.22, and every
    prompt edit re-runs the acceptance corpus by design.

Two defects reported earlier in this investigation that turned out not to exist

Recorded because both were mine and both flattered the conclusion:

  • "saved_usd is gross and nothing nets the component's own spend." dash/query.go:876
    computes NetUSD = SavedUSD − LLMCostUSD in the deployed binary, and dash/ui/app.js:2602-2611
    already flags any component whose amortised and first-removal verdicts disagree in sign — which is
    exactly this component's situation, spelled out for the operator. The figure was on the dashboard
    the whole time.
  • "269 acted rows are silently unpriced." All 269 are non-200 requests (401/408/429/502/503) with
    every token column and cost_usd at zero. The upstream never billed them, so Event.Price's
    early return is correct. I priced a "$5.69 correction" by falling through to the fresh-input branch
    on rows with no token data; a reviewer independently produced the same error at $0.90. It is a
    property of the column — saved_gross > 0 on an unbilled request — not of either of us.

Scope note

extract_llm_sweep was removed from all nine tenant configs on 2026-09-03, so changes 1 and 2 have
no effect on current production traffic. They are shipped because the code is still reachable, and
because change 2 is the instrument anyone would need before deciding whether to re-enable it — a
decision the measurement above says should be no.

Net cash impact of this PR: $0. Its value is that a component reporting a decline as a mutation,
and a tripwire that could not see the harm it was built for, are how the sweep ran for three days
unexamined.

Osher-Elhadad added 5 commits September 5, 2026 00:37
… on a write

ExtractSweep.Offload has two return paths and only the tail one set rep.Skipped, so
every turn that declined at the inventory floor -- which is every turn outside the
pre-expiry window, where phase 1 collects no candidates at all -- reported itself as
having mutated the request. dash/event.go computes Mutated as !Reverted && !Skipped,
and GateN is a no-op at n<=0, so those turns recorded neither a gate nor a skip:
29,321 of 29,372 production rows read mutated=1, acted=0, skipped=0. No dollar figure
is affected (acted derives from saved_gross and is correct); the mutated census is.
Guard the outcome once in a defer so both exits and any future one are covered.

Separately, sweep_prefix_cache_read_ZERO tested only for a zero cache read, so a
PARTIAL hit silenced it however large the write. Three production asks re-created the
agent's own prefix for 704,925 opus cache_write tokens, $3.42 and 73% of this
component's entire spend -- and two of the three had read 39,805 tokens of a stable
sub-prefix, leaving $2.26 of that uncounted. A read is not evidence against a write:
count them separately, since they name different failures with different fixes.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
The proxy writes cg.db and cg-control.db with default names into whatever
directory it is started from, so a proxy run inside a checkout leaves
production-shaped tenant data in the working tree -- untracked, but one
git add -A away from being committed to a repo with three remotes.

Found while preparing this branch: cg.db and cg-control.db were both sitting
untracked in the working tree. Both were 0 bytes with no tables, so nothing
leaked this time.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
The question "would the free character window have done as well as the paid model
leg?" had never been measured, and it is the one arm that answers it without
spending anything: with model == nil, strategyOrder collapses to
["deterministic"] and no call is issued. Skips without CG_H0_DIR, so it costs CI
nothing, and the doc comment carries the export query so the corpus can be
rebuilt from any dashboard DB.

Two things the comment records because they decide how the output may be read.
before_gz is capped at defaultContentCap = 16<<10 BYTES, so 46.4 % of production
blobs are truncated and must be excluded. And the untruncated population tops out
at 6,405 candidate tokens, so the 8-15k band -- the only size band whose measured
cost per MTok saved clears break-even -- has no representation at all, and 62 % of
the corpus sits below 3k where the paid leg is already underwater. The comparison
is therefore biased toward the free leg and cannot speak to the paid leg's one
winning case.

Filename note in the source too, because it cost real time: a file ending
_arm_test.go is read by Go as the GOARCH filename build constraint, lands in
IgnoredGoFiles everywhere else, and 'go test' then reports "no tests to run" and
PASSES. The first version of this file was silently never compiled.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Widening the prefix-counter condition to admit a partial hit that WROTE also put
the block_fallback branch on that path, where it still said "the prefix ask read
nothing from cache" -- false when CacheRead is 39,805 and the write is 260,604.
Branch the reason so the two cases stay distinguishable.

Two things the new test records rather than fixes, because both are pre-existing
and neither is covered by the evidence behind this branch. The Rejection string is
unreachable today: on this path the fallback deliberately never runs, so
foldFallback returns early, rec.Component stays "" and Offload discards the whole
row on its `call.rec.Component != ""` guard -- the same missing-row class the
comment at foldFallback documents for the no-asker path. Adding a row where none
existed is a behaviour change, so the test asserts the counters instead, which are
what a reader actually gets.

And state the behaviour change plainly: block_fallback: true previously declined
only on a zero read and now also declines on any partial hit that wrote. A write
on the agent own prefix bills at 1.25x fresh on the agent model rather than a
tenth of it, so declining is the point of the switch -- but it is a policy change
to a config-gated brake, not just a counter split.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Carries the per-band cost measurement (873 production calls) and the per-session
replay multiplier, and derives the floor from them, so the next edit has to argue
with the data. No config literal changes: the shipped 3000 already clears the
derived bound of 2000.

Two corrections are baked into the guard because the first version of it had both.
It named a constant `intervalTop` and gave it $14.71, which is the MEDIAN token
value, not the ceiling; the p90 is $31.25, and with the median standing in for the
top the same rule selects 5000 instead of 2000 -- it reversed the direction of a
config change. That number has now moved three times on the same 873 calls: an
imported k=12 from another corpus, k=12 again through a mislabelled band, and the
median through a constant NAME.

And it asserts a LOWER BOUND rather than an equality. A higher floor only ever
declines so it cannot hurt the agent, and the bands are non-monotonic in cost
(2-3k at $14.27 is cheaper than 3-5k at $15.76), which puts the ordering inside
the noise of 25-359 calls per band. Equality would pin the literal to a rule
fitting that noise.

The open-ended top band is skipped explicitly: treating its sentinel as a token
count would derive a floor of that size and silently disable the component.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
@OsherElhadad

Copy link
Copy Markdown
Collaborator Author

Review: revert-verified all three fixes, plus two findings the tests do not cover

Reviewed at f633429a (base is current main, 0 commits behind) in a dedicated worktree. Verdict: approve with nits. All three defects are real, correctly fixed, and each regression test fails — naming its own subject — when the original defect is reintroduced in its exact original form. Two things are worth acting on: a new panic half-state, and an "unreachable" rationale in a commit message that is factually wrong for the case it is given for.

Data boundary: every production figure in the description (the 220,933-request / 16-day corpus, the 29,321/29,372 row split, the $3.42 / 704,925-token prefix write, the band economics) is not verifiable within the data boundary — no reviewer here read the service database or any export of it. Those numbers are treated as given; what was checked is the mechanism behind each, against the code.

Real run results

Command Result
gofmt -l . / go vet ./... clean
go build ./... clean, 1.9s
go test -count=1 ./... all 29 packages ok, 1m23s (dash 76.5s, proxy 41.4s)
CGO_ENABLED=1 go test -race ./components/offload/... ./config/... ./internal/extract/... ./dash/... ./proxy/... all 5 ok, 0 races, 7m57s — 1,723 === RUN, 0 FAIL, 19 SKIP

Per-package counts that actually RAN (non-race): components/offload 373, config 267, internal/extract 101.

The filename trap does not recur. go list -f '{{.TestGoFiles}} {{.IgnoredGoFiles}}':

  • internal/extractzz_freeleg_test.go is in TestGoFiles, IgnoredGoFiles=[]. Compiled, as its name intends.
  • components/offloadIgnoredGoFiles=[skeleton.go skeleton_measure_test.go skeleton_safety_test.go], pre-existing behind cg_skeleton, untouched here.
  • configIgnoredGoFiles=[].

Of the 19 SKIPs, one is new: TestDeterministicOnlyArmOverCapturedBodies needs CG_H0_DIR pointing at a corpus exported from extraction_calls.before_gz. That is production data and not synthesisable inside the boundary, so it is reported as unverified, not as a pass — it skipped in every run. Worth knowing that this test can only ever be exercised by someone with corpus access.

Revert-verification — each defect restored to its original form

Defect Mutation Observed failure
1 · Skipped unset on the early return removed the defer guard and restored the pre-PR shape exactly: bare return keys, nil at the inventory floor plus the old inline if changed==0 {rep.Skipped=true} back before the tail return TestSweepDeclinedAtInventoryFloorReportsSkippedboth subtests FAIL: a component that declined reports Skipped=false, so dash reads mutated=1 for a turn that changed nothing (gates: map[below_output_floor:1 sweep_inventory_below_min:1])
2 · counter blind to partial-hit writes restored if !fellBack && usage.CacheRead == 0 { as the sole condition TestSweepPrefixWriteTripsTheCounterEvenOnAPartialHit FAILS: an ask that re-created the agent's prefix for 260604 tokens recorded nothing (gates: map[below_output_floor:1 sweep_kept:1])
3 · block_fallback said "read nothing" on a write same widened-condition revert (it is what routes the write case into block_fallback at all) TestBlockFallbackDistinguishesAZeroReadFromAWrite FAILS on partial_hit_that_wrote: block_fallback did not decline (gates: map[below_output_floor:1]); the zero_read subtest correctly still passes
bonus · the housellm floor guard set extract_llm.min_tokens to 1000, below the derived floor TestHousellmFloorClearsTheBandsThatLoseAtEveryK FAILS: housellm ships min_tokens: 1000, below the measured floor of 2000. Bands under 2000 cost $32.41-$79.75 per MTok saved against $31.25 on even a top-decile session

That last one answers the question worth asking of any test that pins a measured number: the floor is genuinely derived, by looping the bands table against p90TokenValue, not copied from a log. The failure message names the computed number. A log goes stale like a build; this does not.

Mechanism spot-checks behind the three: GateN/EventN (components/component.go:646-654,690-698) are confirmed no-ops at n<=0 — the exact mechanism blamed for the silent miscount; dash/event.go:525 is row.Mutated = !r.Reverted && !r.Skipped as stated; and the fixtures use the exact production values (CacheRead=39805 / CacheWrite=260604 for request 151760), not approximations.

Findings

Minor · components/offload/extract_sweep.go:281-285 — the deferred guard also fires during panic unwind, creating a Reverted && Skipped half-state that did not exist before.

Go runs deferred functions during a panic. If Offload panics before changed is incremented — the common case, since changed++ happens deep in phases 1/3 — the defer sets rep.Skipped=true, and then the panic reaches pipeline.runOne's recover() (components/pipeline.go:101-109), which sets rep.Reverted=true on the same Report without clearing Skipped. Pre-PR this was essentially unreachable: the old inline check sat immediately before the clean tail return, with nothing after it that could panic.

Traced every consumer: dash/event.go's Mutated is unaffected (false either way), and apply/apply.go:88-98's logDecisions switch tests Reverted before Skipped, so the human-readable verdict stays right. But the raw skipped column (dash/store.go:416, dash/query.go) will now read true on some panic rows where it never could before — a future "count skipped rows" query that does not also exclude Reverted would misread it. Not a blocker, and no current derived metric is corrupted. Flagging it because it is precisely the ambiguous-half-state class this PR is otherwise hunting. Cheapest fix: have the defer skip when rep.Reverted is set, or clear Skipped in runOne's recover.

Minor · commit 549714c's rationale for defect 3 is wrong, and the consequence is an untested published field.

The commit message says the new Rejection string is "unreachable today" because foldFallback returns early and rec.Component stays "", and the shipped test therefore asserts only gates. That is false for the case it is given for, and I verified the mechanism myself rather than relaying it:

  • r.rec = components.ModelCall{…} is assigned at extract_sweep.go:812.
  • The !fellBack && (usage.CacheRead == 0 || usage.CacheWrite > 0) branch that writes the new string is at :876-894after 812.
  • rep.Component is always non-empty in production: components/pipeline.go:81 builds Report{Component: comp.Name(), …}.
  • And :763's if r.rec.Component == "" is not a discard — it fills Component in, precisely so the caller's call.rec.Component != "" guard cannot drop the row. Its own comment says so.

So the row is built, retained, and carries the new string. An independent probe test (written, run, deleted) confirmed it: with rep.Component set as production sets it, len(rep.Calls) == 1 and Calls[0].Rejection is exactly the new "re-created the agent's prefix instead of reading it" text. The claim is correct for the genuinely-early paths (no-asker, ask-failed — those give len(rep.Calls) == 0), and the commit conflates those with this reachable one.

Net: the string is live in production and has zero test coverage of its content, because the coverage was skipped on a premise that does not hold. Given this PR's own words that this column "has already been mis-analysed twice, so a wrong reason here becomes a wrong finding later," that is the one gap I would close — a rep.Calls-asserting test, and a corrected rationale so the wrong reason does not stand in history.

Minor · docs · docs/components/extract_llm_sweep.md (~264-276) is now stale. Its "## Counters" section is an explicitly exhaustive, operator-facing enumeration of the gates this component raises. It names sweep_prefix_cache_read_ZERO and not the new sweep_prefix_cache_write. This diff touches no docs. To be clear about what does not need changing: the counter needs no code wiring — metrics/metrics.go and proxy/promexport.go both iterate range cs.Gates/r.Gates generically, so /stats and cg_component_gate_declines_total already carry it — and docs/reference/routes.md correctly needs no row, since this is a per-component gate rather than a snapshot field. It is one line in the component page.

Nit · the "both exits" coverage claim does not hold. Both subtests of TestSweepDeclinedAtInventoryFloorReportsSkipped reach the same early return (:500) under two different gate reasons. Neither exercises the other exit (:568, the tail return after adjudicate kept everything) — which is the path whose duplicated inline check this PR deleted. Nothing in the suite regression-tests that the tail exit still sets Skipped: TestSweepCountsKeepEverythingSeparatelyFromAFailure walks that exact shape but never asserts rep.Skipped. Low risk, since one defer now covers both by construction — but the coverage claim is stronger than the fixture.

Nit · the headline production shape is untested. Request 175582 had CacheRead=0 and CacheWrite=240309 — the one row that fired under the old counter. Probed directly (temporary test, reverted): both sweep_prefix_cache_read_ZERO and sweep_prefix_cache_write fire together, which reads as the intended two-distinct-facts design rather than a bug. Untested, though, and it is the shape the description leads with.

Checked and clean

  • Security: nothing new logs content, keys, paths, session or tenant ids. The new gate name and the branched Rejection strings are static literals, never built from request content — no cardinality or log-injection surface. No unbounded growth.
  • .gitignore: git ls-files | grep -E '\.db(-wal|-shm)?$' is empty and no .db* file appears anywhere in history, so the new rule cannot mask a needed file and there is no ignored-but-tracked half-state.
  • config/config_more_test.go guards a real, derived lower bound against the shipped housellm preset's extract_llm.min_tokens: 3000 — not a tautology (see the bonus mutation above).
  • Consumers of the corrected mutated semantics — all get better numbers with no changes of their own: dash/event.go:525 (the derivation), dash/store.go:416, dash/query.go:233,249,636,752,767,777 (note ActedStructural = Mutated - Acted at :777 inherits the correction directly), metrics/metrics.go:234,339,438,517-518,920-922, proxy/promexport.go:553-577 (cg_component_runs_total{outcome="mutated"}), dash/ui/app.js:2339,2736,2845,2848,3393,3753 (verdict badges and the "ran" filter). deploy/grafana/dashboards/context-guru.json also references it, but that file is not what production mounts, so no prod effect.

Cross-PR note (#212, #211, #196)

Unlike the previous trio, these three do not collide: merged onto main in order with zero conflicts, and on the merged tree gofmt/go vet/go build are clean with kvcache, components/offload, config, internal/extract and deploy/harbor all passing. #211 is the only one of the three already based on current main (#212 is 70 commits behind, #196 is 20).

…c half-state

549714c's rationale for skipping a test on the new Rejection string was wrong:
it claimed rec.Component stays "" on this decline path, so Offload's
`call.rec.Component != ""` guard drops the row. That premise only holds for
the existing unit tests' own `rep := &components.Report{}`. In production,
pipeline.go's runOne sets rep.Component = comp.Name() before any component
runs, and extract_sweep.go assigns r.rec.Component from rep.Component before
this decline branch is reached -- so the row survives, lands in rep.Calls,
and the branched Rejection string is exactly what an operator reads on
extraction_calls.rejection. Add a test that sets rep.Component the way the
pipeline does and asserts the published string, and correct the stale
"row is discarded" claim left on the neighboring test's doc comment.

Also close a half-state the deferred Skipped-guard (introduced earlier in
this branch) made reachable: that defer can run during the same panic
unwind that reaches runOne's recover, which sets Reverted without clearing
Skipped, so a Report could end up with both true -- which dash/event.go's
Mutated derivation and any future "skipped rows" query would misread.
Clear Skipped alongside Reverted in the one place all components' panics
route through.

And add sweep_prefix_cache_write to the counters list in
docs/components/extract_llm_sweep.md, which 549714c's own change had left
stale.

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
@OsherElhadad

Copy link
Copy Markdown
Collaborator Author

Addressed the review at pull/211#issuecomment-5558475749. Pushed 798d40e to fix/extract-llm-waste-removal.

Verified the finding first

Re-derived the mechanism at this HEAD before changing anything: components/pipeline.go:81 sets
rep.Component = comp.Name() before any component runs; extract_sweep.go's
r.rec = components.ModelCall{Component: rep.Component, ...} (around line 812) fills rec.Component
before the block_fallback decline branch (~876-894) is ever reached; and the "IDENTITY FIRST" block
in foldFallback fills Component in when empty, it does not discard the row. 549714c's rationale
("rec.Component stays "", so the row is discarded") only holds for the existing tests' own
rep := &components.Report{} construction — not for production. Confirmed the review's read was
correct.

What changed

  • components/offload/extract_sweep_test.go: added
    TestBlockFallbackDecliningAWritePublishesTheCorrectRejection. It sets rep.Component the way
    pipeline.go actually sets it, then asserts rep.Calls has exactly one row with a non-empty
    Component and the exact Rejection text. Also rewrote the stale doc-comment on the neighboring
    TestBlockFallbackDistinguishesAZeroReadFromAWrite that repeated the wrong "row is DROPPED" claim.
  • components/pipeline.go / components/pipeline_test.go: fixed the other thing the review
    called "worth acting on" — the panic half-state. A component's own deferred Skipped-guard (the
    pattern this branch's fix feat: Go context-engineering proxy/library (tree-sitter, TOON, Starlark extractor, config, metrics, real integrations) #1 introduced) can run during the same panic unwind that reaches
    runOne's recover, which sets Reverted without clearing Skipped. Added rep.Skipped = false
    alongside rep.Reverted = true in runOne's recover — the one place every component's panic routes
    through — and a regression test, TestPanicClearsSkippedSetByTheComponentsOwnDefer.
  • docs/components/extract_llm_sweep.md: added sweep_prefix_cache_write to the Counters list
    (the "Minor · docs" nit — one line, no code wiring needed, metrics/metrics.go and
    proxy/promexport.go already iterate gates generically).
  • PR body: corrected section 3's "unreachable today" claim in place, and updated the Tests section.

Revert-verification (source mutated, never the test)

New Rejection test — reverted the branched reason to the pre-PR flat string in
extract_sweep.go (removed the usage.CacheWrite > 0 branch entirely, restoring exactly what
549714c fixed):

go test ./components/offload/... -run TestBlockFallbackDecliningAWritePublishesTheCorrectRejection -v

FAILED, naming its own subject:

extract_sweep_test.go:1066: Rejection = "the prefix ask read nothing from cache and block_fallback is
set; declining rather than paying again for a full-price transcript read", want "the prefix ask
re-created the agent's prefix instead of reading it and block_fallback is set; declining rather than
paying again for a full-price transcript read"

Restored the file, rebuilt, reran: PASS.

Panic half-state test — removed the new rep.Skipped = false line from pipeline.go's recover:

go test ./components/ -run TestPanicClearsSkippedSetByTheComponentsOwnDefer -v

FAILED, naming its own subject:

pipeline_test.go:139: Reverted && Skipped both true: {Component:boomafterskipped Kind:reformat
TokensBefore:3 TokensAfter:3 ... Skipped:true Reverted:true ... Err:panic: kaboom ...}

Restored the file, rebuilt, reran: PASS.

Ready-checklist

  • gofmt -l . — clean.
  • go vet ./... — clean.
  • go build ./... — clean.
  • go test -count=1 ./...all 29 packages ok. Verbose run: 2,548 === RUN, 0 FAIL, 28 SKIP.
    components/offload specifically: 374 === RUN (was 373 in the review's own non-race count,
    +1 for the new test). components: 33 === RUN (+1 top-level for the new panic-half-state
    test, plus its assertions).
  • CGO_ENABLED=1 go test -race -count=1 ./components/offload/...ok, 0 races, 106.2s.
  • git merge-tree --write-tree HEAD origin/main — resolves cleanly to a tree (no conflict markers);
    still merges onto main with no conflicts.
  • Pushed as a normal fast-forward: f633429..798d40e fix/extract-llm-waste-removal on
    osher-fork (no rebase, no force).

Decided NOT to change, and why

The review's two remaining "Nit" items (as opposed to the "Minor" items above) are left as-is:

  • The "both exits" coverage claim (TestSweepDeclinedAtInventoryFloorReportsSkipped only
    exercises the early-return exit, not the tail-return exit TestSweepCountsKeepEverythingSeparatelyFromAFailure
    walks). The reviewer calls this "low risk, since one defer now covers both by construction" and
    flags it as a documentation-strength mismatch, not a defect — no behavior to fix, only a stronger
    test to write. Left for a follow-up rather than expanding this PR's test surface further.
  • The headline production shape untested (CacheRead=0 and CacheWrite=240309 together, request
    175582). The reviewer already probed this directly (temporary test, reverted) and confirmed both
    gates fire together as intended — a design confirmation, not a bug. Same call: worth a permanent
    test eventually, not a blocker here.

Both are testing-coverage suggestions the review itself marks as lower severity than the two "worth
acting on" items (the wrong rationale and the panic half-state), which this push addresses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New/ToDo

Development

Successfully merging this pull request may close these issues.

2 participants