feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting (#982)

[maxmilian]

The ranking half of #982, kept deliberately separate from the corpus-frequency discount in #1462 — the issue asks for the two to stay distinct because they have different semantics, and measuring confirmed they cover different shapes.

Why a second lever

matchesNonProductionDir hardcodes example/sample/fixture/benchmark/demo, so a peripheral tree only the project knows about — optional-skills/, scripts/, a generator's output dir — gets no de-prioritization at all. When helpers there carry generic symbol names (usage, status, run), an exact name match hands them a large bonus and they crowd out the product code that answers the query.

{
  "deprioritize": ["optional-skills/", "scripts/"]
}

Gitignore-style patterns, matched against project-root-relative paths — same contract as exclude, include, includeIgnored, and validated by the same warn-and-skip rules (a non-array value or a blank/non-string entry is dropped with a warning, never thrown).

This is a relevance lever, not a recall lever. exclude takes content out of the index; deprioritize leaves it fully indexed and findable and only stops it from winning. There's a test pinning exactly that — after de-prioritizing, getNodesByName('usage') still returns both helpers and a direct search for usage still surfaces them.

The part I got wrong first, and what the measurement showed

My first cut only added the patterns to the existing −15 path penalty, matching the built-in non-production dirs. It did not fix the issue's repro, and the numbers say why:

	usage() helper	top product symbol
baseline	74.8	51.2
−15 path penalty only	59.8	51.2

The path penalty is additive, and the signal it has to counter — the exact-name bonus — is additive and larger. No magnitude of a hand-tuned additive penalty fixes that shape robustly; it just moves the threshold.

So the lever now also targets the mechanism directly: a path the project de-prioritized is saying its symbol names are not the answer, so the exact-name bonus is damped there as well (0.75× — derived below; it started as a guessed 0.25×). Damped rather than zeroed, so the tree still ranks when it genuinely is what you asked for — the same "discount, don't erase" rule as the path penalty.

Worth flagging as a finding regardless of this PR: the built-in example//fixture/ de-prioritization has the same weakness. A usage() inside a hardcoded examples/ dir would out-rank product code today for exactly the same reason. This PR fixes it for user-declared paths; say the word and I'll extend the damping to the built-in set in a follow-up (I left it out to keep the diff to one behaviour).

Relationship to #1462

Complementary, not overlapping — this is the point of keeping them distinct:

	keys on	fixes #982's 8-file repro?
#1462 corpus-frequency discount	the name being common across the corpus	No — only 2 symbols are named usage there, so the name is rare and IDF is ~0.8, near-inert by design
this PR	the path being one the project declared peripheral	Yes — that repro is the fixture here

Neither subsumes the other. They're on independent branches off main and touch scorePathRelevance's signature in the same region, so whichever lands second needs a trivial rebase — happy to do that in either order.

Two defects a review pass caught, now fixed

Both were real, both are pinned by new tests.

1. The setting needed a process restart to take effect. The matcher was built once in wireLayers(), which runs only from the constructor and reopenIfReplaced(). src/mcp/tools.ts keeps one CodeGraph per project root alive for the whole server lifetime — so a user editing codegraph.json, re-indexing, and seeing nothing change would reasonably conclude the feature was broken. exclude and include don't behave that way. The predicate now reads loadDeprioritizePatterns() per call (mtime-cached — one stat) and memoizes the compiled matcher on the pattern array's identity. There's a test that writes the config after opening the project; I confirmed it fails against the old code.

2. explore only half-honoured the setting — and explore is the surface this issue reports on. Both scorePathRelevance call sites in src/context/index.ts passed two arguments, so the lever never reached them; explore improved only incidentally, via the two sub-searches that route through searchNodes. Rows B, C and D of #982's own reproduction table are all codegraph explore, so shipping it search-only would have missed the reported case. Both sites now pass it.

What I deliberately did not join: explore's hard early-continue filters and its non-production budget cap. Those remove content from the result set, and deprioritize is a ranking lever by definition — exclude is the lever for putting something out of reach. Mixing the two would collapse the distinction the issue asks us to preserve.

The README claim was also overstated — it said this "extends" the built-in example//fixture/ list, when the built-ins get five mechanisms and this gets one. Narrowed to state exactly what it does.

Smaller items from the same pass: scorePathRelevance now takes a boolean rather than a predicate (the caller had already evaluated it, and it was being invoked twice per result); the predicate body is exception-guarded so a malformed path can never take a search down with it; the misplaced module const moved out from between two imports; and two test assertions that could pass vacuously on an empty result set were tightened.

Tests

__tests__/deprioritize-config.test.ts, 16 tests:

• parsing — default empty with no config; patterns kept verbatim and trimmed; non-array warns-and-skips; blank/non-string entries dropped while the rest survive; other config keys undisturbed
• ranking — a control asserting that without the config the helpers still take the top two ranks (the reported status quo), then that with the config product code outranks them
• recall preserved — the helpers stay indexed and directly findable
• no collateral — packages/core/util/strings.ts, which no pattern names, scores identically with and without the config
• config written after open — fails on the eager implementation, verified
• explore — the matcher is reachable from the context builder and classifies both trees correctly
• a query that genuinely targets the tree — bodyfat calc usage still ranks the de-prioritized helper first with product code competing, which is explore/query relevance: a generic token's exact name-match overboosts in peripheral dirs — follow-up to #746 #982's "discount, don't erase" edge case
• the damping constant is derived — 80 × SCALE − 15 must stay above the prefix arm's ceiling; one of the two tests fails at the originally proposed 0.25
• the two deliberate asymmetries — a path that is both test-like and de-prioritized is docked once, not twice; and the user penalty is not waived by a test-y query the way the inferred built-in one is (a standing project declaration outranks a heuristic — asserted so it reads as a decision, not an accident)

Regression suites, all green: context-ranking, explore-corroboration-ranking, explore-nl-stopword-collision, explore-result-count, explore-blast-radius, explore-output-budget, context, symbol-lookup, same-name-disambiguation, field-name-retrieval, search-query-parser, exclude-config, include-config, include-ignored-config, is-test-file — 165 passed. tsc --noEmit clean. README documents the key alongside exclude/include.

The A/B this repo gates on — now run, and what it did and didn't show

CLAUDE.md requires a scorer change to be A/B'd with scripts/agent-eval/*, ≥2 runs/arm. That gate was the honest blocker on this PR, so it is now run: ab-new-vs-baseline.sh (both arms codegraph-on — the only harness that isolates a retrieval change) plus run-all.sh for the with/without pass bar. sonnet / high, RUNS=2, baseline ref = this branch's merge-base (572d22b), target = a 62k-node django index carrying a codegraph.json the baseline build simply doesn't understand.

Headline: the agent-level A/B is a null result. The deterministic evidence is real and is what this section actually offers. Details below, including the two attempts that measured nothing.

Deterministic probes — the lever does what it claims, on a real corpus

django, 62,080 nodes, deprioritize: ["tests/"] (that tree is 68.4% of all nodes — a fair stand-in for a project whose peripheral tree dwarfs its product code).

40 queries, auto-generated from the 20 symbol names most concentrated in tests/ that also exist in first-party code — i.e. exactly the crowd-out condition, not queries chosen to flatter the change:

	peripheral slots in top-10 (40 queries)	queries improved	queries made worse
config absent	88	—	—
deprioritize: ["tests/"]	49	15	0

Instrument control: same build, same index, config present but with a pattern that matches nothing (no-such-tree/) → 0 of 40 queries changed, byte-identical rankings. So the deltas above are the lever, not run-to-run anything.

A concrete one — setup handling during request processing, top 10, 6 peripheral slots → 1:

rank	off	on
4	setUp@django/test/utils.py	setUp@django/test/utils.py
5–9	five more copies of setUp@tests/admin_inlines/tests.py	request@django/test/client.py ×4, setup@docs/_ext/djangodocs.py
10	setUp@tests/admin_inlines/tests.py	setUp@tests/admin_inlines/tests.py

codegraph_explore moves too, which matters because explore is the surface #982 reports on. With a core-dev-shaped config (["django/contrib/", "tests/"]), where is urlpatterns configured at startup returns 5 files, 3 of them django/contrib/** URL modules on the baseline; with the lever, 3 files, 1 peripheral — django/conf/__init__.py and django/core/management/commands/listurls.py take the freed budget, and the response drops 24.7k → 23.2k chars.

And the case that should not improve, stated plainly: how does the test runner set up the database before running tests — a query that genuinely targets the de-prioritized tree — still returns it (5 of 7 files, alongside django/test/runner.py). That is "discount, don't erase" working, not a miss.

0.25× → 0.75×: the number you asked me not to guess, now derived

You asked me to set these two numbers rather than guess. One of them turned out to be measurable, so I stopped guessing it and pushed the change.

Sweeping the damping constant on the same 40 queries, plus a recall check on 15 symbol names that exist only inside the de-prioritized tree (an on-target query for those has no first-party alternative, so the tree should still win):

scale	peripheral slots cleared (of 88)	on-target names still at rank 1 (of 15)
1.0 (path penalty only)	14	15
0.75	39	15
0.5	40	15
0.25 (originally proposed)	56	10
0.1	68	9
0 (erase)	74	9

Two things fall out:

1. 1.0 — the −15 path penalty alone — clears 14 of 88. That is the quantified version of the "additive penalty can't beat an additive, larger bonus" argument above, on a real corpus rather than the fixture.
2. 0.25 breaks the PR's own rule. The names that lose rank 1 there lose it to prefix matches: child → children@django/test/utils.py, parent → all_parents@django/db/models/options.py, method → method_decorator@django/utils/decorators.py. That is indefensible ranking behaviour regardless of this feature.

There's a bound that predicts it: nameMatchBonus's prefix arm tops out below 10 + 30 = 40, and a de-prioritized node also takes the −15 path penalty, so 80 × SCALE − 15 > 40 is what keeps a damped whole-query exact match ahead of a mere prefix match at any corpus shape. 0.75 clears it (45); 0.25 doesn't (5). And since 0.75 and 0.5 clear virtually the same crowd-out (39 vs 40 of 88), the deeper discount was buying almost nothing.

So the constant is now 0.75, derived, with two tests pinning the bound — one of them fails at the old 0.25, so it can't drift back silently. The other number, the −15 path penalty, I left alone: it's shared with the built-ins and changing it would move behaviour this PR doesn't own.

Agent A/B — null result, reported as such

ab-new-vs-baseline.sh, django + ["django/contrib/", "tests/"], question chosen after verifying with the probe above that it actually surfaces the de-prioritized tree (where are urlpatterns configured at startup…), RUNS=2 per arm, on the shipped 0.75 build:

	new (this PR)	baseline (572d22b)
duration	43s [40–46]	42s [42–43]
codegraph calls	3	3 [2–3]
Read / Grep	0 / 0	0 / 0
retrieval residual	23.6k tok [21.8–25.4k]	17.5k tok [15.0–20.0k]

Control repo (excalidraw, no codegraph.json → the lever is inert by construction, so any gap is pure variance): duration 38s [35–41] vs 41s [37–45], residual 31.5k [31.1–31.9k] vs 24.4k [17.8–30.9k]. That control puts the noise floor at roughly ±7k tokens of residual and ±5s of duration at n=2 — as large as every gap in the table above. An earlier pass of the same question on the interim 0.25 build even came out the other way (38s [33–43] vs 47s [45–49], residual 15.5k vs 21.6k). Two runs of the same question disagreeing on direction is the definition of a null result, and I'd rather say so than quote the favourable half.

Pass bar (run-all.sh, with vs without codegraph, same question, 2 invocations): with = 3–4 tool calls, 0–1 Read, 0 Grep, 2 explore calls, 39s/55s; without = 6–8 tool calls, 4–5 Read, 2–3 Bash, 40s/41s. The Read/Grep half of the bar is met comfortably; the "faster than without" half is a wash on this question (39 vs 40, then 55 vs 41). That measures codegraph as a whole, not this lever.

Two measurements I threw away, and why

Both are the same trap, and it's worth writing down: "the instrument didn't see it" is not "it isn't there."

1. First attempt: both arms made 0 codegraph calls. Given a flow question with Read/Bash available, the agent solved it by grep and Read in both arms — 8 tool calls, no explore. Any difference between those arms is about grep, not about ranking. Both arms then got an identical instruction to use codegraph_explore and not shell out; from then on every run is 2–3 explore calls and 0 Grep.
2. Second attempt: the question never surfaced the de-prioritized tree at all. "How does Django turn a URL into a view" returns handler and resolver files in both arms — no tests/, no contrib/. The lever had nothing to act on, so the arms differed only by noise (new looked worse: 88% vs 96% allocation efficiency). Discarded, and the question replaced with one the deterministic probe had already shown to be contaminated in the baseline.

Where that leaves the merge decision

The gate is procedurally satisfied — ab-new-vs-baseline.sh and run-all.sh, ≥2 runs/arm, isolated arms, disclosed methodology. But I'm not going to claim the agent-level win it was designed to detect: at n=2 on one repo the arms are inside the control's noise. What the measurement did buy is concrete: it caught that my own damping constant broke the "discount, don't erase" invariant, and replaced it with a derived one.

If a null agent A/B is disqualifying for a ranking lever that is off by default, say so and I'll close this. If the deterministic evidence is enough — the effect is user-opt-in, zero-diff when the key is absent, and provably no-op when the patterns match nothing — then the remaining decision is just whether you want the same damping extended to the built-in example//fixture/ set, which has the identical weakness.

Two small things I've deliberately left for the merge moment rather than doing now: a rebase onto current main, and the CHANGELOG entry (adding it to this branch's stale [Unreleased] section is a guaranteed conflict — I checked). Both are one command each once you tell me this is going in; I kept the head at the exact tree the numbers above were measured on.

[maxmilian]

Ran the A/B this repo gates scorer changes on, since that was the one thing I'd flagged as missing here. Body updated with the full numbers; the short version, including the part that didn't work:

The agent-level A/B is a null result. ab-new-vs-baseline.sh (both arms codegraph-on, so it isolates the ranking change rather than measuring adoption), sonnet/high, 2 runs/arm, on a 62k-node django index: duration 43s [40–46] vs 42s [42–43], 3 explore calls vs 3 [2–3], 0 Read and 0 Grep in every run. A control repo with no codegraph.json — where the lever is inert by construction — puts the noise floor at about ±7k tokens of residual context and ±5s at n=2, which is as large as every gap I measured. An earlier pass of the same question on an interim build even came out the other way round. So I'm not claiming an agent-level win. run-all.sh (with vs without codegraph) clears the Read/Grep half of the pass bar comfortably (0–1 Read vs 4–5) and is a wash on the "faster" half.

The deterministic evidence is real, and it's what I'd ask you to judge this on. 40 queries auto-generated from the symbol names most concentrated in a de-prioritized tree (django's tests/, which is 68% of that index's nodes): peripheral occupancy of the top 10 drops from 88 slots to 49, 15 queries improve, 0 get worse. Control: the same config with a pattern that matches nothing produces byte-identical rankings on all 40, so the deltas are the lever and not variance.

The measurement also caught a real defect in my own patch, which is the honest highlight here. You asked me not to guess the two constants. Sweeping the name-bonus damping showed 0.25× breaks the "discount, don't erase" rule this PR is built on: symbols that exist only inside the de-prioritized tree started losing rank 1 to prefix matches — child → children, parent → all_parents, method → method_decorator. There's a bound that predicts it (nameMatchBonus's prefix arm tops out below 40, and a de-prioritized node also takes the −15 path penalty, so 80 × SCALE − 15 > 40), and 0.75 clears it while clearing essentially the same crowd-out as 0.5 (39 vs 40 of 88 slots). So the constant is now 0.75, derived, with two tests pinning the bound — one fails at the old 0.25, so it can't drift back silently. The other constant, −15, I left alone: it's shared with the built-ins.

Also worth recording, because it's the trap this kind of eval sets: my first two A/B attempts measured nothing. In the first, both arms made zero codegraph calls (the agent grepped its way to the answer, so any difference was about grep). In the second, the question never surfaced the de-prioritized tree at all, so the lever had nothing to act on and the arms differed only by noise. Both were discarded; the final question was picked only after a deterministic probe confirmed the baseline was actually contaminated on it. Sequence and numbers are in the body.

Head is 8de1034; 16 tests in the feature suite, 165 across the ranking regression set, tsc --noEmit clean. I deliberately did not rebase or add the CHANGELOG entry yet — six upstream commits have touched these ranking files since this branch's merge-base, so rebasing now would mean the numbers above no longer describe the tree you'd be looking at (and the CHANGELOG entry conflicts against this branch's stale [Unreleased] section; I checked). Both are one command each the moment you say this is going in.

If a null agent A/B disqualifies a ranking lever that is off by default and provably no-op when its patterns match nothing, say so and I'll close it — the derivation above was worth the run either way.

[colbymchenry]

Merged, along with the rest of the batch (#1547, #1215, #594, #1542, #1548, #1551). This set was unusually easy to trust because every claim came with something falsifiable attached — the verification pass reproduced your measurements everywhere it checked (mutation tests, the regex blowup, the end-to-end fixture behavior).

Closing out the questions you left open:

The null agent A/B was not disqualifying. The A/B gate exists to catch regressions in default retrieval behavior. An opt-in key that's provably inert when absent — your matching-nothing control, byte-identical rankings — can't regress a default, and your own control run put the noise floor (~±7k residual tokens at n=2) above any plausible effect size for a ranking-only lever. The deterministic probes were the right instrument, and 88→49 peripheral slots with zero queries made worse was convincing. Reporting the two discarded measurement attempts instead of quoting the favourable half is exactly the eval discipline this repo tries to hold itself to.

The 0.75 derivation with a bound test pinning it is how constants should land here. That the sweep caught your own initial 0.25 breaking the discount-don't-erase rule is the best argument for having run it.

Yes to the follow-up: extend the same name-bonus damping to the built-in example//sample//fixture//benchmark/ set — you're right that it has the identical weakness. Same shape as this PR: keep the diff to that one behavior, same bound test, deterministic probes as the evidence bar (no agent A/B needed).

Housekeeping: the rebase you were holding turned out unnecessary (applied clean), and the CHANGELOG entry is written as part of the batch bookkeeping, so nothing is left on your side here.

One heads-up that affects #1462, detailed there: now that this and #1542 are both in, the two exact-name discounts compose, and the composition breaks the invariant each PR pins individually.