Make the resolution stage 34% faster

[dersam]

Summary

Makes the resolution stage of Rubydex faster on a hyper scale codebase. Measured
against the Shopify monolith, resolve goes from 12.318 s to 8.126 s
(−34.0 %), and the whole run from 18.74 s to 13.93 s (−25.7 %).

A re-measurement after the review fixes gave 7.889 s. The headline keeps the
more careful 8.126 s, because the load of the shared machine moves the figure
by about half a second between sessions.

The resolution stage was 66.9 % of the total time at the start, and is 58.3 %
now. There is no change to the behaviour: the graph that the resolution builds
is identical, down to the count of the descendant relationships.

What made it faster

Fifteen changes, each measured on its own. Three patterns give almost all of the
gain.

1. Do not copy a chain that the caller can read. linearize_ancestors
returned an owned copy of the ancestor chain, although most callers only needed
to know whether the chain was complete. The function now splits into
linearize_ancestors_state, which stores the chain and returns a small
ChainState, and a thin wrapper that copies only for the callers that need an
owned chain. ancestors_state_of gives the same split to the entry point.

2. Pool the temporary buffers. The linearization recurses in strict last in,
first out order, thus a plain list of spare buffers on the Resolver is enough.
Parent chains, mixin lists, the work deques and the result chain all come from a
pool now, and the LinearizationContext gets reused instead of rebuilt. A
changed chain still goes into a list of the exact size, so that the graph never
holds the spare capacity of a pooled buffer.

3. Do not repeat work that cannot have changed. The convergence loop
linearizes the same declaration many times, and 94 % of those tries give back
the chain that the graph already holds. The new chain now goes into a pooled
buffer and gets compared with the stored one. When they match, there is no
allocation, no store, and no descendant work.

The descendant bookkeeping also changed shape. It used to walk the descendants
of a declaration across its whole chain on every cached lookup, which cost
93.5 million hash inserts. Each declaration now records itself once on the
entries of its own chain, which gives the same result because the chain of a
declaration contains the chain of each of its parents.

Behaviour check

Unit tests do not catch a resolution that stops short of convergence, because
their graphs converge in a single pass. A chain state audit against the
benchmark workspace was used for every risky change, and the final state matches
the baseline exactly:

complete=140343 cyclic=0 partial=171700 descendants=3883377

autoresearch.md holds the audit recipe and the reasoning.

One test changed

DeclarationTest#test_descendants compared the descendants of a declaration as
an ordered list. Descendants are a set, and the C API iterates it without a
sort, thus the order follows the placement in the hash table, which follows the
order of the inserts. The new descendant bookkeeping changes that order for one
of the three assertions, although the contents stay the same. The test now
sorts before the comparison, so that it checks the contents only. This is the
disposition that the operator chose.

Question for the reviewer

The four autoresearch.* files at the root are the artifacts of the
optimization session. autoresearch.md holds findings that are worth keeping,
such as the chain state audit and four approaches that look attractive but do
not work. The other three are session machinery. Tell me which of these you
prefer, and I will apply it:

• keep all four as they are
• move the findings into docs/ and drop the machinery
• drop all four

Dead ends worth knowing

• A cache for partial chains looks like the last large win, and it is not. The
  fast form reaches 6.471 s but leaves 109 thousand namespaces short of a
  complete chain, because the repeated linearization is how a resolved name
  spreads through the graph. A correct form, guarded by a generation counter,
  gives parity.
• An Arc around the ancestor chain is slower than a copy, because the chains
  are short.
• A hash set for the deduplication in linearize_mixins is slower than the
  linear scan it replaces.

Review fixes

An automated review of the first draft found a defect and a set of smaller
faults. All of them are fixed in this branch.

The defect. linearize_superclass_into lost a step that the code it
replaced ended with: it turned a cyclic chain of the superclass into a partial
one before the caller read the state. Without that step, a class with an
unresolved superclass reference and a superclass whose chain is cyclic got a
cyclic chain of its own. A cyclic chain counts as settled, thus the resolver
never tried that class again and the unresolved entry stayed in its chain for
the life of the process. A lookup of a constant or a method through that class
then missed the members of the real superclass.

unresolved_superclass_beats_a_cyclic_superclass_chain covers it. The class has
to sit outside the cycle, because a class inside the cycle gets a cyclic chain
from the cycle itself.

The smaller faults. resolve_alias_chains returned a dangling id as a valid
target for a missing declaration, where the code it replaced panicked.
search_ancestors and its slow twin disagreed about a missing declaration, one
skipping and one panicking; both skip now, through a shared helper. The
benchmark script built a python3 program by string interpolation of a path,
restored its config on EXIT only, and reported a parse failure as value=0,
which is a perfect score for a metric where lower is better.

Correction to an earlier claim. An earlier version of this description said
the two collapsible_if clippy errors were already on main. That was wrong.
The check behind it used git stash, which leaves HEAD on the branch, thus
both runs measured the same code. Base 108c3638 reports no clippy output at
all. Both errors came from this work and are now fixed.

Testing

• cargo test --workspace: 1144 tests pass
• bundle exec rake test: 303 Ruby tests, 1365 assertions, 0 failures
• bundle exec rubocop: no offenses
• cargo clippy --workspace: no errors and no warnings, which matches base
  108c3638
• cargo fmt --check: clean

The suite gained a convergence_tests module. Almost every existing fixture
settles in one pass, thus none of them ran the two skips that this work adds.
The new fixtures force several passes through a chain of forward references,
and they pin the exact descendant set instead of testing for containment.
The exact sets match base 108c3638, which shows the new descendant
bookkeeping keeps the old semantics.

Co-authored-by: Claude Opus 5 noreply@anthropic.com
Orchestrated-by: ae noreply@shopify.com