Skip to content

Update the graph when the workspace changes - #979

Draft
paracycle wants to merge 1 commit into
uk-add-client-server-for-clifrom
uk-graph-partial-updates
Draft

Update the graph when the workspace changes#979
paracycle wants to merge 1 commit into
uk-add-client-server-for-clifrom
uk-graph-partial-updates

Conversation

@paracycle

Copy link
Copy Markdown
Member

Status: draft

This is not ready for review. The deletion path knowingly carries two defects, described under
"Known gap" below, and they are the reason this work was split out of #869. This pull request should
own the fix before it opens for review.

Goal

Stacked on #869. That pull request adds the client/server infrastructure and serves the graph as a
boot snapshot, so a file edited after the server starts is still answered from the state it had at
boot. This one reconciles the graph with the workspace before answering a query, which makes
rdx query --server agree with rdx query.

Review #869 first. The diff here is only the reconciliation layer.

What changes

A walk records the mtime of every indexable file. Comparing it with the previous walk gives three
sets: added or changed files go to the indexer, deleted files leave the graph, and the rest are
untouched.

Mirroring the indexer

The walk decides what gets re-indexed, so it has to see the workspace the way the indexer does.
File.directory? follows symlinks and the Rust walker does not, so a link pointing at an ancestor
made the walk descend into the same subtree repeatedly: one ln -s .. sub/loop recorded a single
file 32 times, under paths up to 301 characters.

The rule differs by depth, and both halves matter:

  • Graph#workspace_paths uses File.directory? on the workspace's own children, so a symlinked
    directory there becomes an explicit root, and the Rust walker traverses an explicit root.
  • Below that, a symlink is never followed.

Four tests pin this against the Rust tests that describe the same behaviour.

Attributing an index failure

index_all reports opaque messages for a whole batch, so a failure cannot be attributed to a file.
The graph cannot answer it either, because a failed update leaves the previous document in place. A
failing batch is therefore halved until each failure sits alone. A batch indexes far faster per file
than single calls do:

Strategy 1081 files, one unreadable
Halving the batch 21 calls, 75 ms
One file at a time 1081 calls, 311 ms

A file that failed keeps its previous mtime, so the next request tries it again. Recording the new
one would call a file the server never read "fresh" for the rest of its life. The same rule covers
the initial index, which is why build_graph returns the errors it saw and drops the roots that do
not exist: workspace_paths names gem directories an install may not have, and those phantom errors
would hide the real ones.

Not erasing what it cannot see

A directory the walk cannot read contributes no entries, and reading that silence as "every file
below here was deleted" erased whole subtrees. Two cases reach it: a directory that cannot be read at
all, and one that is readable but not searchable, where each_child lists the names and every
lstat is refused. Both keep their entries. A directory that is really gone still takes its files
with it.

The rescues name the errors that describe one path, including ELOOP, which a self-referential
symlink raises from File.mtime after lstat called it a plain file. A resource failure such as
EMFILE is deliberately absent: swallowing one would empty the walk, and the refresh would then
erase the graph.

Known gap: deletion and URIs

This is what keeps the pull request in draft.

Graph#delete_document is keyed by the document's canonical URI, and the server has a filesystem
path. uri_for reproduces that mapping in Ruby, and it does so imperfectly:

  • It does not canonicalise, so under a symlinked path the strings differ, remove finds nothing, and
    the deletion silently does nothing. Deleted files keep answering queries until the server restarts.
  • URI::File.build raises URI::InvalidComponentError for a path containing a space, so the first
    refresh that sees a deletion fails the query.

Both are confirmed, not suspected.

This is the only place the server needs a URI at all. Rust builds the document URI from a path it has
already canonicalised and simplified (path_helpers::resolved, then Url::from_file_path), so the
correct fix belongs at the graph API: either delete_document_at(path), or an exposed canonical
path-to-URI function. Test::Helpers::Context#uri_to carries the same two lines and the comment
TODO: This has to go away once we have a proper URI abstraction, which is the same gap.

Tests

test/server/core_test.rb covers the walk, the attribution and the preservation rules;
test/server/integration_test.rb covers a change and a deletion through the real executable.

@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from beda794 to 0255914 Compare August 4, 2026 18:51
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 7665e79 to 2a0811a Compare August 4, 2026 18:53
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 0255914 to 2e05f1c Compare August 4, 2026 18:59
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 2a0811a to 4fdac7c Compare August 4, 2026 18:59
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch 2 times, most recently from c7fde98 to 43bebb4 Compare August 4, 2026 19:10
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch 3 times, most recently from f839e9a to 2842fab Compare August 4, 2026 19:50
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 43bebb4 to 583658c Compare August 4, 2026 19:50
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 2842fab to d4dcd65 Compare August 4, 2026 19:54
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch 2 times, most recently from 96a8e47 to bbc1a69 Compare August 4, 2026 19:58
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch 2 times, most recently from 772b946 to 2fd98ce Compare August 4, 2026 20:02
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from bbc1a69 to 2be9a3e Compare August 4, 2026 20:02
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 2fd98ce to dd31f1d Compare August 4, 2026 20:06
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 2be9a3e to a72da34 Compare August 4, 2026 20:06
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from dd31f1d to c9ee992 Compare August 4, 2026 20:20
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch 2 times, most recently from 562ec55 to 1ddcecd Compare August 4, 2026 20:25
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch 2 times, most recently from aa9c8df to 51c15f3 Compare August 4, 2026 20:39
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 1ddcecd to 254366b Compare August 4, 2026 20:39
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 51c15f3 to a9f00eb Compare August 4, 2026 21:09
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch 2 times, most recently from 5ee43dd to 57aa496 Compare August 4, 2026 21:14
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch 2 times, most recently from 9794016 to a26f626 Compare August 4, 2026 21:39
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch 2 times, most recently from 93bd725 to c8d9108 Compare August 4, 2026 22:19
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from a26f626 to c3a6b51 Compare August 4, 2026 22:19
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from c8d9108 to df872ce Compare August 4, 2026 22:31
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from c3a6b51 to 0b67ee3 Compare August 4, 2026 22:31
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from df872ce to 720e4e6 Compare August 4, 2026 22:42
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 0b67ee3 to 6e4bbe2 Compare August 4, 2026 22:42
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 720e4e6 to d71de2a Compare August 4, 2026 22:50
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch 2 times, most recently from adf1a2d to 5f42b74 Compare August 4, 2026 23:02
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from d71de2a to 9d24c41 Compare August 4, 2026 23:02
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from 5f42b74 to a2bbb2c Compare August 4, 2026 23:09
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 9d24c41 to 1dd0a1f Compare August 4, 2026 23:09
The server built its graph at boot and never looked again, so a file edited
after the server started was still answered from the state it had at boot.
This reconciles the graph with the workspace before answering a query.

A walk records the mtime of every indexable file. Comparing it with the
previous walk gives three sets: added or changed files go to the indexer,
deleted files leave the graph, and the rest are untouched.

The walk decides what gets re-indexed, so it has to see the workspace the
way the indexer does. `File.directory?` follows symlinks and the Rust
walker does not, which made a link pointing at an ancestor walk the same
subtree again and again: one `ln -s .. sub/loop` recorded a single file 32
times, under ever longer paths, until the platform refused.

The rule differs by depth, and both halves matter. `Graph#workspace_paths`
uses `File.directory?` on the workspace's own children, so a symlinked
directory there becomes an explicit root and the Rust walker traverses it.
Below that, a symlink is never followed.

`index_all` reports opaque messages for a whole batch, so a failure cannot
be attributed to a file. The graph cannot answer it either, because a
failed update leaves the previous document in place. So a failing batch is
halved until each failure sits alone. A batch indexes far faster per file
than single calls do: measured on 1081 files with one unreadable among
them, 21 calls in 75ms against 1081 calls in 311ms.

A file that failed keeps its previous mtime, so the next request tries it
again. Recording the new one would call a file the server never read
"fresh" for the rest of its life. The same rule covers the initial index,
which is why `build_graph` now returns the errors it saw and drops the
roots that do not exist: `workspace_paths` names gem directories an install
may not have, and those phantom errors would hide the real ones.

A directory the walk cannot read contributes no entries, and reading that
silence as "every file below here was deleted" erased whole subtrees. Two
cases reach it: a directory that cannot be read at all, and one that is
readable but not searchable, where `each_child` lists the names and every
`lstat` is refused. Both keep their entries. A directory that is really
gone still takes its files with it.

The rescues name the errors that describe one path, including `ELOOP`,
which a self-referential symlink raises from `File.mtime` after `lstat`
called it a plain file. A resource failure such as `EMFILE` is deliberately
absent: swallowing one would empty the walk, and the refresh would then
erase the graph.

`Graph#delete_document` is keyed by the document's canonical URI, and the
server has a filesystem path. `uri_for` reproduces the mapping in Ruby, and
it does so imperfectly: it does not canonicalise, so it misses under a
symlinked path, and `URI::File.build` raises for a path containing a space.
This is the one place the server needs a URI at all, and closing that gap
belongs with the graph API rather than here.
@paracycle
paracycle force-pushed the uk-graph-partial-updates branch from a2bbb2c to b615dfe Compare August 4, 2026 23:16
@paracycle
paracycle force-pushed the uk-add-client-server-for-cli branch from 1dd0a1f to 055b037 Compare August 4, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant