Skip to content

fix(index): compile in-process when the serving image has no guest rootfs - #12

Closed
philocalyst wants to merge 21 commits into
jimmy/turso-index-unification-7a89from
jimmy/nix-oci-backend-7a89
Closed

philocalyst wants to merge 21 commits into
jimmy/turso-index-unification-7a89from
jimmy/nix-oci-backend-7a89

Conversation

@philocalyst

@philocalyst philocalyst commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Linux serving image has no Smolvm guest rootfs, so every package failed in the cage and stayed unindexed. Compile strategy now follows whether NUDOX_GUEST_ROOTFS is provisioned:

  • unset or empty: run the in-process language producer on the materialized tree
  • non-empty on Linux: keep the cage, and still fail the job if the cage fails

The image entrypoint creates /tmp before the first tempfile, and now also puts cargo, rustc, and the gcc cc wrapper on PATH with a writable CARGO_HOME. In-process Rust indexing runs cargo metadata and build-script cargo check. Without the linker, serde failed in Compiling with linker cc not found.

tests/docker/lib_rs_indexed.sh ingests serde, waits until it is Stored, and requires a package card plus a Serialize symbol hit.

Tests

RUSTC_BOOTSTRAP=1 cargo test -p index --features server --lib select_compile_strategy — 3 passed.

Against a container with cargo, rustc, and cc on PATH, tests/docker/lib_rs_indexed.sh passed: serde Stored, description "A generic serialization/deserialization framework", keywords, license mit or apache-2.0, downloads present, and a Serialize symbol hit. The image rebuild bakes those tools in so the container no longer depends on a host mount.

Open in Web Open in Cursor 

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.

cursoragent and others added 21 commits September 25, 2026 16:05
buildRustPackage refuses to vendor turso_versioning-0.8.0-pre.7 without
an outputHashes entry, so the nix2container backend image cannot build.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
smolvm's submodules are declared as git@github.com URLs. The sandboxed
fetch has no ssh client, so cargo vendoring of the backend image failed.
A git insteadOf rewrite sends those clones to https://github.com/.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The HTTPS submodule checkout changes the smolfile/smolvm git tree.
Pin the fixed-output hash Nix computed for that tree.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The lockfile has a crates.io trustfall and a git fork at the same
version. importCargoLock linked both to one directory name, and the
second link failed inside the read-only first crate. A name collision
keeps the later tree under its store path.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
importCargoLock cannot name both trustfall 0.8.1 crates, and cargo's
git fetch missed the HTTPS rewrite that fetchgit uses. Vendor with
cargoHash and point GIT_CONFIG_GLOBAL at the GitHub insteadOf config
so smolvm submodules clone over HTTPS.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The serving binary is feature-gated. The image package now passes
buildFeatures = [ "server" ] through mkRustService.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The serving crate graph uses #![feature] on the fenix stable toolchain.
RUSTC_BOOTSTRAP=1 is the same switch the workspace tests already use.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The host LLVM cannot select avx512.vpdpbusd when crates target the
native CPU. Pin RUSTFLAGS to x86-64-v3 so the serving binary links.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
Thin LTO re-lowers avx512.vpdpbusd after the target feature is dropped,
and this LLVM cannot select that intrinsic. Release LTO stays off for
the nix2container serving image.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
qdrant-edge compiles its vnni kernels on their own. Thin LTO and a
forced x86-64-v3 target both ask LLVM to select vpdpbusd without that
feature. Drop the CPU pin and leave release LTO off.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
This LLVM cannot lower vpdpbusd even in per-function target_feature
objects. Set qdrant_edge_no_avx512_vnni for mkRustService and skip the
VNNI kernels at compile time; runtime dispatch keeps the AVX2 path.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
…hits

A lib.rs-style card needs the signals that ordered the hit. Package search
already ranks with dependents, and the wire hit now carries that count plus
keywords and license from the same facets.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The script checks health, an empty query, a serde search page, and that
dependents do not rise as scores fall. It exits 2 when nothing is listening.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
Three indexed crates with a shared keyword sort by dependents.
A dependents mass that stays under the popularity clamp beats a
download count just below that mass. An exact name beats a keyword
when popularity ties.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The gitignored doltlite.c amalgamation is pinned in manifest.toml but absent
from the flake source snapshot. Add a fixed-output fetch and copy the three
engine files into workspace/vendor/doltlite during mkRustService postPatch so
nudox-serve links the real DoltLite catalog engine.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The server requires Qdrant at assembly time. Pin the upstream musl release
binary (nixpkgs qdrant fails AVX-512 codegen here) and add nudox-backend-run
to start Qdrant then nudox-serve with durable paths under /var/lib/nudox.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
Linux serving images have no Smolvm golden. Choosing the cage only
because the host is Linux left every package Unindexed. Use the
in-process producer unless NUDOX_GUEST_ROOTFS is set, and create /tmp
before the first tempfile.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
The live probe now ingests serde, polls until the package is Stored,
and requires a package card with description, keywords, or license
plus a Serialize symbol hit.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
In-process Rust indexing runs cargo metadata and build-script checks.
The live serde job reached Stored only after those tools and a linker
were on PATH. The image now carries them, and the container probe
reads the HTTP status from the response instead of a lost subshell
variable.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
Indexing spawns tantivy merge threads. With the SIGPROF sampler
armed, nudox-serve aborts during serde compile. OTEL_SDK_DISABLED
leaves logs and /metrics up.

Co-authored-by: Miles Wirht <philocalyst@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants