Tecnix dep tracking + eval caching

[joshheinrichs-shopify]

robo summary 🤖

Tecnix makes Nix evaluation itself input-addressed: each target's evaluated
result is certified by its source closure — the fingerprinted set of source
paths (including directories listed and files found absent) that evaluation
actually observed. The closure is simultaneously the target's exact dependency
set and the proof of cache validity: a stored result is reused at any commit
where every path in one stored closure still fingerprints the same, and is
recomputed otherwise. Warm runs skip evaluation entirely and pay only for
fingerprint validation.

Design docs (authoritative; read these first):
plans/tecnix-target-eval-caching/explainer.md — problem, design, structures
plans/tecnix-target-eval-caching/walkthrough.md — one request traced end to end
plans/tecnix-target-eval-caching/guardrails.md — review checklist / invariants

Public surface

• builtins.tecnixTargetNames { gitDir, resolver, rev, args, ... }
  -> [ target-id ... ], optionally + { dependencies } (discovery closure)
• builtins.tecnixTargets { ..., targets, includeDependencies, includeTargets }
  -> attrset of values, or [ { target, value?, dependencies } ... ]
• The entire repo contract is one resolver file exporting
  { allTargetNames; resolve = id: ...; }. The builtins are generic over any
  git repository; target-id syntax belongs to the resolver.
• Settings: tecnix-eval-cache (default true; the persistent cache engages only
  under pure-eval), tecnix-parallel-dependencies (default true).
• Internal scope builtins __tecnixInternalSourceDepsScope/Attrs/List:
  evaluate a value under a bracketed region whose collected source accesses
  become the value's label (identity outside tracked eval; the engine uses
  the same brackets to replay provenance through its own caches).
• scripts/measure-tecnix-eval.sh: manual measurement driver.

Tracking core (src/libexpr/tecnix/, include/nix/expr/tecnix/)

• Labels live in a sparse two-level value-label table keyed by cell
  address: a constant-initialized BSS directory (top address bits) pointing
  at lazily-mapped NORESERVE chunks, one u32 slot per 16-byte-aligned cell.
  Value keeps upstream's exact 16-byte layout (value-layout-tripwire.hh
  static_asserts it); untracked processes never allocate a chunk, tracked
  evaluation pays ~4 bytes per labeled cell (clears elide the 0-over-0
  store, so pages commit only where labels are actually published), and
  the directory's hot entries cover the whole heap in a few cache lines. Chunk pointers are published by
  CAS-release over kernel-zeroed pages and read relaxed through an address
  dependency (the rcu_dereference pattern).
• Paths and path sets are interned in an append-only
  EvalSourceAccessSetGraph: lock-free path->id hits, singleton table,
  pair-union cache (lazy eval overwhelmingly unions exactly two labels), one
  mutex for graph-touching publishes. Zero allocations except on first sight
  of a path/set.
• forceValue: with no tracking context, one thread-local read + branch
  (initial-exec TLS, like the evaluator's other hot thread-locals). Under
  tracking, finished values contribute their label with
  two integer ops (this is the answer to memoization hiding reads); thunks
  get a small stack frame that interns on finish — frames suppress only
  consecutive duplicate entries, since the interner canonicalizes at publish
  anyway. The label is stored before the cell is marked finished, so no
  thread can observe a finished value without its label.
• Value hooks (value-hooks.hh, called from ValueStorage): every cell becomes
  a finished value through the single finish() chokepoint, and its label
  slot is cleared there unconditionally — a recycled GC cell or reused stack
  slot can never leak a stale label; a label describes current contents,
  never history. Copies of
  finished values propagate labels, so provenance survives the evaluator's
  movement of values between cells: force, call-time capture forcing, and
  copy are indistinguishable channels, and contents never move without their
  label.
• Copy semantics: Value now has user-declared copy ctors/assignment funneling
  through ValueStorage::operator= so hooks see every copy. Consequences kept
  deliberately: addConstant does a raw Storage memcpy + label clear (operator=
  barfs on thunks; constants predate tracked eval), mkThunk_ routes through a
  helper, and builtins.break now forces its argument before returning it
  (behavior change, unit-tested: break (1+2) == 3).
• Tracking contexts are thread-confined: created, recorded into, snapshotted,
  destroyed on one thread; no locks on recording. Constructing a context
  enables the state's graph (hot paths never re-check), and publishing is
  typed against the context's own graph — a label cannot be interned into
  any graph but the one that will later resolve it. Tracked evaluation must not
  spawn parallel work — builtins.parallel and toJSON's deep-force prefetch
  skip spawning under tracking (consumer forces sequentially; identical
  results), and EvalState::makeWork throws under a tracking context: work
  items capture only owned state, but a tracking context is a non-owning
  pointer into another thread's stack, and detached work would make closures
  scheduling-dependent. The only cross-thread dependency channel is the
  published label on a finished value.
• Flatten + fingerprint is deferred until all per-target evaluations finish;
  it is a pure function of the snapshots, which is why sequential, parallel,
  isolated, and warm-cache evaluation produce identical closures (the test
  suite's oracle).
• EvalState::resetFileCache never clears the graph: labels are graph-local
  IDs, values carrying them survive the reset, and a cleared graph would
  re-mint IDs and silently mis-resolve surviving labels. The graph is
  append-only precisely so IDs stay valid forever (regression-tested).

Source observation

• All tracked reads flow through one TecnixSourceAccessor: pluggable clean
  backend (libgit2 tree, or worldtree FUSE projection) + dirty checkout
  overlay + per-path fingerprints + repo-relative access recording. Repo-root
  access is unrepresentable in the closure format and fails closed.
• Fingerprint vocabulary (identical across backends, so TXDC rows validate
  regardless of producer):
  git:;mode= clean file/dir at the pinned rev
  ...;dirty= + hash of dirty content beneath the path
  absent[;dirty=...] negative lookups are first-class deps
• The clean view is the raw committed tree: no export-ignore or LFS
  filtering (fingerprints describe raw tree objects; a filtered view would
  be an uncertified input channel — regression-tested).
• Dirty overlay: one git status --porcelain -z --untracked-files=all per
  evaluation, with GIT_DIR/GIT_WORK_TREE/GIT_COMMON_DIR/GIT_INDEX_FILE
  scrubbed so ambient git contexts (hooks, rebase exec) cannot redirect status
  to the wrong repo/index. Failure to get status is a hard error for the new
  path (the overlay is load-bearing for closure validity); the legacy
  dirty-zones caller keeps its warn-and-continue policy explicitly.
• Recording convention (documented at maybeLstat and in the guardrails):
  reads self-record in the accessor; stats are evaluator plumbing and do NOT
  record — existence/type observations record at their semantic call sites
  (prim_pathExists, prim_readFileType via recordEvalAccess). Any new primop
  observing existence/type without a read must record itself.
• dumpPath suppression: readFile/readDirectory/readLink tracking is gated on
  a thread-local dumpPathDepth — a materialized tree is covered by its
  directory-level fingerprint (src = ./. depends on the dir, not every child).
  callPathFilter lifts the suppression for the duration of a filter call:
  filters run arbitrary Nix code inside dumpPath, and a readFile there (e.g.
  an ignore list) is a real dependency whose first read would otherwise be
  cached unlabeled, under-tracking every later consumer.
• Every evaluator cache that can absorb a physical read either has a separate
  tracked-domain instance or replays provenance on hit:
  • trackedFileEvalCache (separate so untracked eval can't prewarm
    unlabeled entries); tracked hits replay the file's label via a scope
  • trackedImportResolutionCache entries carry the source-deps label
    recorded while resolving (symlinks, default.nix) and replay it on hit
  • shared parsedFileCache (parse results carry no provenance; safe)
  • srcToStore / copyPathToStore record the source path before the hit
• libutil/libfetchers hooks: SourceAccessor gains tracksEvalAccesses /
  recordEvalAccess virtuals + dumpPathDepth; Mounted/Union/Filtering/
  Forwarding accessors forward them. ForwardingSourceAccessor now also
  forwards getFingerprint — previously OverrideProvenanceSourceAccessor
  silently dropped fingerprints, so a filtered tree's root access was never
  recorded (stale cache hit on edit) and fetchToStore caching was lost for
  provenance-wrapped sources (an upstream loss too).
• fetchToStore2 always calls getFingerprint (so accessors can record) but
  skips the persistent source-path cache when a filter is present: the filter
  predicate is not part of the cache key (functionally tested with two
  filters over one path name).

Persistent cache (tecnix/eval-cache.cc)

• One SQLite db, ~/.cache/nix/tecnix-eval-cache-v1.sqlite, one table:
  DependencyShards(gitDir, resolver, argsKey, shard) -> TXDC blob. 256 shards
  by FNV-1a of target name. No commit in the key: validity is proven by
  re-fingerprinting a stored candidate against the current tree, never
  trusted from row identity — structurally so: the cache layer is addressed
  by a (gitDir, resolver, argsKey) scope and cannot see rev or checkoutPath
  at all, and builtins consume hits as opaque validated-blob handles over
  the row bytes. Malformed/foreign/corrupt rows fail to open and
  are misses, never errors; deleting the db is always safe.
• TXDC v1 blob ("TecniX Dependency Closure"): header + string tables
  (targets/paths/fingerprints/payloads) + target records + candidate records
  • flat (pathId, fingerprintId) pair streams. Opening = bounds-checking the
    sections; validation and output walk the row bytes in place (no JSON parse,
    no heap graph). Limits: <=1024 targets/shard, <=32 candidates/target,
    <=64 MiB/row. Candidates are bounded newest-first history; identical
    closures dedupe; both branches of a flip-flop (e.g. absent<->present) can
    coexist so branch switching doesn't thrash.
• When a merged shard exceeds the target cap (rename churn — target names are
  never individually evicted), warn and rebuild the shard from the current
  evaluation's entries alone; evicted targets re-enter on next use.
• Discovery is a reserved key (__tecnixTargetNames, rejected as a caller
  target id) in the same rows, its candidate payload carrying the discovered
  name list — same lookup, validation, history, and compaction as targets
  (so a valid empty target list is an ordinary cache hit).
• argsKey is a canonical JSON encoding computed by canonicalJsonFromValue,
  which must stay injective and coercion-free — deliberately NOT
  printValueAsJSON (derivation->outPath and __toString coercions collide
  distinct args onto one key, i.e. stale hits; path coercion copies to the
  store; floats are ambiguous). Only null/bool/int/context-free
  string/list/attrset are accepted.
• Ambient inputs a pure eval can still observe (builtins.nixVersion,
  storeDir) are deliberately not in the cache key, matching the flake eval
  cache; evaluator/semantics changes are handled by bumping the version in
  the cache filename (see the explainer's §8 footnote). Unshipped dev
  format: no migrations, ever; incompatible rows miss or the db is wiped.
• Cache writes are an optimization: failures warn and continue. Multi-target
  upserts batch into one immediate transaction. Per-run fingerprint memo
  (generation-stamped thread-local map) makes validation cost scale with
  unique paths, not total closure entries; it is the only fingerprint-caching
  layer, so cached fingerprints live exactly as long as the dirty snapshot
  they describe.
• Warm hits with includeTargets still resolve fresh target values: a
  dependency-cache hit is never a target-value cache hit.

Worldtree backend (reconciling the wt-single-mount merge)

• With tectonix-worldtree-socket set there is no git repo to read; the clean
  tree is the daemon's immutable FUSE projection
  /tecnix//, mapped through the committed manifest
  (served by the reserved W-000000 pseudo-zone, never the checkout copy).
  The socket is control-plane only (dirty_zones, zone_tree_shas).
• WorldtreeFuseSourceAccessor keeps the exact git fingerprint vocabulary:
  directories read the committed tree oid from the user.worldtree.tree-oid
  xattr (guaranteed by the daemon; absence is a contract violation); regular
  files read the user.worldtree.blob-oid xattr when the daemon serves it,
  else hash content as a git blob ("blob \0" framing — a blob oid is a
  pure function of content), memoized in memory per accessor (the projection
  is immutable). Symlinks never try the xattr: getxattr would follow the link
  and answer for its target. Mode is reconstructed from the stat (the
  projection preserves the executable bit).
• Dirty worktrees: the daemon serves no dirty data to the tracked path.
  The dirty set still comes from git status in the materialized checkout
  (interim boundary until a full repo dirty-path RPC / scoped.status), and
  dirty reads come from the checkout on disk, overlaid on the projection.
• Zone-granularity caveats: committed paths outside every visible zone
  observe as absent; synthesized zone-ancestor directories get a composite
  worldtree-union: fingerprint (their listing genuinely differs from the
  full git tree, so cross-backend misses there are correct).
• Legacy tectonix builtins take wt-single-mount's behavior verbatim, moved
  out of eval.cc into tecnix/source-accessors.cc as free functions over a
  TecnixEvalData pImpl (eval.cc/eval.hh shrink by ~1000 lines net). The
  worldtree helpers are intentionally duplicated between the legacy file and
  tecnix/repo-accessor.cc so the legacy implementation stays byte-identical
  to wt-single-mount and the new API owns everything it needs.
• Removed with the socket data plane: the per-zone session connection pool,
  WorldtreeSourceAccessor, WorldtreeCleanTreeSourceAccessor, the
  worldtree-pool unit test, the socket-worldtree functional test, and the
  fake-git-worldtreed mock daemon.
• configureTectonixContext pins (gitDir, rev, checkoutPath) once per
  EvalState; a second call with a different context errors rather than
  silently mixing content from two commits.

The module layout makes the explainer's pipeline physical: the builtins
(primops/tecnix.cc), source observation (tecnix/repo-accessor.cc), the
persistent cache (tecnix/eval-cache.cc), and the tracking core
(tecnix/source-deps.cc) are one file per layer, and each file's includes are
that layer's actual dependencies. The interning containers live outside the
eval.hh include chain, keeping eval.hh on upstream's forward include.

Other upstream-file changes

• GitRepo: new getPathInfo (oid + git filemode by full relative path; mode is
  in the fingerprint because materialization observes executable bits);
  getSubtreeSha rewritten over git_tree_entry_bypath (empty name returns the
  tree itself; non-directory entries error).
• SQLiteStmt::Use::getBlob (zero-copy string_view column read).
• AsyncPathWriter now batches queryValidPaths and skips re-adding
  already-valid store paths; new wasAdded() lets nix provenance verify
  count deduplicated writes as instantiated instead of reporting a false
  "evaluation did not re-instantiate path".
• Legacy unsafeTectonixInternal* builtins that expose checkout-local state
  (sparse roots, dirty zones, zone root) throw under tracking (fail closed);
  the representable ones record their zone path / manifest access.

Tests and benchmarks

• tests/functional/tecnix/: synthetic-repo suite for the public builtins.
  Covers the correctness oracle (isolated == shared sequential == shared
  parallel == warm cache, run in separate evals), cross-target contamination
  (scope owner collisions), replay of shared forced values / imports /
  readFile thunks (string, bool, list), multi-hop AccessSet closure
  flattening, negative and positive pathExists + readFileType deps, readDir
  as a directory-level dep, symlinked imports (both link and destination),
  untracked prewarm vs tracked domains, dumpPath suppression + path-filter
  lift, builtins.parallel / toJSON prefetch gating, dirty overlays (zone,
  resolver, nested untracked files), absent->present invalidation and
  bounded-history flip-flop hits, repo-root fail-closed, context
  reconfiguration errors, git-status fail-closed, the rev argument's
  checkout-HEAD default (same output as an explicit rev; fails with guidance
  when the checkout cannot answer), and cold/warm persistent cache roundtrips
  under pure eval with an isolated XDG_CACHE_HOME asserting actual "cache
  hit" log lines.
• src/libexpr-tests/tecnix-dependency-tracking.cc: graph interning, label
  lifecycle (overwrite clears, copy publishes, tryEval paths), file-cache
  reset preserves labels, makeWork-under-tracking throws.
• src/libexpr-tests/tecnix-worldtree.cc: fakes the FUSE projection with
  plain directories + real xattrs; content mapping, exact fingerprints
  against an independent libgit2 oracle, absent/unzoned observations,
  zone-ancestor composites, blob-oid xattr preference + hash fallback, and
  in-memory-only memoization semantics.
• src/libexpr-tests/tecnix-eval-bench.cc (-Dbenchmarks=true): synthetic
  world, ~20-path closures (15 shared libs exercising inheritance + the
  pair-union cache), deep narrow directory geometry matching the measured
  monorepo (median width 2, p99 44). Geometry matters: a flat 10k-file layout
  made warm validation quadratic because git tree objects >4 KiB fall outside
  libgit2's default object cache (34s warm vs 59ms with the limit raised);
  GIT_OPT_SET_CACHE_OBJECT_LIMIT is the known one-line remedy if that shape
  ever becomes real. Reference numbers: cold 1k/10k = 216ms/2.26s, warm =
  8.1ms/78ms, linear in target count. On the 7,254-target monorepo workload:
  warm all-target dependency graph well under a second; cold tracked eval
  ~tens of seconds vs tens of minutes for untracked per-target isolation.

[EsterKais]

Approving this. The design is clean, and the tests are genuinely great — they
check the hard stuff (shared values, negative lookups, dirty files, symlinks,
cross-commit invalidation), and they compare a target evaluated on its own against
the same target in a shared and a parallel run.

I'm approving rather than asking for changes first, and that's on purpose. Merging
this into tecnix doesn't actually change anything in World — we pin a specific
tecnix commit, so nothing moves until a separate bump PR. And everything I raise
below only matters on the dependency-graph path, which isn't driving any real
decision yet. Normal builds never touch the cache. So I'd rather land this now and
handle these as follow-ups before we turn the dependency graph on, instead of
growing the PR.

A few things my agent raised that we discussed — they seemed like concern points,
or at least clarification points, for me:

1. A .gitattributes change can fool the cache.
Git has a way to say "pretend this file isn't here" (export-ignore, via
.gitattributes). The cache remembers each file it read plus a fingerprint of
that file's contents. The catch is the fingerprint only tells us whether the
file's contents changed — it says nothing about whether the "pretend it isn't
here" rule changed. So if someone adds a rule that hides a file a target read, the
file's contents haven't changed, its fingerprint still matches, and the cache says
"nothing changed" and hands back a stale answer — even though a real re-run can't
see that file anymore.

In the code: getTecnixRepoAccessor turns on export-ignore but doesn't pass the
attribute info the way the zone accessors do in makeZoneAccessorOptions, and
GitPathFingerprintSourceAccessor::getFingerprint only records the raw object id
and mode, nothing about attributes.

The reassuring part: it's only risky in one direction (hiding a file that was
read; un-hiding is already caught), and some cases get caught by a directory
fingerprint. What are your thoughts?

2. Every value gets a little bigger the moment we bump World.
Merging here is harmless, but once World runs on this evaluator, every value is 8
bytes bigger (24 → 32), for all evaluation, whether tracking is on or not — plus a
small check when values finish and a branch in the force path. My worry is memory
and cache pressure on big evals, since value size is exactly the thing upstream
works hard to keep small. Is this something we need to worry about?

3. A shared function can quietly lose its file list — nice-to-have safety net, not a blocker.
accessSetForCopiedValue deliberately drops a function's file list when it's
reused and carries more than one file, so shared helpers don't smear their whole
history onto every target. I think that's the right call and I wouldn't change it.
The catch is it depends on resolver authors remembering to wrap file-reading
shared helpers in a scope, and nothing catches it if they forget. Instead of
changing the drop, could we add a CI test that evaluates targets both on their own
and in a shared/parallel run and diffs the results? A forgotten scope would show
up as the two runs disagreeing — so it turns "hope nobody forgets" into an
automatic check.

4. Just checking on the one lock.
The access-set graph uses a single lock for interning and for merging into each
target's total. The common operations stay off it, and the expensive flatten
happens after the workers finish, so I'd guess it's fine — but under heavy
parallel tracked eval, is that one lock going to hold back scaling, or is it
clearly off the hot path? Just want your read, not asking for a change.

These are all follow-ups or questions, not blockers — landing the foundation makes
sense to me.

[joshheinrichs-shopify]

1. A .gitattributes change can fool the cache.

we don't use export-ignore and i dont foresee us making use of it in the future. ill look at explicitly disabling it.

2. Every value gets a little bigger the moment we bump World.

i think we go from 16 bytes to 32 bytes. yeah this is unfortunate but i haven't found a more performant solution yet. i might try taking one final stab at this taking some inspiration from asan's shadow memory. in testing tracking appears to add ~10% overhead atm so it's not untenable given we get a >99.5% hit rate.

edit: this is a bit more performant but would conflict with asan itself -- not worth it imo but could be convinced otherwise

edit: i took another stab at a lookup table and it's performant enough that we can avoid keeping labels with values

3. A shared function can quietly lose its file list — nice-to-have safety net, not a blocker.

good callout! i looked back at this and it is totally unnecessary. i ripped it out and we're producing the same source closures. more correct and less fragile!

4. Just checking on the one lock.

tbh parallel eval's performance gains appear to be nebulous at best. if we want high performance eval in the uncached case, we'll probably want to use nix-eval-jobs.