diff --git a/plans/tecnix-target-eval-caching/explainer.md b/plans/tecnix-target-eval-caching/explainer.md new file mode 100644 index 0000000000..e95197e3b7 --- /dev/null +++ b/plans/tecnix-target-eval-caching/explainer.md @@ -0,0 +1,492 @@ +# Tecnix: Exact Target Dependencies and Cross-Commit Eval Caching + +**Abstract.** Nix builds are input-addressed: a derivation's inputs are hashed, and identical inputs need never be rebuilt. Tecnix extends the same idea one level up, to evaluation itself. Each target's evaluation result is addressed by the content hashes of the source files that evaluation actually read — an artifact we call the target's *source closure*. The closure serves simultaneously as the target's exact dependency set and as a proof of cache validity: a stored result may be reused at *any* commit whose relevant sources still match, and must be recomputed at any commit where they do not. This document explains why that requires cooperation from the evaluator, how dependency information is threaded through Nix's lazy, memoized evaluation at near-zero cost, and how the resulting cache reduces warm evaluation time by orders of magnitude. + +*Reading guide: users of the system need §§1–5 and §9. §§6–8 cover the evaluator internals. §10 concerns maintenance of the fork and may be skipped by everyone else. A glossary appears at the end. For the system in motion — one request traced from call to answer — see the companion document, 'The Life of a Target Evaluation'.* + +--- + +## 1. The Problem + +Consider a repository with many thousands of build targets. A developer changes one file. Which targets must CI re-test? Which merge-queue candidates actually conflict? Is anything in the developer's environment now stale? Today, answering any of these questions requires evaluating the repository's build logic — which is the very cost the answers were supposed to help avoid. + +Two capabilities resolve this, and they turn out to be two views of one artifact: + +**The target dependency graph.** For every target, the precise set of source paths on which its computed result depends: + +```json +{ "//app/web:server": [ "app/web/target.nix", "lib/common.nix", ... ], ... } +``` + +**Target evaluation caching.** The ability to skip computing a target entirely when the sources it depends on have not changed. + +A single requirement shapes the design: both must remain valid **across commits**. The system runs on every push, every merge-queue candidate, and every developer checkout after every pull. Dependency information tied to the commit that produced it would have little value; the value lies in reusing previous work against a new tree. + +Exactness is equally fundamental, because dependency information is a substrate for many downstream decisions: precise test selection in CI, concurrent evaluation of merge-queue candidates, local staleness detection ("your development environment is stale after that pull — your `Gemfile.lock` changed"), and evaluation cheap enough that correctness can be verified on every invocation, in the manner of `bundle exec`. + +### 1.1 Three ways to fail + +It is useful to fix, at the outset, the three failure modes available to any design in this space. Every mechanism in this document exists to defend against one of them. + +| failure | consequence | +|---|---| +| **Under-tracking** — a real dependency is missed | *Wrong answers.* A stale cached result is served as current. This is the fatal failure mode, and the design treats any instance of it as a correctness defect. | +| **Over-tracking** — spurious dependencies are included | *Useless answers.* If every target depends on everything, every commit invalidates everything; hit rates collapse and affected-sets balloon. Sound, but pointless. | +| **Slowness** — exact answers at impractical cost | *Unused answers.* The gold standard — evaluate each target in a fresh evaluator and log its reads — is exact and hopelessly slow. | + +The labels of §6 defend against under-tracking; their per-value granularity — a target inherits only the production history of the values it actually touches — bounds over-tracking; nearly everything else defends against slowness. + +One subproblem is contained within the problem. Before targets can be computed, the system must determine which targets *exist* — and in a monorepo the target list is itself derived from the source tree, so it is subject to the same staleness question as the targets. Discovery proves to be the same problem one level up, solved by the same machinery; it is treated in §9 alongside the public interface. + +--- + +## 2. The Setting: Targets Are Programs + +Targets in this repository are not declared in a static manifest. Their meaning is computed by Nix evaluation: repository code imports files, reads configuration, tests for the presence of optional overrides, and ultimately produces each target's result, which is a derivation. + +### 2.1 Nix evaluation as a memoized value graph + +The model on which the entire design rests is the following. Evaluating a Nix expression does not proceed from top to bottom. The evaluator lazily expands a graph of *thunks* (suspended computations) into *values*, on demand, and memoizes every result. A file is imported once; a shared helper is computed once; thousands of targets that use the same library all refer to the same finished value cells. + +Tecnix's central move can now be stated in one sentence: **it records file accesses alongside this graph.** As the evaluator expands values, every source read, directory listing, and existence check is attached, in the form of a compact label, to the value whose production caused it. Dependency information thereby becomes part of the memoized graph itself and flows to consumers in exactly the way values do. + +### 2.2 Why the mechanism belongs in the evaluator rather than in derivations + +Nix already possesses an input-addressed caching mechanism: the derivation. A `.drv` file hashes its build inputs, so identical inputs yield identical builds. It is natural to ask whether this mechanism suffices. + +It does not, because the derivation is the *output* of evaluation, not its input. Which derivation a target produces is determined by evaluation-time behavior — which files are imported, what `readFile` returns, whether `pathExists` finds an optional override — and none of these reads is recorded anywhere in the resulting derivation. Derivation-level caching answers the question "have these inputs been built before?"; it cannot answer "would evaluation produce the same derivation?". Answering the latter by evaluating is precisely the cost to be avoided. + +Tecnix therefore makes evaluation itself input-addressed, with the source closure (§4) playing the role for evaluation that the input hash plays for builds. + +Pure evaluation is load-bearing for this construction. The addressing is sound only if every input to evaluation flows through a channel that can be fingerprinted: the pinned git tree, the overlay of uncommitted changes, and the declared arguments. Impure evaluation may consult environment variables, the clock, or arbitrary filesystem paths, none of which a source closure can certify. The persistent cache therefore engages only under `pure-eval`. Impure evaluation continues to function and is still tracked within a run, but its results are never trusted across runs. + +### 2.3 Design constraints + +The implementation was required to satisfy the following constraints simultaneously; they recur throughout the document. + +- **High performance.** Dependency evaluation over all targets must be practical, and warm runs must be faster than cold runs by orders of magnitude. +- **Approximately zero allocations during evaluation.** The tracking machinery executes inside the evaluator's innermost loop. Most of the techniques described below are, at bottom, allocation-avoidance techniques. +- **Minimal effect on parallel evaluation; minimal locking.** The parallel evaluator must continue to scale; tracking may not introduce contended locks on hot paths. +- **Simplicity.** Complexity is admitted only where it changes what the system can do. Speculative machinery is removed rather than retained. +- **Minimal invasiveness.** The codebase is a fork that merges upstream Nix regularly. The tracking system resides in its own module, and upstream files carry only a small, stable set of hooks. +- **Genericity.** The new builtins encode no knowledge of any repository's conventions. They operate on any git repository, which also permits correctness to be tested against small synthetic repositories, independently of the monorepo. + +--- + +## 3. The Central Idea + +### 3.1 Why simpler designs fail + +Consider the obvious design first: while target *T* is being evaluated, log every file read. This design is defeated by memoization. + +```mermaid +sequenceDiagram + participant A as Target A + participant T as shared value
(import lib/common.nix) + participant FS as sources + participant B as Target B + A->>T: force + T->>FS: read lib/common.nix (the only physical read that ever occurs) + T-->>A: finished value (memoized) + B->>T: force + T-->>B: same value — no read occurs + Note over B: a read log records nothing for B +``` + +Target B depends on `lib/common.nix` in every meaningful sense: if that file changes, B's result may change. But B never touches the filesystem, because the graph already contains the answer. A per-target read log therefore under-tracks every target that arrives after the first — failure mode one, in its purest form. + +The remedy follows directly from the model of §2.1. If values are the unit of sharing, then values must carry the dependency information: + +``` +a source read during the production of a value → the value receives a label +a target forces a value → the target inherits the value's label +a target's dependencies → the union of inherited labels + and its own direct reads +``` + +> **Why not…?** +> +> *…trace at the syscall layer (strace, fanotify)?* The same memoization argument applies — the read simply never happens for the second consumer — and syscall traces additionally cannot attribute a read to a *target*, only to a process. +> +> *…hash the whole repository as one input?* That is maximal over-tracking: every commit invalidates every target. Sound and useless (failure mode two). +> +> *…restrict what evaluation may read, and treat the allowed set as the dependency set?* Restriction is not attribution. Knowing what evaluation *may* read says nothing about what a particular target *did* read. +> +> *…isn't this flake evaluation caching?* Flake caching keys an entire evaluation by the hash of its locked inputs — all-or-nothing, and keyed by revision. Tecnix operates per target, and reuses results by validating content rather than trusting keys, which is what makes cross-commit reuse possible (§4.1). + +### 3.2 The correctness contract + +The design carries an executable specification, enforced by the test suite over deliberately adversarial sharing patterns: + +> **Oracle.** Evaluating a target in isolation, in a shared evaluator, in a shared parallel evaluator, and answering from the warm cache must all yield identical dependency sets. Under-tracking and cross-target contamination are correctness defects, not performance trade-offs. + +Because the builtins are generic (§2.3), the oracle is exercised against small synthetic git repositories constructed by the tests themselves. No monorepo is involved in verifying correctness. + +### 3.3 The target-evaluation pipeline + +The remainder of the document descends through the layers of Tecnix target evaluation and caching. It may help to hold the whole pipeline in view first: + +```mermaid +flowchart TB + subgraph API["§9 Public interface"] + B1["tecnixTargetNames
(discovery)"] + B2["tecnixTargets
(evaluation)"] + end + subgraph CACHE["§8 Persistent cache (SQLite)"] + LOOK["stored source closures,
validated against the current tree (§4)"] + end + subgraph EVAL["§6 Tracked evaluation"] + LBL["labels on values
frames on the stack
interned path/set identifiers"] + end + subgraph SRC["§7 Source observation"] + ACC["git tree at pinned commit
+ dirty-checkout overlay"] + end + B1 & B2 --> CACHE + CACHE -- "closure matches:
hit, skip evaluation" --> OUT["result"] + CACHE -- "miss" --> EVAL + EVAL --> SRC + EVAL -- "flatten + fingerprint
fresh closure" --> CACHE + EVAL --> OUT +``` + +--- + +## 4. Source Closures + +The tracked output for a target is its **source closure**: the set of source paths that certify its evaluated result, each paired with a *fingerprint* of that path's current state. + +A note on terminology: this use of "closure" is unrelated to the runtime closure of a store path. Here the word refers to the set of source files that close over an evaluation result — everything that could have influenced it. + +```json +{ + "//app/web:server": { + "app/web/target.nix": "git:8a1f…;mode=100644", + "lib/common.nix": "git:03bc…;mode=100644", + "app/web/vendor": "git:77e2…;mode=040000", + "app/web/local.nix": "absent" + } +} +``` + +Three properties of this structure deserve attention. + +**Fingerprints are git-native.** The fingerprint of a clean path is its git object identifier plus the git file mode observed at that path. The object identifier is content-addressed and cheap to obtain from the repository; the mode is included because Nix source materialization observes executable bits even when file contents are unchanged. The fingerprint of a *directory* currently uses its tree object identifier and tree mode, which changes whenever anything beneath the directory changes; consequently, "the target listed this directory" is captured by a single record that remains conservatively correct. Uncommitted changes extend the fingerprint with a content hash of the modified files: + +``` +git:;mode= clean file or directory, at this commit's tree +git:;mode=;dirty= git base, plus a hash of uncommitted content beneath the path +absent the path does not currently exist +absent;dirty= no git object exists, but uncommitted content does +``` + +This path-level fingerprinting is intentionally conservative. There is room to refine the closure model from "this path's full fingerprint" toward "the specific property of this path that evaluation observed." For example, a directory listing depends on the complete child name/type map it returned — which also proves that no other child names were present — not necessarily on the full tree object ID or on every descendant's content. The current tree fingerprint is exact enough to avoid stale cache hits, but it can over-invalidate; a future closure format could record operation-specific observations such as existence, file type, executable bit, symlink target, directory child name/type sets, or file contents separately. + +**Negative lookups are dependencies.** The entry `"app/web/local.nix": "absent"` records that evaluation checked for a file that was not present — an optional import, or a `pathExists` call. Should the file appear in a later tree, the closure ceases to match and the target is re-evaluated. Omitting this class of dependency is a classic source of staleness bugs; here it is a first-class citizen of the closure. + +**The closure serves as the cache's proof of validity.** Validity is established by content, not by name, as the next section describes. + +### 4.1 Cache reuse: validation rather than trust + +The persistent cache stores bounded historical closure candidates for each target. The essential point is that rows are *not keyed by commit*. A cached answer is reused if and only if one complete stored candidate still matches the current tree — that is, if the current fingerprint of every path in that candidate equals the stored fingerprint: + +```mermaid +flowchart TB + Q["look up target"] --> ROW{"stored candidates
exist?"} + ROW -- no --> MISS + ROW -- yes --> CHK["search candidates:
does one complete closure match?"] + CHK -- "yes" --> HIT["hit: reuse the result
evaluation is skipped entirely"] + CHK -- "no" --> MISS["miss: evaluate
store a fresh candidate"] +``` + +This is the precise sense in which the system works across commits. After a commit, rebase, pull, or merge, any target whose relevant files are byte-identical in the new tree still has a matching closure, and its evaluation is skipped. A typical commit touches a small number of paths in a very large repository, so a typical warm run validates nearly everything and re-evaluates nearly nothing. The cache follows content rather than history — the evaluation-side analogue of the input-addressed builds discussed in §2.2. + +The same property makes the system fail-safe. A row is never believed on the strength of its key; it is believed only when proven. Format changes, corruption, and storage defects therefore all degrade into cache misses, never into wrong answers. In practice, the difference between a cold run and a warm run is the difference between roughly a minute of full tracked evaluation and well under a second — several orders of magnitude, with the gap consisting precisely of "all of evaluation" versus "fingerprint checks and output construction." + +--- + +## 5. A Worked Example + +Consider a small synthetic repository, of the kind the correctness suite itself constructs. The directory `build/resolver/` contains the repository's target-definition entry point, called the *resolver*; its contract is given in §9. + +``` +repo/ +├── build/resolver/resolve.nix # enumerates and evaluates this repo's targets +├── services/ +│ ├── api/target.nix # defines //services/api +│ └── web/target.nix # defines //services/web +└── lib/util.nix # imported by api only +``` + +**Run 1, cold, at commit C₁.** Discovery evaluates the resolver, which lists `services/` and reads each `target.nix` to enumerate targets. Evaluating `//services/api` imports its `target.nix`, which in turn imports `lib/util.nix` and probes for `services/api/local.nix`, an optional override that is not present. All of this is recorded: + +``` +discovery closure: build/resolver/resolve.nix → git:…;mode=100644, services → git:…;mode=040000, + services/api/target.nix → git:…;mode=100644, services/web/target.nix → git:…;mode=100644 +//services/api closure: services/api/target.nix → git:…;mode=100644, lib/util.nix → git:…;mode=100644, + services/api/local.nix → absent +//services/web closure: services/web/target.nix → git:…;mode=100644 +``` + +Subsequent runs, one commit each: + +| commit | change | discovery | `//services/api` | `//services/web` | +|---|---|---|---|---| +| C₂ | edit `lib/util.nix` | **hit** — closure doesn't mention it | **miss** — `lib/util.nix` fingerprint stale; re-evaluated | **hit** | +| C₃ | add `services/api/local.nix` | **hit** | **miss** — `absent` entry no longer matches; re-evaluated, now reading the override | **hit** | +| C₄ | edit `README.md` only | **hit** | **hit** | **hit** | + +C₃ deserves emphasis: a *newly created* file correctly invalidated a target that had merely *looked for* it — the negative-lookup machinery operating as designed. And C₄ shows the steady state: a run whose cost is fingerprint validation and nothing more, independent of repository size or target count. + +--- + +## 6. How Labels Propagate Through the Value Graph + +This section descends one level, to the mechanics of attaching and propagating labels, and to the reasons the mechanism is nearly free. + +### 6.1 A label is one integer + +Storing a set of path strings on every Nix value would be prohibitively expensive in both memory and time. Instead, paths and *sets of paths* are interned into 32-bit identifiers in a per-evaluator, append-only structure: + +``` +"lib/util.nix" → AccessId 7 (interned once, on first sight) +{7} → AccessSetId 3 (canonical: equal sets share one ID) +{7, 9} → AccessSetId 5 +union(3, {9}) → 5 (resolved by a small pair-keyed cache) +``` + +A value's label is thus a single `uint32_t`. Comparing labels is integer comparison; inheriting a label is copying an integer; and taking the union of two labels is a cache lookup keyed by the pair of identifiers. This last case dominates because lazy evaluation overwhelmingly combines exactly two labels at a time, and evaluation is repetitive enough that the same pairs recur constantly. + +The label is stored not in the value itself but in a sparse two-level table keyed by the cell's address: a small constant-initialized directory pointing at demand-paged chunks, one 32-bit slot per 16-byte-aligned cell. `Value` keeps its exact upstream size and layout — a `static_assert` enforces this — and the table's physical footprint is proportional to use: a chunk is allocated only by the first labeled value in its address region, so evaluation that never tracks pays nothing, and tracked evaluation pays about four bytes per value cell. The directory's hot entries cover the entire heap in a handful of cache lines, so reading a label is address arithmetic and two loads. + +### 6.2 The force path: where memoization is answered + +`forceValue` is the innermost operation of the evaluator. When tracking is inactive, Tecnix adds a single thread-local read and a well-predicted branch to it. When tracking is active, the behavior is easiest to see in a concrete trace. Suppose a target's thunk imports a library file: + +``` +force lib thunk frame F₁ opens + read lib/util.nix AccessId 7 recorded in F₁ +lib finishes F₁ interned → SetId 3 = {7}; lib.label ← 3 +force target thunk frame F₂ opens + force lib (finished) lib.label (3) recorded in F₂ ← inheritance: no read occurs + read app/target.nix AccessId 9 recorded in F₂ +target finishes F₂ interned → SetId 5 = {7, 9}; target.label ← 5 +``` + +The general shape: + +```mermaid +flowchart TB + F["forceValue(v)"] --> FIN{"is v already
finished?"} + FIN -- "yes (memoized)" --> INH["read v's label — one integer load —
and record it into the current frame.
This step is the answer to §3.1."] + FIN -- "no (thunk)" --> FRAME["push a stack frame for v's production"] + FRAME --> RUN["evaluate: every read and every
inherited label lands in this frame"] + RUN --> PUB["v finishes: the frame is interned,
published as v's label, and
handed to the parent frame"] +``` + +A **frame** is a small accumulator on the call stack, holding the identifiers of directly-read paths and the labels inherited from forced values. Frames nest with evaluation, forming a stack per thread. When a value finishes, its frame is interned and the resulting set identifier is published as part of the finish operation itself, ordered such that the label is guaranteed visible to other threads before the value appears finished; no consumer can observe a finished value whose label is missing. + +A small number of hooks at the evaluator's value-mutation points maintain the labels' single invariant: *a label describes a cell's current contents, never its history.* Every cell becomes a finished value through a single chokepoint, and the label slot is cleared there unconditionally — whatever the slot held for a previous occupant of that address (a recycled heap cell, a reused stack slot), a finished value starts empty and receives its label from the publish that follows. Unconditional clearing at the one point every finished value passes through is what guarantees that a label is always either empty or accurate. Value copies propagate labels, so provenance survives the evaluator's pervasive movement of values between cells: force, call-time capture, and copy are indistinguishable channels, and contents never move without their label. + +### 6.3 Scopes: labels for work the evaluator caches + +Some provenance is produced in one place and consumed from a cache. The evaluator memoizes more than values: a file is evaluated once and its result cached; an import path is resolved once — through symlinks and `default.nix` selection — and the resolution cached; the resolver is imported once and applied to every target. On a cache hit, the reads that produced the cached artifact do not recur, so the artifact itself must carry them. + +A **scope** is the bracket that makes this work: a region of evaluation — the evaluation of a file, the resolution of an import, the import of the resolver, or a region marked by internal builtins — whose collected accesses are interned into a single set identifier when the region closes. That identifier is published as the produced value's label (or stored beside the cache entry, for caches that do not store values), so a later hit replays the provenance through ordinary label inheritance, exactly as if the consumer had performed the production itself. The resolver's scope label is additionally seeded into every target's context, so every target depends on the resolver's own sources without importing it repeatedly. + +This is one instance of the general rule of §7: any cache capable of skipping a physical read must replay the provenance of that read. + +### 6.4 The cost discipline: approximately zero allocations, approximately zero locks + +The tracking hot path is held to an explicit invariant: + +> Recording, forcing, and publishing allocate memory only on the first sight of a path or set — never on a hit. + +In the steady state, this cashes out as follows. Recording an access is a lock-free hash lookup on a borrowed view of a path string the accessor already holds; no string is constructed. Frames live on the stack, with small inline storage sized to the empirically measured distribution (nearly all frames hold between zero and two entries). Publishing an empty frame touches nothing, and publishing a frame with a single inherited child reuses the child's identifier without consulting the graph at all. Label reads are address arithmetic into the value-label table, whose hot directory entries and quarter-density slot lines stay cache-resident. + +One mutex remains, and it is worth being precise about what it covers. The interning graph's single global mutex guards first-sight path interning and every publish that must actually consult the graph: singleton lookup, pair-union lookup — *including hits* — and full set interning, as well as the per-target flatten at finalization. This is the system's one global serialization point, and therefore its most plausible parallel-scaling bottleneck. Two properties bound it. First, the critical sections are tiny — a hash probe or a small append. Second, its acquisition frequency is proportional to thunks that finish having accumulated real dependencies (one direct access, or two or more inherited children), not to total forces; the empty and single-child fast paths drain the overwhelming majority of publishes before the lock. Should measurement ever show contention here, the structures are append-only by design, so the known remedies — per-thread intern memos, sharded intern maps, lock-free readers over atomically published sizes, a thread-local cache in front of the pair-union table — can be applied without changing the model. + +Process-level parallelism sidesteps this analysis entirely, and for large all-target runs it is likely the better scaling axis. A `nix-eval-jobs`-style driver that shards targets across independent evaluator processes gives each worker its own interning graph — and its own mutex — so no global serialization point exists at any worker count; the cost is that values shared between targets (the standard library, common helpers) are evaluated once per process rather than once overall. The oracle of §3.2 is what makes this sound: closures are identical across evaluation modes, so a target's closure does not depend on which process produced it, and the processes can share one persistent cache because rows are validated by content, never by producer. + +Parallel evaluation receives a stronger treatment: **tracking contexts are thread-confined.** Each parallel work item that evaluates a target owns its context outright — created, recorded into, snapshotted, and destroyed on one thread — so recording never locks and no frame is ever shared between threads. The only cross-thread dependency channel is the published label on a finished value. Tracked evaluation is therefore forbidden from spawning parallel work of its own: the evaluator's detached prefetch sites (`toJSON`'s deep force, `builtins.parallel`) skip prefetching under tracking — the consumer forces sequentially, producing identical results — and the work-item factory fails loudly if anything else tries, because a work item may capture only owned state and a tracking context is a non-owning pointer into another thread's stack. The comparatively expensive step — flattening identifier sets into paths and fingerprinting them — is deferred until all work items have finished, at which point it is a pure function of the recorded snapshots. This is why sequential and parallel evaluation produce identical closures, and why the oracle of §3.2 may legitimately demand that they do. + +--- + +## 7. Observing the Sources + +All tracked reads flow through a single source accessor that composes the git tree at the pinned commit with the working checkout. Routing between the two is decided by a set of dirty files computed once per evaluation from `git status`: + +```mermaid +flowchart LR + READ["readFile / readDirectory /
readLink / pathExists"] --> ACC{"is the path
dirty?"} + ACC -- no --> GIT["git object store
(content at the pinned commit)"] + ACC -- yes --> DISK["working checkout"] + ACC -. every access, including misses .-> REC["record the repo-relative path
into the current frame"] +``` + +Clean paths are served from git's object store — no checkout is required, and evaluation can run against a bare repository. Dirty paths are served from disk. Every access records the repository-relative path. Existence checks are recorded at the primop layer (`pathExists`, `readFileType`), which is how negative lookups enter the closure despite no read occurring. + +In a worldtree sandbox (`tectonix-worldtree-socket` set) there is no git repository to read; the clean tree is instead the daemon's immutable FUSE projection of the pinned commit, with repo-relative paths mapped through the committed manifest to per-zone views. The fingerprint vocabulary is unchanged: directories read their exact committed tree oid from the projection's `user.worldtree.tree-oid` xattr, and regular files read the daemon's `user.worldtree.blob-oid` xattr when it is served — one O(1) metadata read each. Symlinks (which cannot carry user xattrs; `getxattr` would follow the link and answer for its target, a different git object) and files under daemons that do not serve blob oids fall back to hashing their bytes as git blobs — a blob oid is a pure function of content — memoized in memory for the accessor's lifetime, which the immutable projection makes sound: each unique file is hashed at most once per evaluation. Because every mechanism emits identical fingerprint strings, closures produced under one backend validate under the other. Two caveats follow from the projection being zone-granular: committed paths outside every visible zone do not exist in this view and observe as `absent`, and zone-ancestor directories are synthesized with a composite `worldtree-union:` fingerprint outside the git vocabulary (their listing genuinely differs from the full git tree, so cross-backend cache misses there are correct, not conservative). + +The evaluator's own caches require care, since any of them could silently absorb a read. Each — the file-evaluation cache, the import-resolution cache, the source-to-store copy cache — either maintains a separate tracked-domain instance or replays its provenance on a hit. The general rule: *any cache capable of skipping a physical read must replay the provenance of that read.* + +The failure policy throughout is to fail closed. If `git status` fails, evaluation raises an error rather than assuming a clean tree, because the dirty overlay is load-bearing for closure validity. If a path cannot be fingerprinted, evaluation raises an error rather than emitting a partial closure. If evaluation reaches something the closure format cannot represent, it raises an error rather than under-tracking. Every failure mode resolves to a cache miss or a visible error; none resolves to a plausible wrong answer. + +--- + +## 8. The Persistent Cache + +The cache is a single SQLite database with one physical row family: + +``` +DependencyShards(gitDir, resolver, argsKey, shard → multi-target history blob) +``` + +Target discovery (§9) is stored in the same rows, under a reserved key whose candidates carry the discovered target list as a payload; discovery thereby shares the lookup, validation, history, and compaction machinery of ordinary targets rather than maintaining a parallel implementation. The key contains no commit. The `argsKey` column holds the canonical JSON encoding of the caller's `args` value; this is sound as a key because the resolver receives that same value, so results can depend on the arguments only through content that is, by construction, the key.[^ambient-inputs] Validity across trees is established entirely by the closure-matching procedure of §4.1. + +[^ambient-inputs]: Ambient inputs that a pure evaluation can still observe — `builtins.nixVersion`, the store directory — are deliberately *not* part of the cache key. This aligns with Nix's existing flake evaluation cache, whose key is likewise content-only. Changes to the evaluator itself, or to Tecnix semantics, are instead handled by bumping the version in the cache's filename (`tecnix-eval-cache-v1.sqlite`), which orphans old rows wholesale rather than mixing results from two evaluator versions in one database. + +A dependency shard row is therefore a physical container for many bounded per-target proof histories, not a log indexed by commits. Each target candidate in that history is a complete source closure: a map from observed source paths to the fingerprints they had when the target was evaluated. A cache hit means that one whole candidate for that target still matches the current tree. The commit at which the candidate was learned may be useful metadata for ordering or eviction, but it is never proof of validity. + +Sharding is a row-size compromise. With `N` targets, `S` shards, and an average source closure of `P` path/fingerprint pairs, the newest-candidate pair payload in a shard is roughly `(N / S) * P` pair records, plus shared dictionaries. Fewer shards improve dictionary sharing and reduce all-target row count, but make each row larger and make each update rewrite more unrelated target history. More shards make point lookup and update rows smaller, but duplicate side tables and increase all-target row overhead. On the measured 7,254-target `aarch64-darwin` workload, the average closure is about 187 path entries; with 256 shards, one-candidate rows average about 79 KiB and max at about 120 KiB, for about 20 MiB total. That is small enough for fast all-target warm lookup while still sharing path and fingerprint strings across many targets. The shard count should move only with measurement; values in the 128–256 range are the plausible region for this workload, while larger counts mostly trade row size for duplicated dictionaries. + +The first implementation deliberately uses flat newest-first candidate history rather than a decision trie. The common case is that the latest candidate still matches, and the history bound is small. In that case, a trie adds another index to build, validate, and explain without reducing the expensive part of validation: computing the current fingerprint once per unique path. The per-run fingerprint memo already makes repeated path checks cheap. A trie over `(pathId, fingerprintId)` predicates could become worthwhile if measurements show many stale candidates per hot target and repeated pair scans dominate warm lookup, but it is not needed for the initial sharded blob design. + +Dependency and discovery blobs begin with the magic bytes `TXDC` (for "TecniX Dependency Closure"). The magic serves as the format's self-identifier: foreign, corrupted, or out-of-date blobs are rejected immediately, and rejection is a cache miss rather than an error. The blob bytes are laid out so they are already the data structure used by validation: + +``` +header: TXDC magic, fixed format marker, counts, section offsets +targets: offset table + concatenated target identifiers +targetRecords: candidate range for each target +paths: offset table + concatenated repo-relative path bytes +fingerprints: offset table + concatenated fingerprint bytes +payloads: offset table + candidate payload bytes +candidates: pair-stream range + payload id for each historical candidate +pairs: flat (path id, fingerprint id) streams for validation and output +``` + +Each candidate record is one complete historical source closure. Validation searches a target's candidates newest-first. For each pair in a candidate, it asks whether the current fingerprint of that path equals the stored fingerprint. If every pair matches, that candidate is a cache hit and its pair stream is walked directly to construct dependency output. For target discovery, the matching candidate also carries the target-list payload. + +Opening a blob is just bounds-checking the section offsets, counts, and ranges, then viewing the arrays in place. There is no JSON parse for dependencies, no heap object graph, no pointer patching, and no decoded index to build before lookup can begin. Bulk queries load all relevant shard rows in a single range scan and search the requested target histories outside the database lock. The search is lazy: it fingerprints a source path only when the candidate currently being checked asks about that path, and a per-run memo eliminates repeated fingerprint computations for paths shared between shards, candidates, and targets. A warm hit thus bypasses the entire tracked-evaluation stack, paying only for shard loading, candidate scanning, fingerprint comparison, and output construction. + +### 8.1 Cache history and lifecycle + +The cache keeps **bounded historical source closures, not per-commit entries.** The target scale is enough recent history to cover ordinary branch switching and merge-queue churn — on the order of 10–32 historical evals — while keeping lookup fast. The current bound is chosen roughly as the number of distinct source-closure changes a hot target might see in about 24 hours, not as a function of commits per day. Reuse across commits still comes from re-proving a candidate closure against the current tree, not from trusting the commit that produced it. + +A fixed candidate count is the simplest first policy. If measurements show that useful histories are mostly time-shaped rather than count-shaped, a future cache could retain candidates by an approximate 24-hour TTL instead: keep all distinct closures learned in the recent window, then evict by age. That would trade a slightly less predictable row size for a policy closer to the product goal of surviving normal daily branch and merge-queue churn. + +Consequently, the cache grows with the logical key space and the bounded history per target, not with repository history. A target's history lives inside the `DependencyShards` row selected by `(gitDir, resolver, argsKey, shard)`, where the shard is a stable hash of the target name; discovery history lives under a reserved key in the same scheme. Within a target history, inserting a freshly evaluated closure deduplicates identical closure content and evicts old candidates by policy when the bound is reached. + +The important behavioral consequence is that switching between divergent trees need not thrash the cache. If two branches produce different but recently seen closures for the same target, both can remain as candidates, and either branch can hit by proving its candidate against the current tree. If the useful candidate has been evicted, the result is only a cold re-evaluation; eviction is a performance policy, not a correctness policy. + +The unbounded dimensions are the key tuples themselves: each distinct `args` value, resolver path, or repository location materializes its own row set, and abandoned tuples are not currently reclaimed. The validation discipline supplies the operational escape hatch: since no row is ever trusted without proof against the current tree, the database is disposable. Deleting it is always safe and costs cold re-evaluation. + +--- + +## 9. The Public Interface, and the Discovery Subproblem + +Constructing the dependency graph presupposes an answer to a prior question: which targets exist? The target list is not a manifest; it emerges from evaluating repository code that walks directories and reads definition files. Discovery is therefore the same kind of computation as target evaluation — Nix evaluation reading the repository — and it is handled identically. Discovery acquires its own source closure, comprising the paths that determine the target list, including the negative space in which no definitions were found; and it is cached under the same validation discipline, in the same rows as target closures, under a reserved discovery key (§8). If a new target definition appears anywhere discovery looked, the closure ceases to match and discovery re-runs; otherwise, a previously computed target list is provably still current. + +The public interface accordingly consists of two builtins, one per problem: + +```nix +# Discovery: which targets exist? +builtins.tecnixTargetNames { + gitDir = "/path/to/repo/.git"; + resolver = "build/resolver/resolve.nix"; # repo-relative resolver file + rev = ""; + args = { systems = [ "x86_64-linux" ]; }; # opaque; must be JSON-canonicalizable +} +# → [ "opaque-target-id" ... ] + +# Evaluation: what do these targets mean, and what do they depend on? +builtins.tecnixTargets { + ... same ...; + targets = [ "opaque-target-id" ]; + includeDependencies = true; # optional: also return source closures +} +``` + +The entire contract between Tecnix and a repository is one file, the **resolver**. The simplest possible resolver makes the contract plain — real resolvers derive the same structure from the source tree: + +```nix +# build/resolver/resolve.nix — evaluates to a function over the caller's args +args: +let + targets = { + "//services/api" = import ../../services/api/target.nix { inherit args; }; + "//services/web" = import ../../services/web/target.nix { inherit args; }; + }; +in { + allTargetNames = builtins.attrNames targets; # discovery + resolve = id: targets.${id}; # evaluation (values expose a drvPath) +} +``` + +Target-identifier syntax, naming conventions, and indeed the very notion of what constitutes a target are decisions belonging to the resolver, not to Tecnix. The engine understands only two operations: arguments in, identifiers out; and identifier in, evaluated value and source closure out. This genericity is deliberate. It makes the machinery applicable to any git repository, and it is what allows the correctness suite to construct small, disposable repositories — such as the one in §5 — and hold the oracle against them. + +The older, repository-specific builtins (the `unsafeTectonixInternal*` family) remain available for existing consumers. They are intentionally quarantined in a separate source file, are deprecated, and are expected to be removed once their consumers migrate. Those among them that expose checkout-local state refuse to run under tracking, in keeping with the fail-closed policy. + +--- + +## 10. Minimal Invasiveness + +*This section concerns the maintenance of the fork itself and may be skipped by readers interested only in the system's behavior.* + +The codebase merges upstream Nix on a regular basis, which makes the merge-conflict surface of upstream files a first-order maintenance cost. The patch is shaped accordingly: the tracking system resides in its own module, and upstream files carry only narrow, stable hooks. + +| upstream file | contents | +|---|---| +| `value.hh` | three one-line hooks at the finish and copy-assignment points; no members added | +| `eval.hh` | a five-line dispatch shim in `forceValue`; work-item context capture; one private data member (the file is smaller, net, than its upstream counterpart) | +| `eval.cc` | cache-domain selection in `evalFile`; one provenance record in `copyPathToStore` | +| `primops.cc` | two one-line existence-check records | +| libutil / libfetchers | a small, generic "access observation" accessor interface | + +Everything else — the interning structure, the frames, the accessors, the cache, and the builtins — is module-local and unaffected by upstream merges. The module carries no exploratory scaffolding: no tuning knobs, per-call-site instrumentation, or speculative API variants. The working rule is that no optimization is admitted without a profile measurement attributing cost to it, and no complexity without a measured improvement; questions of timing are answered by a profiler rather than by permanent counters. + +--- + +## 11. Limitations + +The following limitations are deliberate and documented. The persistent cache requires `pure-eval` (§2.2). Dirty-file state is captured once per evaluation, so mutating the checkout during a query is outside the contract. Access to the repository root is not representable in the closure format and fails closed. The cache has no key-tuple eviction policy; abandoned `(gitDir, resolver, argsKey)` row sets accumulate until the database is deleted, which is always safe (§8.1). + +**Future work.** In a worldtree sandbox, directory and regular-file fingerprints are already single O(1) xattr reads when the daemon serves `user.worldtree.blob-oid` beside `user.worldtree.tree-oid` (§7). The remaining hash fallback covers symlinks — which cannot carry user xattrs at all — and daemons that predate the blob-oid xattr; it is memoized in memory per evaluation. A daemon-side answer for symlink oids (for example serving the parent's raw tree object, whose `(mode, name, oid)` entries are exactly what libgit2 itself reads) would delete the fallback entirely; because every mechanism emits identical fingerprint strings, that change invalidates no stored closure. Additionally, the projection is zone-granular: committed paths outside every visible zone are not observable historically, and mutable-sandbox dirty discovery still assumes a local `git status`, whose worldtree replacement is the daemon's `scoped.status`. + +--- + +## Glossary + +| term | meaning | +|---|---| +| **source closure** | the set of source paths, with fingerprints, that certify one evaluation result; unrelated to a store path's runtime closure | +| **fingerprint** | a path's current observed state: a git object ID plus git mode, optionally extended with a hash of uncommitted changes, or `absent` | +| **label** | the compact identifier, carried on a value, naming the set of source paths that produced the value's current contents | +| **frame** | a stack-resident accumulator collecting the reads and inherited labels of one value's production | +| **scope** | a bracketed evaluation region whose collected accesses are interned once and published as the produced value's label; how provenance replays through the evaluator's caches | +| **resolver** | the repository-owned Nix function defining discovery (`allTargetNames`) and evaluation (`resolve`) | +| **discovery** | determining which targets exist; the same tracked, cached computation as target evaluation, one level up | +| **negative lookup** | an observed *absence* (e.g. `pathExists` returning false), recorded as a dependency so a new file invalidates correctly | +| **TXDC** | the magic bytes ("TecniX Dependency Closure") identifying the binary closure format in the cache | + +--- + +## Appendix: The System in Summary + +``` +Goal: the exact target dependency graph, and the ability to skip target + evaluation across commits. +Setting: Nix evaluation is a lazily-expanded, memoized graph of values; the + derivation is its output, so evaluation itself must be made + input-addressed by the sources it reads. Pure evaluation is what + makes that addressing sound. +Mechanism: file accesses are recorded alongside the value graph as labels on + values; targets inherit labels from every value they force, so + memoization cannot conceal a dependency. +Product: a target's label set, fingerprinted, is its source closure — + including directories listed and files found absent. +Caching: a stored closure that still matches the tree proves a cached result + valid, at any commit. Discovery is the same problem one level up: + the target list has a closure too. +Cost: interned integers, stack frames, lock-free hits, and no allocations + on the hot path; validation everywhere, trust nowhere. +``` diff --git a/plans/tecnix-target-eval-caching/guardrails.md b/plans/tecnix-target-eval-caching/guardrails.md new file mode 100644 index 0000000000..7aa6aaf600 --- /dev/null +++ b/plans/tecnix-target-eval-caching/guardrails.md @@ -0,0 +1,98 @@ +# Tecnix target-eval cache guardrails + +Use this as a review checklist for source-dependency tracking and target-eval cache changes. A change should satisfy every guardrail below, or explicitly amend the guardrail as part of the same work. + +## Correctness oracle + +- **Dependency output must be identical across evaluation modes.** + - Isolated single-target deps, shared sequential deps, shared parallel deps, warm-cache deps, and target-discovery deps must agree. + +- **No under-tracking. No cross-target contamination. No unproven over-tracking.** + - A target's closure must contain exactly the source observations that can affect that target. + - Today those observations are represented as path fingerprints; future formats may represent narrower observed properties when they can prove the same oracle. + +- **Do not hide tracking bugs by disabling or resetting evaluator behavior.** + - Fix provenance propagation rather than avoiding memoization, sharing, or parallelism. + +## Source-dependency model + +- **Values carry source-deps labels; targets inherit labels.** + - Read logs are not sufficient because Nix memoization can reuse a value without repeating the read that produced it. + +- **Every source observation that can affect evaluation is a dependency.** + - File reads, directory reads, existence checks, symlink reads, and negative lookups must be represented. + - Recording happens at the lowest layer that knows the observation is semantic: reads self-record in the Tecnix source accessor; existence/type checks are recorded by their primop call sites (`pathExists`, `readFileType`) via `recordEvalAccess`, because accessor-level stat tracking would over-track plumbing (symlink/import resolution, store copies). A new primop observing existence or type without a read must record the access itself. + - Present clean paths fingerprint as `git:;mode=`; dirty present paths add `;dirty=`. + - Negative lookups fingerprint as `absent` or `absent;dirty=`. + - Directory listings may eventually be tracked by their complete returned child name/type map, including absence of other child names, rather than a full tree fingerprint; cache validation must still prove the exact observed result. + +- **Value labels must describe current value contents, never stale history.** + - Allocation, overwrite, copy, move, and force/memoization paths must preserve or clear labels correctly. + +- **Tracking contexts are thread-confined.** + - A context is created, recorded into, snapshotted, and destroyed on one thread; the only cross-thread dependency channel is the published label on a finished value. + - Tracked evaluation must not spawn parallel evaluation work: detached prefetch sites skip spawning under tracking, and the work-item factory fails loudly otherwise (work items capture only owned state). + +- **Dirty source state must fail closed.** + - Dirty status is captured as one coherent evaluation snapshot. + - If dirty status or fingerprinting cannot certify the source state, do not accept a cache hit. + +- **Repo-root source access remains unrepresentable unless the closure format grows an explicit representation.** + - Until then, repo-root access must fail closed. + +## Cache validity + +- **A persistent cache hit requires one complete stored closure candidate to match current fingerprints.** + - Partial matches are misses. + - Candidate validation may short-circuit on mismatch, but acceptance requires the whole candidate. + +- **Never trust commit identity for cache acceptance.** + - No cache validity by `rev`. + - No per-commit cache key. + - No per-commit/rev fast path. + - Changed-path or tree-diff data may filter affected-target output, but must not accept cache rows. + +- **Unknown or malformed cache data is a miss.** + - Cache data is an optimization; bad rows must not produce stale answers. + +- **Miss evaluation must learn a fresh proof.** + - A miss evaluates under tracking, finalizes the observed closure, fingerprints it, and stores it for future validation. + +## Cache history and storage + +- **History is bounded candidate history, not repository history.** + - Candidate slots represent distinct source-closure alternatives for a logical key, not commits. + - Duplicate closure content should not consume another slot. + +- **Logical correctness remains per target even when physical storage is shared.** + - Sharding and shared dictionaries must not let one target's closure or payload satisfy another target. + +- **Target discovery is cached with the same proof rules as target dependencies.** + - A stale target list is as wrong as a stale target closure. + +- **The cache hit path must use the stored row directly.** + - SQLite blob bytes should open into a bounds-checked view used for validation. + - Do not rebuild a heap object graph, decoded index, old packed JSON trie, or normalized SQL dependency graph on the hit path. + +- **Public dependency output remains path-to-fingerprint data.** + - Internal storage may use IDs/dictionaries, but the public dependency shape remains compact `path = fingerprint` entries. + +## Public API boundaries + +- **Keep cache history out of the public Tecnix API.** + - Public builtins remain `builtins.tecnixTargetNames` and `builtins.tecnixTargets`. + +- **Dependency-only queries must not force target values.** + - Continue to support `includeDependencies = true; includeTargets = false;`. + +- **A dependency-cache hit is not a target-value cache hit.** + - If callers ask for target values, those values still need evaluation unless a separate value-cache proof exists. + +- **Keep legacy `unsafeTectonixInternal*` compatibility isolated from the new Tecnix cache/history design.** + +## Development cache policy + +- **Do not add migrations for unshipped development cache formats.** + - During development, incompatible local rows should miss or be wiped and rebuilt. + +- **Keep the current development blob marker fixed unless the cache format becomes a shipped compatibility contract.** diff --git a/plans/tecnix-target-eval-caching/walkthrough.md b/plans/tecnix-target-eval-caching/walkthrough.md new file mode 100644 index 0000000000..63f3b44430 --- /dev/null +++ b/plans/tecnix-target-eval-caching/walkthrough.md @@ -0,0 +1,172 @@ +# The Life of a Target Evaluation + +*A step-by-step trace through Tecnix.* + +*This document is a companion to the explainer ("Tecnix: Exact Target Dependencies and Cross-Commit Eval Caching"). Where the explainer describes the system at rest — the problem, the design, and the structures — this document describes it in motion: a caller asks Tecnix to evaluate one target, and we follow the request from the builtin call to the answer. Each data structure is introduced at the point where it first participates, together with the reason it exists and its cost at that moment.* + +*We assume the target's identifier is already known. Discovery (`tecnixTargetNames`) travels the identical road — the same cache question, the same tracked evaluation, the same closure — with the target list as its result rather than a target's value. The example repository throughout is the same synthetic repository as the explainer's §5 worked example.* + +--- + +## 1. The Goal + +Consider the following call: + +```nix +builtins.tecnixTargets { + gitDir = "/path/to/repo/.git"; + resolver = "build/resolver/resolve.nix"; + rev = ""; + args = { systems = [ "x86_64-linux" ]; }; + targets = [ "//services/api" ]; + includeDependencies = true; +} +``` + +The caller expects two things in return: the target's **evaluated value**, and its **source closure** — the fingerprinted set of paths that certify the result, including directories that were listed and files that were found absent. The caller further expects the call to be inexpensive whenever nothing relevant has changed, even if the current commit differs from the one at which the target was last evaluated. + +## 2. The Journey at a Glance + +Every request follows the same path, which contains two significant branch points. The numbered stations correspond to Steps ① through ⑤ in the sections that follow: + +```mermaid +flowchart TB + CALL["① the call:
pin repository context,
derive the cache key"] --> CACHE{"② the cache question:
does a stored closure exist,
and does it still match the tree?"} + CACHE -- "yes: cache hit" --> OUT1["answer built from the stored row
evaluation does not run"] + CACHE -- "no: cache miss" --> EVAL["③ tracked evaluation begins:
resolver, context, dirty overlay"] + EVAL --> LOOP["④ inside evaluation:
many forceValue calls, each either a
memoization hit or a memoization miss"] + LOOP --> FIN["⑤ finishing up:
snapshot, flatten, fingerprint,
store, answer"] +``` + +The first branch — cache hit or miss — determines whether evaluation runs at all. The second branch is taken a very large number of times *within* evaluation: each time a value is demanded, it has either been computed already (a memoization hit) or it has not (a memoization miss). The remainder of this document walks the path in order. At each branch we take the expensive side, so that every mechanism is visited, and we note what the inexpensive side would have cost instead. + +## 3. Step ①: The Call + +Argument handling is largely routine. Two decisions made at this stage matter later. + +First, **the repository context is pinned.** The `gitDir`, `rev`, and checkout path configure the evaluator's source accessors, and they do so exactly once per evaluator instance. A second call with a different `rev` produces an error rather than a silent reconfiguration. The reason is that the accessors, fingerprints, and cached content constructed downstream are all built lazily against a single commit; permitting reconfiguration would allow content from two commits to mix without any indication that it had. + +Second, **the `args` value becomes part of the cache key.** It is converted to a canonical JSON encoding, called the `argsKey`. This is sound because the resolver receives the same value: results can depend on the arguments only through content that is, by construction, the key. It is worth observing what the cache key does *not* contain: the commit. Validity across commits is established by proof rather than by key, as the next step describes. + +## 4. Step ②: The Cache Question + +> **Structure: `TecnixEvalCache`.** A SQLite database holding shard rows keyed by `(gitDir, resolver, argsKey, shard)`. Each shard row contains bounded source-closure histories for the targets assigned to that shard. It exists because skipping evaluation requires remembering what would certify the skipped result. + +The shard containing `//services/api` is loaded. A single target uses a point lookup for its shard; when many targets are requested, one range scan retrieves the relevant shard rows, and their validation proceeds outside the database lock. Each row's blob begins with the magic bytes `TXDC` (explainer §8). + +> **Structure: the `TXDC` blob and its `DependencyBlobView`.** The blob is not a serialization that is parsed into objects; it is itself the data structure. Opening a view validates every section, offset, and range once. The view then exposes target, path, fingerprint, and payload dictionaries plus flat candidate and pair records as borrowed spans over the row's own bytes. A malformed or out-of-date row fails to open and is treated as a cache miss rather than an error. + +Validation checks historical candidates newest-first until one complete closure still matches the current tree: + +```mermaid +sequenceDiagram + participant P as lookup + participant BV as blob view + participant FP as fingerprint memo + participant GIT as accessor / git + P->>BV: open(row bytes)
validate sections once + loop candidates newest-first + BV-->>P: candidate pair range + loop candidate pairs + BV-->>P: path and stored fingerprint + P->>FP: current fingerprint of path? + alt already computed this run + FP-->>P: memoized result + else first sight this run + FP->>GIT: git object ID and mode at path
plus dirty-overlay hash if modified + GIT-->>FP: fingerprint, memoized + end + P->>P: compare with stored fingerprint + end + end + P->>P: first fully matching candidate is a hit +``` + +> **Structure: the per-run fingerprint memo.** A thread-local table from path to current fingerprint. It exists because the closures of many targets, and the historical candidates for one target, share most of their paths; each unique path is fingerprinted once per run — a git object-identifier and file-mode read, which is itself inexpensive — and every subsequent occurrence is a hash lookup. Validation cost therefore scales with the number of *unique* paths candidate scanning asks about, not with the total number of historical closure entries. (In a worldtree sandbox the clean tree is the daemon's immutable FUSE projection rather than a git repository: directory fingerprints come from a tree-oid xattr, and regular-file fingerprints from a blob-oid xattr when the daemon serves one, falling back to hashing content as a git blob, memoized in memory for the run — identical fingerprint strings, identical validation; see the explainer's §7.) + +If one candidate fully matches, the journey ends here: that candidate identifies the matching historical closure, the output is built directly from its pair stream, and everything described in the remaining sections is skipped — the resolver, every force, every frame, all interning. This asymmetry accounts for the difference of several orders of magnitude between warm and cold runs. The `absent` entries participate in the search as well: a path that the target once probed and did not find is checked to still be absent, so a newly created file fails the proof in exactly the way an edited one does. + +For the purposes of this walkthrough, suppose no complete candidate matches. The lookup is a miss, and evaluation must run — under observation. + +## 5. Step ③: Tracked Evaluation Begins + +> **Structure: `TrackingContext`.** One per target, owning a root accumulator. It exists so that each target's dependencies remain isolated from those of every other target sharing the evaluator. +> +> **Structure: `TecnixThreadState`.** A single thread-local record holding the active context, the top of the frame stack, and the current publish target. It exists so that tracking state is reachable from the evaluator's innermost loop at the cost of one thread-local access, without threading arguments through upstream code. + +Two preparations precede the target itself. + +**The resolver is imported, once, under a scope.** The repository's `resolve.nix` is evaluated inside a *source-deps scope*: a bracketed region whose collected accesses are interned into a single set identifier and published as the resolver value's label (explainer §6.3). That identifier is then seeded into every target's context, so that all targets depend on the resolver's own sources without importing it repeatedly. + +**The dirty overlay is established.** A single `git status` invocation partitions the tree: clean paths will be served from the git object store at the pinned commit, and modified paths from disk. If `git status` fails, evaluation fails. Assuming a clean tree in that situation would allow stale rows to validate against a tree that does not reflect reality, so the failure is made visible instead. + +The resolver is then applied to `"//services/api"`, the resulting value's `drvPath` is forced, and control descends into the evaluator. + +## 6. Step ④: Inside Evaluation + +Every value demanded during evaluation passes through `forceValue`, which under tracking asks a single question: has this value been computed already? + +### 6.1 A memoization miss: producing a value + +Suppose the target imports `lib/util.nix`, and nothing has evaluated that file yet. + +> **Structure: `TrackedSourceDepsFrame`.** A small, stack-resident accumulator opened for the value under production, holding the identifiers of paths read and labels inherited while producing it. It exists because something must delimit "the reads that occurred while producing this value," and a stack mirrors the shape of evaluation exactly. Its inline storage covers the empirically common case of zero to two entries, so no heap allocation occurs. +> +> **Structure: `EvalSourceAccessSetGraph`.** The interning structure: a path becomes a 32-bit `AccessId`; a set of paths becomes a canonical 32-bit `AccessSetId`, with equal sets sharing one identifier. It exists because everything downstream must operate on integers rather than strings: comparing labels is integer comparison, and inheriting a label is copying an integer. + +```mermaid +sequenceDiagram + participant E as forceValueTracked + participant FR as frame F₁ + participant ACC as source accessor + participant G as interning graph + participant V as value cell + E->>FR: push frame
set publish target to v + E->>V: v.force() + V->>ACC: readFile("lib/util.nix") + ACC->>G: internAccess(path view) + G-->>ACC: AccessId 7
lock-free for previously seen paths + ACC->>FR: record AccessId 7 + V->>V: finish()
intern frame as SetId 3
set label to 3
mark finished + V->>FR: hand SetId 3 to the parent frame + E->>E: pop frame +``` + +The costs along this path are as follows. The accessor records a borrowed view of the repository-relative path, so no string is constructed. `internAccess` is a lock-free hash lookup for any path seen before; the graph mutex is taken only on a path's first sighting in the process. Appending to the frame is an integer store. When the value finishes, the frame is interned through fast paths matched to the common frame shapes: an empty frame publishes nothing, and a frame containing a single inherited child reuses that child's identifier — neither consults the graph. A frame with two children is resolved by the **pair-union cache**, a table from pairs of set identifiers to their union, which exists because lazy evaluation combines exactly two labels far more often than any other number, and because evaluation is repetitive enough that the same pairs recur constantly. The pair-union and singleton lookups do take the graph's global mutex — the system's one global serialization point; its cost profile and the process-level alternative are discussed in the explainer's §6.4. + +One ordering detail deserves attention: the label is stored on the value *before* the cell is marked finished. Consequently no thread — including a parallel worker awaiting this thunk — can observe a finished value whose label is missing. + +> **Structure: the value-label table.** A sparse two-level table holding one 32-bit slot per 16-byte-aligned value cell: a constant-initialized directory indexed by the top address bits, pointing at demand-paged chunks that are installed by the first labeled value in their region. It exists so that labels cost nothing when tracking is off and about four bytes per cell when it is on, while `Value` keeps its exact upstream layout — a `static_assert` guards the size so any change revisits the decision explicitly. + +### 6.2 A memoization hit: inheriting a label + +Now the same import is demanded again, whether later in this target or from a different target altogether. The value is finished. + +``` +forceValue(v): v is finished + → load v's label (one atomic integer read) + → record it into the frame (one integer store) +``` + +This is the complete flow, and it is the flow that executes most often — many millions of times in a large evaluation. No file is read, no frame is pushed, and nothing is allocated or locked; yet the dependency is fully inherited. This is the resolution of the memoization problem (explainer §3.1), reduced to two integer operations. When the recording frame later publishes, the union of the inherited label with the frame's other contents is, in the common case, a single lookup in the pair-union cache. + +## 7. Step ⑤: Finishing Up + +Evaluation of the target completes. The context's root frame now holds, as integers, everything the target touched, whether directly or by inheritance. + +**Snapshot.** The root frame is copied: a vector of identifiers and nothing more. Tracking contexts are thread-confined — when many targets evaluate in parallel, each executor work item owns its target's context outright, so recording never locks and the snapshot is an ordinary read on the owning thread; worker threads handle only integers. + +**Flatten and fingerprint.** After all workers have finished, each snapshot is flattened — a generation-stamped traversal of the interning graph that yields unique path identifiers and then paths — and each path is fingerprinted through the same per-run memo used by the cache lookup. Present clean paths include both git object ID and git mode; dirty paths add the overlay hash. This step is deliberately deferred and pure: it is a function of the snapshots alone, which is why sequential and parallel evaluation produce identical closures, and why the test suite's oracle is entitled to require that they do. If any path cannot be fingerprinted, this step raises an error rather than emitting a partial closure, since a partial closure stored today becomes a stale cache hit tomorrow. + +**Store and answer.** The closure is inserted into the row's `TXDC` history — batched into a single transaction when several targets missed. Identical closures are deduplicated, and old candidates may be evicted when the bounded history is full; the cache's size and lifecycle characteristics are discussed in the explainer (§8.1). The caller receives the target's value and, because `includeDependencies` was set, its closure. The next request for this target, on this commit or on any future commit in which these paths are unchanged, takes the short branch at Step ②. + +## 8. The Road, Costed + +| where | branch | frequency | steady-state cost | +|---|---|---|---| +| Step ② | cache hit | the common case across commits | row load, plus one fingerprint per *unique* path | +| Step ② | cache miss | changed targets only | everything below, once; then cached | +| Step ④ | memoization hit | many millions per evaluation | approximately two integer operations | +| Step ④ | memoization miss | once per thunk | integer operations, plus one interning per *new* path or set | + +The table exhibits the design's economics: the more frequently a branch executes, the fewer structures it touches. Allocation is confined to first-sight events — a new path, a new set, a new closure — and locking to those events plus the union-producing publishes, all of which occur in proportion to *change* and to distinct dependency structure, while the hit paths, which occur in proportion to *scale*, touch almost nothing. The zero-allocation discipline described in the explainer is therefore visible here not as an optimization applied afterward, but as the organizing principle of the architecture. diff --git a/scripts/measure-tecnix-eval.sh b/scripts/measure-tecnix-eval.sh new file mode 100755 index 0000000000..2c4ada5f00 --- /dev/null +++ b/scripts/measure-tecnix-eval.sh @@ -0,0 +1,665 @@ +#!/usr/bin/env bash +# Evaluate Tecnix targets. The Tecnix SQLite eval cache is disabled by default; +# pass --eval-cache to measure the pure-eval cached path. +# +# By default this measures builtins.tecnixTargets by producing target records +# with drvPaths. Other modes are available for target values, dependency +# tracking, and target-name discovery. + +set -euo pipefail + +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P) +repo_root=$(cd "$script_dir/.." && pwd -P) + +default_world_git="$HOME/world/git" + +usage() { + cat >&2 <<'EOF' +Usage: measure-tecnix-eval.sh [options] + +Options: + --git-dir PATH World git directory (default: $GIT_DIR, ~/world/git if it exists, or WORLD/.git) + --world PATH World checkout path, used only to derive git-dir/rev unless --checkout-path is set + --checkout-path PATH Pass checkoutPath to Tecnix args for dirty-overlay/source-available eval + --resolver PATH Repo-relative resolver file (default: $RESOLVER or system/tectonix/resolve.nix) + --system SYSTEM System string (default: $SYSTEM or builtins.currentSystem) + --platform SYSTEM Platform/system for --mode platform-target-dependencies or + platform-target-dependency-paths. May be repeated. + --rev REV Git revision to evaluate (default: $REV or git HEAD from git-dir/world) + --target TARGET Target to evaluate. May be repeated. If omitted, discovers all target names. + --include-dependencies + Add a dependencies attr to target-records output + --mode MODE target-records-jsonl, target-records, target-drvs, targets, + target-dependencies, target-dependency-paths, + platform-target-dependencies, + platform-target-dependency-paths, or target-names + (default: target-records-jsonl) + --nix PATH nix executable (default: $NIX_BIN, ./build/src/nix/nix, or nix) + --eval-cache Enable the Tecnix SQLite eval cache (implies --pure) + --no-eval-cache Disable the Tecnix SQLite eval cache (default) + --parallel Enable parallel eval workers with --eval-cores 0 + --eval-cores N Pass --eval-cores N (0 means auto; implies --parallel) + -v, --verbose Pass --verbose to nix (may be repeated) + --debug Pass --debug to nix + --log-format FORMAT Pass --log-format FORMAT to nix + --pure Use --pure-eval instead of --impure + --impure Use --impure (default; matches tec eval) + --print-result Print the final eval result to stdout (default) + --output PATH Write the final eval result to PATH instead of stdout + --discard-output Force the result but redirect stdout to /dev/null + -h, --help Show this help + +Default behavior measures builtins.tecnixTargets by producing one JSON target +record per line ({ target, drvPath }) for each selected target. Pass +--include-dependencies to add source dependencies to each record. Dependency +modes use builtins.tecnixTargets with includeDependencies enabled. Use --mode +target-dependency-paths for the compact target -> [repo-relative path] graph, or +--mode target-dependencies for the target -> { path = fingerprint; } graph: + nix eval --raw --lazy-trees --impure \ + --extra-experimental-features 'nix-command parallel-eval wasm-builtin' \ + --option tecnix-eval-cache false + +By default the final expression deepSeqs the discovered target list and selected +result, then prints the result. Use --discard-output for timing-only runs or +--output PATH to save it. The Tecnix eval cache is disabled by default, but normal +in-process evaluator caches remain enabled where the evaluator chooses to use +them. Pass --eval-cache to enable the Tecnix SQLite eval cache and run with +--pure-eval, which is required for cache hits. +EOF +} + +world=${WORLD:-} +checkout_path=${CHECKOUT_PATH:-} +git_dir=${GIT_DIR:-} +resolver=${RESOLVER:-system/tectonix/resolve.nix} +system=${SYSTEM:-} +rev=${REV:-} +nix_bin=${NIX_BIN:-} +mode=${MODE:-target-records-jsonl} +eval_cores=${EVAL_CORES:-} +pure_eval=0 +print_result=1 +output_path=${OUTPUT:-} +include_dependencies=${INCLUDE_DEPENDENCIES:-0} +tecnix_eval_cache=${TECNIX_EVAL_CACHE:-0} +targets=() +platforms=() +nix_log_args=() + +while [[ $# -gt 0 ]]; do + case "$1" in + --git-dir) + git_dir=$2 + shift 2 + ;; + --world) + world=$2 + shift 2 + ;; + --checkout-path) + checkout_path=$2 + shift 2 + ;; + --resolver) + resolver=$2 + shift 2 + ;; + --system) + system=$2 + shift 2 + ;; + --platform) + platforms+=("$2") + shift 2 + ;; + --rev) + rev=$2 + shift 2 + ;; + --target) + targets+=("$2") + shift 2 + ;; + --mode) + mode=$2 + shift 2 + ;; + --include-dependencies) + include_dependencies=1 + shift + ;; + --nix) + nix_bin=$2 + shift 2 + ;; + --eval-cache) + tecnix_eval_cache=1 + pure_eval=1 + shift + ;; + --no-eval-cache) + tecnix_eval_cache=0 + shift + ;; + --parallel) + eval_cores=0 + shift + ;; + --eval-cores) + eval_cores=$2 + shift 2 + ;; + -v|--verbose) + nix_log_args+=(--verbose) + shift + ;; + --debug) + nix_log_args+=(--debug) + shift + ;; + --log-format) + nix_log_args+=(--log-format "$2") + shift 2 + ;; + --pure) + pure_eval=1 + shift + ;; + --impure) + pure_eval=0 + shift + ;; + --print-result) + print_result=1 + output_path= + shift + ;; + --output) + output_path=$2 + print_result=0 + shift 2 + ;; + --discard-output) + print_result=0 + output_path= + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + usage + exit 2 + ;; + esac +done + +case "$mode" in + target-records|target-records-jsonl|target-dependencies|target-dependency-paths|platform-target-dependencies|platform-target-dependency-paths|targets|target-drvs|target-names) ;; + *) + echo "invalid --mode '$mode' (expected target-records, target-records-jsonl, target-dependencies, target-dependency-paths, platform-target-dependencies, platform-target-dependency-paths, targets, target-drvs, or target-names)" >&2 + exit 2 + ;; +esac + +if [[ ${#platforms[@]} -gt 0 && "$mode" != platform-target-dependencies && "$mode" != platform-target-dependency-paths ]]; then + echo "--platform only applies to --mode platform-target-dependencies or platform-target-dependency-paths" >&2 + exit 2 +fi + +if [[ "$include_dependencies" == 1 ]]; then + case "$mode" in + target-records|target-records-jsonl) ;; + *) + echo "--include-dependencies only applies to target-records and target-records-jsonl modes" >&2 + exit 2 + ;; + esac +fi + +case "$tecnix_eval_cache" in + 1|true|yes|on) + tecnix_eval_cache=1 + ;; + 0|false|no|off) + tecnix_eval_cache=0 + ;; + *) + echo "invalid TECNIX_EVAL_CACHE/--eval-cache value '$tecnix_eval_cache'" >&2 + exit 2 + ;; +esac + +if [[ "$tecnix_eval_cache" == 1 && "$pure_eval" != 1 ]]; then + echo "--eval-cache requires pure evaluation; pass --pure or let --eval-cache imply it, and do not also pass --impure" >&2 + exit 2 +fi + +if [[ -z "$nix_bin" ]]; then + if [[ -x "$repo_root/build/src/nix/nix" ]]; then + nix_bin="$repo_root/build/src/nix/nix" + else + nix_bin=nix + fi +fi + +if [[ -n "$world" ]]; then + world=$(cd "$world" && pwd -P) +elif [[ -d "$HOME/world" ]]; then + world=$(cd "$HOME/world" && pwd -P) +fi + +if [[ -z "$git_dir" ]]; then + if [[ -d "$default_world_git" ]]; then + git_dir=$default_world_git + elif [[ -n "$world" ]]; then + git_dir=$(git -C "$world" rev-parse --absolute-git-dir) + else + echo "could not determine git dir; pass --git-dir or --world" >&2 + exit 2 + fi +fi + +git_dir=$(cd "$git_dir" && pwd -P) + +if [[ -z "$rev" ]]; then + if rev=$(git --git-dir "$git_dir" rev-parse HEAD 2>/dev/null); then + : + elif [[ -n "$world" ]]; then + rev=$(git -C "$world" rev-parse HEAD) + else + echo "could not determine rev; pass --rev" >&2 + exit 2 + fi +fi + +if ! git --git-dir "$git_dir" cat-file -e "$rev:$resolver" 2>/dev/null; then + cat >&2 <"$target_names_json" + python3 - "$target_names_json" <<'PY' +import json +import sys +with open(sys.argv[1]) as f: + for target in json.load(f): + print(target) +PY + rm -f "$target_names_json" +} + +targets_discovered_separately=0 +if [[ ${#targets[@]} -eq 0 ]]; then + case "$mode" in + target-dependencies|target-dependency-paths) + # Discover target names in a separate Nix process for dependency + # modes. Discovering names in the same evaluator can pre-force broad + # resolver/module values and distort the per-target dependency graph + # that the second phase is trying to measure. + while IFS= read -r target; do + targets+=("$target") + done < <(discover_target_names_for_system "$system") + targets_discovered_separately=1 + ;; + esac +fi + +include_dependencies_nix=false +if [[ "$include_dependencies" == 1 ]]; then + include_dependencies_nix=true +fi + +system_label=$system + +if [[ "$mode" == platform-target-dependencies || "$mode" == platform-target-dependency-paths ]]; then + if [[ ${#platforms[@]} -eq 0 ]]; then + platforms=("$system") + fi + + platforms_expr=$(target_list_expr "${platforms[@]}") + + if [[ ${#targets[@]} -gt 0 ]]; then + target_refs_expr=$(target_ref_list_expr_for_platforms) + targets_label="${#targets[@]} explicit target ref(s)" + else + target_refs_expr='builtins.tecnixTargetNames baseArgs' + targets_label="all discovered target refs for platforms (${platforms[*]})" + fi + + if [[ "$mode" == platform-target-dependencies ]]; then + result_expr='targetDependencyPathSets' + else + result_expr='targetDependencyPaths' + fi + seq_expr='builtins.deepSeq platforms (builtins.deepSeq targetRefs (builtins.deepSeq result result))' + discard_seq_expr='builtins.deepSeq platforms (builtins.deepSeq targetRefs (builtins.deepSeq result "ok"))' + + if [[ "$print_result" == 1 || -n "$output_path" ]]; then + final_expr=$seq_expr + eval_output_flag=--json + else + final_expr=$discard_seq_expr + eval_output_flag=--raw + fi + + expr=$(cat <&2 <&2 + printf '%q ' "${nix_log_args[@]}" >&2 + printf '\n' >&2 +fi + +if [[ -n "$checkout_path" ]]; then + echo " checkoutPath: $checkout_path" >&2 +fi + +nix_args=( + eval + "$eval_output_flag" + --lazy-trees + --extra-experimental-features 'nix-command parallel-eval wasm-builtin' + --option tecnix-eval-cache "$tecnix_eval_cache_nix" + --option tectonix-git-dir "$git_dir" + --option tectonix-git-sha "$rev" +) + +if [[ ${#nix_log_args[@]} -gt 0 ]]; then + nix_args+=("${nix_log_args[@]}") +fi + +nix_args+=(--expr "$expr") + +if [[ -n "$checkout_path" ]]; then + nix_args+=(--option tectonix-checkout-path "$checkout_path") +fi + +if [[ -n "$eval_cores" ]]; then + nix_args+=(--eval-cores "$eval_cores") +fi + +if [[ "$pure_eval" == 1 ]]; then + nix_args+=(--pure-eval) +else + nix_args+=(--impure) +fi + +if [[ -n "$output_path" ]]; then + mkdir -p "$(dirname "$output_path")" + exec /usr/bin/time -p "$nix_bin" "${nix_args[@]}" >"$output_path" +elif [[ "$print_result" == 1 ]]; then + exec /usr/bin/time -p "$nix_bin" "${nix_args[@]}" +else + exec /usr/bin/time -p "$nix_bin" "${nix_args[@]}" >/dev/null +fi diff --git a/src/libexpr-tests/meson.build b/src/libexpr-tests/meson.build index 65c88a744b..7e4ab96ea9 100644 --- a/src/libexpr-tests/meson.build +++ b/src/libexpr-tests/meson.build @@ -65,6 +65,8 @@ sources = files( 'nix_api_value_internal.cc', 'primops.cc', 'search-path.cc', + 'tecnix-dependency-tracking.cc', + 'tecnix-worldtree.cc', 'tectonix.cc', 'trivial.cc', 'value/context.cc', @@ -105,6 +107,7 @@ if get_option('benchmarks') 'dynamic-attrs-bench.cc', 'get-drvs-bench.cc', 'regex-cache-bench.cc', + 'tecnix-eval-bench.cc', ) benchmark_exe = executable( diff --git a/src/libexpr-tests/primops.cc b/src/libexpr-tests/primops.cc index ca13f64875..e1bf149565 100644 --- a/src/libexpr-tests/primops.cc +++ b/src/libexpr-tests/primops.cc @@ -60,6 +60,12 @@ TEST_F(PrimOpTest, abort) ASSERT_THROW(eval("abort \"abort\""), Abort); } +TEST_F(PrimOpTest, breakForcesThunkBeforeReturning) +{ + auto v = eval("builtins.break (1 + 2)"); + ASSERT_THAT(v, IsIntEq(3)); +} + TEST_F(PrimOpTest, ceil) { auto v = eval("builtins.ceil 1.9"); diff --git a/src/libexpr-tests/tecnix-dependency-tracking.cc b/src/libexpr-tests/tecnix-dependency-tracking.cc new file mode 100644 index 0000000000..44f89c569a --- /dev/null +++ b/src/libexpr-tests/tecnix-dependency-tracking.cc @@ -0,0 +1,411 @@ +#include +#include +#include +#include +#include +#include + +#include + +#include "nix/expr/eval.hh" +#include "nix/expr/parallel-eval.hh" +#include "nix/expr/tecnix/access-set-graph.hh" +#include "nix/expr/tests/libexpr.hh" +#include "nix/util/serialise.hh" + +namespace nix { + +class ScopedEnv +{ + std::string name; + std::optional oldValue; + +public: + ScopedEnv(std::string name, std::string value) + : name(std::move(name)) + { + if (auto * existing = std::getenv(this->name.c_str())) + oldValue = std::string(existing); + setenv(this->name.c_str(), value.c_str(), 1); + } + + ~ScopedEnv() + { + if (oldValue) + setenv(name.c_str(), oldValue->c_str(), 1); + else + unsetenv(name.c_str()); + } +}; + +static bool containsPath(const std::vector & paths, std::string_view expected) +{ + return std::find(paths.begin(), paths.end(), expected) != paths.end(); +} + +static std::vector +flattenFrame(const ref & graph, const TrackedSourceDepsFrame & frame) +{ + std::vector direct( + frame.directSourceAccessSetAccesses.begin(), frame.directSourceAccessSetAccesses.end()); + std::vector children(frame.childSourceAccessSets.begin(), frame.childSourceAccessSets.end()); + return graph->flatten(direct, children); +} + +class TrackingMemorySourceAccessor : public SourceAccessor +{ + std::map files; + + static std::string key(const CanonPath & path) + { + return path.isRoot() ? std::string{} : std::string(path.rel()); + } + + void track(const CanonPath & path) const + { + if (path.isRoot()) + return; + if (auto * ctx = currentTecnixThreadState.trackingContext) + ctx->recordAccess(key(path)); + } + +public: + explicit TrackingMemorySourceAccessor(std::map files) + : files(std::move(files)) + { + setPathDisplay("memory:"); + } + + bool tracksEvalAccesses(const CanonPath &) override + { + return true; + } + + void recordEvalAccess(const CanonPath & path) override + { + track(path); + } + + void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override + { + track(path); + auto it = files.find(key(path)); + if (it == files.end()) + throw FileNotFound("path '%s' does not exist", path.abs()); + sizeCallback(it->second.size()); + sink(it->second); + } + + std::optional maybeLstat(const CanonPath & path) override + { + if (path.isRoot()) + return Stat{.type = tDirectory}; + + auto pathKey = key(path); + if (auto it = files.find(pathKey); it != files.end()) + return Stat{.type = tRegular, .fileSize = it->second.size()}; + + auto dirPrefix = pathKey + "/"; + for (auto & [file, _] : files) + if (file.starts_with(dirPrefix)) + return Stat{.type = tDirectory}; + + return std::nullopt; + } + + DirEntries readDirectory(const CanonPath & path) override + { + track(path); + if (!maybeLstat(path) || maybeLstat(path)->type != tDirectory) + throw FileNotFound("path '%s' does not exist", path.abs()); + + DirEntries entries; + auto dirKey = key(path); + auto prefix = dirKey.empty() ? std::string{} : dirKey + "/"; + for (auto & [file, _] : files) { + if (!file.starts_with(prefix)) + continue; + auto rest = file.substr(prefix.size()); + auto slash = rest.find('/'); + auto name = slash == std::string::npos ? rest : rest.substr(0, slash); + entries.emplace(name, slash == std::string::npos ? DirEntry{tRegular} : DirEntry{tDirectory}); + } + return entries; + } + + std::string readLink(const CanonPath & path) override + { + track(path); + throw NotASymlink("path '%s' is not a symlink", path.abs()); + } +}; + +class ScopedTrackingContext +{ + ActiveTrackingContext active; + +public: + explicit ScopedTrackingContext(TrackingContext & context) + : active(context) + { + } +}; + +class ScopedTrackedValueForceFrame +{ + TrackedSourceDepsFrame * oldFrame; + const void * oldPublishValue; + +public: + explicit ScopedTrackedValueForceFrame(TrackedSourceDepsFrame & frame) + : oldFrame(currentTecnixThreadState.sourceDepsFrame) + , oldPublishValue(currentTecnixThreadState.valueDependencyPublishValue) + { + currentTecnixThreadState.sourceDepsFrame = &frame; + currentTecnixThreadState.valueDependencyPublishValue = frame.value; + } + + ~ScopedTrackedValueForceFrame() + { + currentTecnixThreadState.sourceDepsFrame = oldFrame; + currentTecnixThreadState.valueDependencyPublishValue = oldPublishValue; + } +}; + +class TecnixValueProvenanceTest : public LibExprTest +{ +protected: + EvalSourceAccessSetId publishSingleAccess(Value & value, std::string_view path) + { + enableSourceAccessSetTracking(state); + auto access = trackedSourceAccessSetGraph(state)->internAccess(std::string(path)); + std::array direct{access}; + return publishTrackedSourceAccessSetDependencies( + *trackedSourceAccessSetGraph(state), + value, + std::span(direct), + std::span()); + } +}; + +TEST(TecnixSourceAccessSetGraph, disabledGraphReturnsEmptyIdsAndNoFlattenedAccesses) +{ + EvalSourceAccessSetGraph graph; + + auto access = graph.internAccess("a.nix"); + EXPECT_EQ(access, emptyEvalSourceAccessId); + + std::vector direct{access}; + std::vector children; + auto set = graph.internAccessSet(direct, children); + EXPECT_EQ(set, emptyEvalSourceAccessSetId); + EXPECT_TRUE(graph.flatten(direct, {set}).empty()); +} + +TEST(TecnixSourceAccessSetGraph, accessIdsAreInternedByPath) +{ + EvalSourceAccessSetGraph graph; + graph.enable(); + + auto read = graph.internAccess("a.nix"); + auto readAgain = graph.internAccess("a.nix"); + auto other = graph.internAccess("b.nix"); + + EXPECT_NE(read, emptyEvalSourceAccessId); + EXPECT_EQ(read, readAgain); + EXPECT_NE(read, other); + + EXPECT_EQ(graph.access(read), "a.nix"); +} + +TEST(TecnixSourceAccessSetGraph, internedSetsAreCanonicalAndFlattenTransitiveClosures) +{ + EvalSourceAccessSetGraph graph; + graph.enable(); + + auto a = graph.internAccess("a.nix"); + auto b = graph.internAccess("b.nix"); + auto c = graph.internAccess("c.nix"); + + auto setAB = graph.internAccessSet(std::vector{b, a, a}, {}); + auto setBA = graph.internAccessSet(std::vector{a, b}, {}); + EXPECT_EQ(setAB, setBA); + + auto setC = graph.internAccessSet(std::vector{c}, {}); + auto setABC = + graph.internAccessSet(std::vector{a}, std::vector{setAB, setC}); + auto setABCAgain = graph.internAccessSet(std::vector{c, b, a}, {}); + EXPECT_EQ(setABC, setABCAgain); + + auto flattened = graph.flatten({}, {setABC}); + EXPECT_EQ(flattened, (std::vector{"a.nix", "b.nix", "c.nix"})); + + auto directPlusEdge = graph.flatten(std::vector{c, a}, {setAB}); + EXPECT_EQ(directPlusEdge, (std::vector{"c.nix", "a.nix", "b.nix"})); +} + +TEST_F(TecnixValueProvenanceTest, overwritingValueClearsStaleAccessSetMapping) +{ + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + + Value value; + value.mkInt(1); + auto originalSet = publishSingleAccess(value, "old-source.nix"); + ASSERT_NE(originalSet, emptyEvalSourceAccessSetId); + ASSERT_EQ(value.trackedSourceAccessSet(), originalSet); + + Value frameValue; + frameValue.mkInt(0); + TrackedSourceDepsFrame frame(trackingCtx, &frameValue, currentTecnixThreadState.sourceDepsFrame); + ScopedTrackedValueForceFrame scopedFrame(frame); + + value.mkInt(2); + + EXPECT_EQ(value.trackedSourceAccessSet(), emptyEvalSourceAccessSetId); +} + +TEST_F(TecnixValueProvenanceTest, copyingFinishedValuePublishesAccessSetOnDestination) +{ + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + + Value source; + source.mkInt(1); + auto sourceSet = publishSingleAccess(source, "copied-source.nix"); + ASSERT_NE(sourceSet, emptyEvalSourceAccessSetId); + + Value destination; + Value frameValue; + frameValue.mkInt(0); + TrackedSourceDepsFrame frame(trackingCtx, &frameValue, currentTecnixThreadState.sourceDepsFrame); + ScopedTrackedValueForceFrame scopedFrame(frame); + + destination = source; + + auto copiedSet = destination.trackedSourceAccessSet(); + ASSERT_NE(copiedSet, emptyEvalSourceAccessSetId); + EXPECT_EQ( + trackedSourceAccessSetGraph(state)->flatten({}, {copiedSet}), (std::vector{"copied-source.nix"})); +} + +TEST_F(TecnixValueProvenanceTest, copyingIntoCurrentForceFramePublishesCopiedAccessSet) +{ + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + + Value source; + source.mkInt(1); + auto sourceSet = publishSingleAccess(source, "current-frame-copy-source.nix"); + ASSERT_NE(sourceSet, emptyEvalSourceAccessSetId); + + Value destination; + TrackedSourceDepsFrame frame(trackingCtx, &destination, currentTecnixThreadState.sourceDepsFrame); + ScopedTrackedValueForceFrame scopedFrame(frame); + + destination = source; + + auto copiedSet = destination.trackedSourceAccessSet(); + ASSERT_NE(copiedSet, emptyEvalSourceAccessSetId); + EXPECT_EQ( + trackedSourceAccessSetGraph(state)->flatten({}, {copiedSet}), + (std::vector{"current-frame-copy-source.nix"})); +} + +TEST_F(TecnixValueProvenanceTest, tryEvalCaughtThrowRecordsSourceAccessesThatInfluenceResult) +{ + enableSourceAccessSetTracking(state); + auto accessor = make_ref(std::map{ + {"main.nix", "builtins.tryEval (import ./throws.nix)"}, + {"throws.nix", "throw (builtins.readFile ./dep.txt)"}, + {"dep.txt", "boom"}, + }); + + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + + Value result; + state.evalFile(SourcePath(accessor, CanonPath("/main.nix")), result, false); + state.forceAttrs(result, noPos, "while testing caught tryEval dependency tracking"); + + auto dependencies = flattenFrame(trackedSourceAccessSetGraph(state), trackingCtx.rootFrame); + + EXPECT_TRUE(containsPath(dependencies, "main.nix")); + EXPECT_TRUE(containsPath(dependencies, "throws.nix")); + EXPECT_TRUE(containsPath(dependencies, "dep.txt")); + + auto resultSet = result.trackedSourceAccessSet(); + ASSERT_NE(resultSet, emptyEvalSourceAccessSetId); + auto resultDependencies = trackedSourceAccessSetGraph(state)->flatten({}, {resultSet}); + EXPECT_TRUE(containsPath(resultDependencies, "main.nix")); + EXPECT_TRUE(containsPath(resultDependencies, "throws.nix")); + EXPECT_TRUE(containsPath(resultDependencies, "dep.txt")); +} + +TEST_F(TecnixValueProvenanceTest, tryEvalShallowSuccessDoesNotRecordUnforcedThrowingAttribute) +{ + enableSourceAccessSetTracking(state); + auto accessor = make_ref(std::map{ + {"main.nix", "builtins.tryEval { x = import ./throws.nix; }"}, + {"throws.nix", "throw (builtins.readFile ./dep.txt)"}, + {"dep.txt", "boom"}, + }); + + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + + Value result; + state.evalFile(SourcePath(accessor, CanonPath("/main.nix")), result, false); + state.forceAttrs(result, noPos, "while testing shallow tryEval dependency tracking"); + + auto dependencies = flattenFrame(trackedSourceAccessSetGraph(state), trackingCtx.rootFrame); + + EXPECT_TRUE(containsPath(dependencies, "main.nix")); + EXPECT_FALSE(containsPath(dependencies, "throws.nix")); + EXPECT_FALSE(containsPath(dependencies, "dep.txt")); + + auto resultSet = result.trackedSourceAccessSet(); + ASSERT_NE(resultSet, emptyEvalSourceAccessSetId); + auto resultDependencies = trackedSourceAccessSetGraph(state)->flatten({}, {resultSet}); + EXPECT_TRUE(containsPath(resultDependencies, "main.nix")); + EXPECT_FALSE(containsPath(resultDependencies, "throws.nix")); + EXPECT_FALSE(containsPath(resultDependencies, "dep.txt")); +} + +TEST_F(TecnixValueProvenanceTest, fileCacheResetPreservesGraphLabels) +{ + // Value labels are graph-local IDs and values survive a file-cache reset + // (e.g. a repl reload). The graph must never be cleared while an EvalState + // is alive, or surviving labels would resolve to re-minted IDs and hence + // the wrong paths. + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + + Value value; + value.mkInt(1); + auto set = publishSingleAccess(value, "kept-source.nix"); + ASSERT_NE(set, emptyEvalSourceAccessSetId); + + state.resetFileCache(); + + auto label = value.trackedSourceAccessSet(); + ASSERT_EQ(label, set); + auto resolved = trackedSourceAccessSetGraph(state)->flatten({}, {label}); + EXPECT_TRUE(containsPath(resolved, "kept-source.nix")); +} + +TEST_F(TecnixValueProvenanceTest, spawningParallelWorkUnderTrackingThrows) +{ + // Tracking contexts are thread-confined and work items capture only owned + // state. Spawning parallel evaluation work under an active tracking + // context must fail loudly instead of capturing a dangling context. + Executor::WorkItems work; + state.addWork(work, 0, []() {}); + EXPECT_EQ(work.size(), 1u); + + TrackingContext trackingCtx(state); + ScopedTrackingContext scopedContext(trackingCtx); + EXPECT_THROW(state.addWork(work, 0, []() {}), Error); +} + +} // namespace nix diff --git a/src/libexpr-tests/tecnix-eval-bench.cc b/src/libexpr-tests/tecnix-eval-bench.cc new file mode 100644 index 0000000000..5fb109d288 --- /dev/null +++ b/src/libexpr-tests/tecnix-eval-bench.cc @@ -0,0 +1,185 @@ +#include + +#include +#include + +#include "nix/expr/eval-settings.hh" +#include "nix/expr/eval.hh" +#include "nix/fetchers/fetch-settings.hh" +#include "nix/store/store-open.hh" +#include "nix/util/file-system.hh" +#include "nix/util/fmt.hh" +#include "nix/util/processes.hh" +#include "nix/util/strings.hh" + +using namespace nix; + +namespace { + +/** + * A synthetic Tecnix world: `targetCount` targets whose source closures each + * contain ~20 paths (own target file, a per-target leaf, 15 shared libs, the + * resolver, and the target index). Shared libs exercise label inheritance and + * the pair-union cache; per-target files exercise first-sight interning and + * per-path fingerprinting at scale. + * + * Targets are spread over a deep, narrow directory hierarchy (~25 entries per + * directory), matching real monorepo geometry (measured median directory + * width 2, p99 44). Directory shape matters: git tree objects over ~4 KiB are + * outside libgit2's default object cache, so fingerprinting many paths under + * one wide directory degrades sharply; that shape deserves its own benchmark + * if it ever becomes representative. + */ +struct SyntheticWorld +{ + std::filesystem::path dir; + std::string rev; + std::string exprString; + + explicit SyntheticWorld(size_t targetCount) + { + dir = createTempDir() + "/world"; + std::filesystem::create_directories(dir / "lib"); + + constexpr size_t libCount = 15; + constexpr size_t targetsPerDir = 13; // 13 target files + 13 leaves per directory + constexpr size_t dirsPerGroup = 25; + for (size_t i = 0; i < libCount; ++i) + std::ofstream(dir / "lib" / fmt("common-%d.nix", i)) << fmt("{ v = %d; }\n", i); + + std::string libImports, libSum; + for (size_t i = 0; i < libCount; ++i) { + libImports += fmt(" l%1% = import ../../../lib/common-%1%.nix;\n", i); + libSum += fmt("%sl%d.v", i == 0 ? "" : " + ", i); + } + + std::string index = "{\n"; + for (size_t i = 0; i < targetCount; ++i) { + auto zone = i / (targetsPerDir * dirsPerGroup); + auto group = (i / targetsPerDir) % dirsPerGroup; + auto rel = fmt("zones/z-%d/g-%d", zone, group); + std::filesystem::create_directories(dir / rel); + std::ofstream(dir / rel / fmt("leaf-%d.txt", i)) << fmt("leaf %d\n", i); + std::ofstream(dir / rel / fmt("target-%d.nix", i)) << fmt( + "{ args }:\n" + "let\n" + "%s" + " leaf = builtins.readFile ./leaf-%d.txt;\n" + "in {\n" + " drvPath = \"/nix/store/00000000000000000000000000000000-t${toString (%s)}-${builtins.hashString " + "\"sha256\" leaf}-%d.drv\";\n" + "}\n", + libImports, + i, + libSum, + i); + index += fmt(" \"target-%1%\" = import ./%2%/target-%1%.nix;\n", i, rel); + } + index += "}\n"; + std::ofstream(dir / "targets-index.nix") << index; + + std::ofstream(dir / "resolve.nix") << "args:\n" + "let targets = import ./targets-index.nix;\n" + "in {\n" + " allTargetNames = builtins.attrNames targets;\n" + " resolve = id: targets.${id} { inherit args; };\n" + "}\n"; + + auto git = [&](Strings args) { + args.insert(args.begin(), {"-C", dir.string()}); + return runProgram("git", true, args); + }; + git({"init", "-q"}); + git({"config", "user.email", "bench@example.com"}); + git({"config", "user.name", "bench"}); + git({"add", "-A"}); + git({"commit", "-q", "-m", "synthetic world"}); + rev = chomp(git({"rev-parse", "HEAD"})); + + exprString = fmt( + "let\n" + " base = { gitDir = \"%s/.git\"; resolver = \"resolve.nix\"; rev = \"%s\"; args = { system = \"bench\"; " + "}; };\n" + " names = builtins.tecnixTargetNames base;\n" + " result = builtins.tecnixTargets (base // { targets = names; includeDependencies = true; includeTargets " + "= false; });\n" + "in builtins.deepSeq result (builtins.length result)", + dir.string(), + rev); + } +}; + +SyntheticWorld & worldForTargetCount(size_t targetCount) +{ + // The per-process Tecnix SQLite cache must live in a fresh directory so + // benchmark runs are isolated; set before the first eval creates it. + static bool cacheIsolated = [] { + setenv("XDG_CACHE_HOME", createTempDir("", "tecnix-bench-cache").c_str(), 1); + return true; + }(); + (void) cacheIsolated; + + static std::map> worlds; + auto & world = worlds[targetCount]; + if (!world) + world = std::make_unique(targetCount); + return *world; +} + +struct BenchEnv +{ + ref store = openStore("dummy://"); + fetchers::Settings fetchSettings{}; + bool readOnlyMode = true; + EvalSettings evalSettings{readOnlyMode}; + + explicit BenchEnv(bool tecnixEvalCache) + { + evalSettings.nixPath = {}; + evalSettings.pureEval = true; + evalSettings.lazyTrees = true; + evalSettings.tecnixEvalCache = tecnixEvalCache; + } + + /** One full tracked evaluation in a fresh EvalState; returns the target count. */ + NixInt::Inner evalOnce(const SyntheticWorld & world) + { + EvalState state(LookupPath{}, store, fetchSettings, evalSettings, nullptr); + auto * expr = state.parseExprFromString(world.exprString, state.rootPath(CanonPath::root)); + Value v; + state.eval(expr, v); + state.forceValue(v, noPos); + return v.integer().value; + } +}; + +void BM_TecnixTrackedEvalUncached(benchmark::State & state) +{ + auto & world = worldForTargetCount(state.range(0)); + BenchEnv env(false); + for (auto _ : state) { + auto n = env.evalOnce(world); + benchmark::DoNotOptimize(n); + if (n != state.range(0)) + state.SkipWithError("unexpected target count"); + } +} + +void BM_TecnixWarmCacheHit(benchmark::State & state) +{ + auto & world = worldForTargetCount(state.range(0)); + BenchEnv env(true); + // Populate the persistent cache outside the timed loop. + env.evalOnce(world); + for (auto _ : state) { + auto n = env.evalOnce(world); + benchmark::DoNotOptimize(n); + if (n != state.range(0)) + state.SkipWithError("unexpected target count"); + } +} + +} // namespace + +BENCHMARK(BM_TecnixTrackedEvalUncached)->Unit(benchmark::kMillisecond)->Arg(1000)->Arg(10000); +BENCHMARK(BM_TecnixWarmCacheHit)->Unit(benchmark::kMillisecond)->Arg(1000)->Arg(10000); diff --git a/src/libexpr-tests/tecnix-worldtree.cc b/src/libexpr-tests/tecnix-worldtree.cc new file mode 100644 index 0000000000..28c4aa025e --- /dev/null +++ b/src/libexpr-tests/tecnix-worldtree.cc @@ -0,0 +1,265 @@ +#include + +#include "nix/expr/eval.hh" +#include "nix/expr/eval-settings.hh" +#include "nix/expr/tests/libexpr.hh" +#include "nix/expr/tecnix/source-accessors.hh" +#include "nix/store/store-open.hh" +#include "nix/store/globals.hh" +#include "nix/util/file-system.hh" +#include "nix/util/hash.hh" +#include "nix/util/source-accessor.hh" +#include "nix/util/util.hh" + +#include +#include +#include +#include + +#include + +namespace nix { + +// ============================================================================ +// The worldtree FUSE projection as the tracked Tecnix clean backend. +// +// These tests fake the immutable projection with a plain directory tree plus +// real `user.worldtree.tree-oid` xattrs, exactly the surface the daemon +// exposes. No socket, no daemon, no git repository. +// ============================================================================ + +static bool setWorldtreeTestTreeOidXattr(const std::filesystem::path & path, std::string_view value) +{ +#ifdef __APPLE__ + return ::setxattr(path.c_str(), "user.worldtree.tree-oid", value.data(), value.size(), 0, 0) == 0; +#else + return ::setxattr(path.c_str(), "user.worldtree.tree-oid", value.data(), value.size(), 0) == 0; +#endif +} + +static bool setWorldtreeTestBlobOidXattr(const std::filesystem::path & path, std::string_view value) +{ +#ifdef __APPLE__ + return ::setxattr(path.c_str(), "user.worldtree.blob-oid", value.data(), value.size(), 0, 0) == 0; +#else + return ::setxattr(path.c_str(), "user.worldtree.blob-oid", value.data(), value.size(), 0) == 0; +#endif +} + +/** + * Independent fingerprint oracle: the git blob oid of `content`, computed from + * the git object spec framing ("blob \0") rather than through the + * accessor's own hashing helpers. + */ +static std::string gitBlobOidOf(std::string_view content) +{ + std::string object = "blob " + std::to_string(content.size()); + object.push_back('\0'); + object.append(content); + return hashString(HashAlgorithm::SHA1, object).gitRev(); +} + +class TecnixWorldtreeTest : public ::testing::Test +{ +protected: + static void SetUpTestSuite() + { + initLibStore(false); + initGC(); + } + + std::unique_ptr delTmpDir; + std::filesystem::path tmpDir; + // The rev only names the projection directory; no repository exists. + std::string commitSha = std::string(40, 'f'); + + void SetUp() override + { + auto tmp = createTempDir(); + delTmpDir = std::make_unique(tmp, true); + tmpDir = tmp; + } + + void TearDown() override + { + delTmpDir.reset(); + } + + std::filesystem::path mountDir() const + { + return tmpDir / "worldtree"; + } + + std::filesystem::path revisionRoot() const + { + return mountDir() / "tecnix" / commitSha; + } + + void writeManifest(std::string_view manifest) + { + auto dir = revisionRoot() / "W-000000"; + std::filesystem::create_directories(dir); + writeFile((dir / "manifest.json").string(), std::string(manifest)); + } + + struct WorldtreeEvalContext + { + bool readOnlyMode = true; + fetchers::Settings fetchSettings{}; + EvalSettings evalSettings{readOnlyMode}; + ref store; + std::unique_ptr state; + + WorldtreeEvalContext(const std::filesystem::path & mount, const std::string & rev) + : store(openStore("dummy://")) + { + evalSettings.nixPath = {}; + evalSettings.tectonixGitSha = rev; + // Deliberately nonexistent: the FUSE-backed accessor never connects. + evalSettings.tectonixWorldtreeSocket = (mount / "unused-worldtree.sock").string(); + evalSettings.tectonixWorldtreeMount = mount.string(); + state = std::make_unique(LookupPath{}, store, fetchSettings, evalSettings, nullptr); + } + }; + + std::unique_ptr createContext() + { + return std::make_unique(mountDir(), commitSha); + } +}; + +TEST_F(TecnixWorldtreeTest, repo_accessor_serves_zone_content_with_git_fingerprints) +{ + writeManifest(R"({ + "//app/zone": { "id": "W-aaaa01" } + })"); + auto zoneDir = revisionRoot() / "W-aaaa01"; + std::filesystem::create_directories(zoneDir / "sub"); + writeFile((zoneDir / "hello.txt").string(), "hello\n"); + writeFile((zoneDir / "sub" / "nested.txt").string(), "nested\n"); + writeFile((zoneDir / "tool.sh").string(), "hi\n"); + std::filesystem::permissions( + zoneDir / "tool.sh", std::filesystem::perms::owner_all, std::filesystem::perm_options::add); + std::filesystem::create_symlink("hello.txt", zoneDir / "link"); + + const std::string zoneTreeOid(40, 'a'); + const std::string subTreeOid(40, 'b'); + if (!setWorldtreeTestTreeOidXattr(zoneDir, zoneTreeOid) + || !setWorldtreeTestTreeOidXattr(zoneDir / "sub", subTreeOid)) + GTEST_SKIP() << "filesystem does not support user xattrs"; + + auto ctx = createContext(); + auto accessor = getTecnixRepoAccessor(*ctx->state); + + // Repo-relative paths are served through the manifest's zone mapping. + ASSERT_EQ(accessor->readFile(CanonPath("/app/zone/hello.txt")), "hello\n"); + ASSERT_EQ(accessor->readFile(CanonPath("/app/zone/sub/nested.txt")), "nested\n"); + ASSERT_EQ(accessor->readLink(CanonPath("/app/zone/link")), "hello.txt"); + auto zoneEntries = accessor->readDirectory(CanonPath("/app/zone")); + ASSERT_TRUE(zoneEntries.count("hello.txt")); + ASSERT_TRUE(zoneEntries.count("sub")); + + // Zone-ancestor directories are synthesized so traversal can reach zones. + auto ancestorStat = accessor->maybeLstat(CanonPath("/app")); + ASSERT_TRUE(ancestorStat && ancestorStat->type == SourceAccessor::Type::tDirectory); + auto ancestorEntries = accessor->readDirectory(CanonPath("/app")); + ASSERT_TRUE(ancestorEntries.count("zone")); + + // Directory fingerprints echo the projection's tree-oid xattr byte for byte. + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/app/zone")).second, "git:" + zoneTreeOid + ";mode=040000"); + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/app/zone/sub")).second, "git:" + subTreeOid + ";mode=040000"); + + // File and symlink fingerprints are exact git blob oids, so TXDC closures + // cross-validate with the libgit2 backend. Pin one absolute value to anchor + // the oracle itself ("hello\n" is the classic known git blob). + auto helloOid = gitBlobOidOf("hello\n"); + ASSERT_EQ(helloOid, "ce013625030ba8dba906f756967f9e9ca394464a"); + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/app/zone/hello.txt")).second, "git:" + helloOid + ";mode=100644"); + ASSERT_EQ( + *accessor->getFingerprint(CanonPath("/app/zone/tool.sh")).second, + "git:" + gitBlobOidOf("hi\n") + ";mode=100755"); + ASSERT_EQ( + *accessor->getFingerprint(CanonPath("/app/zone/link")).second, + "git:" + gitBlobOidOf("hello.txt") + ";mode=120000"); + + // Negative lookups, and paths outside every visible zone, observe absence. + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/app/zone/missing.nix")).second, "absent"); + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/README.md")).second, "absent"); + + // Synthesized directories (the root and zone ancestors) have no single git + // object; their composite fingerprint stays outside the git vocabulary. + auto ancestorFp = accessor->getFingerprint(CanonPath("/app")).second; + ASSERT_TRUE(ancestorFp && hasPrefix(*ancestorFp, "worldtree-union:")); + auto rootFp = accessor->getFingerprint(CanonPath::root).second; + ASSERT_TRUE(rootFp && hasPrefix(*rootFp, "worldtree-union:")); +} + +TEST_F(TecnixWorldtreeTest, blob_fingerprints_memoize_in_memory_only) +{ + writeManifest(R"({ + "//app/zone": { "id": "W-aaaa02" } + })"); + auto zoneDir = revisionRoot() / "W-aaaa02"; + std::filesystem::create_directories(zoneDir); + writeFile((zoneDir / "marker.txt").string(), "one\n"); + + auto fingerprintOf = [&](WorldtreeEvalContext & ctx) { + return *getTecnixRepoAccessor(*ctx.state)->getFingerprint(CanonPath("/app/zone/marker.txt")).second; + }; + + auto ctx1 = createContext(); + ASSERT_EQ(fingerprintOf(*ctx1), "git:" + gitBlobOidOf("one\n") + ";mode=100644"); + + // Rewrite the bytes. A real projection cannot do this — it is immutable — + // so the stale answer through the same accessor proves fingerprints are + // memoized rather than re-hashed per query. + writeFile((zoneDir / "marker.txt").string(), "two\n"); + ASSERT_EQ(fingerprintOf(*ctx1), "git:" + gitBlobOidOf("one\n") + ";mode=100644"); + + // A fresh evaluator re-hashes: the memo lives in memory, never on disk. + auto ctx2 = createContext(); + ASSERT_EQ(fingerprintOf(*ctx2), "git:" + gitBlobOidOf("two\n") + ";mode=100644"); +} + +TEST_F(TecnixWorldtreeTest, blob_fingerprints_prefer_blob_oid_xattr_and_fall_back_to_hashing) +{ + writeManifest(R"({ + "//app/zone": { "id": "W-aaaa03" } + })"); + auto zoneDir = revisionRoot() / "W-aaaa03"; + std::filesystem::create_directories(zoneDir); + writeFile((zoneDir / "served.txt").string(), "one\n"); + writeFile((zoneDir / "tool.sh").string(), "hi\n"); + std::filesystem::permissions( + zoneDir / "tool.sh", std::filesystem::perms::owner_all, std::filesystem::perm_options::add); + writeFile((zoneDir / "unserved.txt").string(), "two\n"); + std::filesystem::create_symlink("served.txt", zoneDir / "link"); + + const std::string servedOid(40, 'e'); + if (!setWorldtreeTestBlobOidXattr(zoneDir / "served.txt", servedOid) + || !setWorldtreeTestBlobOidXattr(zoneDir / "tool.sh", servedOid)) + GTEST_SKIP() << "filesystem does not support user xattrs"; + + auto ctx = createContext(); + auto accessor = getTecnixRepoAccessor(*ctx->state); + + // A served blob oid wins over content hashing: the sentinel oid is not the + // content's hash, so any hashing would produce a different answer. The + // mode still comes from the filesystem stat. + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/app/zone/served.txt")).second, "git:" + servedOid + ";mode=100644"); + ASSERT_EQ(*accessor->getFingerprint(CanonPath("/app/zone/tool.sh")).second, "git:" + servedOid + ";mode=100755"); + + // No xattr: fall back to hashing the bytes. + ASSERT_EQ( + *accessor->getFingerprint(CanonPath("/app/zone/unserved.txt")).second, + "git:" + gitBlobOidOf("two\n") + ";mode=100644"); + + // A symlink must never attempt the xattr: getxattr() follows links, and + // its target carries the sentinel blob oid — answering with it would + // fingerprint the wrong git object. The link hashes its target string. + ASSERT_EQ( + *accessor->getFingerprint(CanonPath("/app/zone/link")).second, + "git:" + gitBlobOidOf("served.txt") + ";mode=120000"); +} + +} // namespace nix diff --git a/src/libexpr-tests/tectonix.cc b/src/libexpr-tests/tectonix.cc index 3410906d93..3bb8e5ee82 100644 --- a/src/libexpr-tests/tectonix.cc +++ b/src/libexpr-tests/tectonix.cc @@ -4,6 +4,7 @@ #include "nix/expr/tests/libexpr.hh" #include "nix/expr/eval.hh" #include "nix/expr/eval-settings.hh" +#include "nix/expr/tecnix/source-accessors.hh" #include "nix/fetchers/git-utils.hh" #include "nix/store/globals.hh" #include "nix/util/file-system.hh" @@ -235,12 +236,12 @@ TEST_F(TectonixTest, historical_worldtree_manifest_and_dirty_set_are_filesystem_ })"; auto ctx = createHistoricalWorldtreeContext(historicalManifest); - ASSERT_EQ(ctx->state->getManifestContent(), historicalManifest); - auto & manifest = ctx->state->getManifestJson(); + ASSERT_EQ(getManifestContent(*ctx->state), historicalManifest); + auto & manifest = getManifestJson(*ctx->state); ASSERT_EQ(manifest.size(), 1u); ASSERT_EQ(manifest.at("//historical/only").at("id"), "W-123456"); - auto & dirty = ctx->state->getTectonixDirtyZones(); + auto & dirty = getTectonixDirtyZones(*ctx->state); ASSERT_EQ(dirty.size(), 1u); ASSERT_FALSE(dirty.at("//historical/only").dirty); } @@ -250,7 +251,7 @@ TEST_F(TectonixTest, historical_worldtree_explains_malformed_manifest) auto ctx = createHistoricalWorldtreeContext("{"); try { - ctx->state->getManifestJson(); + getManifestJson(*ctx->state); FAIL() << "expected malformed manifest to fail"; } catch (const Error & e) { ASSERT_THAT(e.what(), testing::HasSubstr("historical World manifest '.meta/manifest.json'")); @@ -265,7 +266,7 @@ TEST_F(TectonixTest, historical_worldtree_explains_missing_manifest) auto ctx = std::make_unique(repoPath, commitSha, false, mount); try { - ctx->state->getManifestContent(); + getManifestContent(*ctx->state); FAIL() << "expected missing manifest to fail"; } catch (const Error & e) { ASSERT_THAT(e.what(), testing::HasSubstr("historical World manifest '.meta/manifest.json'")); @@ -279,7 +280,7 @@ TEST_F(TectonixTest, historical_worldtree_rejects_noncanonical_revision) auto mount = repoPath / "worldtree"; auto ctx = std::make_unique(repoPath, "../escape", false, mount); - ASSERT_THROW(ctx->state->getManifestContent(), Error); + ASSERT_THROW(getManifestContent(*ctx->state), Error); } TEST_F(TectonixTest, historical_worldtree_rejects_noncanonical_zone_id) @@ -288,7 +289,7 @@ TEST_F(TectonixTest, historical_worldtree_rejects_noncanonical_zone_id) "//historical/only": { "id": "../escape" } })"); - ASSERT_THROW(ctx->state->getZoneStorePath("//historical/only"), Error); + ASSERT_THROW(getLegacyTectonixZoneStorePath(*ctx->state, "//historical/only"), Error); } // ============================================================================ @@ -505,15 +506,15 @@ TEST_F(TectonixTest, dirtyZones_empty_without_checkout) TEST_F(TectonixTest, getWorldRepo_returns_repo) { auto ctx = createTectonixContext(); - auto repo = ctx->state->getWorldRepo(); + auto repo = getWorldRepo(*ctx->state); ASSERT_NE(&*repo, nullptr); } TEST_F(TectonixTest, getWorldRepo_caches_instance) { auto ctx = createTectonixContext(); - auto repo1 = ctx->state->getWorldRepo(); - auto repo2 = ctx->state->getWorldRepo(); + auto repo1 = getWorldRepo(*ctx->state); + auto repo2 = getWorldRepo(*ctx->state); // Should return same instance ASSERT_EQ(&*repo1, &*repo2); } @@ -525,7 +526,7 @@ TEST_F(TectonixTest, getWorldRepo_caches_instance) TEST_F(TectonixTest, getWorldTreeSha_returns_hash) { auto ctx = createTectonixContext(); - auto hash = ctx->state->getWorldTreeSha("//areas/tools/dev"); + auto hash = getWorldTreeSha(*ctx->state, "//areas/tools/dev"); // SHA1 hash is 40 hex chars ASSERT_EQ(hash.gitRev().size(), 40u); } @@ -533,8 +534,8 @@ TEST_F(TectonixTest, getWorldTreeSha_returns_hash) TEST_F(TectonixTest, getWorldTreeSha_caches_results) { auto ctx = createTectonixContext(); - auto hash1 = ctx->state->getWorldTreeSha("//areas/tools/dev"); - auto hash2 = ctx->state->getWorldTreeSha("//areas/tools/dev"); + auto hash1 = getWorldTreeSha(*ctx->state, "//areas/tools/dev"); + auto hash2 = getWorldTreeSha(*ctx->state, "//areas/tools/dev"); ASSERT_EQ(hash1, hash2); } @@ -542,7 +543,7 @@ TEST_F(TectonixTest, getWorldTreeSha_root_returns_commit_tree) { auto ctx = createTectonixContext(); // Root path should work - auto hash = ctx->state->getWorldTreeSha("//"); + auto hash = getWorldTreeSha(*ctx->state, "//"); ASSERT_EQ(hash.gitRev().size(), 40u); } @@ -553,7 +554,7 @@ TEST_F(TectonixTest, getWorldTreeSha_root_returns_commit_tree) TEST_F(TectonixTest, getManifestJson_parses_correctly) { auto ctx = createTectonixContext(); - auto & json = ctx->state->getManifestJson(); + auto & json = getManifestJson(*ctx->state); ASSERT_TRUE(json.contains("//areas/tools/dev")); ASSERT_EQ(json["//areas/tools/dev"]["id"], "W-000001"); @@ -562,8 +563,8 @@ TEST_F(TectonixTest, getManifestJson_parses_correctly) TEST_F(TectonixTest, getManifestJson_caches_result) { auto ctx = createTectonixContext(); - auto & json1 = ctx->state->getManifestJson(); - auto & json2 = ctx->state->getManifestJson(); + auto & json1 = getManifestJson(*ctx->state); + auto & json2 = getManifestJson(*ctx->state); // Should return same instance ASSERT_EQ(&json1, &json2); } @@ -575,13 +576,13 @@ TEST_F(TectonixTest, getManifestJson_caches_result) TEST_F(TectonixTest, isTectonixSourceAvailable_false_without_checkout) { auto ctx = createTectonixContext(false); - ASSERT_FALSE(ctx->state->isTectonixSourceAvailable()); + ASSERT_FALSE(isTectonixSourceAvailable(*ctx->state)); } TEST_F(TectonixTest, isTectonixSourceAvailable_true_with_checkout) { auto ctx = createTectonixContext(true); - ASSERT_TRUE(ctx->state->isTectonixSourceAvailable()); + ASSERT_TRUE(isTectonixSourceAvailable(*ctx->state)); } // ============================================================================ @@ -599,7 +600,7 @@ TEST_F(TectonixTest, missing_git_dir_throws) EvalState evalState(LookupPath{}, openStore("dummy://"), fetchSettings, evalSettings, nullptr); - ASSERT_THROW(evalState.getWorldRepo(), Error); + ASSERT_THROW(getWorldRepo(evalState), Error); } TEST_F(TectonixTest, missing_git_sha_throws) @@ -613,7 +614,7 @@ TEST_F(TectonixTest, missing_git_sha_throws) EvalState evalState(LookupPath{}, openStore("dummy://"), fetchSettings, evalSettings, nullptr); - ASSERT_THROW(evalState.getWorldGitAccessor(), Error); + ASSERT_THROW(getWorldGitAccessor(evalState), Error); } TEST_F(TectonixTest, missing_git_sha_tree_sha_throws) @@ -627,7 +628,7 @@ TEST_F(TectonixTest, missing_git_sha_tree_sha_throws) EvalState evalState(LookupPath{}, openStore("dummy://"), fetchSettings, evalSettings, nullptr); - ASSERT_THROW(evalState.getWorldTreeSha("//areas/tools/dev"), Error); + ASSERT_THROW(getWorldTreeSha(evalState, "//areas/tools/dev"), Error); } TEST_F(TectonixTest, invalid_sha_throws) @@ -641,7 +642,7 @@ TEST_F(TectonixTest, invalid_sha_throws) EvalState evalState(LookupPath{}, openStore("dummy://"), fetchSettings, evalSettings, nullptr); - ASSERT_THROW(evalState.getWorldGitAccessor(), Error); + ASSERT_THROW(getWorldGitAccessor(evalState), Error); } // ============================================================================ @@ -657,7 +658,7 @@ TEST_F(TectonixTest, concurrent_manifest_access) // Multiple threads calling getManifestJson for (size_t i = 0; i < 8; i++) { - threads.emplace_back([&, i]() { results[i] = &ctx->state->getManifestJson(); }); + threads.emplace_back([&, i]() { results[i] = &getManifestJson(*ctx->state); }); } for (auto & t : threads) { @@ -679,7 +680,7 @@ TEST_F(TectonixTest, concurrent_tree_sha_computation) // Multiple threads computing tree SHAs for same path for (size_t i = 0; i < 8; i++) { - threads.emplace_back([&, i]() { results[i] = ctx->state->getWorldTreeSha("//areas/tools/dev").gitRev(); }); + threads.emplace_back([&, i]() { results[i] = getWorldTreeSha(*ctx->state, "//areas/tools/dev").gitRev(); }); } for (auto & t : threads) { @@ -701,7 +702,7 @@ TEST_F(TectonixTest, concurrent_world_repo_access) // Multiple threads calling getWorldRepo for (size_t i = 0; i < 8; i++) { - threads.emplace_back([&, i]() { results[i] = &*ctx->state->getWorldRepo(); }); + threads.emplace_back([&, i]() { results[i] = &*getWorldRepo(*ctx->state); }); } for (auto & t : threads) { @@ -727,7 +728,7 @@ TEST_F(TectonixTest, concurrent_different_tree_shas) for (const auto & path : zonePaths) { for (int i = 0; i < 3; i++) { threads.emplace_back([&, path]() { - auto hash = ctx->state->getWorldTreeSha(path); + auto hash = getWorldTreeSha(*ctx->state, path); auto hashStr = hash.gitRev(); std::lock_guard lock(resultsMutex); auto it = results.find(path); @@ -820,7 +821,7 @@ TEST_F(TectonixTest, zone_path_traversal_throws) auto ctx = createTectonixContext(); // Path traversal attempt should fail - ASSERT_THROW(ctx->state->getWorldTreeSha("//areas/../.git"), Error); + ASSERT_THROW(getWorldTreeSha(*ctx->state, "//areas/../.git"), Error); } TEST_F(TectonixTest, nonexistent_git_dir_throws) @@ -834,7 +835,7 @@ TEST_F(TectonixTest, nonexistent_git_dir_throws) EvalState evalState(LookupPath{}, openStore("dummy://"), fetchSettings, evalSettings, nullptr); - ASSERT_THROW(evalState.getWorldRepo(), Error); + ASSERT_THROW(getWorldRepo(evalState), Error); } TEST_F(TectonixTest, treeSha_for_nonexistent_subpath_throws) @@ -842,7 +843,7 @@ TEST_F(TectonixTest, treeSha_for_nonexistent_subpath_throws) auto ctx = createTectonixContext(); // Path that doesn't exist in the tree - ASSERT_THROW(ctx->state->getWorldTreeSha("//areas/tools/dev/nonexistent/deep/path"), Error); + ASSERT_THROW(getWorldTreeSha(*ctx->state, "//areas/tools/dev/nonexistent/deep/path"), Error); } TEST_F(TectonixTest, manifest_missing_id_field_throws) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 6b401465bc..86466dc9ce 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -1,4 +1,5 @@ #include "nix/expr/eval.hh" +#include "tecnix/eval-data.hh" #include "nix/expr/eval-error.hh" #include "nix/expr/eval-settings.hh" #include "nix/expr/primops.hh" @@ -8,7 +9,6 @@ #include "nix/util/exit.hh" #include "nix/util/types.hh" #include "nix/util/util.hh" -#include "nix/util/worldtree-client.hh" #include "nix/util/environment-variables.hh" #include "nix/store/store-api.hh" #include "nix/store/derivations.hh" @@ -26,18 +26,14 @@ #include "nix/fetchers/fetch-to-store.hh" #include "nix/fetchers/tarball.hh" #include "nix/fetchers/input-cache.hh" -#include "nix/fetchers/git-utils.hh" #include "nix/util/current-process.hh" -#include "nix/util/processes.hh" #include "nix/store/async-path-writer.hh" #include "nix/expr/parallel-eval.hh" #include "parser-tab.hh" #include -#include #include -#include #include #include #include @@ -45,15 +41,11 @@ #include #include #include -#include #include #include #include #include -#include -#include -#include #ifndef _WIN32 // TODO use portable implementation # include @@ -65,6 +57,12 @@ using json = nlohmann::json; namespace nix { +struct ParsedFileCacheEntry +{ + std::once_flag once; + Expr * expr = nullptr; +}; + /** * Just for doc strings. Not for regular string values. */ @@ -337,12 +335,11 @@ EvalState::EvalState( , trylevel(0) , asyncPathWriter(AsyncPathWriter::make(store)) , srcToStore(make_ref()) - , importResolutionCache(make_ref()) , fileEvalCache(make_ref()) , positionToDocComment(make_ref()) , lookupPathResolved(make_ref()) , regexCache(makeRegexCache()) - , worldTreeShaCache(make_ref()) + , tecnixData(std::make_unique()) #if NIX_USE_BOEHMGC , baseEnvP(std::allocate_shared(traceable_allocator(), &mem.allocEnv(BASE_ENV_SIZE))) , baseEnv(**baseEnvP) @@ -404,6 +401,16 @@ EvalState::EvalState( EvalState::~EvalState() {} +EvalState::TecnixEvalData & EvalState::tecnixEvalData() +{ + return *tecnixData; +} + +const EvalState::TecnixEvalData & EvalState::tecnixEvalData() const +{ + return *tecnixData; +} + void EvalState::allowPathLegacy(const std::string & path) { if (auto rootFS2 = rootFS.dynamic_pointer_cast()) @@ -434,943 +441,6 @@ void EvalState::allowAndSetStorePathString(const StorePath & storePath, Value & mkStorePathString(storePath, v); } -ref EvalState::getWorldRepo() const -{ - std::call_once(worldRepoFlag, [this]() { - auto gitDir = settings.tectonixGitDir.get(); - if (gitDir.empty()) - throw Error("--tectonix-git-dir must be specified to use tectonix builtins"); - - // Expand ~ to home directory - if (hasPrefix(gitDir, "~/")) - gitDir = getHome() + gitDir.substr(1); - - worldRepo = GitRepo::openRepo(std::filesystem::path(gitDir), {.bare = true}); - debug("opened world repo at %s", gitDir); - }); - return *worldRepo; -} - -const std::string & EvalState::requireTectonixGitSha() const -{ - auto & sha = settings.tectonixGitSha.get(); - if (sha.empty()) - throw Error("--tectonix-git-sha must be specified to use tectonix builtins"); - return sha; -} - -ref EvalState::getWorldGitAccessor() const -{ - std::call_once(worldGitAccessorFlag, [this]() { - auto & sha = requireTectonixGitSha(); - - auto repo = getWorldRepo(); - auto hash = Hash::parseNonSRIUnprefixed(sha, HashAlgorithm::SHA1); - - if (!repo->hasObject(hash)) - throw Error("tectonix-git-sha '%s' not found in repository", sha); - - // Validate that the SHA is a commit by trying to get its tree. - // This gives a clear error if someone accidentally passes a tree or blob SHA. - try { - repo->getCommitTree(hash); - } catch (Error & e) { - throw Error("tectonix-git-sha '%s' does not appear to be a valid commit: %s", sha, e.what()); - } - - // exportIgnore=false: The world accessor is used for path validation and tree SHA - // computation, where we need to see all files. Zone accessors (mountZoneByTreeSha, - // getZoneStorePath) use exportIgnore=true to honor .gitattributes for actual content. - GitAccessorOptions opts{.exportIgnore = false, .smudgeLfs = false}; - worldGitAccessor = repo->getAccessor(hash, opts, "world"); - debug("created world accessor at commit %s", sha); - }); - return *worldGitAccessor; -} - -bool EvalState::isTectonixSourceAvailable() const -{ - return !settings.tectonixCheckoutPath.get().empty(); -} - -// Helper to normalize zone paths: strip leading // prefix -// Zone paths in manifest have // prefix (e.g., //areas/tools/dev) -// Filesystem operations need paths without // (e.g., areas/tools/dev) -static std::string normalizeZonePath(std::string_view zonePath) -{ - std::string path(zonePath); - if (hasPrefix(path, "//")) - path = path.substr(2); - return path; -} - -static GitAccessorOptions -makeZoneAccessorOptions(ref repo, const Hash & commitHash, const std::string & zonePath) -{ - std::string attrFp; - for (auto & h : repo->getGitAttributesAlongPath(commitHash, zonePath)) - attrFp += h.gitRev(); - return { - .exportIgnore = true, - .smudgeLfs = true, - .attrCommitRev = commitHash, - .attrPathPrefix = zonePath, - .attrFingerprint = std::move(attrFp), - }; -} - -// Helper to sanitize zone path for use in store path names. -// Store paths only allow: a-zA-Z0-9 and +-._?= -// Replaces / with - and any other invalid chars with _ -static std::string sanitizeZoneNameForStore(std::string_view zonePath) -{ - auto zone = normalizeZonePath(zonePath); - std::string result; - result.reserve(zone.size()); - for (char c : zone) { - if (c == '/') { - result += '-'; - } else if ( - (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '+' || c == '-' - || c == '.' || c == '_' || c == '?' || c == '=') { - result += c; - } else { - result += '_'; - } - } - return result; -} - -// ============================================================================ -// Worldtree daemon integration (design §5.1 / §5.1a) -// -// When `tectonix-worldtree-socket` is set, mutable-checkout metadata moves off the -// libgit2 repo+checkout walk and onto O(changes) RPCs against the bound workspace: -// * getTectonixDirtyZones() -> `dirty_zones` — the tracked-dirty set; -// * getWorldTreeSha() -> `zone_tree_shas` — the working-tree subtree oid -// (the committed oid when clean, the synthesized frontier oid when dirty; -// the zone's build-cache key); -// * getZoneStorePath() reads source bytes from FUSE in both regimes: `root` for the -// mutable workspace and `tecnix//` for immutable history. -// Fail-loud contract: when the socket is SET, the daemon is the sole source of truth — a -// worldtree sandbox has no git repo to fall back to. A daemon that is unreachable, or that -// refuses/errors a request, is a hard failure (the error propagates), never a silent -// downgrade. libgit2 / the checkout walk are reached ONLY when the socket is UNSET (plain -// local, non-worldtree eval). Historical reads create no daemon connection. -/** Reinterpret a 20-byte worldtree object id as a Nix SHA-1 Hash. */ -static Hash oidToHash(const worldtree::Oid & oid) -{ - Hash h(HashAlgorithm::SHA1); - assert(h.hashSize == oid.size()); - std::memcpy(h.hash, oid.data(), oid.size()); - return h; -} - -/** - * The scoped socket is only the mutable root-checkout control plane. Historical - * committed source is ordinary filesystem input beneath - * /mnt/worldtree/tecnix//. - */ -struct WorldtreeConn -{ - uint64_t ws; - std::mutex mutex; - worldtree::Client client; - - WorldtreeConn(worldtree::Client && client, uint64_t ws) - : ws(ws) - , client(std::move(client)) - { - } - - /** The dirty set with each zone's changed files (for full ZoneDirtyInfo). */ - std::vector dirtyZoneEntries() - { - std::lock_guard lock(mutex); - return client.dirtyZoneEntries(ws); - } - - /** One zone's working-tree subtree oid, or nullopt when absent or out of scope. */ - std::optional zoneTreeSha(std::string_view worldPath) - { - std::string wp = hasPrefix(worldPath, "//") ? std::string(worldPath) : "//" + std::string(worldPath); - std::lock_guard lock(mutex); - auto resp = client.zoneTreeShas(ws, {wp}); - if (resp.empty() || !resp.front().treeSha) - return std::nullopt; - return oidToHash(*resp.front().treeSha); - } -}; - -static constexpr std::string_view WORLDTREE_TREE_OID_XATTR = "user.worldtree.tree-oid"; - -static std::filesystem::path worldtreeRevisionRoot(const EvalSettings & settings) -{ - auto revision = Hash::parseNonSRIUnprefixed(settings.tectonixGitSha.get(), HashAlgorithm::SHA1); - return std::filesystem::path(settings.tectonixWorldtreeMount.get()) / "tecnix" / revision.gitRev(); -} - -static std::string requireWorldtreeZoneId(const std::string & id) -{ - auto isLowerHex = [](char c) { return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'); }; - if (id.size() != 8 || !id.starts_with("W-") || !std::ranges::all_of(std::string_view(id).substr(2), isLowerHex)) - throw Error("worldtree: invalid zone id '%s' in manifest", id); - return id; -} - -/** Find the manifest zone containing worldPath and return (zone id, path within zone). */ -static std::pair -worldtreeZoneLocation(const nlohmann::json & manifest, std::string_view worldPath) -{ - auto clean = normalizeZonePath(worldPath); - while (!clean.empty() && clean.front() == '/') - clean.erase(clean.begin()); - while (!clean.empty() && clean.back() == '/') - clean.pop_back(); - for (auto & component : tokenizeString>(clean, "/")) - if (component.empty() || component == "." || component == "..") - throw Error("invalid world path '%s'", worldPath); - - const nlohmann::json * best = nullptr; - std::string bestPath; - for (auto & [candidateWorldPath, value] : manifest.items()) { - auto candidate = normalizeZonePath(candidateWorldPath); - bool contains = - clean == candidate - || (clean.size() > candidate.size() && hasPrefix(clean, candidate) && clean[candidate.size()] == '/'); - if (contains && candidate.size() > bestPath.size()) { - best = &value; - bestPath = std::move(candidate); - } - } - if (!best || !best->is_object() || !best->contains("id") || !best->at("id").is_string()) - throw Error("worldtree: path '%s' is not contained by a visible World zone", worldPath); - - auto id = requireWorldtreeZoneId(best->at("id").get()); - auto relative = clean.size() == bestPath.size() ? std::string() : clean.substr(bestPath.size() + 1); - return {std::move(id), std::move(relative)}; -} - -static Hash readWorldtreeTreeOid(const std::filesystem::path & path) -{ - std::array value{}; -#ifdef __APPLE__ - auto size = ::getxattr(path.c_str(), WORLDTREE_TREE_OID_XATTR.data(), value.data(), value.size(), 0, 0); -#else - auto size = ::getxattr(path.c_str(), WORLDTREE_TREE_OID_XATTR.data(), value.data(), value.size()); -#endif - if (size < 0) - throw Error("worldtree: cannot read tree identity for '%s': %s", path.string(), std::strerror(errno)); - if (size != static_cast(value.size())) - throw Error("worldtree: invalid tree identity on '%s'", path.string()); - return Hash::parseNonSRIUnprefixed(std::string(value.data(), value.size()), HashAlgorithm::SHA1); -} - -Hash EvalState::getWorldTreeSha(std::string_view worldPath) const -{ - // The mutable root checkout needs its synthesized working-tree oid from the control - // plane. A historical evaluation is already pinned by its FUSE path and reads the - // exact committed tree oid from that directory's synthetic xattr instead. - if (isTectonixSourceAvailable()) { - auto control = worldtreeControlConn(); - if (!control) - goto local_git; - if (auto sha = control->zoneTreeSha(worldPath)) - return *sha; - throw Error("worldtree: world path '%s' is absent or outside this workspace's visibility scope", worldPath); - } - if (!settings.tectonixWorldtreeSocket.get().empty()) { - auto revision = worldtreeRevisionRoot(settings); - if (normalizeZonePath(worldPath).empty()) - return readWorldtreeTreeOid(revision); - auto [zoneId, relative] = worldtreeZoneLocation(getManifestJson(), worldPath); - auto path = revision / zoneId; - if (!relative.empty()) - path /= relative; - return readWorldtreeTreeOid(path); - } - -local_git: - auto path = normalizeZonePath(worldPath); - - // Check cache first - if (auto cached = getConcurrent(*worldTreeShaCache, path)) { - debug("getWorldTreeSha cache hit for '%s'", path); - return *cached; - } - - // Compute by walking from root - auto repo = getWorldRepo(); - auto & sha = requireTectonixGitSha(); - auto commitSha = Hash::parseNonSRIUnprefixed(sha, HashAlgorithm::SHA1); - - // Get the root tree SHA from the commit - auto rootTreeSha = repo->getCommitTree(commitSha); - - // Walk path components, caching intermediate results - Hash currentSha = rootTreeSha; - std::string currentPath; - - // Reuse cached accessor for path validation - auto accessor = getWorldGitAccessor(); - - for (auto & component : tokenizeString>(path, "/")) { - if (component.empty()) - continue; - if (component == ".." || component == ".") - throw Error("invalid path component '%s' in world path '%s'", component, worldPath); - - std::string nextPath = currentPath.empty() ? component : currentPath + "/" + component; - - // Check if this level is cached - if (auto cached = getConcurrent(*worldTreeShaCache, nextPath)) { - currentSha = *cached; - currentPath = nextPath; - continue; - } - - // Need to compute: get tree entry for this component - auto fullPath = CanonPath("/" + nextPath); - auto stat = accessor->maybeLstat(fullPath); - - if (!stat || stat->type != SourceAccessor::Type::tDirectory) - throw Error("path '%s' does not exist or is not a directory in world", nextPath); - - // Get the tree SHA for this subtree - currentSha = repo->getSubtreeSha(currentSha, component); - - // Cache this level. Note: concurrent threads may compute and insert the same - // path simultaneously. This is benign because they will compute the same SHA - // (deterministic from git tree), so either insertion succeeds or finds an - // equivalent value. We use try_emplace which is atomic for concurrent_flat_map. - worldTreeShaCache->try_emplace(nextPath, currentSha); - currentPath = nextPath; - } - - debug("getWorldTreeSha computed '%s' -> %s", path, currentSha.gitRev()); - return currentSha; -} - -const std::set & EvalState::getTectonixSparseCheckoutRoots() const -{ - std::call_once(tectonixSparseCheckoutRootsFlag, [this]() { - if (isTectonixSourceAvailable()) { - auto checkoutPath = settings.tectonixCheckoutPath.get(); - - // Read .git to find the actual git directory - // It can be either a directory or a file containing "gitdir: " - auto dotGitPath = std::filesystem::path(checkoutPath) / ".git"; - std::filesystem::path gitDir; - - if (std::filesystem::is_directory(dotGitPath)) { - gitDir = dotGitPath; - } else if (std::filesystem::is_regular_file(dotGitPath)) { - auto gitdirContent = readFile(dotGitPath.string()); - // Parse "gitdir: \n" - if (hasPrefix(gitdirContent, "gitdir: ")) { - auto path = trim(gitdirContent.substr(8)); - gitDir = std::filesystem::path(path); - // Handle relative paths - if (gitDir.is_relative()) - gitDir = std::filesystem::path(checkoutPath) / gitDir; - } - } - - if (!gitDir.empty()) { - // Read sparse-checkout-roots - auto sparseRootsPath = gitDir / "info" / "sparse-checkout-roots"; - if (std::filesystem::exists(sparseRootsPath)) { - auto content = readFile(sparseRootsPath.string()); - for (auto & line : tokenizeString>(content, "\n")) { - auto trimmed = trim(line); - if (!trimmed.empty()) - tectonixSparseCheckoutRoots.insert(std::string(trimmed)); - } - } - } - } - }); - return tectonixSparseCheckoutRoots; -} - -const std::map & EvalState::getTectonixDirtyZones() const -{ - std::call_once(tectonixDirtyZonesFlag, [this]() { - // A historical FUSE view is immutable by construction. Preserve the usual full - // manifest-shaped result (every visible zone present and clean) without opening a - // control connection or manufacturing an ephemeral daemon workspace. - if (!isTectonixSourceAvailable() && !settings.tectonixWorldtreeSocket.get().empty()) { - for (auto & [zonePath, value] : getManifestJson().items()) - if (value.is_object() && value.contains("id") && value.at("id").is_string()) - tectonixDirtyZones[zonePath] = {}; - return; - } - - // Worldtree mode: the daemon is authoritative for the tracked-dirty set (design - // §5.1), derived from the materialization frontier in O(changes) — no O(working-tree) - // `git status` scan. Reconstruct a *full* ZoneDirtyInfo so every consumer (notably - // the `__unsafeTectonixInternalDirtyZones` primop) sees every manifest zone with an - // accurate flag, matching the libgit2 path's shape: - // (1) init every manifest zone clean — the immutable FUSE view supplies it in - // historical mode and the checkout supplies it in mutable mode (see - // getManifestContent); either way it enumerates the full visible zone set; - // (2) overlay the daemon's per-zone dirty files for the bound mutable workspace. - if (auto control = worldtreeControlConn()) { - const nlohmann::json * manifest; - try { - manifest = &getManifestJson(); - } catch (nlohmann::json::parse_error & e) { - warn("failed to parse manifest for dirty zone detection: %s", e.what()); - return; - } catch (Error &) { - // Manifest unavailable (e.g. daemon refused) — fail loud rather than report a - // misleading empty/partial dirty set. - throw; - } - for (auto & [zonePath, value] : manifest->items()) - if (value.is_object() && value.contains("id") && value.at("id").is_string()) - tectonixDirtyZones[zonePath] = {}; - for (auto & entry : control->dirtyZoneEntries()) { - // A dirty zone the daemon names but the manifest omits still surfaces (a zone - // added in this workspace) — insert-or-update keeps the two sources unioned. - auto & info = tectonixDirtyZones[entry.zone]; - info.dirty = true; - for (auto & f : entry.files) - info.dirtyFiles.insert(f); - } - return; - } - - if (!isTectonixSourceAvailable()) - return; - - // Get sparse checkout roots (zone IDs) - auto & sparseRoots = getTectonixSparseCheckoutRoots(); - if (sparseRoots.empty()) - return; - - // Get manifest (uses cached parsed JSON) - const nlohmann::json * manifest; - try { - manifest = &getManifestJson(); - } catch (nlohmann::json::parse_error & e) { - warn("failed to parse manifest for dirty zone detection: %s", e.what()); - return; - } catch (Error &) { - // Manifest file not available (e.g., not in world repo) - return; - } - - // Build map of zone ID -> zone path for sparse roots only - std::map zoneIdToPath; - for (auto & [path, value] : manifest->items()) { - if (!value.contains("id") || !value.at("id").is_string()) { - warn("zone '%s' in manifest has missing or non-string 'id' field", path); - continue; - } - auto & id = value.at("id").get_ref(); - if (sparseRoots.count(id)) - zoneIdToPath[id] = path; - } - - // Initialize all sparse-checked-out zones as not dirty - for (auto & [zoneId, zonePath] : zoneIdToPath) { - tectonixDirtyZones[zonePath] = {}; - } - - // Create git command environment with environment variables - // GIT_DIR/GIT_WORK_TREE/GIT_COMMON_DIR removed since they affect - // git repository discovery - StringMap gitEnvironment = getEnv(); - gitEnvironment.erase("GIT_DIR"); - gitEnvironment.erase("GIT_WORK_TREE"); - gitEnvironment.erase("GIT_COMMON_DIR"); - - // Get dirty files via git status with -z for NUL-separated output - // This handles filenames with special characters correctly - auto checkoutPath = settings.tectonixCheckoutPath.get(); - auto [gitStatusCode, gitStatusOutput] = runProgram( - {.program = "git", - .args = {"-C", checkoutPath, "status", "--porcelain", "-z"}, - .environment = gitEnvironment}); - if (!statusOk(gitStatusCode)) { - // If git status fails, treat all zones as clean (fallback) - // This ensures call_once completes and we don't retry with partial state - warn( - "failed to get git status for dirty zone detection in '%s': program 'git' %s; treating all zones as clean", - checkoutPath, - statusToString(gitStatusCode)); - return; - } - - // Parse NUL-separated output - // Format with -z: XY SP path NUL [orig-path NUL for renames/copies] - size_t pos = 0; - while (pos < gitStatusOutput.size()) { - // Find the next NUL - auto nulPos = gitStatusOutput.find('\0', pos); - if (nulPos == std::string::npos) - break; - - auto entry = gitStatusOutput.substr(pos, nulPos - pos); - pos = nulPos + 1; - - // Git porcelain format: "XY PATH" where XY is 2-char status, then space, then path - // Minimum valid entry is "X P" (4 chars): status + space + 1-char path - if (entry.size() < 4) - continue; - - // XY is first 2 chars, then space, then path - char xy0 = entry[0]; - std::string rawPath = entry.substr(3); - - // Collect paths to check - destination path is always included - std::vector pathsToCheck; - pathsToCheck.push_back("/" + rawPath); - - // For renames (R) and copies (C), also process the original path - // Both source and destination zones should be marked dirty - if (xy0 == 'R' || xy0 == 'C') { - auto nextNul = gitStatusOutput.find('\0', pos); - if (nextNul != std::string::npos) { - auto origPath = gitStatusOutput.substr(pos, nextNul - pos); - pathsToCheck.push_back("/" + origPath); - pos = nextNul + 1; - } - } - - for (const auto & filePath : pathsToCheck) { - for (auto & [zonePath, info] : tectonixDirtyZones) { - auto normalized = "/" + normalizeZonePath(zonePath); - if (hasPrefix(filePath, normalized + "/") || filePath == normalized) { - info.dirty = true; - info.dirtyFiles.insert(filePath.substr(1)); - break; - } - } - } - } - - size_t dirtyCount = 0; - for (const auto & [_, info] : tectonixDirtyZones) - if (info.dirty) - dirtyCount++; - debug("computed dirty zones: %d of %d zones are dirty", dirtyCount, tectonixDirtyZones.size()); - }); - return tectonixDirtyZones; -} - -// Path to the tectonix manifest file within the world repository -static constexpr std::string_view TECTONIX_MANIFEST_PATH = "/.meta/manifest.json"; - -[[noreturn]] static void throwHistoricalWorldManifestError(const EvalSettings & settings, std::string_view detail) -{ - throw Error( - "worldtree: historical World manifest '.meta/manifest.json' for commit '%s' is missing or malformed: %s", - settings.tectonixGitSha.get(), - detail); -} - -const std::string & EvalState::getManifestContent() const -{ - // Cached for the lifetime of evaluation. This is intentional: evaluation is - // bound to a specific git SHA (tectonix-git-sha), so the manifest content is - // immutable for this EvalState instance. - std::call_once(tectonixManifestFlag, [this]() { - auto fullPath = CanonPath(TECTONIX_MANIFEST_PATH); - - // Mode A (`tec `, materialized checkout): the working tree is the source of - // truth and may carry uncommitted manifest edits (a zone added/removed in this - // sandbox), so read the local file — never a stale committed copy. - if (isTectonixSourceAvailable()) { - auto manifestPath = std::filesystem::path(settings.tectonixCheckoutPath.get()) / ".meta" / "manifest.json"; - if (std::filesystem::exists(manifestPath)) { - tectonixManifestContent = readFile(manifestPath); - debug("loaded manifest from checkout: %s", manifestPath.string()); - return; - } - } - - // Mode B (`tec --ref`, no checkout): manifest metadata is an ordinary immutable - // file in the FUSE projection. W-000000 is the reserved manifest pseudo-zone; it - // follows the same workspace visibility as the root checkout and needs no socket. - if (!settings.tectonixWorldtreeSocket.get().empty()) { - auto manifestPath = worldtreeRevisionRoot(settings) / "W-000000" / "manifest.json"; - std::error_code ec; - if (!std::filesystem::is_regular_file(manifestPath, ec)) - throwHistoricalWorldManifestError(settings, ec ? ec.message() : "file does not exist"); - try { - tectonixManifestContent = readFile(manifestPath); - } catch (const Error & e) { - throwHistoricalWorldManifestError(settings, e.what()); - } - debug("loaded manifest from immutable worldtree view: %s", manifestPath.string()); - return; - } - - // Socket unset (plain local eval): read the committed manifest via libgit2. - auto accessor = getWorldGitAccessor(); - if (!accessor->pathExists(fullPath)) - throw Error("manifest.json does not exist at %s in world", TECTONIX_MANIFEST_PATH); - - tectonixManifestContent = accessor->readFile(fullPath); - debug("loaded manifest from git at %s", fullPath); - }); - return tectonixManifestContent; -} - -const nlohmann::json & EvalState::getManifestJson() const -{ - std::call_once(tectonixManifestJsonFlag, [this]() { - try { - tectonixManifestJson = std::make_unique(nlohmann::json::parse(getManifestContent())); - } catch (const nlohmann::json::parse_error & e) { - if (!settings.tectonixWorldtreeSocket.get().empty() && !isTectonixSourceAvailable()) - throwHistoricalWorldManifestError(settings, e.what()); - throw; - } - }); - return *tectonixManifestJson; -} - -StorePath EvalState::getZoneStorePath(std::string_view zonePath) -{ - // A worldtree sandbox has two source regimes but only one filesystem accessor: - // the mutable root checkout path for ordinary evaluation, or the immutable - // commit/zone path for --ref. Only the former needs control RPCs for its dirty - // frontier identity. - if (!settings.tectonixWorldtreeSocket.get().empty()) { - if (isTectonixSourceAvailable()) { - auto control = worldtreeControlConn(); - if (!control) - throw Error("worldtree: mutable checkout has no control connection"); - auto treeSha = control->zoneTreeSha(zonePath); - if (!treeSha) - throw Error("worldtree: zone '%s' is absent or outside this workspace's visibility scope", zonePath); - auto fullPath = std::filesystem::path(settings.tectonixCheckoutPath.get()) / normalizeZonePath(zonePath); - if (!std::filesystem::is_directory(fullPath)) - throw Error("worldtree: zone '%s' is not materialized at '%s'", zonePath, fullPath.string()); - return worldtreeMountAccessor(*treeSha, zonePath, makeFSSourceAccessor(fullPath)); - } - - auto manifestIt = getManifestJson().find(std::string(zonePath)); - if (manifestIt == getManifestJson().end() || !manifestIt->is_object() || !manifestIt->contains("id") - || !manifestIt->at("id").is_string()) - throw Error("worldtree: zone '%s' is absent from the visible manifest", zonePath); - auto zoneId = requireWorldtreeZoneId(manifestIt->at("id").get()); - auto fullPath = worldtreeRevisionRoot(settings) / zoneId; - if (!std::filesystem::is_directory(fullPath)) - throw Error("worldtree: immutable zone '%s' is unavailable at '%s'", zonePath, fullPath.string()); - auto treeSha = readWorldtreeTreeOid(fullPath); - return worldtreeMountAccessor(treeSha, zonePath, makeFSSourceAccessor(fullPath)); - } - - // Check dirty status using original zonePath (with // prefix) since - // tectonixDirtyZones keys come directly from manifest with // prefix - const ZoneDirtyInfo * dirtyInfo = nullptr; - if (isTectonixSourceAvailable()) { - auto & dirtyZones = getTectonixDirtyZones(); - auto it = dirtyZones.find(std::string(zonePath)); - if (it != dirtyZones.end() && it->second.dirty) - dirtyInfo = &it->second; - } - - if (dirtyInfo) { - debug("getZoneStorePath: %s is dirty, using checkout", zonePath); - return getZoneFromCheckout(zonePath, &dirtyInfo->dirtyFiles); - } - - // Clean zone: get tree SHA - auto treeSha = getWorldTreeSha(zonePath); - - if (!settings.lazyTrees) { - debug("getZoneStorePath: %s clean, eager copy from git (tree %s)", zonePath, treeSha.gitRev()); - auto repo = getWorldRepo(); - auto commitHash = Hash::parseNonSRIUnprefixed(requireTectonixGitSha(), HashAlgorithm::SHA1); - auto opts = makeZoneAccessorOptions(repo, commitHash, normalizeZonePath(zonePath)); - auto accessor = repo->getAccessor(treeSha, opts, "zone"); - - std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); - auto storePath = - fetchToStore(fetchSettings, *store, SourcePath(accessor, CanonPath::root), FetchMode::Copy, name); - - allowPath(storePath); - return storePath; - } - - debug("getZoneStorePath: %s clean, lazy mount (tree %s)", zonePath, treeSha.gitRev()); - return mountZoneByTreeSha(treeSha, zonePath); -} - -StorePath EvalState::mountZoneByTreeSha(const Hash & treeSha, std::string_view zonePath) -{ - // Double-checked locking pattern for concurrent zone mounting: - // 1. Read lock check (fast path - allows concurrent readers) - { - auto cache = tectonixZoneCache_.readLock(); - auto it = cache->find(treeSha); - if (it != cache->end()) { - debug("zone cache hit for tree %s", treeSha.gitRev()); - return it->second; - } - } // Read lock released - - // 2. Write lock check (catch races between read unlock and write lock) - { - auto cache = tectonixZoneCache_.lock(); - auto it = cache->find(treeSha); - if (it != cache->end()) { - debug("zone cache hit for tree %s (after lock upgrade)", treeSha.gitRev()); - return it->second; - } - } // Write lock released - expensive work happens without holding lock - - // 3. Perform expensive git operations without holding lock. - // This allows concurrent mounts of different zones. Multiple threads may - // race to mount the same zone, but we check again before inserting. - auto repo = getWorldRepo(); - auto commitHash = Hash::parseNonSRIUnprefixed(requireTectonixGitSha(), HashAlgorithm::SHA1); - auto opts = makeZoneAccessorOptions(repo, commitHash, std::string(zonePath)); - auto accessor = repo->getAccessor(treeSha, opts, "zone"); - - // Generate name from zone path (sanitized for store path requirements) - std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); - - // Create virtual store path - auto storePath = StorePath::random(name); - - // 4. Re-acquire write lock and check again before mounting - auto cache = tectonixZoneCache_.lock(); - auto it = cache->find(treeSha); - if (it != cache->end()) { - // Another thread mounted while we were working - use their result - debug("zone cache hit for tree %s (after work)", treeSha.gitRev()); - return it->second; - } - - // Mount accessor at this path first, then allow the path. - // This order ensures we don't leave allowed paths without mounts on exception. - storeFS->mount(CanonPath(store->printStorePath(storePath)), accessor); - allowPath(storePath); - - // Insert into cache (we hold the lock, so this will succeed) - cache->emplace(treeSha, storePath); - - debug("mounted zone %s (tree %s) at %s", zonePath, treeSha.gitRev(), store->printStorePath(storePath)); - - return storePath; -} - -std::shared_ptr EvalState::connectWorldtree() const -{ - auto socketPath = settings.tectonixWorldtreeSocket.get(); - // The socket being unset is the *only* non-worldtree signal — plain local eval, where - // libgit2 is the source. Returning nullptr here routes callers to that path. - if (socketPath.empty()) - return nullptr; - // Fail-loud (the load-bearing invariant): with the socket SET there is no git repo to - // fall back to, so an unreachable daemon is a hard error — let `Client::connect`'s - // `ProtocolError` propagate rather than silently degrading to libgit2 (which would read - // the wrong content, or none). - auto ws = settings.tectonixWorldtreeWorkspace.get(); - return std::make_shared(worldtree::Client::connect(socketPath), ws); -} - -std::shared_ptr EvalState::worldtreeControlConn() const -{ - // Historical --ref evaluations are filesystem-only. Returning null here is not a - // libgit2 fallback: their callers branch on worldtree mode before consulting this - // control seam. - if (!isTectonixSourceAvailable()) - return nullptr; - std::call_once(worldtreeControlConnFlag, [this]() { worldtreeControlConn_ = connectWorldtree(); }); - return worldtreeControlConn_; -} - -StorePath -EvalState::worldtreeMountAccessor(const Hash & treeSha, std::string_view zonePath, ref accessor) -{ - std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); - - if (!settings.lazyTrees) { - // Eager: copy the zone content into the store now (content-addressed by content). - auto storePath = - fetchToStore(fetchSettings, *store, SourcePath(accessor, CanonPath::root), FetchMode::Copy, name); - allowPath(storePath); - return storePath; - } - - // Lazy-trees: mount at a virtual store path, deduplicated by the daemon's working-tree - // oid so a zone evaluated twice in one EvalState mounts once (same shape as - // mountZoneByTreeSha — the two share tectonixZoneCache_'s tree-oid keyspace). - { - auto cache = tectonixZoneCache_.readLock(); - if (auto it = cache->find(treeSha); it != cache->end()) - return it->second; - } - - auto storePath = StorePath::random(name); - - auto cache = tectonixZoneCache_.lock(); - if (auto it = cache->find(treeSha); it != cache->end()) - return it->second; - - storeFS->mount(CanonPath(store->printStorePath(storePath)), accessor); - allowPath(storePath); - cache->emplace(treeSha, storePath); - - debug("worldtree: mounted zone %s (tree %s) at %s", zonePath, treeSha.gitRev(), store->printStorePath(storePath)); - - return storePath; -} - -/** - * Overlays dirty files from disk on top of a clean git tree accessor. - */ -struct DirtyOverlaySourceAccessor : SourceAccessor -{ - ref base, disk; - boost::unordered_flat_set dirtyFiles, dirtyDirs; - - DirtyOverlaySourceAccessor( - ref base, ref disk, boost::unordered_flat_set && dirtyFiles) - : base(base) - , disk(disk) - , dirtyFiles(std::move(dirtyFiles)) - { - for (auto & f : this->dirtyFiles) { - for (auto p = CanonPath(f); !p.isRoot();) { - p.pop(); - if (!dirtyDirs.insert(p.rel().empty() ? "" : std::string(p.rel())).second) - break; - } - } - } - - bool isDirty(const CanonPath & path) - { - return dirtyFiles.contains(std::string(path.rel())); - } - - std::optional maybeLstat(const CanonPath & path) override - { - if (path.isRoot()) - return base->maybeLstat(path); - if (isDirty(path)) - return disk->maybeLstat(path); - auto s = base->maybeLstat(path); - if (s || !dirtyDirs.contains(std::string(path.rel()))) - return s; - return disk->maybeLstat(path); - } - - void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override - { - return (isDirty(path) ? disk : base)->readFile(path, sink, sizeCallback); - } - - std::string readLink(const CanonPath & path) override - { - return (isDirty(path) ? disk : base)->readLink(path); - } - - std::optional getPhysicalPath(const CanonPath & path) override - { - return (isDirty(path) ? disk : base)->getPhysicalPath(path); - } - - DirEntries readDirectory(const CanonPath & path) override - { - auto rel = path.isRoot() ? "" : std::string(path.rel()); - if (!path.isRoot() && !dirtyDirs.contains(rel)) - return base->readDirectory(path); - - DirEntries entries; - try { - entries = base->readDirectory(path); - } catch (...) { - } - - auto prefix = rel.empty() ? "" : rel + "/"; - for (auto & f : dirtyFiles) { - if (!f.starts_with(prefix)) - continue; - auto rest = std::string_view(f).substr(prefix.size()); - if (rest.find('/') != std::string_view::npos) - continue; - auto stat = disk->maybeLstat(path / rest); - if (stat) - entries[std::string(rest)] = stat->type; - else - entries.erase(std::string(rest)); - } - for (auto & d : dirtyDirs) { - if (!d.starts_with(prefix)) - continue; - auto rest = std::string_view(d).substr(prefix.size()); - if (rest.find('/') != std::string_view::npos || rest.empty()) - continue; - if (!entries.count(std::string(rest))) - entries[std::string(rest)] = Type::tDirectory; - } - return entries; - } -}; - -StorePath -EvalState::getZoneFromCheckout(std::string_view zonePath, const boost::unordered_flat_set * dirtyFiles) -{ - auto zone = normalizeZonePath(zonePath); - std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); - auto checkoutPath = settings.tectonixCheckoutPath.get(); - auto fullPath = std::filesystem::path(checkoutPath) / zone; - - auto makeDirtyAccessor = [&]() -> ref { - auto repo = getWorldRepo(); - auto commitHash = Hash::parseNonSRIUnprefixed(requireTectonixGitSha(), HashAlgorithm::SHA1); - auto zoneOpts = makeZoneAccessorOptions(repo, commitHash, zone); - auto baseAccessor = repo->getAccessor(getWorldTreeSha(zone), zoneOpts, "zone"); - boost::unordered_flat_set zoneDirtyFiles; - if (dirtyFiles) { - auto zonePrefix = zone + "/"; - for (auto & f : *dirtyFiles) - if (f.starts_with(zonePrefix)) - zoneDirtyFiles.insert(f.substr(zonePrefix.size())); - } - return make_ref( - baseAccessor, makeFSSourceAccessor(fullPath), std::move(zoneDirtyFiles)); - }; - - if (!settings.lazyTrees) { - auto accessor = makeDirtyAccessor(); - auto storePath = - fetchToStore(fetchSettings, *store, SourcePath(accessor, CanonPath::root), FetchMode::Copy, name); - allowPath(storePath); - return storePath; - } - - { - auto cache = tectonixCheckoutZoneCache_.readLock(); - auto it = cache->find(std::string(zonePath)); - if (it != cache->end()) - return it->second; - } - - auto cache = tectonixCheckoutZoneCache_.lock(); - auto it = cache->find(std::string(zonePath)); - if (it != cache->end()) - return it->second; - - if (!std::filesystem::exists(fullPath)) - throw Error("zone '%s' not found in checkout at '%s'", zonePath, fullPath.string()); - - auto storePath = StorePath::random(name); - storeFS->mount(CanonPath(store->printStorePath(storePath)), makeDirtyAccessor()); - allowPath(storePath); - cache->emplace(std::string(zonePath), storePath); - return storePath; -} - inline static bool isJustSchemePrefix(std::string_view prefix) { return !prefix.empty() && prefix[prefix.size() - 1] == ':' @@ -1434,8 +504,11 @@ void EvalState::checkURI(const std::string & uri0) Value * EvalState::addConstant(const std::string & name, Value & v, Constant info) { Value * v2 = allocValue(); - // Do a raw copy since `operator =` barfs on thunks. - memcpy((char *) v2, (char *) &v, sizeof(Value)); + // Do a raw storage copy since `operator =` barfs on thunks. Do not copy + // Value-level tracking certificates; constants are installed before any + // Tecnix dependency-producing evaluation owns their contents. + memcpy(static_cast(v2), static_cast(&v), sizeof(Value::Storage)); + v2->clearTrackedSourceAccessSet(); addConstant(name, v2, info); return v2; } @@ -1918,7 +991,7 @@ Value * EvalState::getBool(bool b) static Counter nrThunks; -static inline void mkThunk(Value & v, Env & env, Expr * expr) +static inline void mkThunk(EvalState &, Value & v, Env & env, Expr * expr) { v.mkThunk(&env, expr); nrThunks++; @@ -1926,7 +999,7 @@ static inline void mkThunk(Value & v, Env & env, Expr * expr) void EvalState::mkThunk_(Value & v, Expr * expr) { - mkThunk(v, baseEnv, expr); + mkThunk(*this, v, baseEnv, expr); } void EvalState::mkPos(Value & v, PosIdx p) @@ -2020,7 +1093,7 @@ void EvalState::mkSingleDerivedPathString(const SingleDerivedPath & p, Value & v Value * Expr::maybeThunk(EvalState & state, Env & env) { Value * v = state.allocValue(); - mkThunk(*v, env, this); + mkThunk(state, *v, env, this); return v; } @@ -2080,7 +1153,8 @@ struct ExprParseFile : Expr { printTalkative("evaluating file '%s'", path); - auto e = state.parseExprFromFile(path); + auto e = path.accessor->tracksEvalAccesses(path.path) ? state.parseExprFromFileCached(path) + : state.parseExprFromFile(path); try { auto dts = @@ -2104,40 +1178,79 @@ struct ExprParseFile : Expr void EvalState::evalFile(const SourcePath & path, Value & v, bool mustBeTrivial) { - auto resolvedPath = getConcurrent(*importResolutionCache, path); - - if (!resolvedPath) { - resolvedPath = resolveExprPath(path); - importResolutionCache->emplace(path, *resolvedPath); + auto trackingCtx = currentTecnixThreadState.trackingContext; + auto useTrackedImportResolutionCache = trackingCtx && path.accessor->tracksEvalAccesses(path.path); + auto & activeImportResolutionCache = useTrackedImportResolutionCache ? *tecnixData->trackedImportResolutionCache + : *tecnixData->importResolutionCache; + + auto cachedResolvedPath = getConcurrent(activeImportResolutionCache, path); + std::optional resolvedEntry; + + if (cachedResolvedPath) { + resolvedEntry = *cachedResolvedPath; + if (useTrackedImportResolutionCache && resolvedEntry->sourceDeps != emptyEvalSourceAccessSetId) + recordTrackedSourceAccessSetDependency(*trackingCtx, resolvedEntry->sourceDeps); + } else if (useTrackedImportResolutionCache) { + TrackedSourceDepsScope sourceDepsScope(*trackingCtx); + resolvedEntry.emplace( + EvalImportResolutionCacheEntry{ + .resolvedPath = resolveExprPath(path), + .sourceDeps = emptyEvalSourceAccessSetId, + }); + resolvedEntry->sourceDeps = sourceDepsScope.finish(); + activeImportResolutionCache.emplace(path, *resolvedEntry); + } else { + resolvedEntry.emplace(EvalImportResolutionCacheEntry{.resolvedPath = resolveExprPath(path)}); + activeImportResolutionCache.emplace(path, *resolvedEntry); } - if (auto v2 = getConcurrent(*fileEvalCache, *resolvedPath)) { + auto useTrackedFileEvalCache = + trackingCtx && resolvedEntry->resolvedPath.accessor->tracksEvalAccesses(resolvedEntry->resolvedPath.path); + auto & activeFileEvalCache = useTrackedFileEvalCache ? *tecnixData->trackedFileEvalCache : *fileEvalCache; + + if (auto v2 = getConcurrent(activeFileEvalCache, resolvedEntry->resolvedPath)) { forceValue(**v2, noPos); v = **v2; return; } Value * vExpr; - ExprParseFile expr{*resolvedPath, mustBeTrivial}; + ExprParseFile expr{resolvedEntry->resolvedPath, mustBeTrivial}; - fileEvalCache->try_emplace_and_cvisit( - *resolvedPath, + activeFileEvalCache.try_emplace_and_cvisit( + resolvedEntry->resolvedPath, nullptr, [&](auto & i) { vExpr = allocValue(); vExpr->mkThunk(&baseEnv, &expr); + nrThunks++; i.second = vExpr; }, [&](auto & i) { vExpr = i.second; }); - forceValue(*vExpr, noPos); + if (useTrackedFileEvalCache && trackingCtx) { + TrackedSourceDepsScope sourceDepsScope(*trackingCtx); + forceValue(*vExpr, noPos); + sourceDepsScope.finish(vExpr); + v = *vExpr; + return; + } + forceValue(*vExpr, noPos); v = *vExpr; } void EvalState::resetFileCache() { - importResolutionCache->clear(); + tecnixData->importResolutionCache->clear(); + tecnixData->trackedImportResolutionCache->clear(); + tecnixData->parsedFileCache->clear(); + // Deliberately NOT cleared: tecnixData->sourceAccessSetGraph. Value labels + // are graph-local IDs, and values carrying labels survive a file-cache + // reset (e.g. across a repl reload). Clearing the graph would re-mint IDs + // and make surviving labels resolve to the wrong paths. The graph is + // append-only, so retained IDs stay correct forever. + tecnixData->trackedFileEvalCache->clear(); fileEvalCache->clear(); inputCache->clear(); positions.clear(); @@ -2243,7 +1356,7 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v) Value * vAttr; if (hasOverrides && i.second.kind != AttrDef::Kind::Inherited) { vAttr = state.allocValue(); - mkThunk(*vAttr, *i.second.chooseByKind(&env2, &env, inheritEnv), i.second.e); + mkThunk(state, *vAttr, *i.second.chooseByKind(&env2, &env, inheritEnv), i.second.e); } else vAttr = i.second.e->maybeThunk(state, *i.second.chooseByKind(&env2, &env, inheritEnv)); env2.values[displ++] = vAttr; @@ -3474,6 +2587,13 @@ BackedStringView EvalState::coerceToString( StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePath & path, PosIdx pos) { + // srcToStore can make repeated path materialization a pure cache hit. The + // source path is still a dependency of the current target, so record it + // before consulting the cache. + if (auto trackingCtx = currentTecnixThreadState.trackingContext; + trackingCtx && path.accessor->tracksEvalAccesses(path.path)) + path.accessor->recordEvalAccess(path.path); + if (nix::isDerivation(path.path.abs())) error("file names are not allowed to end in '%1%'", drvExtension).debugThrow(); @@ -4114,6 +3234,22 @@ Expr * EvalState::parseExprFromFile(const SourcePath & path, const std::shared_p return parse(buffer.data(), buffer.size(), Pos::Origin(path), path.parent(), staticEnv); } +Expr * EvalState::parseExprFromFileCached(const SourcePath & path) +{ + path.accessor->recordEvalAccess(path.path); + + auto entry = getConcurrent(*tecnixData->parsedFileCache, path); + if (!entry) { + auto newEntry = std::make_shared(); + tecnixData->parsedFileCache->emplace(path, newEntry); + entry = getConcurrent(*tecnixData->parsedFileCache, path).value_or(newEntry); + } + + std::call_once((*entry)->once, [&]() { (*entry)->expr = parseExprFromFile(path); }); + assert((*entry)->expr); + return (*entry)->expr; +} + Expr * EvalState::parseExprFromString( std::string s_, const SourcePath & basePath, const std::shared_ptr & staticEnv) { diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index a7e65b3837..19ad2299fd 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -505,6 +505,29 @@ struct EvalSettings : Config Note that enabling the debugger (`--debugger`) disables multi-threaded evaluation. )"}; + Setting tecnixEvalCache{ + this, + true, + "tecnix-eval-cache", + R"( + Whether to use the Tecnix evaluation cache for target-dependency and + target-name discovery. + + Disabling this forces Tecnix dependency discovery to re-evaluate + instead of reusing results whose recorded source fingerprints still + match. + )"}; + + Setting tecnixParallelDependencies{ + this, + true, + "tecnix-parallel-dependencies", + R"( + Whether Tecnix target dependency evaluation may evaluate independent + target misses/cache validations in parallel when parallel evaluation is + enabled. + )"}; + Setting tectonixGitDir{ this, "~/world/git", diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index a32a7c865e..e8aec9d1e0 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -6,6 +6,7 @@ #include "nix/expr/eval-profiler.hh" #include "nix/util/types.hh" #include "nix/expr/value.hh" +#include "nix/expr/tecnix/source-deps.hh" #include "nix/expr/nixexpr.hh" #include "nix/expr/symbol-table.hh" #include "nix/util/configuration.hh" @@ -16,6 +17,7 @@ #include "nix/expr/search-path.hh" #include "nix/expr/repl-exit-status.hh" #include "nix/util/ref.hh" +#include "nix/util/finally.hh" #include "nix/expr/counter.hh" // For `NIX_USE_BOEHMGC`, and if that's set, `GC_THREADS` @@ -27,13 +29,17 @@ #include +#include +#include #include #include #include #include #include #include +#include #include +#include namespace nix { @@ -53,8 +59,9 @@ struct Input; } // namespace fetchers struct EvalSettings; class EvalState; -/** A connection to a worldtree daemon, defined in eval.cc (it owns the C++ worldtree - * client). Held by pointer here so the worldtree client header stays out of eval.hh. */ +/** A connection to a worldtree daemon, defined with the Tecnix source accessors + * (it owns the C++ worldtree client). Held by pointer here so the worldtree + * client header stays out of eval.hh. */ struct WorldtreeConn; class StorePath; struct SingleDerivedPath; @@ -237,9 +244,10 @@ struct StaticEvalSymbols { Symbol with, outPath, drvPath, type, meta, name, value, system, overrides, outputs, outputName, ignoreNulls, file, line, column, functor, toString, right, wrong, structuredAttrs, json, allowedReferences, allowedRequisites, - disallowedReferences, disallowedRequisites, maxSize, maxClosureSize, builder, args, contentAddressed, impure, - outputHash, outputHashAlgo, outputHashMode, recurseForDerivations, description, self, epsilon, startSet, - operator_, key, path, prefix, outputSpecified, __meta; + disallowedReferences, disallowedRequisites, maxSize, maxClosureSize, builder, args, gitDir, resolver, rev, + checkoutPath, targets, contentAddressed, impure, outputHash, outputHashAlgo, outputHashMode, + recurseForDerivations, description, self, epsilon, startSet, operator_, key, path, prefix, outputSpecified, + __meta; Expr::AstSymbols exprSymbols; @@ -277,6 +285,11 @@ struct StaticEvalSymbols .maxClosureSize = alloc.create("maxClosureSize"), .builder = alloc.create("builder"), .args = alloc.create("args"), + .gitDir = alloc.create("gitDir"), + .resolver = alloc.create("resolver"), + .rev = alloc.create("rev"), + .checkoutPath = alloc.create("checkoutPath"), + .targets = alloc.create("targets"), .contentAddressed = alloc.create("__contentAddressed"), .impure = alloc.create("__impure"), .outputHash = alloc.create("outputHash"), @@ -473,17 +486,20 @@ public: */ std::map> evalCaches; -private: + struct ZoneDirtyInfo + { + bool dirty = false; + boost::unordered_flat_set dirtyFiles; + }; - /* Cache for calls to addToStore(); maps source paths to the store - paths. */ - const ref> srcToStore; + struct TecnixEvalData; + +private: /** - * A cache that maps paths to "resolved" paths for importing Nix - * expressions, i.e. `/foo` to `/foo/default.nix`. + * A cache for source paths copied to the store. */ - const ref> importResolutionCache; + const ref> srcToStore; /** * A cache from resolved paths to values. @@ -512,109 +528,10 @@ private: */ const ref regexCache; - /** Lazy-initialized git repository for world builtins (thread-safe via once_flag) */ - mutable std::once_flag worldRepoFlag; - mutable std::optional> worldRepo; - - /** Lazy-initialized source accessor for world git content (thread-safe via once_flag) */ - mutable std::once_flag worldGitAccessorFlag; - mutable std::optional> worldGitAccessor; - - /** Cache: world path → tree SHA (lazy computed, cached at each path level) */ - const ref> worldTreeShaCache; - - /** Lazy-initialized set of zone IDs in sparse checkout (thread-safe via once_flag) */ - mutable std::once_flag tectonixSparseCheckoutRootsFlag; - mutable std::set tectonixSparseCheckoutRoots; - - /** Per-zone dirty status: whether the zone is dirty, and if so, which - * repo-relative file paths are dirty (from git status). */ - struct ZoneDirtyInfo - { - bool dirty = false; - boost::unordered_flat_set dirtyFiles; // repo-relative paths - }; - - /** Lazy-initialized map of zone path → dirty info (thread-safe via once_flag) */ - mutable std::once_flag tectonixDirtyZonesFlag; - mutable std::map tectonixDirtyZones; - - /** Cached manifest content (thread-safe via once_flag) */ - mutable std::once_flag tectonixManifestFlag; - mutable std::string tectonixManifestContent; - - /** Cached parsed manifest JSON (thread-safe via once_flag) */ - mutable std::once_flag tectonixManifestJsonFlag; - mutable std::unique_ptr tectonixManifestJson; - - /** - * Cache tree SHA → virtual store path for lazy zone mounts. - * Thread-safe for eval-cores > 1. - */ - mutable SharedSync> tectonixZoneCache_; - - /** - * Cache zone path → virtual store path for lazy checkout zone mounts. - * Thread-safe for eval-cores > 1. - */ - mutable SharedSync> tectonixCheckoutZoneCache_; - - /** - * Lazily-connected worldtree daemon control connection (zone tree shas + the dirty - * set), or null when the socket is unset or the evaluation targets an immutable - * historical FUSE view rather than the mutable root checkout. - * With the socket set an unreachable daemon THROWS rather than yielding null — there - * is no null-on-unreachable and no libgit2 fallback (fail-loud; see the - * `tectonix-worldtree-socket` setting doc). Connected at most once; thread-safe via - * once_flag. - */ - mutable std::once_flag worldtreeControlConnFlag; - mutable std::shared_ptr worldtreeControlConn_; - - /** - * Mount a zone by tree SHA, returning a (potentially virtual) store path. - * Caches by tree SHA for deduplication across world revisions. - */ - StorePath mountZoneByTreeSha(const Hash & treeSha, std::string_view zonePath); - - /** - * The mutable root-checkout control connection. Historical evaluations use the - * immutable FUSE projection and deliberately return null here without falling back - * to libgit2. A configured socket that is needed for a mutable checkout still fails - * loud when unreachable. - */ - std::shared_ptr worldtreeControlConn() const; - - /** - * Open a fresh connection to the configured worldtree socket + workspace for the - * mutable root checkout. Returns null when the socket is unset; a configured socket - * failure propagates. - */ - std::shared_ptr connectWorldtree() const; - - /** - * Devirtualization tail shared by both worldtree source paths (own-workspace root - * and immutable historical FUSE view): copy `accessor` to the store (eager) or mount - * it at a virtual store path (lazy-trees), deduplicating by the zone's `treeSha` - * (the daemon's working-tree oid — committed when clean, synthesized when dirty). - * The caller picks `accessor`; this owns the store-path identity and cache. - */ - StorePath worldtreeMountAccessor(const Hash & treeSha, std::string_view zonePath, ref accessor); - - /** - * Get zone store path from checkout (for dirty zones). - * With lazy-trees enabled, mounts lazily and caches by zone path. - */ - StorePath - getZoneFromCheckout(std::string_view zonePath, const boost::unordered_flat_set * dirtyFiles = nullptr); + const std::unique_ptr tecnixData; public: - /** - * Return the configured tectonix git SHA, or throw if unset. - */ - const std::string & requireTectonixGitSha() const; - /** * @param lookupPath Only used during construction. * @param store The store to use for instantiation @@ -644,48 +561,8 @@ public: return lookupPath; } - /** Get the world git repository, initializing lazily */ - ref getWorldRepo() const; - - /** - * Get accessor for world git content at worldSha. - * - * exportIgnore policy for tectonix accessors: - * - World accessor (getWorldGitAccessor): exportIgnore=false - * Used for path validation and tree SHA computation; needs to see all files - * - Zone accessors (mountZoneByTreeSha, getZoneStorePath): exportIgnore=true - * Used for actual zone content; honors .gitattributes for filtered output - * - Raw tree accessor (__unsafeTectonixInternalTree): exportIgnore=false - * Low-level access by SHA; provides unfiltered content - */ - ref getWorldGitAccessor() const; - - /** Get tree SHA for a world path, with lazy caching */ - Hash getWorldTreeSha(std::string_view worldPath) const; - - /** Check if we're in source-available mode */ - bool isTectonixSourceAvailable() const; - - /** Get set of zone IDs in sparse checkout (source-available mode only) */ - const std::set & getTectonixSparseCheckoutRoots() const; - - /** Get map of zone path → dirty status (only for sparse-checked-out zones) */ - const std::map & getTectonixDirtyZones() const; - - /** Get cached manifest content (thread-safe, lazy-loaded) */ - const std::string & getManifestContent() const; - - /** Get cached parsed manifest JSON (thread-safe, lazy-loaded) */ - const nlohmann::json & getManifestJson() const; - - /** - * Get a zone's store path, handling dirty detection and lazy mounting. - * - * For clean zones with lazy-trees enabled: mounts accessor lazily - * For dirty zones: currently eager-copies from checkout (extension point) - * For lazy-trees disabled: eager-copies from git - */ - StorePath getZoneStorePath(std::string_view zonePath); + TecnixEvalData & tecnixEvalData(); + const TecnixEvalData & tecnixEvalData() const; /** * Return a `SourcePath` that refers to `path` in the root @@ -749,6 +626,12 @@ public: Expr * parseExprFromFile(const SourcePath & path); Expr * parseExprFromFile(const SourcePath & path, const std::shared_ptr & staticEnv); + /** + * Parse a normal imported file through the shared parsed-file cache. + * Only suitable for imports using the evaluator's static base environment. + */ + Expr * parseExprFromFileCached(const SourcePath & path); + /** * Parse a Nix expression from the specified string. */ @@ -817,7 +700,13 @@ public: */ inline void forceValue(Value & v, const PosIdx pos) { - v.force(*this, pos); + auto * trackingCtx = currentTecnixThreadState.trackingContext; + if (!trackingCtx) { + v.force(*this, pos); + return; + } + + forceValueTracked(*this, v, pos, *trackingCtx); } void tryFixupBlackHolePos(Value & v, PosIdx pos); @@ -1306,10 +1195,19 @@ public: /** * Create a work item that propagates the current evaluation context. + * + * Tecnix tracked evaluation must not spawn work: work items capture only + * owned state, but a tracking context is a non-owning pointer into + * another thread's stack, and contexts are thread-confined by design (the + * only cross-thread dependency channel is the published label on a + * finished value). Detached prefetch sites skip spawning under tracking; + * anything else fails loudly here instead of dangling. */ template auto makeWork(T && t) { + if (currentTecnixThreadState.trackingContext) + throw Error("Tecnix tracked evaluation must not spawn parallel evaluation work"); return [this, t{std::move(t)}, evalContext(evalContext)]() { this->evalContext = evalContext; t(); @@ -1396,4 +1294,5 @@ struct PushProvenance } // namespace nix +#include "nix/expr/tecnix/force-value.hh" #include "nix/expr/eval-inline.hh" diff --git a/src/libexpr/include/nix/expr/meson.build b/src/libexpr/include/nix/expr/meson.build index 3191fd2dc1..35de84aefd 100644 --- a/src/libexpr/include/nix/expr/meson.build +++ b/src/libexpr/include/nix/expr/meson.build @@ -36,6 +36,14 @@ headers = [ config_pub_h ] + files( 'search-path.hh', 'static-string-data.hh', 'symbol-table.hh', + 'tecnix/access-set-graph.hh', + 'tecnix/force-value.hh', + 'tecnix/source-accessors.hh', + 'tecnix/source-deps.hh', + 'tecnix/thread-state.hh', + 'tecnix/value-hooks.hh', + 'tecnix/value-layout-tripwire.hh', + 'tecnix/value-methods.hh', 'value-to-json.hh', 'value-to-xml.hh', 'value.hh', diff --git a/src/libexpr/include/nix/expr/tecnix/access-set-graph.hh b/src/libexpr/include/nix/expr/tecnix/access-set-graph.hh new file mode 100644 index 0000000000..54eef0115d --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/access-set-graph.hh @@ -0,0 +1,79 @@ +#pragma once +///@file +/// +/// The interning graph behind Tecnix source-deps labels: paths become 32-bit +/// access ids, sets of paths become canonical 32-bit set ids (equal sets +/// share one id). Append-only for the EvalState's lifetime — labels held by +/// surviving values stay resolvable forever, so the graph is never cleared. +/// +/// Split from source-deps.hh so the tracking machinery that eval.hh inlines +/// (frames, contexts, forceValueTracked) does not pull the interning +/// containers into every evaluator translation unit. + +#include "nix/expr/tecnix/source-deps.hh" +#include "nix/util/strings.hh" + +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace nix { + +struct EvalSourceAccessSetNode +{ + uint32_t first = 0; + uint32_t count = 0; + uint32_t nextWithSameHash = 0; + uint64_t hash = 0; +}; + +class EvalSourceAccessSetGraph +{ + std::atomic enabled{false}; + mutable std::mutex mutex; + boost::concurrent_flat_map> accessIds; + std::vector accesses; + std::vector accessSets; + std::vector accessSetItems; + std::vector singletonAccessSets; + boost::unordered_flat_map accessSetIdsByHash; + boost::unordered_flat_map pairUnionAccessSets; + mutable std::vector seenAccessGenerations; + mutable uint32_t nextFlattenGeneration = 1; + + bool accessSetEquals(EvalSourceAccessSetId id, const std::vector & items) const; + +public: + EvalSourceAccessSetGraph(); + + bool isEnabled() const + { + return enabled.load(std::memory_order_acquire); + } + + void enable(); + + EvalSourceAccessId internAccess(std::string_view path); + EvalSourceAccessSetId internAccessSet( + std::span directAccesses, std::span children); + EvalSourceAccessSetId internAccessSet( + const std::vector & directAccesses, const std::vector & children); + std::string access(EvalSourceAccessId id) const; + + /** + * Resolve direct accesses plus the transitive members of `accessSetEdges` + * to unique repo-relative paths, in first-seen order. + */ + std::vector flatten( + const std::vector & directAccesses, + const std::vector & accessSetEdges) const; + EvalSourceAccessSetStats stats() const; +}; + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/eval-cache.hh b/src/libexpr/include/nix/expr/tecnix/eval-cache.hh new file mode 100644 index 0000000000..46e86e7420 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/eval-cache.hh @@ -0,0 +1,117 @@ +#pragma once +///@file +/// +/// The persistent Tecnix evaluation cache: bounded per-key source-closure +/// histories in SQLite, validated against current fingerprints (never trusted +/// by key; see plans/tecnix-target-eval-caching/). This header is the narrow +/// boundary the builtins use; the TXDC blob format, sharding, and SQLite +/// schema are implementation details of tecnix/eval-cache.cc. + +#include "nix/util/ref.hh" + +#include +#include +#include +#include +#include +#include +#include + +namespace nix { + +class EvalState; +struct SourceAccessor; +struct Value; + +struct DependencyEntry +{ + std::string path; + std::string fingerprint; +}; + +using DependencyClosure = std::vector; + +/** + * Per-run token for the thread-local fingerprint memo: fingerprints are + * memoized per unique path for the lifetime of one of these, so validation + * cost scales with unique paths, not total closure entries. + */ +struct DependencyFingerprintCache +{ + uint64_t generation; + + DependencyFingerprintCache(); +}; + +std::optional +dependencyFingerprint(ref accessor, std::string_view path, DependencyFingerprintCache & cache); + +/** Fingerprint every path, throwing if any path cannot be certified. */ +DependencyClosure dependencyFingerprints( + ref accessor, const std::vector & paths, DependencyFingerprintCache & cache); + +/** + * Reserved cache key under which target discovery stores its closure and + * discovered-name payload. Rejected as a caller-supplied target id. + */ +constexpr std::string_view tecnixTargetNamesCacheKey = "__tecnixTargetNames"; + +/** The (gitDir, resolver, argsKey) row family a lookup or upsert addresses. */ +struct TecnixCacheScope +{ + std::string_view gitDir; + std::string_view resolver; + std::string_view argsKey; +}; + +struct TecnixDependencyUpsert +{ + std::string_view target; + const DependencyClosure * dependencies; + /** Stored as the candidate's payload (e.g. discovery's target-name JSON); empty for none. */ + std::string payload; +}; + +/** + * A proven cache hit: one stored candidate whose complete closure matched + * current fingerprints. A move-only handle over the stored row bytes; output + * is built directly from the row (no decoded object graph). + */ +class ValidatedDependencyBlob +{ +public: + struct Impl; + + explicit ValidatedDependencyBlob(std::unique_ptr impl); + ValidatedDependencyBlob(ValidatedDependencyBlob &&) noexcept; + ValidatedDependencyBlob & operator=(ValidatedDependencyBlob &&) noexcept; + ~ValidatedDependencyBlob(); + + /** Dependency output (`path = fingerprint` attrs) built from the matched candidate's pair stream. */ + Value * toValue(EvalState & state) const; + + /** The matched candidate's payload (e.g. discovery's target-name JSON), if any. */ + std::optional payload() const; + +private: + std::unique_ptr impl; +}; + +/** + * Look up cached dependency rows for `keys` (target IDs or the discovery key) + * and validate their candidates against current fingerprints. An entry is set + * on a proven hit and nullopt on a miss. + */ +std::vector> lookupValidatedDependencyBlobs( + EvalState & state, + const TecnixCacheScope & scope, + std::span keys, + DependencyFingerprintCache & fingerprintCache); + +/** + * Persist freshly learned closures. Cache writes are an optimization: + * failures warn and continue, and never fail the evaluation. + */ +void upsertDependencyClosures(const TecnixCacheScope & scope, const std::vector & entries); + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/force-value.hh b/src/libexpr/include/nix/expr/tecnix/force-value.hh new file mode 100644 index 0000000000..124d2f6aa6 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/force-value.hh @@ -0,0 +1,45 @@ +#pragma once + +///@file + +namespace nix { + +[[gnu::always_inline]] inline void +forceValueTracked(EvalState & state, Value & v, const PosIdx pos, TrackingContext & trackingCtx) +{ + auto recordPublishedDependencies = [&]() { + auto sourceAccessSet = v.trackedSourceAccessSet(); + if (sourceAccessSet != emptyEvalSourceAccessSetId) + recordTrackedSourceAccessSetDependency(trackingCtx, sourceAccessSet); + }; + + if (v.isFinished()) { + recordPublishedDependencies(); + if (v.isFailed()) + v.force(state, pos); + return; + } + + if (!(v.isThunk() || v.isApp())) { + v.force(state, pos); + recordPublishedDependencies(); + return; + } + + TrackedSourceDepsFrame frame(trackingCtx, &v, currentTecnixThreadState.sourceDepsFrame); + auto * previousFrame = currentTecnixThreadState.sourceDepsFrame; + auto * previousPublishValue = currentTecnixThreadState.valueDependencyPublishValue; + currentTecnixThreadState.sourceDepsFrame = &frame; + currentTecnixThreadState.valueDependencyPublishValue = &v; + Finally restoreTrackedValueForceFrame([&]() { + currentTecnixThreadState.sourceDepsFrame = previousFrame; + currentTecnixThreadState.valueDependencyPublishValue = previousPublishValue; + mergeUnpublishedTrackedSourceDepsFrame(frame); + }); + + v.force(state, pos); + if (!frame.published) + recordPublishedDependencies(); +} + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/source-accessors.hh b/src/libexpr/include/nix/expr/tecnix/source-accessors.hh new file mode 100644 index 0000000000..7899883da2 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/source-accessors.hh @@ -0,0 +1,26 @@ +#pragma once +///@file + +#include "nix/expr/eval.hh" + +namespace nix { + +void configureTectonixContext(EvalState & state, std::string gitDir, std::string rev, std::string checkoutPath); +ref getWorldRepo(const EvalState & state); +const std::string & requireTectonixGitSha(const EvalState & state); +ref getWorldGitAccessor(const EvalState & state); +Hash getWorldTreeSha(const EvalState & state, std::string_view worldPath); +bool isTectonixSourceAvailable(const EvalState & state); +const std::set & getTectonixSparseCheckoutRoots(const EvalState & state); +const std::map & getTectonixDirtyZones(const EvalState & state); +const std::string & getManifestContent(const EvalState & state); +const nlohmann::json & getManifestJson(const EvalState & state); +StorePath getLegacyTectonixZoneStorePath(EvalState & state, std::string_view zonePath); +ref getTecnixRepoAccessor(EvalState & state); + +/** Resolve a checkout's HEAD to a commit SHA (throws if it has none). */ +std::string resolveCheckoutHeadRev(const std::string & checkoutPath); +StorePath mountTecnixRepoAccessor(EvalState & state); +std::string getTecnixRepoPath(EvalState & state, std::string_view repoRelPath); + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/source-deps.hh b/src/libexpr/include/nix/expr/tecnix/source-deps.hh new file mode 100644 index 0000000000..16de669620 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/source-deps.hh @@ -0,0 +1,131 @@ +#pragma once +///@file + +#include "nix/expr/tecnix/thread-state.hh" +#include "nix/util/pos-idx.hh" +#include "nix/util/ref.hh" + +#include + +#include +#include +#include +#include +#include +#include + +namespace nix { + +class EvalState; +struct Value; + +using EvalSourceAccessId = uint32_t; +using EvalSourceAccessSetId = uint32_t; +static constexpr EvalSourceAccessId emptyEvalSourceAccessId = 0; +static constexpr EvalSourceAccessSetId emptyEvalSourceAccessSetId = 0; + +class EvalSourceAccessSetGraph; +struct TrackingContext; + +struct EvalSourceAccessSetStats +{ + size_t accesses = 0; + size_t accessSets = 0; + size_t accessSetItems = 0; +}; + +void enableSourceAccessSetTracking(EvalState & state); +EvalSourceAccessSetStats trackedSourceAccessSetStats(const EvalState & state); +ref trackedSourceAccessSetGraph(const EvalState & state); +EvalSourceAccessSetId publishTrackedSourceAccessSetDependencies( + EvalSourceAccessSetGraph & graph, + Value & v, + std::span directAccesses, + std::span children); +void recordTrackedSourceAccessSetAccess(EvalSourceAccessId access); +void recordTrackedSourceAccessSetDependency(TrackingContext & trackingCtx, EvalSourceAccessSetId accessSet); +void mergeUnpublishedTrackedSourceDepsFrame(TrackedSourceDepsFrame & frame); +[[gnu::always_inline]] inline void +forceValueTracked(EvalState & state, Value & v, PosIdx pos, TrackingContext & trackingCtx); + +std::vector parseGitPorcelainZDirtyPaths(std::string_view output); + +using EvalSourceAccessIdFrameVector = boost::container::small_vector; +using EvalSourceAccessSetIdFrameVector = boost::container::small_vector; + +/** + * A stack-resident accumulator for one bracketed region of evaluation: the + * force of one value (`value` set) or a source-deps scope / target root + * (`value` null). Collects direct path accesses and inherited child labels; + * interned into one set id when the region publishes. + */ +struct TrackedSourceDepsFrame +{ + TrackingContext & trackingCtx; + Value * value = nullptr; + EvalSourceAccessIdFrameVector directSourceAccessSetAccesses; + EvalSourceAccessSetIdFrameVector childSourceAccessSets; + EvalSourceAccessSetId accessSet = emptyEvalSourceAccessSetId; + TrackedSourceDepsFrame * previous = nullptr; + TrackedSourceDepsFrame * nearestValueForceFrame = nullptr; + bool published = false; + + TrackedSourceDepsFrame( + TrackingContext & trackingCtx, Value * value = nullptr, TrackedSourceDepsFrame * previous = nullptr); +}; + +/** + * Tracks file/directory accesses during Tecnix target resolution and + * target-name discovery for cache invalidation. Paths are repo-relative + * (e.g. "areas/core/shopify/default.nix"). + * + * Tracking contexts are thread-confined: a context is created, recorded + * into, snapshotted, and destroyed on one thread, so it needs no locking. + * Tracked evaluation must not spawn parallel evaluation work (enforced in + * EvalState::makeWork); the only cross-thread dependency channel is the + * published label on a finished value. + * + * Tracking contexts must use the EvalState-owned source-access graph. Inline + * Value labels are graph-local IDs, so constructing a context with a private + * graph would silently interpret copied/forced value labels as the wrong paths. + * + * Constructing a context enables the graph, establishing the invariant the + * hot paths rely on: a live context implies an enabled graph. + */ +struct TrackingContext +{ + ref sourceAccessSetGraph; + TrackedSourceDepsFrame rootFrame; + + // Always captures the EvalState-owned source-access graph; no foreign graph constructor exists. + explicit TrackingContext(EvalState & state); + + void recordAccess(std::string_view path); +}; + +struct ActiveTrackingContext +{ + TrackingContext & trackingCtx; + TrackingContext * previousTrackingCtx = nullptr; + TrackedSourceDepsFrame * previousFrame = nullptr; + + explicit ActiveTrackingContext(TrackingContext & trackingCtx); + ActiveTrackingContext(const ActiveTrackingContext &) = delete; + ActiveTrackingContext & operator=(const ActiveTrackingContext &) = delete; + ~ActiveTrackingContext(); +}; + +struct TrackedSourceDepsScope +{ + TrackedSourceDepsFrame frame; + TrackedSourceDepsFrame * previousFrame = nullptr; + + explicit TrackedSourceDepsScope(TrackingContext & trackingCtx); + TrackedSourceDepsScope(const TrackedSourceDepsScope &) = delete; + TrackedSourceDepsScope & operator=(const TrackedSourceDepsScope &) = delete; + ~TrackedSourceDepsScope(); + + EvalSourceAccessSetId finish(Value * publishValue = nullptr); +}; + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/thread-state.hh b/src/libexpr/include/nix/expr/tecnix/thread-state.hh new file mode 100644 index 0000000000..4a83e3a498 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/thread-state.hh @@ -0,0 +1,19 @@ +#pragma once + +///@file + +namespace nix { + +struct TrackingContext; +struct TrackedSourceDepsFrame; + +struct TecnixThreadState +{ + TrackingContext * trackingContext = nullptr; + TrackedSourceDepsFrame * sourceDepsFrame = nullptr; + const void * valueDependencyPublishValue = nullptr; +}; + +[[gnu::tls_model("initial-exec")]] extern thread_local TecnixThreadState currentTecnixThreadState; + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/value-hooks.hh b/src/libexpr/include/nix/expr/tecnix/value-hooks.hh new file mode 100644 index 0000000000..4ad6e85fe6 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/value-hooks.hh @@ -0,0 +1,113 @@ +#pragma once +///@file + +#include "nix/expr/tecnix/thread-state.hh" + +#include +#include +#include + +namespace nix { + +/** + * Tecnix source-access labels live outside Value, in a two-level sparse + * table: a constant-initialized directory indexed by the top address bits, + * pointing at lazily-mapped shadow chunks holding one 32-bit slot per + * 16-byte-aligned value cell. Chunks are allocated only by the first + * nonzero label store in their 4 GiB region, so a process that never runs + * tracked evaluation allocates nothing; loads and clears on absent chunks + * are no-ops (absent means all-zero). The directory lives in zeroed BSS, + * so no dynamic initializer is involved and values finished during static + * initialization are handled correctly by construction. + */ +constexpr size_t tecnixValueLabelDirSize = size_t{1} << 16; // addr >> 32; covers [0, 2^48) + +extern std::atomic tecnixValueLabelDir[tecnixValueLabelDirSize]; + +uint32_t * tecnixInstallValueLabelChunk(size_t dirIndex); +[[noreturn]] void tecnixValueLabelOutOfRange(const void * value); + +[[gnu::always_inline]] inline uint32_t tecnixValueLabelLoad(const void * value, std::memory_order order) noexcept +{ + auto addr = reinterpret_cast(value); + auto dirIndex = addr >> 32; + if (dirIndex >= tecnixValueLabelDirSize) [[unlikely]] + return 0; // nonzero stores to uncovered addresses abort, so no label can exist here + // Relaxed is sound: chunks are kernel-zeroed before their pointer is + // CAS-released into the directory, and all slot access is through an + // address dependency on the loaded pointer (the rcu_dereference pattern). + auto * chunk = tecnixValueLabelDir[dirIndex].load(std::memory_order_relaxed); + if (!chunk) + return 0; + return std::atomic_ref(chunk[(addr & 0xffffffff) >> 4]).load(order); +} + +[[gnu::always_inline]] inline void +tecnixValueLabelStore(const void * value, uint32_t accessSet, std::memory_order order) noexcept +{ + auto addr = reinterpret_cast(value); + auto dirIndex = addr >> 32; + if (dirIndex >= tecnixValueLabelDirSize) [[unlikely]] { + if (accessSet != 0) + tecnixValueLabelOutOfRange(value); + return; + } + auto * chunk = tecnixValueLabelDir[dirIndex].load(std::memory_order_relaxed); + if (!chunk) { + if (accessSet == 0) + return; // clearing an absent chunk: already zero + chunk = tecnixInstallValueLabelChunk(dirIndex); + } + std::atomic_ref(chunk[(addr & 0xffffffff) >> 4]).store(accessSet, order); +} + +/** + * Clear-if-set: reads of untouched demand-zero pages map the shared zero + * page and commit nothing, so eliding the 0-over-0 store keeps the table's + * physical footprint proportional to labels actually published rather than + * to every value ever finished inside a chunk's region. + */ +[[gnu::always_inline]] inline void tecnixValueLabelClear(const void * value) noexcept +{ + auto addr = reinterpret_cast(value); + auto dirIndex = addr >> 32; + if (dirIndex >= tecnixValueLabelDirSize) [[unlikely]] + return; + auto * chunk = tecnixValueLabelDir[dirIndex].load(std::memory_order_relaxed); + if (!chunk) + return; + auto slot = std::atomic_ref(chunk[(addr & 0xffffffff) >> 4]); + if (slot.load(std::memory_order_relaxed) != 0) + slot.store(0, std::memory_order_relaxed); +} + +void publishTrackedValueDependencies(const void * value); +void copyTrackedValueDependencies(void * dst, const void * src); +void publishCopiedValueDependencies(void * dst, const void * src); + +/** + * ValueStorage::finish is the single chokepoint through which every cell + * becomes a finished value, so clearing the label slot here is what + * guarantees a label is never stale: whatever the slot held for a previous + * occupant of this address (a reused GC cell, a reused stack slot), the + * finished value starts empty and receives its label from the publish that + * follows, ordered before the cell is observable as finished. + */ +inline void tecnixValueFinishHook(const void * value) +{ + tecnixValueLabelClear(value); + if (currentTecnixThreadState.valueDependencyPublishValue == value) + publishTrackedValueDependencies(value); +} + +inline void tecnixValueCopyBeforeFinish(void * dst, const void * src) +{ + copyTrackedValueDependencies(dst, src); +} + +inline void tecnixValueCopyAfterFinish(void * dst, const void * src) +{ + publishCopiedValueDependencies(dst, src); +} + +} // namespace nix diff --git a/src/libexpr/include/nix/expr/tecnix/value-layout-tripwire.hh b/src/libexpr/include/nix/expr/tecnix/value-layout-tripwire.hh new file mode 100644 index 0000000000..43714edf4d --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/value-layout-tripwire.hh @@ -0,0 +1,6 @@ +#pragma once +///@file + +static_assert( + sizeof(void *) != 8 || sizeof(Value) == 16, + "Tecnix value labels live in the sparse label table; Value must stay at upstream's pointer-pair size"); diff --git a/src/libexpr/include/nix/expr/tecnix/value-methods.hh b/src/libexpr/include/nix/expr/tecnix/value-methods.hh new file mode 100644 index 0000000000..d072a34222 --- /dev/null +++ b/src/libexpr/include/nix/expr/tecnix/value-methods.hh @@ -0,0 +1,40 @@ +#pragma once + +///@file + +inline Value::Value(const Value & v) +{ + Value::Storage::operator=(v); +} + +inline Value::Value(Value && v) noexcept +{ + Value::Storage::operator=(v); +} + +inline Value & Value::operator=(const Value & v) +{ + Value::Storage::operator=(v); + return *this; +} + +inline Value & Value::operator=(Value && v) noexcept +{ + Value::Storage::operator=(v); + return *this; +} + +inline uint32_t Value::trackedSourceAccessSet() const noexcept +{ + return tecnixValueLabelLoad(this, std::memory_order_acquire); +} + +inline void Value::setTrackedSourceAccessSet(uint32_t accessSet) noexcept +{ + tecnixValueLabelStore(this, accessSet, std::memory_order_release); +} + +inline void Value::clearTrackedSourceAccessSet() noexcept +{ + tecnixValueLabelClear(this); +} diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index f41f7f89af..edeb1d6bf9 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -18,6 +18,7 @@ #include "nix/expr/value/context.hh" #include "nix/util/source-path.hh" #include "nix/expr/print-options.hh" +#include "nix/expr/tecnix/value-hooks.hh" #include "nix/util/checked-arithmetic.hh" #include @@ -680,6 +681,8 @@ class alignas(16) void finish(PackedPointer p0_, PackedPointer p1_) { + tecnixValueFinishHook(this); + // Note: p1 *must* be updated before p0. p1 = p1_; p0_ = p0.exchange(p0_, std::memory_order_release); @@ -925,7 +928,9 @@ protected: auto pd = static_cast(p0_ & discriminatorMask); if (pd == pdThunk || pd == pdPending || pd == pdAwaited) unreachable(); + tecnixValueCopyBeforeFinish(this, &v); finish(p0_, p1_); + tecnixValueCopyAfterFinish(this, &v); return *this; } @@ -1165,8 +1170,22 @@ static_assert(std::random_access_iterator); struct Value : public ValueStorage { + using Storage = ValueStorage; + friend std::string showType(const Value & v); +public: + Value() = default; + + Value(const Value & v); + Value(Value && v) noexcept; + Value & operator=(const Value & v); + Value & operator=(Value && v) noexcept; + + uint32_t trackedSourceAccessSet() const noexcept; + void setTrackedSourceAccessSet(uint32_t accessSet) noexcept; + void clearTrackedSourceAccessSet() noexcept; + /** * Empty list constant. * @@ -1532,6 +1551,9 @@ public: } }; +#include "nix/expr/tecnix/value-methods.hh" +#include "nix/expr/tecnix/value-layout-tripwire.hh" + typedef std::vector> ValueVector; typedef boost::unordered_flat_map< Symbol, diff --git a/src/libexpr/meson.build b/src/libexpr/meson.build index b99cbb6d6c..793fdbbd08 100644 --- a/src/libexpr/meson.build +++ b/src/libexpr/meson.build @@ -186,6 +186,10 @@ sources = files( 'provenance.cc', 'search-path.cc', 'symbol-table.cc', + 'tecnix/eval-cache.cc', + 'tecnix/repo-accessor.cc', + 'tecnix/source-accessors.cc', + 'tecnix/source-deps.cc', 'value-to-json.cc', 'value-to-xml.cc', 'value.cc', diff --git a/src/libexpr/parallel-eval.cc b/src/libexpr/parallel-eval.cc index 50e91e6ee3..9b9bebcaa5 100644 --- a/src/libexpr/parallel-eval.cc +++ b/src/libexpr/parallel-eval.cc @@ -285,7 +285,10 @@ static void prim_parallel(EvalState & state, const PosIdx pos, Value ** args, Va { state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.parallel"); - if (state.executor->enabled) { + /* Tecnix: tracking contexts are thread-confined, so tracked evaluation + must not spawn detached work; under tracking this primop degrades to + sequential evaluation. */ + if (state.executor->enabled && !currentTecnixThreadState.trackingContext) { Executor::WorkItems work; for (auto value : args[0]->listView()) if (!value->isFinished()) diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index b34fffb81f..e71300c791 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -978,7 +978,10 @@ static RegisterPrimOp primop_break( state.runDebugRepl(&error); } - // Return the value we were passed. + // Return the value we were passed. Since `Value::operator=` cannot + // copy thunks, force first; this also waits if another parallel worker + // is already forcing the same argument. + state.forceValue(*args[0], pos); v = *args[0]; }}); @@ -2042,6 +2045,9 @@ static void prim_pathExists(EvalState & state, const PosIdx pos, Value ** args, auto symlinkResolution = mustBeDir ? SymlinkResolution::Full : SymlinkResolution::Ancestors; auto path = state.realisePath(pos, arg, symlinkResolution); + if (currentTecnixThreadState.trackingContext && path.accessor->tracksEvalAccesses(path.path)) + path.accessor->recordEvalAccess(path.path); + auto st = path.maybeLstat(); auto exists = st && (!mustBeDir || st->type == SourceAccessor::tDirectory); v.mkBool(exists); @@ -2455,6 +2461,9 @@ static const Value & fileTypeToString(EvalState & state, SourceAccessor::Type ty static void prim_readFileType(EvalState & state, const PosIdx pos, Value ** args, Value & v) { auto path = state.realisePath(pos, *args[0], std::nullopt); + if (currentTecnixThreadState.trackingContext && path.accessor->tracksEvalAccesses(path.path)) + path.accessor->recordEvalAccess(path.path); + /* Retrieve the directory entry type and stringize it. */ v = fileTypeToString(state, path.lstat().type); } @@ -2882,6 +2891,15 @@ static RegisterPrimOp primop_toFile({ bool EvalState::callPathFilter(Value * filterFun, const SourcePath & path, PosIdx pos) { + /* Tecnix: path filters run arbitrary Nix code from inside dumpPath, where + source-access tracking is suppressed (the dumped tree is covered by its + directory-level fingerprint). The filter's own source reads are real + dependencies — e.g. a readFile of an ignore list — and a first read + here would otherwise be cached unlabeled, under-tracking every later + consumer. Lift the suppression for the duration of the call. */ + auto savedDumpPathDepth = std::exchange(SourceAccessor::dumpPathDepth, 0); + Finally restoreDumpPathDepth([&]() { SourceAccessor::dumpPathDepth = savedDumpPathDepth; }); + auto st = path.lstat(); /* Call the filter function. The first argument is the path, the diff --git a/src/libexpr/primops/meson.build b/src/libexpr/primops/meson.build index 3ec85a9d41..50f510a3c0 100644 --- a/src/libexpr/primops/meson.build +++ b/src/libexpr/primops/meson.build @@ -9,6 +9,7 @@ sources += files( 'fetchMercurial.cc', 'fetchTree.cc', 'fromTOML.cc', + 'tecnix.cc', 'tectonix.cc', ) diff --git a/src/libexpr/primops/tecnix.cc b/src/libexpr/primops/tecnix.cc new file mode 100644 index 0000000000..8f1065c227 --- /dev/null +++ b/src/libexpr/primops/tecnix.cc @@ -0,0 +1,950 @@ +/** + * The public Tecnix builtins (`builtins.tecnixTargets`, + * `builtins.tecnixTargetNames`, and the internal source-deps scope + * builtins): argument parsing, canonical args-key JSON, and per-target + * orchestration over the tracked accessors (tecnix/repo-accessor.cc) and + * the persistent cache (tecnix/eval-cache.cc). + */ + +#include "nix/expr/eval-inline.hh" +#include "nix/expr/eval-settings.hh" +#include "nix/expr/parallel-eval.hh" +#include "nix/expr/primops.hh" +#include "nix/expr/tecnix/access-set-graph.hh" +#include "nix/expr/tecnix/eval-cache.hh" +#include "nix/expr/tecnix/source-accessors.hh" +#include "nix/util/strings.hh" +#include "nix/util/util.hh" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace nix { + +// ============================================================================ +// builtins.tecnixInternalSourceDepsScope value +// Evaluates a lazy value under a reusable source-deps scope for Tecnix +// dependency tracking. Outside tracking eval, this is an identity. +// ============================================================================ +static void prim_tecnixInternalSourceDepsScope(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) { + TrackedSourceDepsScope scope(*ctx); + state.forceValue(*args[0], pos); + v = *args[0]; + scope.finish(&v); + return; + } + + state.forceValue(*args[0], pos); + v = *args[0]; +} + +static RegisterPrimOp primop_tecnixInternalSourceDepsScope({ + .name = "__tecnixInternalSourceDepsScope", + .args = {"value"}, + .doc = R"( + Mark `value` as being evaluated under a reusable Tecnix source-deps scope. + + This is an internal exact-dependency-tracking primitive. It behaves like + `value`, but while tracked Tecnix evaluation forces the wrapper, source + accesses are collected into a label that later reuse of the already-forced + wrapper can inherit. + )", + .impl = prim_tecnixInternalSourceDepsScope, +}); + +static Value * makeTecnixSourceDepsScopeApplication(EvalState & state, Value * value) +{ + auto * scoped = state.allocValue(); + scoped->mkApp(&state.getBuiltin("tecnixInternalSourceDepsScope"), value); + return scoped; +} + +// ============================================================================ +// builtins.tecnixInternalSourceDepsAttrs attrs +// Returns an attrset whose values are lazily wrapped in source-deps scopes. +// ============================================================================ +static void prim_tecnixInternalSourceDepsAttrs(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + state.forceAttrs(*args[0], pos, "while evaluating the 'attrs' argument to builtins.tecnixInternalSourceDepsAttrs"); + + auto bindings = state.buildBindings(args[0]->attrs()->size()); + for (auto & attr : *args[0]->attrs()) + bindings.insert(attr.name, makeTecnixSourceDepsScopeApplication(state, attr.value), attr.pos); + v.mkAttrs(bindings); +} + +static RegisterPrimOp primop_tecnixInternalSourceDepsAttrs({ + .name = "__tecnixInternalSourceDepsAttrs", + .args = {"attrs"}, + .doc = R"( + Internal Tecnix helper: return an attrset whose values are lazily wrapped + in `tecnixInternalSourceDepsScope`. + )", + .impl = prim_tecnixInternalSourceDepsAttrs, +}); + +// ============================================================================ +// builtins.tecnixInternalSourceDepsList list +// Returns a list whose elements are lazily wrapped in source-deps scopes. +// ============================================================================ +static void prim_tecnixInternalSourceDepsList(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + state.forceList(*args[0], pos, "while evaluating the 'list' argument to builtins.tecnixInternalSourceDepsList"); + + auto list = state.buildList(args[0]->listSize()); + size_t index = 0; + for (auto * elem : args[0]->listView()) + list[index++] = makeTecnixSourceDepsScopeApplication(state, elem); + v.mkList(list); +} + +static RegisterPrimOp primop_tecnixInternalSourceDepsList({ + .name = "__tecnixInternalSourceDepsList", + .args = {"list"}, + .doc = R"( + Internal Tecnix helper: return a list whose elements are lazily wrapped in + `tecnixInternalSourceDepsScope`. + )", + .impl = prim_tecnixInternalSourceDepsList, +}); + +// ============================================================================ +// Shared helpers for Tecnix target evaluation and dependency-path tracking. +// ============================================================================ + +/** + * Resolve the git SHA to use: explicit rev attr > checkout HEAD > error. + */ +static std::string +resolveRev(EvalState & state, const PosIdx pos, const Bindings & attrs, const std::string & checkoutPath) +{ + // Check for explicit rev attr + auto revAttr = attrs.get(state.s.rev); + if (revAttr) { + auto sha = state.forceStringNoCtx(*revAttr->value, pos, "while evaluating the 'rev' argument"); + if (!sha.empty()) + return std::string(sha); + } + + // Try to read HEAD from checkout. + if (!checkoutPath.empty()) { + try { + return resolveCheckoutHeadRev(checkoutPath); + } catch (Error & e) { + state + .error( + "could not determine git SHA from checkoutPath '%s': %s; set 'rev' or provide a valid 'checkoutPath'", + checkoutPath, + e.what()) + .atPos(pos) + .debugThrow(); + } + } + + state.error("could not determine git SHA: set 'rev' or provide a valid 'checkoutPath'") + .atPos(pos) + .debugThrow(); +} + +struct TecnixArgs +{ + std::string gitDir; + std::string resolver; + std::string rev; + std::string checkoutPath; + Value * resolverArgs = nullptr; + std::string argsKey; + std::vector targets; +}; + +/** The persistent-cache row family these arguments address. */ +static TecnixCacheScope cacheScope(const TecnixArgs & args) +{ + return {args.gitDir, args.resolver, args.argsKey}; +} + +static const Bindings & forceTecnixBuiltinAttrs(EvalState & state, const PosIdx pos, Value ** args) +{ + state.forceAttrs(*args[0], pos, "while evaluating the argument to a tecnix builtin"); + return *args[0]->attrs(); +} + +static void parseTecnixRepoArgs(EvalState & state, const PosIdx pos, const Bindings & attrs, TecnixArgs & result) +{ + auto gitDirAttr = attrs.get(state.s.gitDir); + if (!gitDirAttr) + state.error("'gitDir' attribute required").atPos(pos).debugThrow(); + result.gitDir = + std::string(state.forceStringNoCtx(*gitDirAttr->value, pos, "while evaluating the 'gitDir' argument")); + + auto resolverAttr = attrs.get(state.s.resolver); + if (!resolverAttr) + state.error("'resolver' attribute required").atPos(pos).debugThrow(); + result.resolver = + std::string(state.forceStringNoCtx(*resolverAttr->value, pos, "while evaluating the 'resolver' argument")); + + auto checkoutPathAttr = attrs.get(state.s.checkoutPath); + if (checkoutPathAttr) + result.checkoutPath = std::string( + state.forceStringNoCtx(*checkoutPathAttr->value, pos, "while evaluating the 'checkoutPath' argument")); + + result.rev = resolveRev(state, pos, attrs, result.checkoutPath); +} + +/** + * Canonical JSON encoding of the caller's `args`, used as the cache key + * (`argsKey`). Caching is sound only because this encoding is injective on + * the values it accepts: the resolver receives the same value, so results can + * depend on `args` only through content that is, by construction, the key. + * + * Deliberately NOT `printValueAsJSON`, whose coercions break injectivity or + * purity: derivation attrsets serialize as their `outPath`, `__toString` + * attrsets coerce to strings (either lets distinct args collide on one key, + * turning a key collision into a stale cache hit), string context is dropped, + * paths are copied to the store as a side effect, and floats serialize + * ambiguously. This function instead rejects everything whose encoding would + * lose information: only null, bool, int, context-free string, list, and + * plain attrset are accepted. + */ +static nlohmann::json canonicalJsonFromValue(EvalState & state, Value & value, const PosIdx pos) +{ + state.forceValue(value, pos); + + switch (value.type()) { + case nNull: + return nullptr; + case nBool: + return value.boolean(); + case nInt: + return value.integer().value; + case nString: { + auto string = state.forceStringNoCtx(value, pos, "while converting the 'args' argument to canonical JSON"); + return std::string(string); + } + case nList: { + auto result = nlohmann::json::array(); + for (auto elem : value.listView()) + result.push_back(canonicalJsonFromValue(state, *elem, pos)); + return result; + } + case nAttrs: { + auto result = nlohmann::json::object(); + for (auto & attr : value.attrs()->lexicographicOrder(state.symbols)) { + result.emplace(state.symbols[attr->name], canonicalJsonFromValue(state, *attr->value, attr->pos)); + } + return result; + } + case nFloat: + case nPath: + case nThunk: + case nFailed: + case nFunction: + case nExternal: + state + .error( + "'args' must be JSON-convertible (null, bool, int, string without context, list, or attrset)") + .atPos(pos) + .debugThrow(); + } + + unreachable(); +} + +static std::pair +parseTecnixResolverArgsValue(EvalState & state, const PosIdx pos, const Bindings & attrs) +{ + auto resolverArgsAttr = attrs.get(state.s.args); + if (!resolverArgsAttr) + state.error("'args' attribute required").atPos(pos).debugThrow(); + + auto canonicalJson = canonicalJsonFromValue(state, *resolverArgsAttr->value, pos).dump(); + return {resolverArgsAttr->value, std::move(canonicalJson)}; +} + +static std::vector parseTecnixTargets(EvalState & state, const PosIdx pos, const Bindings & attrs) +{ + auto targetsAttr = attrs.get(state.s.targets); + if (!targetsAttr) + state.error("'targets' attribute required").atPos(pos).debugThrow(); + + state.forceList(*targetsAttr->value, pos, "while evaluating the 'targets' argument"); + std::vector targets; + targets.reserve(targetsAttr->value->listSize()); + for (auto elem : targetsAttr->value->listView()) { + auto targetId = state.forceStringNoCtx(*elem, pos, "while evaluating a target id"); + if (targetId == tecnixTargetNamesCacheKey) + state.error("tecnix target id '%s' is reserved", targetId).atPos(pos).debugThrow(); + targets.push_back(std::string(targetId)); + } + return targets; +} + +static TecnixArgs parseTecnixArgs(EvalState & state, const PosIdx pos, Value ** args, bool withTargets) +{ + auto & attrs = forceTecnixBuiltinAttrs(state, pos, args); + auto [resolverArgs, argsKey] = parseTecnixResolverArgsValue(state, pos, attrs); + + TecnixArgs result; + parseTecnixRepoArgs(state, pos, attrs, result); + result.resolverArgs = resolverArgs; + result.argsKey = std::move(argsKey); + if (withTargets) + result.targets = parseTecnixTargets(state, pos, attrs); + return result; +} + +static bool getTecnixBoolAttr( + EvalState & state, + const PosIdx pos, + Value ** args, + const Symbol & name, + std::string_view context, + bool defaultValue = false) +{ + auto & attrs = forceTecnixBuiltinAttrs(state, pos, args); + if (auto attr = attrs.get(name)) + return state.forceBool(*attr->value, pos, context); + return defaultValue; +} + +static void requireTecnixTargets(EvalState & state, const PosIdx pos, const TecnixArgs & args) +{ + if (args.targets.empty()) + state.error("'targets' attribute must contain at least one target reference") + .atPos(pos) + .debugThrow(); +} + +/** + * Configure the repository context used by Tecnix evaluation. + * + * The settings names are still `tectonix-*` for CLI compatibility, but new + * Tecnix primops use the full-repo accessor mounted from this context. + * + * Must be called before getResolveFunction(). + * + * A single EvalState has lazy, cached repository accessors, so all Tecnix calls + * in that state must use the same repository context. + */ +static void configureTecnixRepoContext(EvalState & state, const TecnixArgs & args) +{ + configureTectonixContext(state, args.gitDir, args.rev, args.checkoutPath); +} + +/** + * Import the explicit resolver file from the git repo and return a value from + * the attrset produced by calling it with `args` (e.g. `resolve` or + * `allTargetNames`). + * + * Requires configureTecnixRepoContext() to have been called first. + */ +static Value & +getTecnixModuleValue(EvalState & state, const PosIdx pos, const TecnixArgs & tArgs, std::string_view attrName) +{ + // Get resolver file path from the lazily-mounted Tecnix repo accessor. + auto resolverPath = getTecnixRepoPath(state, tArgs.resolver); + auto modulePath = SourcePath(state.rootFS, CanonPath(resolverPath)); + + // Import the resolver file (a function taking the opaque `args` value) and call it. + auto * moduleFn = state.allocValue(); + state.evalFile(modulePath, *moduleFn); + + if (!tArgs.resolverArgs) + state.error("missing Tecnix resolver args").atPos(pos).debugThrow(); + + auto * moduleVal = state.allocValue(); + state.callFunction(*moduleFn, *tArgs.resolverArgs, *moduleVal, pos); + state.forceAttrs(*moduleVal, pos, "while evaluating tecnix module"); + + auto attr = moduleVal->attrs()->get(state.symbols.create(attrName)); + if (!attr) + state.error("tecnix module must have a '%s' attribute", attrName).atPos(pos).debugThrow(); + + return *attr->value; +} + +static Value & getResolveFunction(EvalState & state, const PosIdx pos, const TecnixArgs & tArgs) +{ + auto & fn = getTecnixModuleValue(state, pos, tArgs, "resolve"); + state.forceFunction(fn, pos, "while evaluating the 'resolve' attribute of tecnix module"); + return fn; +} + +struct SourceAccessSetSnapshot +{ + std::vector directAccesses; + std::vector accessSetEdges; +}; + +static SourceAccessSetSnapshot snapshotSourceAccessSetTracking(const TrackingContext & ctx) +{ + // Tracking contexts are thread-confined: the snapshot runs on the thread + // that owns the context, after its evaluation has completed. + SourceAccessSetSnapshot snapshot; + snapshot.directAccesses.assign( + ctx.rootFrame.directSourceAccessSetAccesses.begin(), ctx.rootFrame.directSourceAccessSetAccesses.end()); + snapshot.accessSetEdges.assign( + ctx.rootFrame.childSourceAccessSets.begin(), ctx.rootFrame.childSourceAccessSets.end()); + return snapshot; +} + +static std::vector collectSourceAccessSetTrackedPaths( + const ref & sourceAccessSetGraph, const SourceAccessSetSnapshot & snapshot) +{ + // `flatten` yields unique paths; sort for deterministic closure output. + auto paths = sourceAccessSetGraph->flatten(snapshot.directAccesses, snapshot.accessSetEdges); + std::sort(paths.begin(), paths.end()); + return paths; +} + +static std::vector collectSourceAccessSetTrackedPaths(const TrackingContext & ctx) +{ + if (!ctx.sourceAccessSetGraph->isEnabled()) + throw Error("Tecnix source access-set tracking was not enabled"); + return collectSourceAccessSetTrackedPaths(ctx.sourceAccessSetGraph, snapshotSourceAccessSetTracking(ctx)); +} + +static Value * dependencyAttrsToValue(EvalState & state, const DependencyClosure & dependencies) +{ + auto attrs = state.buildBindings(dependencies.size()); + for (auto & dependency : dependencies) { + auto * fingerprintValue = state.allocValue(); + fingerprintValue->mkString(dependency.fingerprint, state.mem); + attrs.insert(state.symbols.create(dependency.path), fingerprintValue); + } + auto * val = state.allocValue(); + val->mkAttrs(attrs); + + return val; +} + +static std::vector evalTargetNamesOnly(EvalState & state, const PosIdx pos, const TecnixArgs & tArgs) +{ + auto & allTargetNames = getTecnixModuleValue(state, pos, tArgs, "allTargetNames"); + state.forceList(allTargetNames, pos, "while evaluating all target names"); + + std::vector targetNames; + for (auto elem : allTargetNames.listView()) { + auto targetName = state.forceStringNoCtx(*elem, pos, "while evaluating a target id"); + targetNames.push_back(std::string(targetName)); + } + return targetNames; +} + +struct TecnixDiscoveryResult +{ + std::vector targetNames; + /** The freshly evaluated closure; set on a cache miss. */ + DependencyClosure dependencies; + /** The proven cached closure; set on a cache hit. */ + std::optional dependencyBlob; +}; + +/** + * Discover target names through the same cache pipeline as target + * dependencies: one reserved key, the same lookup and validation, the same + * tracked evaluation on a miss, and the same upsert, with the discovered + * names carried as the candidate payload. + */ +static TecnixDiscoveryResult discoverTecnixTargetNames( + EvalState & state, const PosIdx pos, const TecnixArgs & tArgs, DependencyFingerprintCache & fingerprintCache) +{ + bool useCache = state.settings.pureEval && state.settings.tecnixEvalCache; + + std::string cacheKey{tecnixTargetNamesCacheKey}; + if (useCache) { + auto hits = lookupValidatedDependencyBlobs( + state, cacheScope(tArgs), std::span{&cacheKey, 1}, fingerprintCache); + if (hits[0]) { + if (auto payload = hits[0]->payload()) { + try { + auto targetNames = nlohmann::json::parse(*payload).get>(); + printTalkative("tecnixTargetNames: discovery cache hit"); + return {std::move(targetNames), {}, std::move(hits[0])}; + } catch (const nlohmann::json::exception &) { + // A malformed payload is a cache miss, never an error. + } + } + } + } + + printTalkative("tecnixTargetNames: discovery cache miss, evaluating"); + TrackingContext trackingCtx(state); + std::vector targetNames; + { + ActiveTrackingContext activeTrackingCtx(trackingCtx); + targetNames = evalTargetNamesOnly(state, pos, tArgs); + } + auto trackedPaths = collectSourceAccessSetTrackedPaths(trackingCtx); + auto dependencies = dependencyFingerprints(getTecnixRepoAccessor(state), trackedPaths, fingerprintCache); + + if (useCache && !dependencies.empty()) { + std::vector upserts; + upserts.push_back({cacheKey, &dependencies, nlohmann::json(targetNames).dump()}); + upsertDependencyClosures(cacheScope(tArgs), upserts); + } + + return {std::move(targetNames), std::move(dependencies), std::nullopt}; +} + +static Value * targetRefToValue(EvalState & state, const std::string & target) +{ + auto * val = state.allocValue(); + val->mkString(target, state.mem); + return val; +} + +// ============================================================================ +// builtins.tecnixTargets { gitDir, resolver, args, targets = [ target-id ... ], ... } +// Resolves opaque target IDs via module contract. +// ============================================================================ +static void finishTecnixFutures(std::vector> && futures) +{ + std::exception_ptr ex; + std::exception_ptr interrupted; + size_t secondaryErrors = 0; + size_t secondaryInterrupts = 0; + + for (auto & future : futures) { + try { + future.get(); + } catch (const Interrupted &) { + if (!interrupted) + interrupted = std::current_exception(); + else + secondaryInterrupts++; + } catch (...) { + if (!ex) + ex = std::current_exception(); + else + secondaryErrors++; + } + } + + if (secondaryErrors || secondaryInterrupts) { + warn( + "tecnix: %d additional parallel evaluation(s) failed and %d were interrupted; rethrowing the first error", + secondaryErrors, + secondaryInterrupts); + } + + if (ex) + std::rethrow_exception(ex); + if (interrupted) + std::rethrow_exception(interrupted); +} + +template +static void +evalTecnixIndices(EvalState & state, const std::vector & indices, EvalOne evalOne, bool allowParallel = true) +{ + if (indices.empty()) + return; + + if (allowParallel && state.executor->enabled && !Executor::amWorkerThread && indices.size() > 1) { + Executor::WorkItems work; + for (auto i : indices) + state.addWork(work, 0, [&, i]() { evalOne(i); }); + finishTecnixFutures(state.executor->spawn(std::move(work))); + return; + } + + for (auto i : indices) + evalOne(i); +} + +static void forceTargetDrvPath(EvalState & state, Value & targetValue, const PosIdx pos) +{ + state.forceValue(targetValue, pos); + if (targetValue.type() != nAttrs) + return; + + auto drvPathAttr = targetValue.attrs()->get(state.s.drvPath); + if (!drvPathAttr) + return; + + NixStringContext context; + state.forceString(*drvPathAttr->value, context, pos, "while evaluating the 'drvPath' attribute of a tecnix target"); +} + +static void prim_tecnixTargetsWithDependencies( + EvalState & state, const PosIdx pos, Value ** args, Value & v, TecnixArgs && tArgs, bool includeTargets); + +static void prim_tecnixTargets(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + auto tArgs = parseTecnixArgs(state, pos, args, true); + requireTecnixTargets(state, pos, tArgs); + configureTecnixRepoContext(state, tArgs); + + auto includeDependencies = getTecnixBoolAttr( + state, + pos, + args, + state.symbols.create("includeDependencies"), + "while evaluating the 'includeDependencies' argument to builtins.tecnixTargets"); + + if (includeDependencies) { + auto includeTargets = getTecnixBoolAttr( + state, + pos, + args, + state.symbols.create("includeTargets"), + "while evaluating the 'includeTargets' argument to builtins.tecnixTargets", + true); + prim_tecnixTargetsWithDependencies(state, pos, args, v, std::move(tArgs), includeTargets); + return; + } + + printTalkative( + "tecnixTargets: evaluating %d target ref(s)%s, eval cores %d", + tArgs.targets.size(), + state.executor->enabled && !Executor::amWorkerThread && tArgs.targets.size() > 1 ? " in parallel" + : " sequentially", + state.executor->evalCores); + + auto & resolveFn = getResolveFunction(state, pos, tArgs); + + std::vector values(tArgs.targets.size()); + for (size_t i = 0; i < tArgs.targets.size(); i++) + values[i] = state.allocValue(); + + std::vector indices; + indices.reserve(tArgs.targets.size()); + for (size_t i = 0; i < tArgs.targets.size(); i++) + indices.push_back(i); + + auto evalTarget = [&](size_t i) { + auto & target = tArgs.targets[i]; + auto started = std::chrono::steady_clock::now(); + printTalkative( + "tecnixTargets: start evaluating '%s' on %s thread", target, Executor::amWorkerThread ? "worker" : "main"); + + auto * targetArg = state.allocValue(); + targetArg->mkString(target, state.mem); + state.callFunction(resolveFn, *targetArg, *values[i], pos); + forceTargetDrvPath(state, *values[i], pos); + + auto elapsedMs = + std::chrono::duration_cast(std::chrono::steady_clock::now() - started).count(); + printTalkative("tecnixTargets: finished '%s' in %d ms", target, elapsedMs); + }; + + evalTecnixIndices(state, indices, evalTarget); + + auto rootAttrs = state.buildBindings(tArgs.targets.size()); + for (size_t i = 0; i < tArgs.targets.size(); i++) + rootAttrs.insert(state.symbols.create(tArgs.targets[i]), values[i]); + v.mkAttrs(rootAttrs); +} + +static RegisterPrimOp primop_tecnixTargets({ + .name = "__tecnixTargets", + .args = {"attrs"}, + .doc = R"( + Resolve Tecnix target references via the resolver module. Input `targets` + is a list of opaque target ID strings. By default, returns an attrset + keyed by those same strings. With `includeDependencies = true`, returns + an ordered list of `{ target, value, dependencies }` records, where + `dependencies` is an attrset of `path = fingerprint`. Add + `includeTargets = false` to omit `value` from each record. + )", + .impl = prim_tecnixTargets, +}); + +struct TargetDependencyResult +{ + DependencyClosure dependencies; + std::optional dependencyBlob; + std::optional sourceAccessSetSnapshot; + Value * targetValue = nullptr; + bool cacheNeedsUpsert = false; +}; + +struct PreparedTrackedResolveFunction +{ + Value * resolveFn; + EvalSourceAccessSetId sourceDeps = emptyEvalSourceAccessSetId; +}; + +static PreparedTrackedResolveFunction +prepareTrackedResolveFunction(EvalState & state, const PosIdx pos, const TecnixArgs & tArgs) +{ + TrackingContext trackingCtx(state); + ActiveTrackingContext activeTrackingCtx(trackingCtx); + + TrackedSourceDepsScope sourceDepsScope(trackingCtx); + auto & resolveFn = getResolveFunction(state, pos, tArgs); + auto sourceDeps = sourceDepsScope.finish(&resolveFn); + + return { + .resolveFn = &resolveFn, + .sourceDeps = sourceDeps, + }; +} + +static TargetDependencyResult evalTargetDependencies( + EvalState & state, + const PosIdx pos, + Value & resolveFn, + EvalSourceAccessSetId resolveSourceDeps, + const std::string & target, + bool keepTargetValue) +{ + auto started = std::chrono::steady_clock::now(); + printTalkative( + "tecnixTargets dependencies: start evaluating '%s' on %s thread", + target, + Executor::amWorkerThread ? "worker" : "main"); + + TrackingContext trackingCtx(state); + if (resolveSourceDeps != emptyEvalSourceAccessSetId) + recordTrackedSourceAccessSetDependency(trackingCtx, resolveSourceDeps); + Value * targetValue = nullptr; + { + ActiveTrackingContext activeTrackingCtx(trackingCtx); + + auto * targetArg = state.allocValue(); + targetArg->mkString(target, state.mem); + auto * resolveResult = state.allocValue(); + state.callFunction(resolveFn, *targetArg, *resolveResult, pos); + forceTargetDrvPath(state, *resolveResult, pos); + if (keepTargetValue) + targetValue = resolveResult; + } + + auto snapshot = snapshotSourceAccessSetTracking(trackingCtx); + auto elapsedMs = + std::chrono::duration_cast(std::chrono::steady_clock::now() - started).count(); + printTalkative( + "tecnixTargets dependencies: finished '%s' in %d ms with access-set dependency snapshot", target, elapsedMs); + return { + .dependencies = {}, + .sourceAccessSetSnapshot = std::move(snapshot), + .targetValue = targetValue, + }; +} + +static void finalizeSourceAccessSetDependencies( + EvalState & state, + std::vector> & results, + DependencyFingerprintCache & fingerprintCache) +{ + if (!trackedSourceAccessSetGraph(state)->isEnabled()) + return; + + for (auto & maybeResult : results) { + if (!maybeResult || !maybeResult->sourceAccessSetSnapshot) + continue; + + auto trackedPaths = collectSourceAccessSetTrackedPaths( + trackedSourceAccessSetGraph(state), *maybeResult->sourceAccessSetSnapshot); + auto dependencies = dependencyFingerprints(getTecnixRepoAccessor(state), trackedPaths, fingerprintCache); + maybeResult->dependencies = std::move(dependencies); + maybeResult->sourceAccessSetSnapshot.reset(); + } +} + +static void printTecnixAccessSetStats(EvalState & state, std::string_view opName) +{ + auto sourceAccessSetStats = trackedSourceAccessSetStats(state); + printTalkative( + "%s: source access-set graph has %d access id(s), %d access set(s), %d access set item(s)", + opName, + sourceAccessSetStats.accesses, + sourceAccessSetStats.accessSets, + sourceAccessSetStats.accessSetItems); +} + +static std::vector> evaluateTecnixTargetDependencies( + EvalState & state, + const PosIdx pos, + const TecnixArgs & args, + DependencyFingerprintCache & fingerprintCache, + bool keepTargetValues = false) +{ + bool useCache = state.settings.pureEval && state.settings.tecnixEvalCache; + printTalkative( + "tecnixTargets dependencies: planning %d target ref(s), dependency cache %s, eval cores %d", + args.targets.size(), + useCache ? "enabled" : "disabled", + state.executor->evalCores); + + std::vector> results(args.targets.size()); + std::vector misses; + size_t cacheHits = 0; + + if (useCache) { + auto hits = lookupValidatedDependencyBlobs( + state, + cacheScope(args), + std::span{args.targets.data(), args.targets.size()}, + fingerprintCache); + for (size_t i = 0; i < hits.size(); i++) { + if (!hits[i]) { + printTalkative("tecnixTargets dependencies: dependency cache miss, evaluating '%s'", args.targets[i]); + continue; + } + printTalkative("tecnixTargets dependencies: dependency cache hit for '%s'", args.targets[i]); + results[i].emplace(); + results[i]->dependencyBlob = std::move(hits[i]); + cacheHits++; + } + } + + for (size_t i = 0; i < args.targets.size(); i++) { + if (!results[i]) + misses.push_back(i); + } + + bool allowParallelDependencies = state.settings.tecnixParallelDependencies && state.executor->enabled + && state.executor->evalCores > 1 && !Executor::amWorkerThread && misses.size() > 1; + printTalkative( + "tecnixTargets dependencies: %d cache hit(s), %d target ref(s) to evaluate%s", + cacheHits, + misses.size(), + allowParallelDependencies ? " in parallel" : " sequentially"); + + if (!misses.empty()) { + auto preparedResolve = prepareTrackedResolveFunction(state, pos, args); + + auto evalMiss = [&](size_t i) { + auto & target = args.targets[i]; + results[i] = evalTargetDependencies( + state, pos, *preparedResolve.resolveFn, preparedResolve.sourceDeps, target, keepTargetValues); + if (results[i]) + results[i]->cacheNeedsUpsert = true; + }; + + evalTecnixIndices(state, misses, evalMiss, allowParallelDependencies); + finalizeSourceAccessSetDependencies(state, results, fingerprintCache); + + if (useCache) { + std::vector upserts; + upserts.reserve(misses.size()); + for (auto i : misses) { + if (results[i] && results[i]->cacheNeedsUpsert) + upserts.push_back({args.targets[i], &results[i]->dependencies, {}}); + } + upsertDependencyClosures(cacheScope(args), upserts); + } + } + + return results; +} + +static void prim_tecnixTargetsWithDependencies( + EvalState & state, const PosIdx pos, Value **, Value & v, TecnixArgs && tArgs, bool includeTargets) +{ + DependencyFingerprintCache fingerprintCache; + auto results = evaluateTecnixTargetDependencies(state, pos, tArgs, fingerprintCache, includeTargets); + printTecnixAccessSetStats(state, "tecnixTargets"); + + if (includeTargets) { + std::vector missingTargetValueIndices; + missingTargetValueIndices.reserve(tArgs.targets.size()); + for (size_t i = 0; i < tArgs.targets.size(); i++) { + assert(results[i]); + if (!results[i]->targetValue) + missingTargetValueIndices.push_back(i); + } + + if (!missingTargetValueIndices.empty()) { + auto & resolveFn = getResolveFunction(state, pos, tArgs); + auto evalTarget = [&](size_t i) { + auto * targetValue = state.allocValue(); + auto * targetArg = state.allocValue(); + targetArg->mkString(tArgs.targets[i], state.mem); + state.callFunction(resolveFn, *targetArg, *targetValue, pos); + forceTargetDrvPath(state, *targetValue, pos); + results[i]->targetValue = targetValue; + }; + evalTecnixIndices(state, missingTargetValueIndices, evalTarget); + } + } + + auto list = state.buildList(tArgs.targets.size()); + for (size_t i = 0; i < tArgs.targets.size(); i++) { + auto & result = *results[i]; + auto * targetValue = state.allocValue(); + targetValue->mkString(tArgs.targets[i], state.mem); + auto * dependenciesValue = result.dependencyBlob ? result.dependencyBlob->toValue(state) + : dependencyAttrsToValue(state, result.dependencies); + + auto attrs = state.buildBindings(includeTargets ? 3 : 2); + attrs.insert(state.symbols.create("target"), targetValue); + if (includeTargets) + attrs.insert(state.symbols.create("value"), result.targetValue); + attrs.insert(state.symbols.create("dependencies"), dependenciesValue); + + auto * recordValue = state.allocValue(); + recordValue->mkAttrs(attrs); + list[i] = recordValue; + } + + v.mkList(list); +} + +// ============================================================================ +// builtins.tecnixTargetNames { gitDir, resolver, args, ... } +// Discovers fully-qualified target names via the Tecnix module contract. +// Caches the discovered names using the same path -> fingerprint validation as +// dependency discovery, so repeated discovery for an unchanged source is cheap. +// ============================================================================ +static void prim_tecnixTargetNames(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + auto dArgs = parseTecnixArgs(state, pos, args, false); + configureTecnixRepoContext(state, dArgs); + + auto includeDependencies = getTecnixBoolAttr( + state, + pos, + args, + state.symbols.create("includeDependencies"), + "while evaluating the 'includeDependencies' argument to builtins.tecnixTargetNames"); + + DependencyFingerprintCache fingerprintCache; + auto result = discoverTecnixTargetNames(state, pos, dArgs, fingerprintCache); + + auto list = state.buildList(result.targetNames.size()); + for (size_t i = 0; i < result.targetNames.size(); i++) + list[i] = targetRefToValue(state, result.targetNames[i]); + + if (!includeDependencies) { + v.mkList(list); + return; + } + + auto * targetsValue = state.allocValue(); + targetsValue->mkList(list); + auto * dependenciesValue = result.dependencyBlob ? result.dependencyBlob->toValue(state) + : dependencyAttrsToValue(state, result.dependencies); + auto rootAttrs = state.buildBindings(2); + rootAttrs.insert(state.symbols.create("targets"), targetsValue); + rootAttrs.insert(state.symbols.create("dependencies"), dependenciesValue); + v.mkAttrs(rootAttrs); +} + +static RegisterPrimOp primop_tecnixTargetNames({ + .name = "__tecnixTargetNames", + .args = {"attrs"}, + .doc = R"( + Discover Tecnix target references by passing `args` to the resolver. By + default, returns a flat list of opaque target ID strings supplied by the + resolver. With `includeDependencies = true`, returns + `{ targets = [ ... ]; dependencies = { path = fingerprint; ... }; }`. + `args` must be JSON-convertible for cache keying. + )", + .impl = prim_tecnixTargetNames, +}); + +} // namespace nix diff --git a/src/libexpr/primops/tectonix.cc b/src/libexpr/primops/tectonix.cc index 30e3b36517..da7066a220 100644 --- a/src/libexpr/primops/tectonix.cc +++ b/src/libexpr/primops/tectonix.cc @@ -1,6 +1,7 @@ #include "nix/expr/primops.hh" #include "nix/expr/eval-inline.hh" #include "nix/expr/eval-settings.hh" +#include "nix/expr/tecnix/source-accessors.hh" #include "nix/fetchers/git-utils.hh" #include "nix/store/store-api.hh" #include "nix/fetchers/fetch-to-store.hh" @@ -13,7 +14,7 @@ namespace nix { // Helper to get cached manifest JSON (avoids repeated parsing) static const nlohmann::json & getManifest(EvalState & state) { - return state.getManifestJson(); + return getManifestJson(state); } // Helper to validate that a zone path exists in the manifest @@ -32,6 +33,9 @@ static void validateZonePath(EvalState & state, const PosIdx pos, std::string_vi // ============================================================================ static void prim_worldManifest(EvalState & state, const PosIdx pos, Value ** args, Value & v) { + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(".meta/manifest.json"); + auto json = getManifest(state); auto attrs = state.buildBindings(json.size()); @@ -67,6 +71,9 @@ static RegisterPrimOp primop_worldManifest({ // ============================================================================ static void prim_worldManifestInverted(EvalState & state, const PosIdx pos, Value ** args, Value & v) { + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(".meta/manifest.json"); + auto json = getManifest(state); // Track seen IDs to detect duplicates @@ -100,6 +107,8 @@ static RegisterPrimOp primop_worldManifestInverted({ .impl = prim_worldManifestInverted, }); +static std::string normalizeTrackedRepoPath(std::string_view path); + // ============================================================================ // builtins.unsafeTectonixInternalTreeSha worldPath // Returns the git tree SHA for a world path @@ -109,7 +118,10 @@ static void prim_unsafeTectonixInternalTreeSha(EvalState & state, const PosIdx p auto worldPath = state.forceStringNoCtx( *args[0], pos, "while evaluating the 'worldPath' argument to builtins.unsafeTectonixInternalTreeSha"); - auto sha = state.getWorldTreeSha(worldPath); + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(normalizeTrackedRepoPath(worldPath)); + + auto sha = getWorldTreeSha(state, worldPath); v.mkString(sha.gitRev(), state.mem); } @@ -137,7 +149,7 @@ static void prim_unsafeTectonixInternalTree(EvalState & state, const PosIdx pos, auto treeSha = state.forceStringNoCtx( *args[0], pos, "while evaluating the 'treeSha' argument to builtins.unsafeTectonixInternalTree"); - auto repo = state.getWorldRepo(); + auto repo = getWorldRepo(state); auto hash = Hash::parseNonSRIUnprefixed(treeSha, HashAlgorithm::SHA1); if (!repo->hasObject(hash)) @@ -178,14 +190,26 @@ static RegisterPrimOp primop_unsafeTectonixInternalTree({ // With lazy-trees enabled, returns a virtual store path that is only // materialized when used as a derivation input. // ============================================================================ +static std::string normalizeTrackedRepoPath(std::string_view path) +{ + std::string result(path); + if (hasPrefix(result, "//")) + result = result.substr(2); + else if (hasPrefix(result, "/")) + result = result.substr(1); + return result; +} + static void prim_unsafeTectonixInternalZoneSrc(EvalState & state, const PosIdx pos, Value ** args, Value & v) { auto zonePath = state.forceStringNoCtx( *args[0], pos, "while evaluating the 'zonePath' argument to builtins.unsafeTectonixInternalZoneSrc"); validateZonePath(state, pos, zonePath); + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(normalizeTrackedRepoPath(zonePath)); - auto storePath = state.getZoneStorePath(zonePath); + auto storePath = getLegacyTectonixZoneStorePath(state, zonePath); state.allowAndSetStorePathString(storePath, v); } @@ -219,8 +243,10 @@ static void prim_unsafeTectonixInternalZonePath(EvalState & state, const PosIdx *args[0], pos, "while evaluating the 'zonePath' argument to builtins.unsafeTectonixInternalZonePath"); validateZonePath(state, pos, zonePath); + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(normalizeTrackedRepoPath(zonePath)); - auto storePath = state.getZoneStorePath(zonePath); + auto storePath = getLegacyTectonixZoneStorePath(state, zonePath); state.allowPath(storePath); v.mkPath(state.storePath(storePath), state.mem); } @@ -249,10 +275,23 @@ static RegisterPrimOp primop_unsafeTectonixInternalZonePath({ // builtins.unsafeTectonixInternalSparseCheckoutRoots // Returns list of zone IDs in sparse checkout // ============================================================================ +static void throwIfTrackedLegacyCheckoutStateBuiltin(EvalState & state, const PosIdx pos, std::string_view builtinName) +{ + if (currentTecnixThreadState.trackingContext) + state + .error( + "legacy compatibility builtin builtins.%s exposes checkout-local state and cannot be used during Tecnix dependency tracking", + builtinName) + .atPos(pos) + .debugThrow(); +} + static void prim_unsafeTectonixInternalSparseCheckoutRoots(EvalState & state, const PosIdx pos, Value ** args, Value & v) { - auto & roots = state.getTectonixSparseCheckoutRoots(); + throwIfTrackedLegacyCheckoutStateBuiltin(state, pos, "unsafeTectonixInternalSparseCheckoutRoots"); + + auto & roots = getTectonixSparseCheckoutRoots(state); auto list = state.buildList(roots.size()); size_t i = 0; @@ -284,7 +323,9 @@ static RegisterPrimOp primop_unsafeTectonixInternalSparseCheckoutRoots({ // ============================================================================ static void prim_unsafeTectonixInternalDirtyZones(EvalState & state, const PosIdx pos, Value ** args, Value & v) { - auto & dirtyZones = state.getTectonixDirtyZones(); + throwIfTrackedLegacyCheckoutStateBuiltin(state, pos, "unsafeTectonixInternalDirtyZones"); + + auto & dirtyZones = getTectonixDirtyZones(state); auto attrs = state.buildBindings(dirtyZones.size()); for (const auto & [zonePath, info] : dirtyZones) { @@ -317,14 +358,16 @@ static RegisterPrimOp primop_unsafeTectonixInternalDirtyZones({ // ============================================================================ static void prim_unsafeTectonixInternalZoneIsDirty(EvalState & state, const PosIdx pos, Value ** args, Value & v) { + throwIfTrackedLegacyCheckoutStateBuiltin(state, pos, "unsafeTectonixInternalZoneIsDirty"); + auto zonePath = state.forceStringNoCtx( *args[0], pos, "while evaluating the 'zonePath' argument to builtins.__unsafeTectonixInternalZoneIsDirty"); validateZonePath(state, pos, zonePath); bool isDirty = false; - if (state.isTectonixSourceAvailable()) { - auto & dirtyZones = state.getTectonixDirtyZones(); + if (isTectonixSourceAvailable(state)) { + auto & dirtyZones = getTectonixDirtyZones(state); auto it = dirtyZones.find(std::string(zonePath)); isDirty = it != dirtyZones.end() && it->second.dirty; } @@ -351,6 +394,8 @@ static RegisterPrimOp primop_unsafeTectonixInternalZoneIsDirty({ // ============================================================================ static void prim_unsafeTectonixInternalZoneRoot(EvalState & state, const PosIdx pos, Value ** args, Value & v) { + throwIfTrackedLegacyCheckoutStateBuiltin(state, pos, "unsafeTectonixInternalZoneRoot"); + auto zonePath = state.forceStringNoCtx( *args[0], pos, "while evaluating the 'zonePath' argument to builtins.__unsafeTectonixInternalZoneRoot"); @@ -396,7 +441,7 @@ static RegisterPrimOp primop_unsafeTectonixInternalZoneRoot({ // ============================================================================ static void prim_unsafeTectonixInternalGitSha(EvalState & state, const PosIdx pos, Value ** args, Value & v) { - auto & sha = state.requireTectonixGitSha(); + auto & sha = requireTectonixGitSha(state); v.mkString(sha, state.mem); } diff --git a/src/libexpr/tecnix/eval-cache.cc b/src/libexpr/tecnix/eval-cache.cc new file mode 100644 index 0000000000..8452fa6244 --- /dev/null +++ b/src/libexpr/tecnix/eval-cache.cc @@ -0,0 +1,1236 @@ +/** + * The persistent Tecnix evaluation cache (see eval-cache.hh for the API and + * plans/tecnix-target-eval-caching/ for the design): DependencyShards rows in + * SQLite holding TXDC blobs — bounded per-key source-closure candidate + * histories, validated against current fingerprints on every use. + */ + +#include "nix/expr/tecnix/eval-cache.hh" + +#include "nix/expr/eval-inline.hh" +#include "nix/expr/tecnix/source-accessors.hh" +#include "nix/store/globals.hh" +#include "nix/store/sqlite.hh" +#include "nix/util/file-system.hh" +#include "nix/util/source-accessor.hh" +#include "nix/util/strings.hh" +#include "nix/util/sync.hh" +#include "nix/util/users.hh" +#include "nix/util/util.hh" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace nix { + +static std::atomic nextDependencyFingerprintCacheGeneration{1}; + +DependencyFingerprintCache::DependencyFingerprintCache() + : generation(nextDependencyFingerprintCacheGeneration.fetch_add(1, std::memory_order_relaxed)) +{ +} + +struct DependencyFingerprintThreadLocalCache +{ + uint64_t generation = 0; + boost::unordered_flat_map, StringViewHash, std::equal_to<>> fingerprints; + + DependencyFingerprintThreadLocalCache() + { + fingerprints.reserve(8192); + } +}; + +static DependencyFingerprintThreadLocalCache & getDependencyFingerprintThreadCache(DependencyFingerprintCache & cache); + +// Unshipped development cache: the schema may change incompatibly at any +// time, with no migrations. Foreign or stale rows are rejected by content +// validation (a miss), and deleting the database is always safe. +static const char * tecnixEvalCacheSchema = R"sql( +create table if not exists DependencyShards ( + gitDir text not null, + resolver text not null, + argsKey text not null, + shard integer not null, + dependencies blob not null, + timestamp integer not null, + primary key (gitDir, resolver, argsKey, shard) +); + +)sql"; + +// Returns a pointer into the thread-local fingerprint memo. Consume it immediately; +// it must not be retained across another memo insertion, which may rehash the map. +static const std::optional * +dependencyFingerprintCached(ref accessor, std::string_view path, DependencyFingerprintCache & cache); + +struct TecnixEvalCache +{ + struct State + { + SQLite db; + SQLiteStmt upsertShard, lookupShard, lookupAllShards; + }; + + Sync _state; + + TecnixEvalCache() + { + auto state(_state.lock()); + + auto dbPath = getCacheDir() / "tecnix-eval-cache-v1.sqlite"; + createDirs(dbPath.parent_path()); + + state->db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL}); + state->db.isCache(); + state->db.exec(tecnixEvalCacheSchema); + + state->upsertShard.create( + state->db, + "insert or replace into DependencyShards(gitDir, resolver, argsKey, shard, dependencies, timestamp) " + "values (?, ?, ?, ?, ?, ?)"); + state->lookupShard.create( + state->db, + "select dependencies from DependencyShards where gitDir = ? and resolver = ? and argsKey = ? and shard = ?"); + state->lookupAllShards.create( + state->db, + "select shard, dependencies from DependencyShards where gitDir = ? and resolver = ? and argsKey = ?"); + } + + static constexpr std::string_view dependencyBlobMagic = "TXDC"; + static constexpr uint32_t dependencyBlobVersion = 1; + static constexpr uint32_t dependencyBlobFlags = 0; + static constexpr size_t dependencyBlobFieldCount = 16; + static constexpr size_t dependencyBlobHeaderSize = 4 + dependencyBlobFieldCount * sizeof(uint32_t); + static constexpr size_t dependencyShardCount = 256; + static constexpr size_t maxDependencyBlobTargets = 1024; + static constexpr size_t maxDependencyBlobCandidates = 32; + static constexpr size_t maxDependencyBlobStrings = 200000; + static constexpr size_t maxDependencyBlobPairs = 1000000; + static constexpr size_t maxDependencyBlobBytes = 64 * 1024 * 1024; + + enum DependencyBlobField : size_t { + blobVersionField = 0, + blobFlagsField, + blobTargetCountField, + blobPathCountField, + blobFingerprintCountField, + blobPayloadCountField, + blobCandidateCountField, + blobPairCountField, + blobTargetOffsetsOffsetField, + blobTargetRecordsOffsetField, + blobPathOffsetsOffsetField, + blobFingerprintOffsetsOffsetField, + blobPayloadOffsetsOffsetField, + blobCandidateRecordsOffsetField, + blobPairsOffsetField, + blobEndOffsetField, + }; + + struct DependencyCandidate + { + DependencyClosure dependencies; + std::string payload; + }; + + struct DependencyShardTarget + { + std::string target; + std::vector candidates; + }; + + static uint32_t dependencyShardForTarget(std::string_view target) + { + uint64_t hash = 1469598103934665603ULL; + for (unsigned char c : target) { + hash ^= c; + hash *= 1099511628211ULL; + } + return static_cast(hash % dependencyShardCount); + } + + static void appendU32(std::string & out, uint32_t value) + { + out.push_back(static_cast(value & 0xff)); + out.push_back(static_cast((value >> 8) & 0xff)); + out.push_back(static_cast((value >> 16) & 0xff)); + out.push_back(static_cast((value >> 24) & 0xff)); + } + + static void writeU32(std::string & out, size_t offset, uint32_t value) + { + assert(offset + sizeof(uint32_t) <= out.size()); + out[offset + 0] = static_cast(value & 0xff); + out[offset + 1] = static_cast((value >> 8) & 0xff); + out[offset + 2] = static_cast((value >> 16) & 0xff); + out[offset + 3] = static_cast((value >> 24) & 0xff); + } + + static std::optional readU32(std::string_view blob, size_t offset) + { + if (offset + sizeof(uint32_t) > blob.size()) + return std::nullopt; + auto * data = reinterpret_cast(blob.data() + offset); + return static_cast(data[0]) | (static_cast(data[1]) << 8) + | (static_cast(data[2]) << 16) | (static_cast(data[3]) << 24); + } + + static size_t align4(size_t value) + { + return (value + 3) & ~size_t{3}; + } + + static void padTo4(std::string & out) + { + while (out.size() != align4(out.size())) + out.push_back('\0'); + } + + static bool u32Fits(size_t value) + { + return value <= std::numeric_limits::max(); + } + + static uint32_t checkedU32(size_t value, std::string_view what) + { + if (!u32Fits(value)) + throw Error("Tecnix dependency cache %s is too large", what); + return static_cast(value); + } + + static void setBlobField(std::string & out, DependencyBlobField field, uint32_t value) + { + writeU32(out, 4 + static_cast(field) * sizeof(uint32_t), value); + } + + struct SQLiteImmediateTxn + { + SQLite & db; + bool active = false; + + explicit SQLiteImmediateTxn(SQLite & db) + : db(db) + { + db.exec("begin immediate;"); + active = true; + } + + void commit() + { + db.exec("commit;"); + active = false; + } + + ~SQLiteImmediateTxn() + { + try { + if (active) + db.exec("rollback;"); + } catch (...) { + ignoreExceptionInDestructor(); + } + } + }; + + struct DependencyBlobView + { + std::string_view blob; + uint32_t targetCount = 0; + uint32_t pathCount = 0; + uint32_t fingerprintCount = 0; + uint32_t payloadCount = 0; + uint32_t candidateCount = 0; + uint32_t pairCount = 0; + uint32_t targetOffsetsOffset = 0; + uint32_t targetRecordsOffset = 0; + uint32_t pathOffsetsOffset = 0; + uint32_t fingerprintOffsetsOffset = 0; + uint32_t payloadOffsetsOffset = 0; + uint32_t candidateRecordsOffset = 0; + uint32_t pairsOffset = 0; + uint32_t endOffset = 0; + + static constexpr uint32_t targetRecordU32s = 2; + static constexpr uint32_t candidateRecordU32s = 3; + static constexpr uint32_t pairRecordU32s = 2; + + static std::optional open(std::string_view blob) + { + if (blob.size() < dependencyBlobHeaderSize || blob.size() > maxDependencyBlobBytes) + return std::nullopt; + if (blob.substr(0, dependencyBlobMagic.size()) != dependencyBlobMagic) + return std::nullopt; + + auto field = [&](DependencyBlobField f) -> std::optional { + return readU32(blob, 4 + static_cast(f) * sizeof(uint32_t)); + }; + + auto version = field(blobVersionField); + auto flags = field(blobFlagsField); + if (!version || *version != dependencyBlobVersion || !flags || *flags != dependencyBlobFlags) + return std::nullopt; + + auto targetCount = field(blobTargetCountField); + auto pathCount = field(blobPathCountField); + auto fingerprintCount = field(blobFingerprintCountField); + auto payloadCount = field(blobPayloadCountField); + auto candidateCount = field(blobCandidateCountField); + auto pairCount = field(blobPairCountField); + auto targetOffsetsOffset = field(blobTargetOffsetsOffsetField); + auto targetRecordsOffset = field(blobTargetRecordsOffsetField); + auto pathOffsetsOffset = field(blobPathOffsetsOffsetField); + auto fingerprintOffsetsOffset = field(blobFingerprintOffsetsOffsetField); + auto payloadOffsetsOffset = field(blobPayloadOffsetsOffsetField); + auto candidateRecordsOffset = field(blobCandidateRecordsOffsetField); + auto pairsOffset = field(blobPairsOffsetField); + auto endOffset = field(blobEndOffsetField); + if (!targetCount || !pathCount || !fingerprintCount || !payloadCount || !candidateCount || !pairCount + || !targetOffsetsOffset || !targetRecordsOffset || !pathOffsetsOffset || !fingerprintOffsetsOffset + || !payloadOffsetsOffset || !candidateRecordsOffset || !pairsOffset || !endOffset) + return std::nullopt; + + DependencyBlobView view; + view.blob = blob; + view.targetCount = *targetCount; + view.pathCount = *pathCount; + view.fingerprintCount = *fingerprintCount; + view.payloadCount = *payloadCount; + view.candidateCount = *candidateCount; + view.pairCount = *pairCount; + view.targetOffsetsOffset = *targetOffsetsOffset; + view.targetRecordsOffset = *targetRecordsOffset; + view.pathOffsetsOffset = *pathOffsetsOffset; + view.fingerprintOffsetsOffset = *fingerprintOffsetsOffset; + view.payloadOffsetsOffset = *payloadOffsetsOffset; + view.candidateRecordsOffset = *candidateRecordsOffset; + view.pairsOffset = *pairsOffset; + view.endOffset = *endOffset; + + if (view.targetCount == 0 || view.targetCount > maxDependencyBlobTargets + || view.pathCount > maxDependencyBlobStrings || view.fingerprintCount > maxDependencyBlobStrings + || view.payloadCount == 0 || view.payloadCount > maxDependencyBlobTargets * maxDependencyBlobCandidates + || view.candidateCount == 0 + || view.candidateCount > maxDependencyBlobTargets * maxDependencyBlobCandidates + || view.pairCount > maxDependencyBlobPairs || view.endOffset != blob.size()) + return std::nullopt; + + auto sectionOk = [&](uint32_t begin, uint32_t end, size_t minBytes = 0) { + return begin >= dependencyBlobHeaderSize && begin <= end && end <= view.endOffset + && static_cast(end - begin) >= minBytes; + }; + auto stringTableOk = [&](uint32_t begin, uint32_t end, uint32_t count) { + if (!sectionOk(begin, end, (static_cast(count) + 1) * sizeof(uint32_t))) + return false; + auto bytesOffset = static_cast(begin) + (static_cast(count) + 1) * sizeof(uint32_t); + if (bytesOffset > end) + return false; + uint32_t previous = 0; + auto byteCount = static_cast(end - bytesOffset); + for (uint32_t i = 0; i <= count; i++) { + auto current = view.arrayValue(begin, i); + if (!current || *current < previous || *current > byteCount) + return false; + previous = *current; + } + return true; + }; + if (!stringTableOk(view.targetOffsetsOffset, view.targetRecordsOffset, view.targetCount) + || !sectionOk( + view.targetRecordsOffset, + view.pathOffsetsOffset, + static_cast(view.targetCount) * targetRecordU32s * 4) + || !stringTableOk(view.pathOffsetsOffset, view.fingerprintOffsetsOffset, view.pathCount) + || !stringTableOk(view.fingerprintOffsetsOffset, view.payloadOffsetsOffset, view.fingerprintCount) + || !stringTableOk(view.payloadOffsetsOffset, view.candidateRecordsOffset, view.payloadCount) + || !sectionOk( + view.candidateRecordsOffset, + view.pairsOffset, + static_cast(view.candidateCount) * candidateRecordU32s * 4) + || !sectionOk( + view.pairsOffset, view.endOffset, static_cast(view.pairCount) * pairRecordU32s * 4)) + return std::nullopt; + + auto exactRecordSection = [](uint32_t begin, uint32_t end, uint32_t count, uint32_t recordU32s) { + return static_cast(end - begin) == static_cast(count) * recordU32s * 4; + }; + if (!exactRecordSection( + view.targetRecordsOffset, view.pathOffsetsOffset, view.targetCount, targetRecordU32s) + || !exactRecordSection( + view.candidateRecordsOffset, view.pairsOffset, view.candidateCount, candidateRecordU32s) + || !exactRecordSection(view.pairsOffset, view.endOffset, view.pairCount, pairRecordU32s)) + return std::nullopt; + + return view; + } + + std::optional arrayValue(uint32_t offset, uint32_t index) const + { + return readU32(blob, static_cast(offset) + static_cast(index) * sizeof(uint32_t)); + } + + std::optional + stringFromTable(uint32_t offsetsOffset, uint32_t count, uint32_t tableEnd, uint32_t id) const + { + if (id >= count) + return std::nullopt; + auto bytesOffset = static_cast(offsetsOffset) + (static_cast(count) + 1) * sizeof(uint32_t); + if (bytesOffset > tableEnd) + return std::nullopt; + auto begin = arrayValue(offsetsOffset, id); + auto end = arrayValue(offsetsOffset, id + 1); + if (!begin || !end || *begin > *end + || static_cast(*end) > static_cast(tableEnd - bytesOffset)) + return std::nullopt; + return std::string_view(blob.data() + bytesOffset + *begin, *end - *begin); + } + + std::optional target(uint32_t id) const + { + return stringFromTable(targetOffsetsOffset, targetCount, targetRecordsOffset, id); + } + + std::optional targetField(uint32_t targetId, uint32_t field) const + { + if (targetId >= targetCount || field >= targetRecordU32s) + return std::nullopt; + return arrayValue(targetRecordsOffset, targetId * targetRecordU32s + field); + } + + std::optional targetCandidateStart(uint32_t targetId) const + { + return targetField(targetId, 0); + } + + std::optional targetCandidateCount(uint32_t targetId) const + { + return targetField(targetId, 1); + } + + std::optional findTarget(std::string_view name) const + { + uint32_t low = 0; + uint32_t high = targetCount; + while (low < high) { + uint32_t mid = low + (high - low) / 2; + auto current = target(mid); + if (!current) + return std::nullopt; + if (*current < name) + low = mid + 1; + else + high = mid; + } + if (low >= targetCount) + return std::nullopt; + auto current = target(low); + if (!current || *current != name) + return std::nullopt; + return low; + } + + std::optional path(uint32_t id) const + { + return stringFromTable(pathOffsetsOffset, pathCount, fingerprintOffsetsOffset, id); + } + + std::optional fingerprint(uint32_t id) const + { + return stringFromTable(fingerprintOffsetsOffset, fingerprintCount, payloadOffsetsOffset, id); + } + + std::optional payload(uint32_t id) const + { + return stringFromTable(payloadOffsetsOffset, payloadCount, candidateRecordsOffset, id); + } + + std::optional candidateField(uint32_t candidateIndex, uint32_t field) const + { + if (candidateIndex >= candidateCount || field >= candidateRecordU32s) + return std::nullopt; + return arrayValue(candidateRecordsOffset, candidateIndex * candidateRecordU32s + field); + } + + std::optional candidatePairStart(uint32_t candidateIndex) const + { + return candidateField(candidateIndex, 0); + } + + std::optional candidatePairCount(uint32_t candidateIndex) const + { + return candidateField(candidateIndex, 1); + } + + std::optional candidatePayload(uint32_t candidateIndex) const + { + return candidateField(candidateIndex, 2); + } + + std::optional pairField(uint32_t pairIndex, uint32_t field) const + { + if (pairIndex >= pairCount || field >= pairRecordU32s) + return std::nullopt; + return arrayValue(pairsOffset, pairIndex * pairRecordU32s + field); + } + + std::optional pairPath(uint32_t pairIndex) const + { + return pairField(pairIndex, 0); + } + + std::optional pairFingerprint(uint32_t pairIndex) const + { + return pairField(pairIndex, 1); + } + + struct CurrentFingerprints + { + ref accessor; + DependencyFingerprintCache * fingerprintCache; + }; + + CurrentFingerprints + currentFingerprints(ref accessor, DependencyFingerprintCache & fingerprintCache) const + { + return CurrentFingerprints{.accessor = accessor, .fingerprintCache = &fingerprintCache}; + } + + std::optional currentFingerprint(uint32_t pathId, CurrentFingerprints & current) const + { + auto p = path(pathId); + if (!p) + return std::nullopt; + auto cached = dependencyFingerprintCached(current.accessor, *p, *current.fingerprintCache); + if (!cached || !*cached) + return std::nullopt; + return std::string_view(**cached); + } + + bool candidateMatches(uint32_t candidateIndex, CurrentFingerprints & currentFingerprints) const + { + auto pairStart = candidatePairStart(candidateIndex); + auto count = candidatePairCount(candidateIndex); + auto payloadId = candidatePayload(candidateIndex); + if (!pairStart || !count || !payloadId) + return false; + if (*pairStart > pairCount || *count > pairCount - *pairStart || *payloadId >= payloadCount) + return false; + for (uint32_t i = 0; i < *count; i++) { + auto pairIndex = *pairStart + i; + auto pathId = pairPath(pairIndex); + auto fingerprintId = pairFingerprint(pairIndex); + if (!pathId || !fingerprintId || *pathId >= pathCount || *fingerprintId >= fingerprintCount) + return false; + auto currentFp = currentFingerprint(*pathId, currentFingerprints); + auto expectedFp = fingerprint(*fingerprintId); + if (!currentFp || !expectedFp || *currentFp != *expectedFp) + return false; + } + return true; + } + + std::optional + findMatchingCandidate(std::string_view targetName, CurrentFingerprints & currentFingerprints) const + { + auto targetId = findTarget(targetName); + if (!targetId) + return std::nullopt; + auto candidateStart = targetCandidateStart(*targetId); + auto count = targetCandidateCount(*targetId); + if (!candidateStart || !count || *candidateStart > candidateCount + || *count > candidateCount - *candidateStart) + return std::nullopt; + for (uint32_t i = 0; i < *count; i++) { + auto candidateIndex = *candidateStart + i; + if (candidateMatches(candidateIndex, currentFingerprints)) + return candidateIndex; + } + return std::nullopt; + } + + std::optional candidatePayloadValue(uint32_t candidateIndex) const + { + auto payloadId = candidatePayload(candidateIndex); + if (!payloadId) + return std::nullopt; + return payload(*payloadId); + } + + DependencyClosure toDependencyClosure(uint32_t candidateIndex) const + { + auto pairStart = candidatePairStart(candidateIndex); + auto count = candidatePairCount(candidateIndex); + if (!pairStart || !count) + throw Error("malformed Tecnix dependency cache row"); + + DependencyClosure result; + result.reserve(*count); + for (uint32_t i = 0; i < *count; i++) { + auto pairIndex = *pairStart + i; + auto pathId = pairPath(pairIndex); + auto fingerprintId = pairFingerprint(pairIndex); + if (!pathId || !fingerprintId) + throw Error("malformed Tecnix dependency cache row"); + auto p = path(*pathId); + auto fp = fingerprint(*fingerprintId); + if (!p || !fp) + throw Error("malformed Tecnix dependency cache row"); + result.push_back({std::string(*p), std::string(*fp)}); + } + return result; + } + + std::vector toShardTargets() const + { + std::vector result; + result.reserve(targetCount); + for (uint32_t targetId = 0; targetId < targetCount; targetId++) { + auto targetName = target(targetId); + auto candidateStart = targetCandidateStart(targetId); + auto count = targetCandidateCount(targetId); + if (!targetName || !candidateStart || !count || *candidateStart > candidateCount + || *count > candidateCount - *candidateStart) + throw Error("malformed Tecnix dependency cache row"); + + DependencyShardTarget targetResult; + targetResult.target = std::string(*targetName); + targetResult.candidates.reserve(*count); + for (uint32_t i = 0; i < *count; i++) { + auto candidateIndex = *candidateStart + i; + auto payloadValue = candidatePayloadValue(candidateIndex); + if (!payloadValue) + throw Error("malformed Tecnix dependency cache row"); + targetResult.candidates.push_back({ + .dependencies = toDependencyClosure(candidateIndex), + .payload = std::string(*payloadValue), + }); + } + result.push_back(std::move(targetResult)); + } + return result; + } + }; + + static void appendStringTable(std::string & out, const std::vector & values) + { + std::vector offsets; + offsets.reserve(values.size() + 1); + size_t bytes = 0; + offsets.push_back(0); + for (auto & value : values) { + bytes += value.size(); + if (!u32Fits(bytes)) + throw Error("Tecnix dependency cache row string table is too large"); + offsets.push_back(static_cast(bytes)); + } + for (auto offset : offsets) + appendU32(out, offset); + for (auto & value : values) + out.append(value.data(), value.size()); + padTo4(out); + } + + static DependencyClosure normalizedDependencyClosure(const DependencyClosure & dependencies) + { + DependencyClosure result = dependencies; + std::sort(result.begin(), result.end(), [](const auto & a, const auto & b) { return a.path < b.path; }); + + DependencyClosure deduped; + deduped.reserve(result.size()); + for (auto & dependency : result) { + if (!deduped.empty() && deduped.back().path == dependency.path) { + if (deduped.back().fingerprint != dependency.fingerprint) + throw Error( + "Tecnix dependency cache closure contains conflicting fingerprints for path '%s'", + dependency.path); + continue; + } + deduped.push_back(std::move(dependency)); + } + return deduped; + } + + static bool dependencyClosuresEqual(const DependencyClosure & a, const DependencyClosure & b) + { + if (a.size() != b.size()) + return false; + for (size_t i = 0; i < a.size(); i++) { + if (a[i].path != b[i].path || a[i].fingerprint != b[i].fingerprint) + return false; + } + return true; + } + + static std::vector candidatesWithInsertedClosure( + std::span existingCandidates, + const DependencyClosure & dependencies, + std::string payload) + { + DependencyCandidate fresh{ + .dependencies = normalizedDependencyClosure(dependencies), + .payload = std::move(payload), + }; + + std::vector result; + result.reserve(maxDependencyBlobCandidates); + result.push_back(fresh); + + for (auto & candidate : existingCandidates) { + auto normalized = normalizedDependencyClosure(candidate.dependencies); + if (candidate.payload == result.front().payload + && dependencyClosuresEqual(normalized, result.front().dependencies)) + continue; + if (result.size() >= maxDependencyBlobCandidates) + break; + result.push_back({ + .dependencies = std::move(normalized), + .payload = candidate.payload, + }); + } + + return result; + } + + static std::string dependencyBlobFromShardTargets(const std::vector & shardTargets) + { + if (shardTargets.empty() || shardTargets.size() > maxDependencyBlobTargets) + throw Error("Tecnix dependency cache shard target count is too large"); + + struct PairId + { + uint32_t pathId; + uint32_t fingerprintId; + }; + + struct TargetRecord + { + uint32_t candidateStart = 0; + uint32_t candidateCount = 0; + }; + + struct CandidateRecord + { + uint32_t pairStart; + uint32_t pairCount; + uint32_t payloadId; + }; + + std::vector normalizedTargets; + normalizedTargets.reserve(shardTargets.size()); + for (auto & shardTarget : shardTargets) { + DependencyShardTarget normalizedTarget; + normalizedTarget.target = shardTarget.target; + normalizedTarget.candidates.reserve(shardTarget.candidates.size()); + for (auto & candidate : shardTarget.candidates) { + normalizedTarget.candidates.push_back({ + .dependencies = normalizedDependencyClosure(candidate.dependencies), + .payload = candidate.payload, + }); + } + if (!normalizedTarget.candidates.empty()) + normalizedTargets.push_back(std::move(normalizedTarget)); + } + std::sort(normalizedTargets.begin(), normalizedTargets.end(), [](const auto & a, const auto & b) { + return a.target < b.target; + }); + normalizedTargets.erase( + std::unique( + normalizedTargets.begin(), + normalizedTargets.end(), + [](const auto & a, const auto & b) { return a.target == b.target; }), + normalizedTargets.end()); + if (normalizedTargets.empty()) + throw Error("Tecnix dependency cache shard has no targets"); + + size_t totalPairs = 0; + size_t totalCandidates = 0; + std::vector targets; + std::vector paths; + std::vector fingerprints; + std::vector payloads; + targets.reserve(normalizedTargets.size()); + for (auto & shardTarget : normalizedTargets) { + targets.push_back(shardTarget.target); + totalCandidates += shardTarget.candidates.size(); + if (shardTarget.candidates.size() > maxDependencyBlobCandidates + || totalCandidates > maxDependencyBlobTargets * maxDependencyBlobCandidates) + throw Error("Tecnix dependency cache candidate history is too large"); + for (auto & candidate : shardTarget.candidates) { + totalPairs += candidate.dependencies.size(); + if (totalPairs > maxDependencyBlobPairs) + throw Error("Tecnix dependency cache closure history is too large"); + payloads.push_back(candidate.payload); + for (auto & dependency : candidate.dependencies) { + paths.push_back(dependency.path); + fingerprints.push_back(dependency.fingerprint); + } + } + } + + auto sortUnique = [](std::vector values) { + std::sort(values.begin(), values.end()); + values.erase(std::unique(values.begin(), values.end()), values.end()); + return values; + }; + paths = sortUnique(std::move(paths)); + fingerprints = sortUnique(std::move(fingerprints)); + payloads = sortUnique(std::move(payloads)); + + if (targets.size() > maxDependencyBlobTargets || paths.size() > maxDependencyBlobStrings + || fingerprints.size() > maxDependencyBlobStrings || payloads.empty() + || payloads.size() > maxDependencyBlobTargets * maxDependencyBlobCandidates) + throw Error("Tecnix dependency cache row string tables are too large"); + + boost::unordered_flat_map> pathIds; + boost::unordered_flat_map> fingerprintIds; + boost::unordered_flat_map> payloadIds; + pathIds.reserve(paths.size()); + fingerprintIds.reserve(fingerprints.size()); + payloadIds.reserve(payloads.size()); + for (uint32_t i = 0; i < paths.size(); i++) + pathIds.emplace(paths[i], i); + for (uint32_t i = 0; i < fingerprints.size(); i++) + fingerprintIds.emplace(fingerprints[i], i); + for (uint32_t i = 0; i < payloads.size(); i++) + payloadIds.emplace(payloads[i], i); + + std::vector targetRecords(targets.size()); + std::vector candidateRecords; + candidateRecords.reserve(totalCandidates); + std::vector pairs; + pairs.reserve(totalPairs); + + auto getPathId = [&](const std::string & path) -> uint32_t { return pathIds.find(path)->second; }; + auto getFingerprintId = [&](const std::string & fingerprint) -> uint32_t { + return fingerprintIds.find(fingerprint)->second; + }; + auto getPayloadId = [&](const std::string & payload) -> uint32_t { return payloadIds.find(payload)->second; }; + + for (uint32_t targetId = 0; targetId < normalizedTargets.size(); targetId++) { + auto & shardTarget = normalizedTargets[targetId]; + auto candidateStart = checkedU32(candidateRecords.size(), "candidate count"); + targetRecords[targetId] = { + .candidateStart = candidateStart, + .candidateCount = checkedU32(shardTarget.candidates.size(), "target candidate count"), + }; + + for (auto & candidate : shardTarget.candidates) { + auto pairStart = checkedU32(pairs.size(), "pair count"); + std::vector candidatePairs; + candidatePairs.reserve(candidate.dependencies.size()); + for (auto & dependency : candidate.dependencies) { + candidatePairs.push_back({ + .pathId = getPathId(dependency.path), + .fingerprintId = getFingerprintId(dependency.fingerprint), + }); + } + std::sort(candidatePairs.begin(), candidatePairs.end(), [](const auto & a, const auto & b) { + if (a.pathId != b.pathId) + return a.pathId < b.pathId; + return a.fingerprintId < b.fingerprintId; + }); + pairs.insert(pairs.end(), candidatePairs.begin(), candidatePairs.end()); + candidateRecords.push_back({ + .pairStart = pairStart, + .pairCount = checkedU32(candidatePairs.size(), "candidate pair count"), + .payloadId = getPayloadId(candidate.payload), + }); + } + } + + std::string out; + out.reserve( + std::min( + maxDependencyBlobBytes, + totalPairs * 8 + totalCandidates * 12 + paths.size() * 32 + targets.size() * 64 + 4096)); + out.append(dependencyBlobMagic.data(), dependencyBlobMagic.size()); + for (size_t i = 0; i < dependencyBlobFieldCount; i++) + appendU32(out, 0); + + setBlobField(out, blobVersionField, dependencyBlobVersion); + setBlobField(out, blobFlagsField, dependencyBlobFlags); + setBlobField(out, blobTargetCountField, checkedU32(targets.size(), "target count")); + setBlobField(out, blobPathCountField, checkedU32(paths.size(), "path count")); + setBlobField(out, blobFingerprintCountField, checkedU32(fingerprints.size(), "fingerprint count")); + setBlobField(out, blobPayloadCountField, checkedU32(payloads.size(), "payload count")); + setBlobField(out, blobCandidateCountField, checkedU32(candidateRecords.size(), "candidate count")); + setBlobField(out, blobPairCountField, checkedU32(pairs.size(), "pair count")); + + padTo4(out); + setBlobField(out, blobTargetOffsetsOffsetField, checkedU32(out.size(), "row")); + appendStringTable(out, targets); + + setBlobField(out, blobTargetRecordsOffsetField, checkedU32(out.size(), "row")); + for (auto & targetRecord : targetRecords) { + appendU32(out, targetRecord.candidateStart); + appendU32(out, targetRecord.candidateCount); + } + + setBlobField(out, blobPathOffsetsOffsetField, checkedU32(out.size(), "row")); + appendStringTable(out, paths); + + setBlobField(out, blobFingerprintOffsetsOffsetField, checkedU32(out.size(), "row")); + appendStringTable(out, fingerprints); + + setBlobField(out, blobPayloadOffsetsOffsetField, checkedU32(out.size(), "row")); + appendStringTable(out, payloads); + + padTo4(out); + setBlobField(out, blobCandidateRecordsOffsetField, checkedU32(out.size(), "row")); + for (auto & candidateRecord : candidateRecords) { + appendU32(out, candidateRecord.pairStart); + appendU32(out, candidateRecord.pairCount); + appendU32(out, candidateRecord.payloadId); + } + + setBlobField(out, blobPairsOffsetField, checkedU32(out.size(), "row")); + for (auto & pair : pairs) { + appendU32(out, pair.pathId); + appendU32(out, pair.fingerprintId); + } + + padTo4(out); + if (!u32Fits(out.size()) || out.size() > maxDependencyBlobBytes) + throw Error("Tecnix dependency cache row is too large"); + setBlobField(out, blobEndOffsetField, static_cast(out.size())); + return out; + } + + using DependencyBlobRef = std::shared_ptr; + + std::vector> + lookupBlobs(const TecnixCacheScope & scope, std::span targets) + { + std::vector> blobs(targets.size()); + if (targets.empty()) + return blobs; + + boost::unordered_flat_map> indicesByShard; + indicesByShard.reserve(std::min(targets.size(), dependencyShardCount)); + for (size_t i = 0; i < targets.size(); i++) + indicesByShard[dependencyShardForTarget(targets[i])].push_back(i); + + auto state(_state.lock()); + if (indicesByShard.size() == 1) { + auto shard = indicesByShard.begin()->first; + auto stmt(state->lookupShard.use()(scope.gitDir)(scope.resolver)(scope.argsKey)(shard)); + if (!stmt.next()) + return blobs; + auto blobView = stmt.getBlob(0); + auto blob = std::make_shared(blobView.data(), blobView.size()); + for (auto index : indicesByShard.begin()->second) + blobs[index] = blob; + return blobs; + } + + auto stmt(state->lookupAllShards.use()(scope.gitDir)(scope.resolver)(scope.argsKey)); + while (stmt.next()) { + auto shard = static_cast(stmt.getInt(0)); + auto indices = indicesByShard.find(shard); + if (indices == indicesByShard.end()) + continue; + auto blobView = stmt.getBlob(1); + auto blob = std::make_shared(blobView.data(), blobView.size()); + for (auto index : indices->second) + blobs[index] = blob; + } + + return blobs; + } + + static std::vector shardTargetsWithUpdates( + std::optional existingBlob, + const std::vector> & updates) + { + std::vector shardTargets; + if (existingBlob) { + if (auto view = DependencyBlobView::open(*existingBlob)) { + try { + shardTargets = view->toShardTargets(); + } catch (const Error &) { + shardTargets.clear(); + } + } + } + + std::sort(shardTargets.begin(), shardTargets.end(), [](const auto & a, const auto & b) { + return a.target < b.target; + }); + + for (auto & [target, dependencies, payload] : updates) { + auto it = std::lower_bound( + shardTargets.begin(), shardTargets.end(), target, [](const auto & entry, const std::string & target) { + return entry.target < target; + }); + if (it == shardTargets.end() || it->target != target) { + DependencyShardTarget inserted; + inserted.target = target; + it = shardTargets.insert(it, std::move(inserted)); + } + it->candidates = candidatesWithInsertedClosure( + std::span{it->candidates.data(), it->candidates.size()}, + *dependencies, + payload); + } + + if (existingBlob && shardTargets.size() > maxDependencyBlobTargets) { + // The blob has no per-target age, so evict everything from + // previous evaluations; dropped targets re-enter on next use. + warn("tecnix: dependency cache shard is full; evicting entries from previous evaluations"); + return shardTargetsWithUpdates(std::nullopt, updates); + } + + return shardTargets; + } + + void upsertMany(const TecnixCacheScope & scope, const std::vector & entries) + { + if (entries.empty()) + return; + + boost:: + unordered_flat_map>> + updatesByShard; + updatesByShard.reserve(std::min(entries.size(), dependencyShardCount)); + for (auto & entry : entries) + updatesByShard[dependencyShardForTarget(entry.target)].push_back( + {std::string(entry.target), entry.dependencies, entry.payload}); + + auto state(_state.lock()); + SQLiteImmediateTxn txn(state->db); + + boost::unordered_flat_map existingBlobs; + existingBlobs.reserve(updatesByShard.size()); + if (updatesByShard.size() == 1) { + auto shard = updatesByShard.begin()->first; + auto stmt(state->lookupShard.use()(scope.gitDir)(scope.resolver)(scope.argsKey)(shard)); + if (stmt.next()) { + auto blobView = stmt.getBlob(0); + existingBlobs.emplace(shard, std::string(blobView.data(), blobView.size())); + } + } else { + auto stmt(state->lookupAllShards.use()(scope.gitDir)(scope.resolver)(scope.argsKey)); + while (stmt.next()) { + auto shard = static_cast(stmt.getInt(0)); + if (updatesByShard.find(shard) == updatesByShard.end()) + continue; + auto blobView = stmt.getBlob(1); + existingBlobs.emplace(shard, std::string(blobView.data(), blobView.size())); + } + } + + std::vector> blobs; + blobs.reserve(updatesByShard.size()); + for (auto & [shard, updates] : updatesByShard) { + std::optional existingBlob; + if (auto existing = existingBlobs.find(shard); existing != existingBlobs.end()) + existingBlob = std::string_view(existing->second); + auto shardTargets = shardTargetsWithUpdates(existingBlob, updates); + blobs.emplace_back(shard, dependencyBlobFromShardTargets(shardTargets)); + } + + auto timestamp = time(nullptr); + for (auto & [shard, blob] : blobs) { + state->upsertShard + .use()(scope.gitDir)(scope.resolver)(scope.argsKey)( + shard) (reinterpret_cast(blob.data()), blob.size())(timestamp) + .exec(); + } + txn.commit(); + } +}; + +static TecnixEvalCache & getTecnixEvalCache() +{ + static TecnixEvalCache cache; + return cache; +} + +static void warnTecnixEvalCacheWriteFailure(const std::exception & e) +{ + warn("tecnix: failed to write eval cache entry; continuing without caching this result: %s", e.what()); +} + +static void warnTecnixEvalCacheWriteFailure() +{ + warn("tecnix: failed to write eval cache entry; continuing without caching this result"); +} + +/** + * Store freshly evaluated closures. Cache writes are an optimization: failures + * warn and continue, they never fail the evaluation that produced the result. + */ +void upsertDependencyClosures(const TecnixCacheScope & scope, const std::vector & entries) +{ + if (entries.empty()) + return; + try { + getTecnixEvalCache().upsertMany(scope, entries); + } catch (const std::exception & e) { + warnTecnixEvalCacheWriteFailure(e); + } catch (...) { + warnTecnixEvalCacheWriteFailure(); + } +} + +/** The blob's string storage is owned by the shared_ptr, so the view's + * borrowed spans stay valid for the Impl's lifetime and moves of the + * public handle move only the pointer. */ +struct ValidatedDependencyBlob::Impl +{ + TecnixEvalCache::DependencyBlobRef blob; + TecnixEvalCache::DependencyBlobView view; + uint32_t candidateIndex; +}; + +ValidatedDependencyBlob::ValidatedDependencyBlob(std::unique_ptr impl) + : impl(std::move(impl)) +{ +} + +ValidatedDependencyBlob::ValidatedDependencyBlob(ValidatedDependencyBlob &&) noexcept = default; +ValidatedDependencyBlob & ValidatedDependencyBlob::operator=(ValidatedDependencyBlob &&) noexcept = default; +ValidatedDependencyBlob::~ValidatedDependencyBlob() = default; + +std::optional ValidatedDependencyBlob::payload() const +{ + return impl->view.candidatePayloadValue(impl->candidateIndex); +} + +/** + * Look up cached dependency rows for `keys` (target IDs or the discovery key) + * and validate their candidates against current fingerprints. An entry is set + * on a proven hit and nullopt on a miss. + */ +std::vector> lookupValidatedDependencyBlobs( + EvalState & state, + const TecnixCacheScope & scope, + std::span keys, + DependencyFingerprintCache & fingerprintCache) +{ + std::vector> results(keys.size()); + auto repoAccessor = getTecnixRepoAccessor(state); + auto cachedBlobs = getTecnixEvalCache().lookupBlobs(scope, keys); + + struct BlobWork + { + TecnixEvalCache::DependencyBlobRef blob; + std::vector indices; + }; + + std::vector blobWork; + blobWork.reserve(std::min(keys.size(), TecnixEvalCache::dependencyShardCount)); + boost::unordered_flat_map blobWorkByBlob; + for (size_t i = 0; i < cachedBlobs.size(); i++) { + if (!cachedBlobs[i]) + continue; + auto key = cachedBlobs[i]->get(); + auto [it, inserted] = blobWorkByBlob.emplace(key, blobWork.size()); + if (inserted) + blobWork.push_back({.blob = *cachedBlobs[i], .indices = {}}); + blobWork[it->second].indices.push_back(i); + } + + for (auto & work : blobWork) { + auto view = TecnixEvalCache::DependencyBlobView::open(*work.blob); + if (!view) + continue; + auto currentFingerprints = view->currentFingerprints(repoAccessor, fingerprintCache); + for (auto i : work.indices) { + if (auto candidate = view->findMatchingCandidate(keys[i], currentFingerprints)) + results[i].emplace( + std::unique_ptr( + new ValidatedDependencyBlob::Impl{work.blob, *view, *candidate})); + } + } + + return results; +} + +static DependencyFingerprintThreadLocalCache & getDependencyFingerprintThreadCache(DependencyFingerprintCache & cache) +{ + static thread_local DependencyFingerprintThreadLocalCache threadCache; + if (threadCache.generation != cache.generation) { + threadCache.generation = cache.generation; + threadCache.fingerprints.clear(); + } + return threadCache; +} + +static const std::optional * +dependencyFingerprintCached(ref accessor, std::string_view path, DependencyFingerprintCache & cache) +{ + auto & threadCache = getDependencyFingerprintThreadCache(cache); + + auto it = threadCache.fingerprints.find(path); + if (it != threadCache.fingerprints.end()) + return &it->second; + + auto [_, fp] = accessor->getFingerprint(CanonPath(path)); + auto inserted = threadCache.fingerprints.emplace(std::string(path), std::move(fp)).first; + return &inserted->second; +} + +std::optional +dependencyFingerprint(ref accessor, std::string_view path, DependencyFingerprintCache & cache) +{ + auto fp = dependencyFingerprintCached(accessor, path, cache); + if (!fp) + return std::nullopt; + return *fp; +} + +DependencyClosure dependencyFingerprints( + ref accessor, const std::vector & paths, DependencyFingerprintCache & cache) +{ + DependencyClosure result; + result.reserve(paths.size()); + for (auto & path : paths) { + auto fp = dependencyFingerprint(accessor, path, cache); + if (!fp) + throw Error("failed to fingerprint Tecnix dependency path '%s'", path); + result.push_back({path, std::move(*fp)}); + } + return result; +} + +Value * ValidatedDependencyBlob::toValue(EvalState & state) const +{ + auto & view = impl->view; + auto candidateIndex = impl->candidateIndex; + auto pairStart = view.candidatePairStart(candidateIndex); + auto pairCount = view.candidatePairCount(candidateIndex); + if (!pairStart || !pairCount) + throw Error("malformed Tecnix dependency cache row"); + + auto attrs = state.buildBindings(*pairCount); + for (uint32_t i = 0; i < *pairCount; i++) { + auto pairIndex = *pairStart + i; + auto pathId = view.pairPath(pairIndex); + auto fingerprintId = view.pairFingerprint(pairIndex); + if (!pathId || !fingerprintId) + throw Error("malformed Tecnix dependency cache row"); + auto path = view.path(*pathId); + auto fingerprint = view.fingerprint(*fingerprintId); + if (!path || !fingerprint) + throw Error("malformed Tecnix dependency cache row"); + + auto * fingerprintValue = state.allocValue(); + fingerprintValue->mkString(*fingerprint, state.mem); + attrs.insert(state.symbols.create(*path), fingerprintValue); + } + auto * val = state.allocValue(); + val->mkAttrs(attrs); + + return val; +} + +} // namespace nix diff --git a/src/libexpr/tecnix/eval-data.hh b/src/libexpr/tecnix/eval-data.hh new file mode 100644 index 0000000000..40c216ccd9 --- /dev/null +++ b/src/libexpr/tecnix/eval-data.hh @@ -0,0 +1,144 @@ +#pragma once +///@file + +#include "nix/expr/eval.hh" +#include "nix/expr/tecnix/access-set-graph.hh" + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace nix { + +struct ParsedFileCacheEntry; +using EvalParsedFileCache = boost::concurrent_flat_map>; + +struct EvalImportResolutionCacheEntry +{ + SourcePath resolvedPath; + EvalSourceAccessSetId sourceDeps = emptyEvalSourceAccessSetId; +}; + +using EvalImportResolutionCache = boost::concurrent_flat_map; +using EvalFileCache = boost::concurrent_flat_map< + SourcePath, + Value *, + std::hash, + std::equal_to, + traceable_allocator>>; +using EvalWorldTreeShaCache = boost::concurrent_flat_map; + +struct EvalState::TecnixEvalData +{ + /** + * A cache that maps paths to "resolved" paths for importing Nix + * expressions, i.e. `/foo` to `/foo/default.nix`. + */ + const ref importResolutionCache = make_ref(); + + /** + * Shared import-resolution cache for tracked Tecnix evaluation. Entries also + * carry the source-deps label recorded while resolving so cache hits can + * replay symlink/default-resolution provenance into each target. + */ + const ref trackedImportResolutionCache = make_ref(); + + /** + * A cache from resolved paths to parsed expressions. This is safe to share + * across tracking contexts because evaluated values/thunks remain isolated. + */ + const ref parsedFileCache = make_ref(); + + /** + * Canonical flat source access-set graph for Tecnix dependency tracking. + */ + const ref sourceAccessSetGraph = make_ref(); + + /** + * Shared evaluated-file cache for tracked Tecnix dependency discovery. + * Kept separate from the normal evaluator cache so untracked eval cannot + * prewarm entries that lack source-deps labels. + */ + const ref trackedFileEvalCache = make_ref(); + + struct TectonixContext + { + std::string gitDir; + std::string rev; + std::string checkoutPath; + }; + + mutable std::mutex tectonixContextMutex; + mutable std::optional tectonixContext; + + /** Lazy-initialized git repository for world builtins (thread-safe via once_flag) */ + mutable std::once_flag worldRepoFlag; + mutable std::optional> worldRepo; + + /** Lazy-initialized source accessor for world git content (thread-safe via once_flag) */ + mutable std::once_flag worldGitAccessorFlag; + mutable std::optional> worldGitAccessor; + + /** + * Repo-wide source accessor with dirty overlay. Lazily created. + * All file reads during Tecnix evaluation go through this single accessor, + * so tracked paths are naturally repo-relative. + */ + mutable std::once_flag tecnixRepoAccessorFlag; + mutable std::optional> tecnixRepoAccessor; + + /** + * Virtual store path where the Tecnix repo-wide accessor is lazily mounted. + * All repo subtree store paths are subpaths of this mount. + */ + mutable std::once_flag tecnixRepoMountFlag; + mutable std::optional tecnixRepoMountStorePath; + + /** Cache: world path → tree SHA (lazy computed, cached at each path level) */ + const ref worldTreeShaCache = make_ref(); + + /** Lazy-initialized set of zone IDs in sparse checkout (thread-safe via once_flag) */ + mutable std::once_flag tectonixSparseCheckoutRootsFlag; + mutable std::set tectonixSparseCheckoutRoots; + + /** Lazy-initialized map of zone path → dirty info (thread-safe via once_flag) */ + mutable std::once_flag tectonixDirtyZonesFlag; + mutable std::map tectonixDirtyZones; + + /** Cached manifest content (thread-safe via once_flag) */ + mutable std::once_flag tectonixManifestFlag; + mutable std::string tectonixManifestContent; + + /** Cached parsed manifest JSON (thread-safe via once_flag) */ + mutable std::once_flag tectonixManifestJsonFlag; + mutable std::unique_ptr tectonixManifestJson; + + /** + * Cache tree SHA → virtual store path for lazy zone mounts. + * Thread-safe for eval-cores > 1. + */ + mutable SharedSync> tectonixZoneCache_; + + /** + * Cache zone path → virtual store path for lazy checkout zone mounts. + * Thread-safe for eval-cores > 1. + */ + mutable SharedSync> tectonixCheckoutZoneCache_; + + /** + * Lazily-connected worldtree daemon control connection (zone tree SHAs + + * dirty set), or null when the socket is unset or the evaluation targets an + * immutable historical FUSE view rather than the mutable root checkout. + */ + mutable std::once_flag worldtreeControlConnFlag; + mutable std::shared_ptr worldtreeControlConn_; +}; + +} // namespace nix diff --git a/src/libexpr/tecnix/repo-accessor.cc b/src/libexpr/tecnix/repo-accessor.cc new file mode 100644 index 0000000000..98ff3734f1 --- /dev/null +++ b/src/libexpr/tecnix/repo-accessor.cc @@ -0,0 +1,792 @@ +/** + * Tecnix source observation (the tracked evaluation path): the repo-wide + * accessor composing a clean backend (libgit2 tree or worldtree FUSE + * projection) with the dirty-checkout overlay, per-path fingerprints, and + * repo-relative access recording. Serves only the new `builtins.tecnix*` + * API; the legacy accessor implementations live in tecnix/source-accessors.cc + * and stay isolated from this code. + */ + +#include "eval-data.hh" +#include "nix/expr/eval-inline.hh" +#include "nix/expr/eval-settings.hh" +#include "nix/expr/tecnix/source-accessors.hh" +#include "nix/fetchers/fetch-to-store.hh" +#include "nix/fetchers/git-utils.hh" +#include "nix/store/store-api.hh" +#include "nix/util/environment-variables.hh" +#include "nix/util/file-system.hh" +#include "nix/util/hash.hh" +#include "nix/util/processes.hh" +#include "nix/util/source-accessor.hh" +#include "nix/util/strings.hh" +#include "nix/util/util.hh" + +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace nix { + +static EvalState::TecnixEvalData * tecnixData(EvalState & state) +{ + return &state.tecnixEvalData(); +} + +/** Strip a leading `//` world prefix: `//areas/x` -> `areas/x`. */ +static std::string tecnixNormalizeRepoPath(std::string_view path) +{ + std::string result(path); + if (hasPrefix(result, "//")) + result = result.substr(2); + return result; +} + +// Worldtree FUSE-projection helpers, deliberately duplicated from the legacy +// accessors so that file stays byte-identical to upstream wt-single-mount. + +static constexpr std::string_view TECNIX_WORLDTREE_TREE_OID_XATTR = "user.worldtree.tree-oid"; +static constexpr std::string_view TECNIX_WORLDTREE_BLOB_OID_XATTR = "user.worldtree.blob-oid"; + +static std::filesystem::path tecnixWorldtreeRevisionRoot(const EvalSettings & settings) +{ + auto revision = Hash::parseNonSRIUnprefixed(settings.tectonixGitSha.get(), HashAlgorithm::SHA1); + return std::filesystem::path(settings.tectonixWorldtreeMount.get()) / "tecnix" / revision.gitRev(); +} + +static std::string tecnixRequireWorldtreeZoneId(const std::string & id) +{ + auto isLowerHex = [](char c) { return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'); }; + if (id.size() != 8 || !id.starts_with("W-") || !std::ranges::all_of(std::string_view(id).substr(2), isLowerHex)) + throw Error("worldtree: invalid zone id '%s' in manifest", id); + return id; +} + +/** A git oid from one of the daemon's synthetic xattrs, or nullopt. */ +static std::optional tecnixReadWorldtreeOidXattr(const std::filesystem::path & path, std::string_view xattrName) +{ + std::array value{}; +#ifdef __APPLE__ + auto size = ::getxattr(path.c_str(), xattrName.data(), value.data(), value.size(), 0, 0); +#else + auto size = ::getxattr(path.c_str(), xattrName.data(), value.data(), value.size()); +#endif + if (size != static_cast(value.size())) + return std::nullopt; + try { + return Hash::parseNonSRIUnprefixed(std::string(value.data(), value.size()), HashAlgorithm::SHA1); + } catch (BadHash &) { + return std::nullopt; + } +} + +/** Directory identity. Unlike blob oids, the daemon guarantees this xattr on + * every directory it serves — zone mounting already relies on it — so absence + * is a contract violation, not a fallback case. */ +static Hash tecnixReadWorldtreeTreeOid(const std::filesystem::path & path) +{ + if (auto oid = tecnixReadWorldtreeOidXattr(path, TECNIX_WORLDTREE_TREE_OID_XATTR)) + return *oid; + throw Error("worldtree: cannot read tree identity for '%s'", path.string()); +} + +[[noreturn]] static void tecnixThrowHistoricalWorldManifestError(const EvalSettings & settings, std::string_view detail) +{ + throw Error( + "worldtree: historical World manifest '.meta/manifest.json' for commit '%s' is missing or malformed: %s", + settings.tectonixGitSha.get(), + detail); +} + +void configureTectonixContext(EvalState & state, std::string gitDir, std::string rev, std::string checkoutPath) +{ + std::lock_guard lock(tecnixData(state)->tectonixContextMutex); + + EvalState::TecnixEvalData::TectonixContext next{ + .gitDir = std::move(gitDir), + .rev = std::move(rev), + .checkoutPath = std::move(checkoutPath), + }; + + if (tecnixData(state)->tectonixContext) { + auto & current = *tecnixData(state)->tectonixContext; + if (current.gitDir != next.gitDir || current.rev != next.rev || current.checkoutPath != next.checkoutPath) + throw Error( + "Tecnix EvalState is already configured for gitDir '%s', rev '%s', checkoutPath '%s'; " + "cannot reconfigure it for gitDir '%s', rev '%s', checkoutPath '%s'", + current.gitDir, + current.rev, + current.checkoutPath, + next.gitDir, + next.rev, + next.checkoutPath); + return; + } + + tecnixData(state)->tectonixContext = next; + + auto & mutableSettings = const_cast(state.settings); + mutableSettings.tectonixGitDir.assign(next.gitDir); + mutableSettings.tectonixGitSha.assign(next.rev); + if (!next.checkoutPath.empty()) + mutableSettings.tectonixCheckoutPath.assign(next.checkoutPath); +} + +static std::string gitFingerprintWithMode(const Hash & oid, uint32_t mode) +{ + std::ostringstream out; + out << "git:" << oid.gitRev() << ";mode=" << std::oct << std::setw(6) << std::setfill('0') << mode; + return out.str(); +} + +/** + * The dirty (modified, added, deleted, renamed, untracked) repo-relative + * paths of `checkoutPath`, per `git status --porcelain -z`. + * + * Scrubs the git discovery environment (GIT_DIR, GIT_WORK_TREE, + * GIT_COMMON_DIR, GIT_INDEX_FILE) so an ambient git context — a hook, a + * rebase exec step, tooling that exports these — cannot redirect status to a + * different repository or index. Throws if git status fails: the dirty set + * is load-bearing for Tecnix source-closure validity, so callers that can + * tolerate an unknown dirty state must catch explicitly. + */ +static std::vector gitStatusDirtyPaths(const std::string & checkoutPath) +{ + StringMap gitEnvironment = getEnv(); + gitEnvironment.erase("GIT_DIR"); + gitEnvironment.erase("GIT_WORK_TREE"); + gitEnvironment.erase("GIT_COMMON_DIR"); + gitEnvironment.erase("GIT_INDEX_FILE"); + + auto [status, output] = runProgram( + {.program = "git", + .args = {"-C", checkoutPath, "--no-optional-locks", "status", "--porcelain", "-z", "--untracked-files=all"}, + .environment = gitEnvironment}); + if (!statusOk(status)) + throw Error("failed to get git status in '%s': program 'git' %s", checkoutPath, statusToString(status)); + + return parseGitPorcelainZDirtyPaths(output); +} + +/** + * Read delegation wrapper that preserves the path-fingerprint semantics Tecnix + * historically used with libgit2: fingerprints are the git object id at the + * requested path, not merely the root accessor fingerprint. + */ +/** + * The Tecnix source accessor: a clean source tree pinned at a rev, an + * optional dirty-checkout overlay, per-path fingerprints, and repo-relative + * source-access tracking, in one place. + * + * Clean content is served by `clean`: a libgit2 tree accessor or, in a + * worldtree sandbox, the immutable FUSE projection at the pinned rev + * (`WorldtreeFuseSourceAccessor`). Clean-path fingerprints come from libgit2 + * (`git`) when set; otherwise `clean` fingerprints its own paths via + * `getFingerprint` (the worldtree FUSE backend does this natively). + * + * `repoPrefix` maps accessor-local paths to the repo-relative paths recorded + * as source dependencies: empty for the repo-wide accessor, or a zone path + * for legacy zone accessors. Repo-root access has no source-path + * representation yet, so with an empty prefix it fails closed under active + * tracking instead of silently under-tracking. + */ +struct TecnixSourceAccessor : SourceAccessor +{ + struct GitCleanFingerprints + { + ref repo; + Hash treeSha; + }; + + /** Transparent hashing so hot-path lookups take `path.rel()` views + * without materializing a std::string per read/stat. */ + using DirtyPathSet = boost::unordered_flat_set>; + + ref clean, disk; + std::optional git; + std::string repoPrefix; + DirtyPathSet dirtyFiles, dirtyDirs; + + TecnixSourceAccessor( + ref clean, + ref disk, + std::optional git, + std::string repoPrefix, + DirtyPathSet && dirtyFiles) + : clean(std::move(clean)) + , disk(std::move(disk)) + , git(std::move(git)) + , repoPrefix(std::move(repoPrefix)) + , dirtyFiles(std::move(dirtyFiles)) + { + for (auto & f : this->dirtyFiles) { + debug("TecnixSourceAccessor: dirty file: '%s'", f); + for (auto p = CanonPath(f); !p.isRoot();) { + p.pop(); + if (!dirtyDirs.emplace(p.rel()).second) + break; + } + } + } + + std::string_view trackedRepoPathForAccess(const CanonPath & path, std::string & scratch) const + { + if (path.isRoot()) { + if (repoPrefix.empty()) + throw Error( + "Tecnix dependency tracking cannot represent repo-root source access yet; use a repo-relative child path instead"); + return repoPrefix; + } + + auto rel = path.rel(); + if (repoPrefix.empty()) + return rel; + + scratch.reserve(repoPrefix.size() + 1 + rel.size()); + scratch.append(repoPrefix); + scratch.push_back('/'); + scratch.append(rel); + return scratch; + } + + bool isDirty(const CanonPath & path) + { + return dirtyFiles.contains(path.rel()); + } + + bool tracksEvalAccesses(const CanonPath &) override + { + return true; + } + + void recordEvalAccess(const CanonPath & path) override + { + trackAccess(path); + } + + void trackAccess(const CanonPath & path) + { + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) { + static thread_local std::string trackedRepoPathScratch; + trackedRepoPathScratch.clear(); + ctx->recordAccess(trackedRepoPathForAccess(path, trackedRepoPathScratch)); + } + } + + /** + * Deliberately untracked. Stats are mostly evaluator plumbing (symlink + * and import resolution, store-copy machinery), and recording every stat + * would drag ancestor directories — up to the unrepresentable repo root — + * into closures. The rule is: record at the lowest layer that knows the + * observation is semantic. Every read is semantic, so reads self-record + * here; for stats only the call site knows, so call sites where existence + * or file type is the observed result must record it via + * `recordEvalAccess` (see `prim_pathExists` and `prim_readFileType`). Any + * new primop that observes existence or type without a read must do the + * same, or it silently under-tracks. + */ + std::optional maybeLstat(const CanonPath & path) override + { + std::optional s; + if (path.isRoot()) + s = clean->maybeLstat(path); + else if (isDirty(path)) + s = disk->maybeLstat(path); + else { + s = clean->maybeLstat(path); + if (!s && dirtyDirs.contains(path.rel())) + s = disk->maybeLstat(path); + } + return s; + } + + void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override + { + if (dumpPathDepth == 0) + trackAccess(path); + return (isDirty(path) ? disk : clean)->readFile(path, sink, sizeCallback); + } + + std::string readLink(const CanonPath & path) override + { + if (dumpPathDepth == 0) + trackAccess(path); + return (isDirty(path) ? disk : clean)->readLink(path); + } + + std::optional getPhysicalPath(const CanonPath & path) override + { + return (isDirty(path) ? disk : clean)->getPhysicalPath(path); + } + + /** The fingerprint of `path` in the clean tree, before the dirty overlay. */ + std::pair> cleanFingerprint(const CanonPath & path) + { + if (!git) + return clean->getFingerprint(path); + + if (path.isRoot()) + return {path, "git:" + git->treeSha.gitRev()}; + + auto pathInfo = git->repo->getPathInfo(git->treeSha, std::string(path.rel())); + if (!pathInfo) + return {path, "absent"}; + return {path, gitFingerprintWithMode(pathInfo->oid, pathInfo->mode)}; + } + + /** + * Deliberately memo-free: Tecnix closure validation and fingerprinting + * already memoize per run (the thread-local dependency-fingerprint cache + * in this file), and other callers (fetchToStore) are bounded by the + * srcToStore cache. One caching layer is enough. + */ + std::pair> getFingerprint(const CanonPath & path) override + { + trackAccess(path); + + auto rel = path.isRoot() ? std::string{} : std::string(path.rel()); + auto [_cleanPath, cleanFp] = cleanFingerprint(path); + + // nullopt means the backend declined to certify this path (a missing + // path reports "absent"); propagate rather than coercing to "absent", + // which would be a fetch-cache key shared by every revision. + if (!cleanFp) + return {path, std::nullopt}; + + auto dirtyPrefix = path.isRoot() ? "" : rel + "/"; + std::vector dirtyUnderPath; + for (auto & f : dirtyFiles) { + if (path.isRoot() || f == rel || f.starts_with(dirtyPrefix)) + dirtyUnderPath.push_back(f); + } + + if (!path.isRoot() && dirtyFiles.contains(rel) && !disk->maybeLstat(path)) + return {path, "absent"}; + + std::string fp = *cleanFp; + if (!dirtyUnderPath.empty()) { + std::sort(dirtyUnderPath.begin(), dirtyUnderPath.end()); + HashSink hashSink{HashAlgorithm::SHA256}; + for (auto & f : dirtyUnderPath) { + hashSink << f; + auto st = disk->maybeLstat(CanonPath(f)); + if (!st) { + hashSink << "D"; + } else if (st->type == Type::tRegular) { + hashSink << (st->isExecutable ? "X" : "F"); + hashSink << disk->readFile(CanonPath(f)); + } else if (st->type == Type::tSymlink) { + hashSink << "L"; + hashSink << disk->readLink(CanonPath(f)); + } + } + fp += ";dirty=" + hashSink.finish().hash.to_string(HashFormat::Base16, false); + } + + return {path, fp}; + } + + DirEntries readDirectory(const CanonPath & path) override + { + if (dumpPathDepth == 0) + trackAccess(path); + + auto rel = path.isRoot() ? "" : std::string(path.rel()); + if (!path.isRoot() && !dirtyDirs.contains(rel)) + return clean->readDirectory(path); + + DirEntries entries; + try { + entries = clean->readDirectory(path); + } catch (...) { + } + + auto dirPrefix = rel.empty() ? "" : rel + "/"; + for (auto & f : dirtyFiles) { + if (!f.starts_with(dirPrefix)) + continue; + auto rest = std::string_view(f).substr(dirPrefix.size()); + if (rest.find('/') != std::string_view::npos) + continue; + auto stat = disk->maybeLstat(path / rest); + if (stat) + entries[std::string(rest)] = stat->type; + else + entries.erase(std::string(rest)); + } + for (auto & d : dirtyDirs) { + if (!d.starts_with(dirPrefix)) + continue; + auto rest = std::string_view(d).substr(dirPrefix.size()); + if (rest.find('/') != std::string_view::npos || rest.empty()) + continue; + if (!entries.count(std::string(rest))) + entries[std::string(rest)] = Type::tDirectory; + } + return entries; + } +}; + +// ============================================================================ +// Tecnix repo-wide accessor (the tracked evaluation path) +// ============================================================================ + +/** + * The Tecnix clean-tree backend for worldtree sandboxes: the committed repo + * view at a pinned revision, served by the worldtree FUSE projection + * (`/tecnix/`). Repo-relative paths map through the committed + * manifest to `/` reads; ancestors of zone roots + * are synthesized, and committed paths outside every visible zone do not + * exist in this view (the daemon's visibility contract). + * + * Fingerprints keep the git vocabulary (`git:;mode=` / `absent`) + * so TXDC rows validate across backends: directories read the tree-oid + * xattr, files the blob-oid xattr when served (hashing the bytes otherwise, + * which reproduces the same oid), synthesized directories compose their + * children. See the explainer §7 for the economics. + */ +struct WorldtreeFuseSourceAccessor : SourceAccessor +{ + std::filesystem::path revisionRoot; + + /** Blob fingerprint memo; the projection is immutable, so entries stay + * valid for the accessor's lifetime. */ + boost::concurrent_flat_map blobFingerprintMemo; + + /** Repo-relative zone path (no leading `//`) → validated zone id. */ + std::map zones; + + /** Per-zone filesystem accessors rooted at `/`. */ + std::mutex zoneFSMutex; + std::map> zoneFS; + + WorldtreeFuseSourceAccessor(std::filesystem::path revisionRoot, const nlohmann::json & manifest) + : revisionRoot(std::move(revisionRoot)) + { + for (auto & [worldPath, value] : manifest.items()) { + if (!value.is_object() || !value.contains("id") || !value.at("id").is_string()) + continue; + auto zonePath = tecnixNormalizeRepoPath(worldPath); + if (zonePath.empty()) + continue; + zones[zonePath] = tecnixRequireWorldtreeZoneId(value.at("id").get()); + } + if (zones.empty()) + throw Error("worldtree: the manifest at '%s' names no zones", this->revisionRoot.string()); + } + + struct InZone + { + std::string zonePath; + std::string zoneId; + std::string rel; // path under the zone root; empty for the zone root itself + }; + + static std::string_view key(const CanonPath & path) + { + return path.isRoot() ? std::string_view() : path.rel(); + } + + /** Longest manifest zone containing `rel`, if any. */ + std::optional resolveZone(std::string_view rel) const + { + std::optional best; + for (auto & [zonePath, zoneId] : zones) { + bool contains = + rel == zonePath + || (rel.size() > zonePath.size() && rel.starts_with(zonePath) && rel[zonePath.size()] == '/'); + if (contains && (!best || zonePath.size() > best->zonePath.size())) + best = InZone{ + .zonePath = zonePath, + .zoneId = zoneId, + .rel = rel.size() == zonePath.size() ? std::string() : std::string(rel.substr(zonePath.size() + 1)), + }; + } + return best; + } + + /** Whether `rel` is the repo root or a proper ancestor of a zone root. */ + bool isZoneAncestor(std::string_view rel) const + { + if (rel.empty()) + return true; + for (auto & [zonePath, _] : zones) + if (zonePath.size() > rel.size() && zonePath.starts_with(rel) && zonePath[rel.size()] == '/') + return true; + return false; + } + + /** Next path components of zone roots strictly below ancestor `rel`. */ + std::set zoneChildNames(std::string_view rel) const + { + std::set names; + for (auto & [zonePath, _] : zones) { + std::string_view tail; + if (rel.empty()) + tail = zonePath; + else if (zonePath.size() > rel.size() && zonePath.starts_with(rel) && zonePath[rel.size()] == '/') + tail = std::string_view(zonePath).substr(rel.size() + 1); + else + continue; + names.insert(std::string(tail.substr(0, tail.find('/')))); + } + return names; + } + + ref zoneAccessor(const InZone & z) + { + std::lock_guard lock(zoneFSMutex); + if (auto it = zoneFS.find(z.zoneId); it != zoneFS.end()) + return it->second; + auto root = revisionRoot / z.zoneId; + std::error_code ec; + if (!std::filesystem::is_directory(root, ec)) + throw Error("worldtree: immutable zone '%s' is unavailable at '%s'", z.zonePath, root.string()); + auto accessor = makeFSSourceAccessor(root); + zoneFS.emplace(z.zoneId, accessor); + return accessor; + } + + static CanonPath zoneRelPath(const InZone & z) + { + return CanonPath(z.rel); + } + + std::optional maybeLstat(const CanonPath & path) override + { + auto rel = key(path); + if (auto z = resolveZone(rel)) + return zoneAccessor(*z)->maybeLstat(zoneRelPath(*z)); + if (isZoneAncestor(rel)) + return Stat{.type = tDirectory}; + return std::nullopt; + } + + DirEntries readDirectory(const CanonPath & path) override + { + auto rel = key(path); + DirEntries entries; + if (auto z = resolveZone(rel)) + entries = zoneAccessor(*z)->readDirectory(zoneRelPath(*z)); + else if (!isZoneAncestor(rel)) + throw Error("worldtree: '%s' is not a directory in the immutable view", showPath(path)); + // Nested zone roots surface even when the enclosing projection omits them. + for (auto & name : zoneChildNames(rel)) + entries.emplace(name, tDirectory); + return entries; + } + + void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override + { + auto z = resolveZone(key(path)); + if (!z) + throw Error("worldtree: '%s' is not a readable file in the immutable view", showPath(path)); + zoneAccessor(*z)->readFile(zoneRelPath(*z), sink, std::move(sizeCallback)); + } + + std::string readLink(const CanonPath & path) override + { + auto z = resolveZone(key(path)); + if (!z) + throw Error("worldtree: '%s' is not a symlink in the immutable view", showPath(path)); + return zoneAccessor(*z)->readLink(zoneRelPath(*z)); + } + + std::optional getPhysicalPath(const CanonPath & path) override + { + if (auto z = resolveZone(key(path))) + return zoneAccessor(*z)->getPhysicalPath(zoneRelPath(*z)); + return std::nullopt; + } + + /** Git blob fingerprint hashed from content; a blob oid is a pure + * function of the bytes, so this reproduces what libgit2 reports. */ + std::string hashBlobFingerprint(const InZone & z, const Stat & st) + { + auto fs = zoneAccessor(z); + auto p = zoneRelPath(z); + HashSink sink(HashAlgorithm::SHA1); + // Git object framing per the object spec: "blob \0". + auto writeBlobPrefix = [&](uint64_t size) { + auto prefix = "blob " + std::to_string(size); + prefix.push_back('\0'); + sink(prefix); + }; + uint32_t mode; + if (st.type == tSymlink) { + auto target = fs->readLink(p); + writeBlobPrefix(target.size()); + sink(target); + mode = 0120000; + } else { + fs->readFile(p, sink, [&](uint64_t size) { writeBlobPrefix(size); }); + mode = st.isExecutable ? 0100755 : 0100644; + } + return gitFingerprintWithMode(sink.finish().hash, mode); + } + + std::string blobFingerprint(const InZone & z, const Stat & st) + { + auto memoKey = z.zoneId + "/" + z.rel; + { + std::string cached; + if (blobFingerprintMemo.visit(memoKey, [&](const auto & entry) { cached = entry.second; })) + return cached; + } + + // Regular files may carry the blob oid as an xattr. Symlinks never do: + // getxattr() would follow the link and answer for its target. + std::string fingerprint; + if (st.type == tRegular) { + auto physical = revisionRoot / z.zoneId; + if (!z.rel.empty()) + physical /= z.rel; + if (auto oid = tecnixReadWorldtreeOidXattr(physical, TECNIX_WORLDTREE_BLOB_OID_XATTR)) + fingerprint = gitFingerprintWithMode(*oid, st.isExecutable ? 0100755 : 0100644); + } + if (fingerprint.empty()) + fingerprint = hashBlobFingerprint(z, st); + + blobFingerprintMemo.insert_or_assign(memoKey, fingerprint); + return fingerprint; + } + + std::pair> getFingerprint(const CanonPath & path) override + { + auto rel = key(path); + if (auto z = resolveZone(rel)) { + auto st = zoneAccessor(*z)->maybeLstat(zoneRelPath(*z)); + if (!st) + return {path, "absent"}; + if (st->type == tDirectory) { + auto physical = revisionRoot / z->zoneId; + if (!z->rel.empty()) + physical /= z->rel; + return {path, gitFingerprintWithMode(tecnixReadWorldtreeTreeOid(physical), 0040000)}; + } + return {path, blobFingerprint(*z, *st)}; + } + + if (!isZoneAncestor(rel)) + return {path, "absent"}; + + // Synthesized directories (the root and zone ancestors) correspond to + // no single git object; compose identity from their children. + HashSink sink(HashAlgorithm::SHA256); + for (auto & name : zoneChildNames(rel)) { + auto [childPath, childFp] = getFingerprint(path / name); + sink << name << childFp.value_or(""); + } + return {path, "worldtree-union:" + sink.finish().hash.to_string(HashFormat::Base16, false)}; + } + + std::optional getLastModified() override + { + return std::nullopt; + } +}; + +ref getTecnixRepoAccessor(EvalState & state) +{ + std::call_once(tecnixData(state)->tecnixRepoAccessorFlag, [&state]() { + auto & sha = requireTectonixGitSha(state); + auto commitHash = Hash::parseNonSRIUnprefixed(sha, HashAlgorithm::SHA1); + + auto [cleanAccessor, gitFingerprints] = + [&]() -> std::pair, std::optional> { + if (!state.settings.tectonixWorldtreeSocket.get().empty()) { + // Clean base: the immutable FUSE projection at the pinned rev, + // mapped through the *committed* manifest (never the checkout copy). + auto revisionRoot = tecnixWorldtreeRevisionRoot(state.settings); + auto manifestPath = revisionRoot / "W-000000" / "manifest.json"; + std::error_code ec; + if (!std::filesystem::is_regular_file(manifestPath, ec)) + tecnixThrowHistoricalWorldManifestError(state.settings, ec ? ec.message() : "file does not exist"); + nlohmann::json manifest; + try { + manifest = nlohmann::json::parse(readFile(manifestPath)); + } catch (const nlohmann::json::parse_error & e) { + tecnixThrowHistoricalWorldManifestError(state.settings, e.what()); + } + debug("created Tecnix repo-wide worldtree FUSE accessor at commit %s", sha); + return {make_ref(revisionRoot, manifest), std::nullopt}; + } + + auto repo = getWorldRepo(state); + auto rootTreeSha = repo->getCommitTree(commitHash); + GitAccessorOptions opts{.exportIgnore = false, .smudgeLfs = false}; + debug("created Tecnix repo-wide libgit2 accessor at commit %s", sha); + return { + repo->getAccessor(rootTreeSha, opts, "repo"), + TecnixSourceAccessor::GitCleanFingerprints{repo, rootTreeSha}}; + }(); + + if (isTectonixSourceAvailable(state)) { + auto checkoutPath = state.settings.tectonixCheckoutPath.get(); + + // Get all dirty files in the repo. This is load-bearing for source + // closure validity: if we cannot determine the dirty overlay, do + // not continue with a clean-tree accessor (gitStatusDirtyPaths + // throws). Until the daemon exposes a full repo dirty-path RPC, + // materialized checkouts keep using git status at this boundary. + TecnixSourceAccessor::DirtyPathSet dirtyFiles; + for (auto & path : gitStatusDirtyPaths(checkoutPath)) + dirtyFiles.insert(std::move(path)); + + tecnixData(state)->tecnixRepoAccessor = make_ref( + cleanAccessor, makeFSSourceAccessor(checkoutPath), gitFingerprints, "", std::move(dirtyFiles)); + } else { + tecnixData(state)->tecnixRepoAccessor = make_ref( + cleanAccessor, cleanAccessor, gitFingerprints, "", TecnixSourceAccessor::DirtyPathSet{}); + } + + debug("created Tecnix repo-wide accessor"); + }); + return *tecnixData(state)->tecnixRepoAccessor; +} + +StorePath mountTecnixRepoAccessor(EvalState & state) +{ + std::call_once(tecnixData(state)->tecnixRepoMountFlag, [&state]() { + auto accessor = getTecnixRepoAccessor(state); + auto storePath = StorePath::random("world-repo"); + state.storeFS->mount(CanonPath(state.store->printStorePath(storePath)), accessor); + state.allowPath(storePath); + tecnixData(state)->tecnixRepoMountStorePath = storePath; + debug("mounted Tecnix repo accessor at %s", state.store->printStorePath(storePath)); + }); + return *tecnixData(state)->tecnixRepoMountStorePath; +} + +std::string resolveCheckoutHeadRev(const std::string & checkoutPath) +{ + return GitRepo::openRepo(checkoutPath, {})->resolveRef("HEAD").gitRev(); +} + +std::string getTecnixRepoPath(EvalState & state, std::string_view repoRelPath) +{ + auto rootStorePath = mountTecnixRepoAccessor(state); + auto path = tecnixNormalizeRepoPath(repoRelPath); + auto result = state.store->printStorePath(rootStorePath) + "/" + path; + debug("getTecnixRepoPath: '%s' -> '%s'", repoRelPath, result); + return result; +} + +} // namespace nix diff --git a/src/libexpr/tecnix/source-accessors.cc b/src/libexpr/tecnix/source-accessors.cc new file mode 100644 index 0000000000..a1e54ea93a --- /dev/null +++ b/src/libexpr/tecnix/source-accessors.cc @@ -0,0 +1,1005 @@ +#include "nix/expr/eval.hh" +#include "nix/expr/tecnix/source-accessors.hh" +#include "tecnix/eval-data.hh" +#include "nix/fetchers/fetch-to-store.hh" +#include "nix/fetchers/git-utils.hh" +#include "nix/store/store-api.hh" +#include "nix/util/current-process.hh" +#include "nix/util/environment-variables.hh" +#include "nix/util/hash.hh" +#include "nix/util/processes.hh" +#include "nix/util/source-accessor.hh" +#include "nix/util/strings-inline.hh" +#include "nix/util/util.hh" +#include "nix/util/worldtree-client.hh" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +namespace nix { + +static EvalState::TecnixEvalData * accessorData(EvalState & state) +{ + return &state.tecnixEvalData(); +} + +static const EvalState::TecnixEvalData * accessorData(const EvalState & state) +{ + return &state.tecnixEvalData(); +} + +ref getWorldRepo(const EvalState & state) +{ + std::call_once(accessorData(state)->worldRepoFlag, [&state]() { + auto gitDir = state.settings.tectonixGitDir.get(); + if (gitDir.empty()) + throw Error("--tectonix-git-dir must be specified to use tectonix builtins"); + + // Expand ~ to home directory + if (hasPrefix(gitDir, "~/")) + gitDir = getHome() + gitDir.substr(1); + + accessorData(state)->worldRepo = GitRepo::openRepo(std::filesystem::path(gitDir), {.bare = true}); + debug("opened world repo at %s", gitDir); + }); + return *accessorData(state)->worldRepo; +} + +const std::string & requireTectonixGitSha(const EvalState & state) +{ + auto & sha = state.settings.tectonixGitSha.get(); + if (sha.empty()) + throw Error("--tectonix-git-sha must be specified to use tectonix builtins"); + return sha; +} + +ref getWorldGitAccessor(const EvalState & state) +{ + std::call_once(accessorData(state)->worldGitAccessorFlag, [&state]() { + auto & sha = requireTectonixGitSha(state); + + auto repo = getWorldRepo(state); + auto hash = Hash::parseNonSRIUnprefixed(sha, HashAlgorithm::SHA1); + + if (!repo->hasObject(hash)) + throw Error("tectonix-git-sha '%s' not found in repository", sha); + + // Validate that the SHA is a commit by trying to get its tree. + // This gives a clear error if someone accidentally passes a tree or blob SHA. + try { + repo->getCommitTree(hash); + } catch (Error & e) { + throw Error("tectonix-git-sha '%s' does not appear to be a valid commit: %s", sha, e.what()); + } + + // exportIgnore=false: The world accessor is used for path validation and tree SHA + // computation, where we need to see all files. Repo/zone accessors used for + // actual content use exportIgnore=true to honor .gitattributes. + GitAccessorOptions opts{.exportIgnore = false, .smudgeLfs = false}; + accessorData(state)->worldGitAccessor = repo->getAccessor(hash, opts, "world"); + debug("created world accessor at commit %s", sha); + }); + return *accessorData(state)->worldGitAccessor; +} + +bool isTectonixSourceAvailable(const EvalState & state) +{ + return !state.settings.tectonixCheckoutPath.get().empty(); +} + +// Helper to normalize paths: strip leading // prefix +// Paths in manifest have // prefix (e.g., //areas/tools/dev) +// Filesystem operations need paths without // (e.g., areas/tools/dev) +static std::string normalizePath(std::string_view path) +{ + std::string result(path); + if (hasPrefix(result, "//")) + result = result.substr(2); + return result; +} + +static std::string normalizeZonePath(std::string_view zonePath) +{ + return normalizePath(zonePath); +} + +static GitAccessorOptions +makeZoneAccessorOptions(ref repo, const Hash & commitHash, const std::string & zonePath) +{ + std::string attrFp; + for (auto & h : repo->getGitAttributesAlongPath(commitHash, zonePath)) + attrFp += h.gitRev(); + return { + .exportIgnore = true, + .smudgeLfs = true, + .attrCommitRev = commitHash, + .attrPathPrefix = zonePath, + .attrFingerprint = std::move(attrFp), + }; +} + +// Helper to sanitize zone path for use in store path names. +// Store paths only allow: a-zA-Z0-9 and +-._?= +// Replaces / with - and any other invalid chars with _ +static std::string sanitizeZoneNameForStore(std::string_view zonePath) +{ + auto zone = normalizeZonePath(zonePath); + std::string result; + result.reserve(zone.size()); + for (char c : zone) { + if (c == '/') { + result += '-'; + } else if ( + (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '+' || c == '-' + || c == '.' || c == '_' || c == '?' || c == '=') { + result += c; + } else { + result += '_'; + } + } + return result; +} + +// ============================================================================ +// Worldtree integration (legacy tectonix builtins) +// +// When `tectonix-worldtree-socket` is set, mutable-checkout metadata moves off the +// libgit2 repo+checkout walk and onto O(changes) RPCs against the bound workspace: +// * getTectonixDirtyZones() -> `dirty_zones` — the tracked-dirty set; +// * getWorldTreeSha() -> `zone_tree_shas` — the working-tree subtree oid +// (the committed oid when clean, the synthesized frontier oid when dirty; +// the zone's build-cache key); +// * getLegacyTectonixZoneStorePath() reads source bytes from FUSE in both regimes: +// the materialized checkout for the mutable workspace and +// `/tecnix//` for immutable history. +// Fail-loud contract: when the socket is SET, the daemon is the sole source of truth — a +// worldtree sandbox has no git repo to fall back to. A daemon that is unreachable, or that +// refuses/errors a request, is a hard failure (the error propagates), never a silent +// downgrade. libgit2 / the checkout walk are reached ONLY when the socket is UNSET (plain +// local, non-worldtree eval). Historical reads create no daemon connection. +// ============================================================================ + +/** Reinterpret a 20-byte worldtree object id as a Nix SHA-1 Hash. */ +static Hash oidToHash(const worldtree::Oid & oid) +{ + Hash h(HashAlgorithm::SHA1); + assert(h.hashSize == oid.size()); + std::memcpy(h.hash, oid.data(), oid.size()); + return h; +} + +/** + * The scoped socket is only the mutable root-checkout control plane. Historical + * committed source is ordinary filesystem input beneath + * `/tecnix//`. + */ +struct WorldtreeConn +{ + uint64_t ws; + std::mutex mutex; + worldtree::Client client; + + WorldtreeConn(worldtree::Client && client, uint64_t ws) + : ws(ws) + , client(std::move(client)) + { + } + + /** The dirty set with each zone's changed files (for full ZoneDirtyInfo). */ + std::vector dirtyZoneEntries() + { + std::lock_guard lock(mutex); + return client.dirtyZoneEntries(ws); + } + + /** One zone's working-tree subtree oid, or nullopt when absent or out of scope. */ + std::optional zoneTreeSha(std::string_view worldPath) + { + std::string wp = hasPrefix(worldPath, "//") ? std::string(worldPath) : "//" + std::string(worldPath); + std::lock_guard lock(mutex); + auto resp = client.zoneTreeShas(ws, {wp}); + if (resp.empty() || !resp.front().treeSha) + return std::nullopt; + return oidToHash(*resp.front().treeSha); + } +}; + +static constexpr std::string_view WORLDTREE_TREE_OID_XATTR = "user.worldtree.tree-oid"; + +static std::filesystem::path worldtreeRevisionRoot(const EvalSettings & settings) +{ + auto revision = Hash::parseNonSRIUnprefixed(settings.tectonixGitSha.get(), HashAlgorithm::SHA1); + return std::filesystem::path(settings.tectonixWorldtreeMount.get()) / "tecnix" / revision.gitRev(); +} + +static std::string requireWorldtreeZoneId(const std::string & id) +{ + auto isLowerHex = [](char c) { return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'); }; + if (id.size() != 8 || !id.starts_with("W-") || !std::ranges::all_of(std::string_view(id).substr(2), isLowerHex)) + throw Error("worldtree: invalid zone id '%s' in manifest", id); + return id; +} + +/** Find the manifest zone containing worldPath and return (zone id, path within zone). */ +static std::pair +worldtreeZoneLocation(const nlohmann::json & manifest, std::string_view worldPath) +{ + auto clean = normalizeZonePath(worldPath); + while (!clean.empty() && clean.front() == '/') + clean.erase(clean.begin()); + while (!clean.empty() && clean.back() == '/') + clean.pop_back(); + for (auto & component : tokenizeString>(clean, "/")) + if (component.empty() || component == "." || component == "..") + throw Error("invalid world path '%s'", worldPath); + + const nlohmann::json * best = nullptr; + std::string bestPath; + for (auto & [candidateWorldPath, value] : manifest.items()) { + auto candidate = normalizeZonePath(candidateWorldPath); + bool contains = + clean == candidate + || (clean.size() > candidate.size() && hasPrefix(clean, candidate) && clean[candidate.size()] == '/'); + if (contains && candidate.size() > bestPath.size()) { + best = &value; + bestPath = std::move(candidate); + } + } + if (!best || !best->is_object() || !best->contains("id") || !best->at("id").is_string()) + throw Error("worldtree: path '%s' is not contained by a visible World zone", worldPath); + + auto id = requireWorldtreeZoneId(best->at("id").get()); + auto relative = clean.size() == bestPath.size() ? std::string() : clean.substr(bestPath.size() + 1); + return {std::move(id), std::move(relative)}; +} + +static Hash readWorldtreeTreeOid(const std::filesystem::path & path) +{ + std::array value{}; +#ifdef __APPLE__ + auto size = ::getxattr(path.c_str(), WORLDTREE_TREE_OID_XATTR.data(), value.data(), value.size(), 0, 0); +#else + auto size = ::getxattr(path.c_str(), WORLDTREE_TREE_OID_XATTR.data(), value.data(), value.size()); +#endif + if (size < 0) + throw Error("worldtree: cannot read tree identity for '%s': %s", path.string(), std::strerror(errno)); + if (size != static_cast(value.size())) + throw Error("worldtree: invalid tree identity on '%s'", path.string()); + return Hash::parseNonSRIUnprefixed(std::string(value.data(), value.size()), HashAlgorithm::SHA1); +} + +[[noreturn]] static void throwHistoricalWorldManifestError(const EvalSettings & settings, std::string_view detail) +{ + throw Error( + "worldtree: historical World manifest '.meta/manifest.json' for commit '%s' is missing or malformed: %s", + settings.tectonixGitSha.get(), + detail); +} + +static std::shared_ptr connectWorldtree(const EvalState & state) +{ + auto socketPath = state.settings.tectonixWorldtreeSocket.get(); + // The socket being unset is the *only* non-worldtree signal — plain local eval, where + // libgit2 is the source. Returning nullptr here routes callers to that path. + if (socketPath.empty()) + return nullptr; + // Fail-loud (the load-bearing invariant): with the socket SET there is no git repo to + // fall back to, so an unreachable daemon is a hard error — let `Client::connect`'s + // `ProtocolError` propagate rather than silently degrading to libgit2 (which would read + // the wrong content, or none). + auto ws = state.settings.tectonixWorldtreeWorkspace.get(); + return std::make_shared(worldtree::Client::connect(socketPath), ws); +} + +static std::shared_ptr worldtreeControlConn(const EvalState & state) +{ + // Historical --ref evaluations are filesystem-only. Returning null here is not a + // libgit2 fallback: their callers branch on worldtree mode before consulting this + // control seam. + if (!isTectonixSourceAvailable(state)) + return nullptr; + std::call_once(accessorData(state)->worldtreeControlConnFlag, [&state]() { + accessorData(state)->worldtreeControlConn_ = connectWorldtree(state); + }); + return accessorData(state)->worldtreeControlConn_; +} + +Hash getWorldTreeSha(const EvalState & state, std::string_view worldPath) +{ + // The mutable root checkout needs its synthesized working-tree oid from the control + // plane. A historical evaluation is already pinned by its FUSE path and reads the + // exact committed tree oid from that directory's synthetic xattr instead. + if (isTectonixSourceAvailable(state)) { + if (auto control = worldtreeControlConn(state)) { + if (auto sha = control->zoneTreeSha(worldPath)) + return *sha; + throw Error("worldtree: world path '%s' is absent or outside this workspace's visibility scope", worldPath); + } + } else if (!state.settings.tectonixWorldtreeSocket.get().empty()) { + auto revision = worldtreeRevisionRoot(state.settings); + if (normalizeZonePath(worldPath).empty()) + return readWorldtreeTreeOid(revision); + auto [zoneId, relative] = worldtreeZoneLocation(getManifestJson(state), worldPath); + auto path = revision / zoneId; + if (!relative.empty()) + path /= relative; + return readWorldtreeTreeOid(path); + } + + auto path = normalizePath(worldPath); + + // Check cache first + if (auto cached = getConcurrent(*accessorData(state)->worldTreeShaCache, path)) { + debug("getWorldTreeSha cache hit for '%s'", path); + return *cached; + } + + // Compute by walking from root + auto repo = getWorldRepo(state); + auto & sha = requireTectonixGitSha(state); + auto commitSha = Hash::parseNonSRIUnprefixed(sha, HashAlgorithm::SHA1); + + // Get the root tree SHA from the commit + auto rootTreeSha = repo->getCommitTree(commitSha); + + // Walk path components, caching intermediate results + Hash currentSha = rootTreeSha; + std::string currentPath; + + // Reuse cached accessor for path validation + auto accessor = getWorldGitAccessor(state); + + for (auto & component : tokenizeString>(path, "/")) { + if (component.empty()) + continue; + if (component == ".." || component == ".") + throw Error("invalid path component '%s' in world path '%s'", component, worldPath); + + std::string nextPath = currentPath.empty() ? component : currentPath + "/" + component; + + // Check if this level is cached + if (auto cached = getConcurrent(*accessorData(state)->worldTreeShaCache, nextPath)) { + currentSha = *cached; + currentPath = nextPath; + continue; + } + + // Need to compute: get tree entry for this component + auto fullPath = CanonPath("/" + nextPath); + auto stat = accessor->maybeLstat(fullPath); + + if (!stat || stat->type != SourceAccessor::Type::tDirectory) + throw Error("path '%s' does not exist or is not a directory in world", nextPath); + + // Get the tree SHA for this subtree + currentSha = repo->getSubtreeSha(currentSha, component); + + // Cache this level. Note: concurrent threads may compute and insert the same + // path simultaneously. This is benign because they will compute the same SHA + // (deterministic from git tree), so either insertion succeeds or finds an + // equivalent value. We use try_emplace which is atomic for concurrent_flat_map. + accessorData(state)->worldTreeShaCache->try_emplace(nextPath, currentSha); + currentPath = nextPath; + } + + debug("getWorldTreeSha computed '%s' -> %s", path, currentSha.gitRev()); + return currentSha; +} + +const std::set & getTectonixSparseCheckoutRoots(const EvalState & state) +{ + std::call_once(accessorData(state)->tectonixSparseCheckoutRootsFlag, [&state]() { + if (isTectonixSourceAvailable(state)) { + auto checkoutPath = state.settings.tectonixCheckoutPath.get(); + + // Read .git to find the actual git directory. It can be either + // a directory or a file containing "gitdir: ". + auto dotGitPath = std::filesystem::path(checkoutPath) / ".git"; + std::filesystem::path gitDir; + + if (std::filesystem::is_directory(dotGitPath)) { + gitDir = dotGitPath; + } else if (std::filesystem::is_regular_file(dotGitPath)) { + auto gitdirContent = readFile(dotGitPath.string()); + if (hasPrefix(gitdirContent, "gitdir: ")) { + auto path = trim(gitdirContent.substr(8)); + gitDir = std::filesystem::path(path); + if (gitDir.is_relative()) + gitDir = std::filesystem::path(checkoutPath) / gitDir; + } + } + + if (!gitDir.empty()) { + auto sparseRootsPath = gitDir / "info" / "sparse-checkout-roots"; + if (std::filesystem::exists(sparseRootsPath)) { + auto content = readFile(sparseRootsPath.string()); + for (auto & line : tokenizeString>(content, "\n")) { + auto trimmed = trim(line); + if (!trimmed.empty()) + accessorData(state)->tectonixSparseCheckoutRoots.insert(std::string(trimmed)); + } + } + } + } + }); + return accessorData(state)->tectonixSparseCheckoutRoots; +} + +const std::map & getTectonixDirtyZones(const EvalState & state) +{ + std::call_once(accessorData(state)->tectonixDirtyZonesFlag, [&state]() { + auto & dirtyZones = accessorData(state)->tectonixDirtyZones; + + // A historical FUSE view is immutable by construction. Preserve the usual full + // manifest-shaped result (every visible zone present and clean) without opening a + // control connection or manufacturing an ephemeral daemon workspace. + if (!isTectonixSourceAvailable(state) && !state.settings.tectonixWorldtreeSocket.get().empty()) { + for (auto & [zonePath, value] : getManifestJson(state).items()) + if (value.is_object() && value.contains("id") && value.at("id").is_string()) + dirtyZones[zonePath] = {}; + return; + } + + // Worldtree mode: the daemon is authoritative for the tracked-dirty set (design + // §5.1), derived from the materialization frontier in O(changes) — no + // O(working-tree) `git status` scan. Reconstruct a *full* ZoneDirtyInfo so every + // consumer (notably the `__unsafeTectonixInternalDirtyZones` primop) sees every + // manifest zone with an accurate flag, matching the libgit2 path's shape: + // (1) init every manifest zone clean — the immutable FUSE view supplies it in + // historical mode and the checkout supplies it in mutable mode (see + // getManifestContent); either way it enumerates the full visible zone set; + // (2) overlay the daemon's per-zone dirty files for the bound mutable workspace. + if (auto control = worldtreeControlConn(state)) { + const nlohmann::json * manifest; + try { + manifest = &getManifestJson(state); + } catch (nlohmann::json::parse_error & e) { + warn("failed to parse manifest for dirty zone detection: %s", e.what()); + return; + } catch (Error &) { + // Manifest unavailable (e.g. daemon refused) — fail loud rather than report + // a misleading empty/partial dirty set. + throw; + } + for (auto & [zonePath, value] : manifest->items()) + if (value.is_object() && value.contains("id") && value.at("id").is_string()) + dirtyZones[zonePath] = {}; + for (auto & entry : control->dirtyZoneEntries()) { + // A dirty zone the daemon names but the manifest omits still surfaces (a + // zone added in this workspace) — insert-or-update keeps the two sources + // unioned. + auto & info = dirtyZones[entry.zone]; + info.dirty = true; + for (auto & f : entry.files) + info.dirtyFiles.insert(f); + } + return; + } + + if (!isTectonixSourceAvailable(state)) + return; + + // Get sparse checkout roots (zone IDs) + auto & sparseRoots = getTectonixSparseCheckoutRoots(state); + if (sparseRoots.empty()) + return; + + // Get manifest (uses cached parsed JSON) + const nlohmann::json * manifest; + try { + manifest = &getManifestJson(state); + } catch (nlohmann::json::parse_error & e) { + warn("failed to parse manifest for dirty zone detection: %s", e.what()); + return; + } catch (Error &) { + // Manifest file not available (e.g., not in world repo) + return; + } + + // Build map of zone ID -> zone path for sparse roots only + std::map zoneIdToPath; + for (auto & [path, value] : manifest->items()) { + if (!value.contains("id") || !value.at("id").is_string()) { + warn("zone '%s' in manifest has missing or non-string 'id' field", path); + continue; + } + auto & id = value.at("id").get_ref(); + if (sparseRoots.count(id)) + zoneIdToPath[id] = path; + } + + // Initialize all sparse-checked-out zones as not dirty + for (auto & [zoneId, zonePath] : zoneIdToPath) { + dirtyZones[zonePath] = {}; + } + + // Create git command environment with environment variables + // GIT_DIR/GIT_WORK_TREE/GIT_COMMON_DIR removed since they affect + // git repository discovery + StringMap gitEnvironment = getEnv(); + gitEnvironment.erase("GIT_DIR"); + gitEnvironment.erase("GIT_WORK_TREE"); + gitEnvironment.erase("GIT_COMMON_DIR"); + + // Get dirty files via git status with -z for NUL-separated output + // This handles filenames with special characters correctly + auto checkoutPath = state.settings.tectonixCheckoutPath.get(); + auto [gitStatusCode, gitStatusOutput] = runProgram( + {.program = "git", + .args = {"-C", checkoutPath, "status", "--porcelain", "-z"}, + .environment = gitEnvironment}); + if (!statusOk(gitStatusCode)) { + // If git status fails, treat all zones as clean (fallback) + // This ensures call_once completes and we don't retry with partial state + warn( + "failed to get git status for dirty zone detection in '%s': program 'git' %s; treating all zones as clean", + checkoutPath, + statusToString(gitStatusCode)); + return; + } + + // Parse NUL-separated output + // Format with -z: XY SP path NUL [orig-path NUL for renames/copies] + size_t pos = 0; + while (pos < gitStatusOutput.size()) { + // Find the next NUL + auto nulPos = gitStatusOutput.find('\0', pos); + if (nulPos == std::string::npos) + break; + + auto entry = gitStatusOutput.substr(pos, nulPos - pos); + pos = nulPos + 1; + + // Git porcelain format: "XY PATH" where XY is 2-char status, then space, then path + // Minimum valid entry is "X P" (4 chars): status + space + 1-char path + if (entry.size() < 4) + continue; + + // XY is first 2 chars, then space, then path + char xy0 = entry[0]; + std::string rawPath = entry.substr(3); + + // Collect paths to check - destination path is always included + std::vector pathsToCheck; + pathsToCheck.push_back("/" + rawPath); + + // For renames (R) and copies (C), also process the original path + // Both source and destination zones should be marked dirty + if (xy0 == 'R' || xy0 == 'C') { + auto nextNul = gitStatusOutput.find('\0', pos); + if (nextNul != std::string::npos) { + auto origPath = gitStatusOutput.substr(pos, nextNul - pos); + pathsToCheck.push_back("/" + origPath); + pos = nextNul + 1; + } + } + + for (const auto & filePath : pathsToCheck) { + for (auto & [zonePath, info] : dirtyZones) { + auto normalized = "/" + normalizeZonePath(zonePath); + if (hasPrefix(filePath, normalized + "/") || filePath == normalized) { + info.dirty = true; + info.dirtyFiles.insert(filePath.substr(1)); + break; + } + } + } + } + + size_t dirtyCount = 0; + for (const auto & [_, info] : dirtyZones) + if (info.dirty) + dirtyCount++; + debug("computed dirty zones: %d of %d zones are dirty", dirtyCount, dirtyZones.size()); + }); + return accessorData(state)->tectonixDirtyZones; +} + +// Path to the tectonix manifest file within the world repository +static constexpr std::string_view TECTONIX_MANIFEST_PATH = "/.meta/manifest.json"; + +const std::string & getManifestContent(const EvalState & state) +{ + // Cached for the lifetime of evaluation. This is intentional: evaluation is + // bound to a specific git SHA (tectonix-git-sha), so the manifest content is + // immutable for this EvalState instance. + std::call_once(accessorData(state)->tectonixManifestFlag, [&state]() { + auto fullPath = CanonPath(TECTONIX_MANIFEST_PATH); + + // Mode A (`tec `, materialized checkout): the working tree is the source of + // truth and may carry uncommitted manifest edits (a zone added/removed in this + // sandbox), so read the local file — never a stale committed copy. + if (isTectonixSourceAvailable(state)) { + auto manifestPath = + std::filesystem::path(state.settings.tectonixCheckoutPath.get()) / ".meta" / "manifest.json"; + if (std::filesystem::exists(manifestPath)) { + accessorData(state)->tectonixManifestContent = readFile(manifestPath); + debug("loaded manifest from checkout: %s", manifestPath.string()); + return; + } + } + + // Mode B (`tec --ref`, no checkout): manifest metadata is an ordinary immutable + // file in the FUSE projection. W-000000 is the reserved manifest pseudo-zone; it + // follows the same workspace visibility as the root checkout and needs no socket. + if (!state.settings.tectonixWorldtreeSocket.get().empty()) { + auto manifestPath = worldtreeRevisionRoot(state.settings) / "W-000000" / "manifest.json"; + std::error_code ec; + if (!std::filesystem::is_regular_file(manifestPath, ec)) + throwHistoricalWorldManifestError(state.settings, ec ? ec.message() : "file does not exist"); + try { + accessorData(state)->tectonixManifestContent = readFile(manifestPath); + } catch (const Error & e) { + throwHistoricalWorldManifestError(state.settings, e.what()); + } + debug("loaded manifest from immutable worldtree view: %s", manifestPath.string()); + return; + } + + // Socket unset (plain local eval): read the committed manifest via libgit2. + auto accessor = getWorldGitAccessor(state); + if (!accessor->pathExists(fullPath)) + throw Error("manifest.json does not exist at %s in world", TECTONIX_MANIFEST_PATH); + + accessorData(state)->tectonixManifestContent = accessor->readFile(fullPath); + debug("loaded manifest from git at %s", fullPath); + }); + return accessorData(state)->tectonixManifestContent; +} + +const nlohmann::json & getManifestJson(const EvalState & state) +{ + std::call_once(accessorData(state)->tectonixManifestJsonFlag, [&state]() { + try { + accessorData(state)->tectonixManifestJson = + std::make_unique(nlohmann::json::parse(getManifestContent(state))); + } catch (const nlohmann::json::parse_error & e) { + if (!state.settings.tectonixWorldtreeSocket.get().empty() && !isTectonixSourceAvailable(state)) + throwHistoricalWorldManifestError(state.settings, e.what()); + throw; + } + }); + return *accessorData(state)->tectonixManifestJson; +} + +static StorePath mountLegacyTectonixZoneByTreeSha(EvalState & state, const Hash & treeSha, std::string_view zonePath); +static StorePath worldtreeMountAccessor( + EvalState & state, const Hash & treeSha, std::string_view zonePath, ref accessor); +static StorePath getLegacyTectonixZoneFromCheckout( + EvalState & state, std::string_view zonePath, const boost::unordered_flat_set * dirtyFiles = nullptr); + +StorePath getLegacyTectonixZoneStorePath(EvalState & state, std::string_view zonePath) +{ + // A worldtree sandbox has two source regimes but only one filesystem accessor: + // the mutable root checkout path for ordinary evaluation, or the immutable + // commit/zone path for --ref. Only the former needs control RPCs for its dirty + // frontier identity. + if (!state.settings.tectonixWorldtreeSocket.get().empty()) { + if (isTectonixSourceAvailable(state)) { + auto control = worldtreeControlConn(state); + if (!control) + throw Error("worldtree: mutable checkout has no control connection"); + auto treeSha = control->zoneTreeSha(zonePath); + if (!treeSha) + throw Error("worldtree: zone '%s' is absent or outside this workspace's visibility scope", zonePath); + auto fullPath = + std::filesystem::path(state.settings.tectonixCheckoutPath.get()) / normalizeZonePath(zonePath); + if (!std::filesystem::is_directory(fullPath)) + throw Error("worldtree: zone '%s' is not materialized at '%s'", zonePath, fullPath.string()); + return worldtreeMountAccessor(state, *treeSha, zonePath, makeFSSourceAccessor(fullPath)); + } + + auto manifestIt = getManifestJson(state).find(std::string(zonePath)); + if (manifestIt == getManifestJson(state).end() || !manifestIt->is_object() || !manifestIt->contains("id") + || !manifestIt->at("id").is_string()) + throw Error("worldtree: zone '%s' is absent from the visible manifest", zonePath); + auto zoneId = requireWorldtreeZoneId(manifestIt->at("id").get()); + auto fullPath = worldtreeRevisionRoot(state.settings) / zoneId; + if (!std::filesystem::is_directory(fullPath)) + throw Error("worldtree: immutable zone '%s' is unavailable at '%s'", zonePath, fullPath.string()); + auto treeSha = readWorldtreeTreeOid(fullPath); + return worldtreeMountAccessor(state, treeSha, zonePath, makeFSSourceAccessor(fullPath)); + } + + // Check dirty status using original zonePath (with // prefix) since + // tectonixDirtyZones keys come directly from manifest with // prefix + const EvalState::ZoneDirtyInfo * dirtyInfo = nullptr; + if (isTectonixSourceAvailable(state)) { + auto & dirtyZones = getTectonixDirtyZones(state); + auto it = dirtyZones.find(std::string(zonePath)); + if (it != dirtyZones.end() && it->second.dirty) + dirtyInfo = &it->second; + } + + if (dirtyInfo) { + debug("getLegacyTectonixZoneStorePath: %s is dirty, using checkout", zonePath); + return getLegacyTectonixZoneFromCheckout(state, zonePath, &dirtyInfo->dirtyFiles); + } + + // Clean zone: get tree SHA + auto treeSha = getWorldTreeSha(state, zonePath); + + if (!state.settings.lazyTrees) { + debug("getLegacyTectonixZoneStorePath: %s clean, eager copy from git (tree %s)", zonePath, treeSha.gitRev()); + auto repo = getWorldRepo(state); + auto commitHash = Hash::parseNonSRIUnprefixed(requireTectonixGitSha(state), HashAlgorithm::SHA1); + auto opts = makeZoneAccessorOptions(repo, commitHash, normalizeZonePath(zonePath)); + auto accessor = repo->getAccessor(treeSha, opts, "zone"); + + std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); + auto storePath = fetchToStore( + state.fetchSettings, *state.store, SourcePath(accessor, CanonPath::root), FetchMode::Copy, name); + + state.allowPath(storePath); + return storePath; + } + + debug("getLegacyTectonixZoneStorePath: %s clean, lazy mount (tree %s)", zonePath, treeSha.gitRev()); + return mountLegacyTectonixZoneByTreeSha(state, treeSha, zonePath); +} + +static StorePath +worldtreeMountAccessor(EvalState & state, const Hash & treeSha, std::string_view zonePath, ref accessor) +{ + std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); + + if (!state.settings.lazyTrees) { + // Eager: copy the zone content into the store now (content-addressed by content). + auto storePath = fetchToStore( + state.fetchSettings, *state.store, SourcePath(accessor, CanonPath::root), FetchMode::Copy, name); + state.allowPath(storePath); + return storePath; + } + + // Lazy-trees: mount at a virtual store path, deduplicated by the daemon's working-tree + // oid so a zone evaluated twice in one EvalState mounts once (same shape as + // mountLegacyTectonixZoneByTreeSha — the two share tectonixZoneCache_'s tree-oid + // keyspace). + { + auto cache = accessorData(state)->tectonixZoneCache_.readLock(); + if (auto it = cache->find(treeSha); it != cache->end()) + return it->second; + } + + auto storePath = StorePath::random(name); + + auto cache = accessorData(state)->tectonixZoneCache_.lock(); + if (auto it = cache->find(treeSha); it != cache->end()) + return it->second; + + state.storeFS->mount(CanonPath(state.store->printStorePath(storePath)), accessor); + state.allowPath(storePath); + cache->emplace(treeSha, storePath); + + debug( + "worldtree: mounted zone %s (tree %s) at %s", + zonePath, + treeSha.gitRev(), + state.store->printStorePath(storePath)); + + return storePath; +} + +static StorePath mountLegacyTectonixZoneByTreeSha(EvalState & state, const Hash & treeSha, std::string_view zonePath) +{ + // Double-checked locking pattern for concurrent zone mounting: + // 1. Read lock check (fast path - allows concurrent readers) + { + auto cache = accessorData(state)->tectonixZoneCache_.readLock(); + auto it = cache->find(treeSha); + if (it != cache->end()) { + debug("zone cache hit for tree %s", treeSha.gitRev()); + return it->second; + } + } // Read lock released + + // 2. Write lock check (catch races between read unlock and write lock) + { + auto cache = accessorData(state)->tectonixZoneCache_.lock(); + auto it = cache->find(treeSha); + if (it != cache->end()) { + debug("zone cache hit for tree %s (after lock upgrade)", treeSha.gitRev()); + return it->second; + } + } // Write lock released - expensive work happens without holding lock + + // 3. Perform expensive git operations without holding lock. + // This allows concurrent mounts of different zones. Multiple threads may + // race to mount the same zone, but we check again before inserting. + auto repo = getWorldRepo(state); + auto commitHash = Hash::parseNonSRIUnprefixed(requireTectonixGitSha(state), HashAlgorithm::SHA1); + auto opts = makeZoneAccessorOptions(repo, commitHash, std::string(zonePath)); + auto accessor = repo->getAccessor(treeSha, opts, "zone"); + + // Generate name from zone path (sanitized for store path requirements) + std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); + + // Create virtual store path + auto storePath = StorePath::random(name); + + // 4. Re-acquire write lock and check again before mounting + auto cache = accessorData(state)->tectonixZoneCache_.lock(); + auto it = cache->find(treeSha); + if (it != cache->end()) { + // Another thread mounted while we were working - use their result + debug("zone cache hit for tree %s (after work)", treeSha.gitRev()); + return it->second; + } + + // Mount accessor at this path first, then allow the path. + // This order ensures we don't leave allowed paths without mounts on exception. + state.storeFS->mount(CanonPath(state.store->printStorePath(storePath)), accessor); + state.allowPath(storePath); + + // Insert into cache (we hold the lock, so this will succeed) + cache->emplace(treeSha, storePath); + + debug("mounted zone %s (tree %s) at %s", zonePath, treeSha.gitRev(), state.store->printStorePath(storePath)); + + return storePath; +} + +/** + * Overlays dirty files from disk on top of a clean git tree accessor. + * Serves only the legacy tectonix zone builtins; the tracked Tecnix + * evaluation path uses TecnixSourceAccessor above. + */ +struct DirtyOverlaySourceAccessor : SourceAccessor +{ + ref base, disk; + boost::unordered_flat_set dirtyFiles, dirtyDirs; + + DirtyOverlaySourceAccessor( + ref base, ref disk, boost::unordered_flat_set && dirtyFiles) + : base(base) + , disk(disk) + , dirtyFiles(std::move(dirtyFiles)) + { + for (auto & f : this->dirtyFiles) { + for (auto p = CanonPath(f); !p.isRoot();) { + p.pop(); + if (!dirtyDirs.insert(p.rel().empty() ? "" : std::string(p.rel())).second) + break; + } + } + } + + bool isDirty(const CanonPath & path) + { + return dirtyFiles.contains(std::string(path.rel())); + } + + std::optional maybeLstat(const CanonPath & path) override + { + if (path.isRoot()) + return base->maybeLstat(path); + if (isDirty(path)) + return disk->maybeLstat(path); + auto s = base->maybeLstat(path); + if (s || !dirtyDirs.contains(std::string(path.rel()))) + return s; + return disk->maybeLstat(path); + } + + void readFile(const CanonPath & path, Sink & sink, fun sizeCallback) override + { + return (isDirty(path) ? disk : base)->readFile(path, sink, sizeCallback); + } + + std::string readLink(const CanonPath & path) override + { + return (isDirty(path) ? disk : base)->readLink(path); + } + + std::optional getPhysicalPath(const CanonPath & path) override + { + return (isDirty(path) ? disk : base)->getPhysicalPath(path); + } + + DirEntries readDirectory(const CanonPath & path) override + { + auto rel = path.isRoot() ? "" : std::string(path.rel()); + if (!path.isRoot() && !dirtyDirs.contains(rel)) + return base->readDirectory(path); + + DirEntries entries; + try { + entries = base->readDirectory(path); + } catch (...) { + } + + auto prefix = rel.empty() ? "" : rel + "/"; + for (auto & f : dirtyFiles) { + if (!f.starts_with(prefix)) + continue; + auto rest = std::string_view(f).substr(prefix.size()); + if (rest.find('/') != std::string_view::npos) + continue; + auto stat = disk->maybeLstat(path / rest); + if (stat) + entries[std::string(rest)] = stat->type; + else + entries.erase(std::string(rest)); + } + for (auto & d : dirtyDirs) { + if (!d.starts_with(prefix)) + continue; + auto rest = std::string_view(d).substr(prefix.size()); + if (rest.find('/') != std::string_view::npos || rest.empty()) + continue; + if (!entries.count(std::string(rest))) + entries[std::string(rest)] = Type::tDirectory; + } + return entries; + } +}; + +static StorePath getLegacyTectonixZoneFromCheckout( + EvalState & state, std::string_view zonePath, const boost::unordered_flat_set * dirtyFiles) +{ + auto zone = normalizeZonePath(zonePath); + std::string name = "zone-" + sanitizeZoneNameForStore(zonePath); + auto checkoutPath = state.settings.tectonixCheckoutPath.get(); + auto fullPath = std::filesystem::path(checkoutPath) / zone; + + auto makeDirtyAccessor = [&]() -> ref { + auto repo = getWorldRepo(state); + auto commitHash = Hash::parseNonSRIUnprefixed(requireTectonixGitSha(state), HashAlgorithm::SHA1); + auto zoneOpts = makeZoneAccessorOptions(repo, commitHash, zone); + auto baseAccessor = repo->getAccessor(getWorldTreeSha(state, zone), zoneOpts, "zone"); + boost::unordered_flat_set zoneDirtyFiles; + if (dirtyFiles) { + auto zonePrefix = zone + "/"; + for (auto & f : *dirtyFiles) + if (f.starts_with(zonePrefix)) + zoneDirtyFiles.insert(f.substr(zonePrefix.size())); + } + return make_ref( + baseAccessor, makeFSSourceAccessor(fullPath), std::move(zoneDirtyFiles)); + }; + + if (!state.settings.lazyTrees) { + auto accessor = makeDirtyAccessor(); + auto storePath = fetchToStore( + state.fetchSettings, *state.store, SourcePath(accessor, CanonPath::root), FetchMode::Copy, name); + state.allowPath(storePath); + return storePath; + } + + { + auto cache = accessorData(state)->tectonixCheckoutZoneCache_.readLock(); + auto it = cache->find(std::string(zonePath)); + if (it != cache->end()) + return it->second; + } + + auto cache = accessorData(state)->tectonixCheckoutZoneCache_.lock(); + auto it = cache->find(std::string(zonePath)); + if (it != cache->end()) + return it->second; + + if (!std::filesystem::exists(fullPath)) + throw Error("zone '%s' not found in checkout at '%s'", zonePath, fullPath.string()); + + auto storePath = StorePath::random(name); + state.storeFS->mount(CanonPath(state.store->printStorePath(storePath)), makeDirtyAccessor()); + state.allowPath(storePath); + cache->emplace(std::string(zonePath), storePath); + return storePath; +} + +} // namespace nix diff --git a/src/libexpr/tecnix/source-deps.cc b/src/libexpr/tecnix/source-deps.cc new file mode 100644 index 0000000000..278b20f112 --- /dev/null +++ b/src/libexpr/tecnix/source-deps.cc @@ -0,0 +1,676 @@ +#include "nix/expr/eval.hh" +#include "nix/expr/tecnix/access-set-graph.hh" +#include "tecnix/eval-data.hh" +#include "nix/util/strings-inline.hh" +#include "nix/util/util.hh" + +#include +#include + +#include + +#ifndef MAP_ANONYMOUS +# define MAP_ANONYMOUS MAP_ANON +#endif + +namespace nix { + +[[gnu::tls_model("initial-exec")]] thread_local TecnixThreadState currentTecnixThreadState; + +/** + * The value-label directory: constant-initialized zeroed storage, so it is + * usable from the very first dynamic initializer without ordering concerns. + * Chunks are 1 GiB sparse mappings covering 4 GiB of address space each + * (one 32-bit slot per 16-byte-aligned value cell), installed on the first + * nonzero label store in their region and never freed. + */ +std::atomic tecnixValueLabelDir[tecnixValueLabelDirSize]; + +uint32_t * tecnixInstallValueLabelChunk(size_t dirIndex) +{ + static_assert(sizeof(void *) == 8, "the Tecnix value-label table requires a 64-bit address space"); + + size_t chunkBytes = (size_t{1} << 32) / 16 * sizeof(uint32_t); + int flags = MAP_PRIVATE | MAP_ANONYMOUS; +#ifdef MAP_NORESERVE + flags |= MAP_NORESERVE; +#endif + void * mem = mmap(nullptr, chunkBytes, PROT_READ | PROT_WRITE, flags, -1, 0); + if (mem == MAP_FAILED) { + fprintf(stderr, "nix: failed to map a Tecnix value-label chunk\n"); + abort(); + } + + auto * chunk = static_cast(mem); + uint32_t * expected = nullptr; + if (!tecnixValueLabelDir[dirIndex].compare_exchange_strong( + expected, chunk, std::memory_order_release, std::memory_order_acquire)) { + munmap(mem, chunkBytes); + return expected; + } + return chunk; +} + +void tecnixValueLabelOutOfRange(const void * value) +{ + fprintf(stderr, "nix: Tecnix value label store outside the covered address range: %p\n", value); + abort(); +} + +std::vector parseGitPorcelainZDirtyPaths(std::string_view output) +{ + std::vector paths; + size_t pos = 0; + while (pos < output.size()) { + auto nulPos = output.find('\0', pos); + if (nulPos == std::string_view::npos) + break; + + auto entry = output.substr(pos, nulPos - pos); + pos = nulPos + 1; + + // Git porcelain v1 -z format is "XY PATH\0", with an extra + // original-path record only when the X column is R/C. Keep both names + // dirty so source reads of either side see the checkout overlay. + if (entry.size() < 4 || entry[2] != ' ') + continue; + + paths.emplace_back(entry.substr(3)); + + if (entry[0] == 'R' || entry[0] == 'C') { + auto nextNul = output.find('\0', pos); + if (nextNul == std::string_view::npos) + break; + + auto originalPath = output.substr(pos, nextNul - pos); + pos = nextNul + 1; + if (!originalPath.empty()) + paths.emplace_back(originalPath); + } + } + return paths; +} + +static uint64_t hashSourceAccessIds(std::span items) +{ + uint64_t hash = 1469598103934665603ULL; + for (auto item : items) { + hash ^= item; + hash *= 1099511628211ULL; + } + return hash; +} + +EvalSourceAccessSetGraph::EvalSourceAccessSetGraph() = default; + +void EvalSourceAccessSetGraph::enable() +{ + if (enabled.load(std::memory_order_acquire)) + return; // enable is one-way; contexts re-enter here on every construction + + std::lock_guard lock(mutex); + if (enabled.load(std::memory_order_acquire)) + return; + + accesses.emplace_back(); + accessSets.push_back(EvalSourceAccessSetNode{}); + enabled.store(true, std::memory_order_release); +} + +EvalSourceAccessId EvalSourceAccessSetGraph::internAccess(std::string_view path) +{ + if (!enabled.load(std::memory_order_acquire)) + return emptyEvalSourceAccessId; + + EvalSourceAccessId id = emptyEvalSourceAccessId; + if (accessIds.cvisit(path, [&](const auto & kv) { id = kv.second; })) + return id; + + std::lock_guard lock(mutex); + if (accessIds.cvisit(path, [&](const auto & kv) { id = kv.second; })) + return id; + + id = static_cast(accesses.size()); + accesses.emplace_back(path); + accessIds.try_emplace_and_cvisit( + accesses.back(), id, [&](const auto & kv) { id = kv.second; }, [&](const auto & kv) { id = kv.second; }); + return id; +} + +bool EvalSourceAccessSetGraph::accessSetEquals( + EvalSourceAccessSetId id, const std::vector & items) const +{ + if (id == emptyEvalSourceAccessSetId || id >= accessSets.size()) + return items.empty(); + + auto node = accessSets[id]; + if (node.count != items.size()) + return false; + return std::equal( + items.begin(), + items.end(), + accessSetItems.begin() + node.first, + accessSetItems.begin() + node.first + node.count); +} + +EvalSourceAccessSetId EvalSourceAccessSetGraph::internAccessSet( + std::span directAccesses, std::span children) +{ + if (!enabled.load(std::memory_order_acquire)) + return emptyEvalSourceAccessSetId; + + size_t directCount = 0; + EvalSourceAccessId singleDirect = emptyEvalSourceAccessId; + for (auto access : directAccesses) { + if (access == emptyEvalSourceAccessId) + continue; + directCount++; + singleDirect = access; + } + + size_t childCount = 0; + EvalSourceAccessSetId singleChild = emptyEvalSourceAccessSetId; + EvalSourceAccessSetId pairChildA = emptyEvalSourceAccessSetId; + EvalSourceAccessSetId pairChildB = emptyEvalSourceAccessSetId; + for (auto child : children) { + if (child == emptyEvalSourceAccessSetId) + continue; + childCount++; + singleChild = child; + if (childCount == 1) + pairChildA = child; + else if (childCount == 2) + pairChildB = child; + } + + if (directCount == 0 && childCount == 0) + return emptyEvalSourceAccessSetId; + if (directCount == 0 && childCount == 1) + return singleChild; + + auto internSingleton = [&](EvalSourceAccessId access) { + if (access == emptyEvalSourceAccessId) + return emptyEvalSourceAccessSetId; + if (access < singletonAccessSets.size()) + if (auto existing = singletonAccessSets[access]; existing != emptyEvalSourceAccessSetId) + return existing; + + auto first = static_cast(accessSetItems.size()); + accessSetItems.push_back(access); + auto id = static_cast(accessSets.size()); + accessSets.push_back(EvalSourceAccessSetNode{.first = first, .count = 1}); + if (access >= singletonAccessSets.size()) + singletonAccessSets.resize(access + 1, emptyEvalSourceAccessSetId); + singletonAccessSets[access] = id; + return id; + }; + + std::lock_guard lock(mutex); + if (childCount == 0 && directCount == 1) + return internSingleton(singleDirect); + + bool hasPairUnionKey = false; + uint64_t pairUnionKey = 0; + if (directCount == 0 && childCount == 2) { + auto a = std::min(pairChildA, pairChildB); + auto b = std::max(pairChildA, pairChildB); + if (a == b) + return a; + pairUnionKey = (uint64_t{a} << 32) | uint64_t{b}; + if (auto existing = pairUnionAccessSets.find(pairUnionKey); existing != pairUnionAccessSets.end()) + return existing->second; + hasPairUnionKey = true; + } + + static thread_local std::vector items; + + auto buildItems = [&] { + size_t itemCount = directCount; + for (auto child : children) + if (child != emptyEvalSourceAccessSetId && child < accessSets.size()) + itemCount += accessSets[child].count; + + items.clear(); + items.reserve(itemCount); + + for (auto access : directAccesses) + if (access != emptyEvalSourceAccessId) + items.push_back(access); + for (auto child : children) { + if (child == emptyEvalSourceAccessSetId || child >= accessSets.size()) + continue; + auto node = accessSets[child]; + items.insert( + items.end(), accessSetItems.begin() + node.first, accessSetItems.begin() + node.first + node.count); + } + + std::sort(items.begin(), items.end()); + items.erase(std::unique(items.begin(), items.end()), items.end()); + return hashSourceAccessIds(items); + }; + + auto lookupAccessSet = [&](uint64_t hash) -> EvalSourceAccessSetId { + if (auto head = accessSetIdsByHash.find(hash); head != accessSetIdsByHash.end()) + for (auto id = head->second; id != emptyEvalSourceAccessSetId; id = accessSets[id].nextWithSameHash) + if (accessSetEquals(id, items)) + return id; + return emptyEvalSourceAccessSetId; + }; + + auto hash = buildItems(); + if (items.empty()) + return emptyEvalSourceAccessSetId; + if (items.size() == 1) + return internSingleton(items.front()); + if (auto existing = lookupAccessSet(hash); existing != emptyEvalSourceAccessSetId) { + if (hasPairUnionKey) + pairUnionAccessSets.emplace(pairUnionKey, existing); + return existing; + } + + auto first = static_cast(accessSetItems.size()); + auto count = static_cast(items.size()); + accessSetItems.insert(accessSetItems.end(), items.begin(), items.end()); + + auto id = static_cast(accessSets.size()); + auto next = emptyEvalSourceAccessSetId; + if (auto head = accessSetIdsByHash.find(hash); head != accessSetIdsByHash.end()) { + next = head->second; + head->second = id; + } else { + accessSetIdsByHash.emplace(hash, id); + } + accessSets.push_back( + EvalSourceAccessSetNode{ + .first = first, + .count = count, + .nextWithSameHash = next, + .hash = hash, + }); + if (hasPairUnionKey) + pairUnionAccessSets.emplace(pairUnionKey, id); + return id; +} + +EvalSourceAccessSetId EvalSourceAccessSetGraph::internAccessSet( + const std::vector & directAccesses, const std::vector & children) +{ + return internAccessSet( + std::span(directAccesses.data(), directAccesses.size()), + std::span(children.data(), children.size())); +} + +std::string EvalSourceAccessSetGraph::access(EvalSourceAccessId id) const +{ + if (!enabled.load(std::memory_order_acquire)) + return {}; + + std::lock_guard lock(mutex); + if (id == emptyEvalSourceAccessId || id >= accesses.size()) + return {}; + return accesses[id]; +} + +std::vector EvalSourceAccessSetGraph::flatten( + const std::vector & directAccesses, + const std::vector & accessSetEdges) const +{ + if (!enabled.load(std::memory_order_acquire)) + return {}; + + std::lock_guard lock(mutex); + + if (nextFlattenGeneration == 0) { + std::fill(seenAccessGenerations.begin(), seenAccessGenerations.end(), 0); + nextFlattenGeneration = 1; + } + auto generation = nextFlattenGeneration++; + + if (seenAccessGenerations.size() < accesses.size()) + seenAccessGenerations.resize(accesses.size(), 0); + + std::vector flattenedAccesses; + flattenedAccesses.reserve(directAccesses.size()); + + auto addAccess = [&](EvalSourceAccessId access) { + if (access == emptyEvalSourceAccessId || access >= seenAccessGenerations.size()) + return; + if (seenAccessGenerations[access] == generation) + return; + seenAccessGenerations[access] = generation; + flattenedAccesses.push_back(access); + }; + + for (auto access : directAccesses) + addAccess(access); + + for (auto accessSet : accessSetEdges) { + if (accessSet == emptyEvalSourceAccessSetId || accessSet >= accessSets.size()) + continue; + auto node = accessSets[accessSet]; + for (uint32_t i = 0; i < node.count; i++) + addAccess(accessSetItems[node.first + i]); + } + + std::vector result; + result.reserve(flattenedAccesses.size()); + for (auto accessId : flattenedAccesses) + result.push_back(accesses[accessId]); // addAccess only admits valid non-empty ids + return result; +} + +EvalSourceAccessSetStats EvalSourceAccessSetGraph::stats() const +{ + if (!enabled.load(std::memory_order_acquire)) + return EvalSourceAccessSetStats{}; + + std::lock_guard lock(mutex); + return EvalSourceAccessSetStats{ + .accesses = accesses.empty() ? 0 : accesses.size() - 1, + .accessSets = accessSets.empty() ? 0 : accessSets.size() - 1, + .accessSetItems = accessSetItems.size(), + }; +} + +/* Frames only suppress *consecutive* duplicates: `internAccessSet` sorts and + fully dedupes at publish time, so per-append deduplication would be + redundant work (and quadratic for large scope frames, e.g. the resolver + import scope). The consecutive check catches the common pattern of a loop + re-reading one path or re-forcing one value. */ +static void addFrameAccess(TrackedSourceDepsFrame & frame, EvalSourceAccessId access) +{ + if (access == emptyEvalSourceAccessId) + return; + + auto size = frame.directSourceAccessSetAccesses.size(); + if (size == 0 || frame.directSourceAccessSetAccesses.data()[size - 1] != access) + frame.directSourceAccessSetAccesses.push_back(access); +} + +static void addFrameChild(TrackedSourceDepsFrame & frame, EvalSourceAccessSetId child) +{ + if (child == emptyEvalSourceAccessSetId) + return; + + auto size = frame.childSourceAccessSets.size(); + if (size == 0 || frame.childSourceAccessSets.data()[size - 1] != child) + frame.childSourceAccessSets.push_back(child); +} + +static void addToParentFrame(TrackedSourceDepsFrame & parent, EvalSourceAccessSetId accessSet) +{ + if (accessSet == emptyEvalSourceAccessSetId) + return; + + addFrameChild(parent, accessSet); +} + +static void addToCurrentFrame(TrackingContext & trackingCtx, EvalSourceAccessSetId accessSet) +{ + if (accessSet == emptyEvalSourceAccessSetId) + return; + + if (auto * frame = currentTecnixThreadState.sourceDepsFrame) { + addFrameChild(*frame, accessSet); + return; + } + + addFrameChild(trackingCtx.rootFrame, accessSet); +} + +static bool frameHasSourceDeps(const TrackedSourceDepsFrame & frame) +{ + return !frame.directSourceAccessSetAccesses.empty() || !frame.childSourceAccessSets.empty(); +} + +static EvalSourceAccessSetId internFrameAccessSet(TrackedSourceDepsFrame & frame) +{ + if (!frameHasSourceDeps(frame)) + return emptyEvalSourceAccessSetId; + + return frame.trackingCtx.sourceAccessSetGraph->internAccessSet( + std::span( + frame.directSourceAccessSetAccesses.data(), frame.directSourceAccessSetAccesses.size()), + std::span(frame.childSourceAccessSets.data(), frame.childSourceAccessSets.size())); +} + +static void mergeFrameIntoParent(TrackedSourceDepsFrame & frame) +{ + auto * parent = frame.previous ? frame.previous : &frame.trackingCtx.rootFrame; + if (parent == &frame) + return; + + for (auto access : frame.directSourceAccessSetAccesses) + addFrameAccess(*parent, access); + for (auto child : frame.childSourceAccessSets) + addFrameChild(*parent, child); +} + +void mergeUnpublishedTrackedSourceDepsFrame(TrackedSourceDepsFrame & frame) +{ + if (!frame.published) + mergeFrameIntoParent(frame); +} + +void recordTrackedSourceAccessSetAccess(EvalSourceAccessId access) +{ + if (access == emptyEvalSourceAccessId) + return; + + if (auto * frame = currentTecnixThreadState.sourceDepsFrame) + addFrameAccess(*frame, access); +} + +void recordTrackedSourceAccessSetDependency(TrackingContext & trackingCtx, EvalSourceAccessSetId accessSet) +{ + addToCurrentFrame(trackingCtx, accessSet); +} + +static TrackedSourceDepsFrame * currentTrackedValueForceFrame(const void * value = nullptr) +{ + auto * frame = currentTecnixThreadState.sourceDepsFrame; + auto * valueFrame = frame ? frame->nearestValueForceFrame : nullptr; + if (!valueFrame || (value && valueFrame->value != value)) + return nullptr; + return valueFrame; +} + +void publishTrackedValueDependencies(const void * value) +{ + auto * frame = currentTrackedValueForceFrame(value); + if (!frame || frame->published) + return; + + auto directCount = frame->directSourceAccessSetAccesses.size(); + auto childCount = frame->childSourceAccessSets.size(); + + if (!frameHasSourceDeps(*frame)) { + frame->published = true; + return; + } + + EvalSourceAccessSetId sourceAccessSet = emptyEvalSourceAccessSetId; + if (directCount == 0 && childCount == 1) { + sourceAccessSet = frame->childSourceAccessSets.data()[0]; + if (sourceAccessSet != emptyEvalSourceAccessSetId) + frame->value->setTrackedSourceAccessSet(sourceAccessSet); + } else { + sourceAccessSet = publishTrackedSourceAccessSetDependencies( + *frame->trackingCtx.sourceAccessSetGraph, + *frame->value, + std::span( + frame->directSourceAccessSetAccesses.data(), frame->directSourceAccessSetAccesses.size()), + std::span( + frame->childSourceAccessSets.data(), frame->childSourceAccessSets.size())); + } + if (sourceAccessSet != emptyEvalSourceAccessSetId) { + if (frame->previous) + addToParentFrame(*frame->previous, sourceAccessSet); + else + addToCurrentFrame(frame->trackingCtx, sourceAccessSet); + } + frame->accessSet = sourceAccessSet; + frame->published = true; +} + +// Copying a finished Value must also copy its provenance. If the copy is the +// value currently being forced, add the source set to that force frame before +// finish() publishes it. Non-current destinations are published after finish() +// by publishCopiedValueDependencies(). +void copyTrackedValueDependencies(void * dst, const void * src) +{ + auto * trackingCtx = currentTecnixThreadState.trackingContext; + if (!trackingCtx || dst == src) + return; + + auto * currentValueFrame = currentTrackedValueForceFrame(dst); + if (!currentValueFrame) + return; + + auto accessSet = static_cast(src)->trackedSourceAccessSet(); + if (accessSet == emptyEvalSourceAccessSetId) + return; + + addFrameChild(*currentValueFrame, accessSet); +} + +void publishCopiedValueDependencies(void * dst, const void * src) +{ + auto * trackingCtx = currentTecnixThreadState.trackingContext; + if (!trackingCtx || dst == src) + return; + + auto * currentValueFrame = currentTrackedValueForceFrame(dst); + if (currentValueFrame) + return; + + auto accessSet = static_cast(src)->trackedSourceAccessSet(); + if (accessSet == emptyEvalSourceAccessSetId) + return; + + auto * dstValue = static_cast(dst); + std::array children{accessSet}; + publishTrackedSourceAccessSetDependencies( + *trackingCtx->sourceAccessSetGraph, + *dstValue, + std::span{}, + std::span(children)); +} + +TrackedSourceDepsFrame::TrackedSourceDepsFrame( + TrackingContext & trackingCtx, Value * value, TrackedSourceDepsFrame * previous) + : trackingCtx(trackingCtx) + , value(value) + , previous(previous) + , nearestValueForceFrame( + value ? this + : previous ? previous->nearestValueForceFrame + : nullptr) +{ +} + +TrackingContext::TrackingContext(EvalState & evalState) + : sourceAccessSetGraph(trackedSourceAccessSetGraph(evalState)) + , rootFrame(*this) +{ + // Establish the invariant every hot path relies on: a live TrackingContext + // implies an enabled graph, so forcing and the value hooks never re-check. + sourceAccessSetGraph->enable(); +} + +void TrackingContext::recordAccess(std::string_view path) +{ + auto accessSetAccessId = sourceAccessSetGraph->internAccess(path); + recordTrackedSourceAccessSetAccess(accessSetAccessId); +} + +ActiveTrackingContext::ActiveTrackingContext(TrackingContext & trackingCtx) + : trackingCtx(trackingCtx) + , previousTrackingCtx(currentTecnixThreadState.trackingContext) + , previousFrame(currentTecnixThreadState.sourceDepsFrame) +{ + currentTecnixThreadState.trackingContext = &trackingCtx; + currentTecnixThreadState.sourceDepsFrame = &trackingCtx.rootFrame; +} + +ActiveTrackingContext::~ActiveTrackingContext() +{ + if (currentTecnixThreadState.sourceDepsFrame == &trackingCtx.rootFrame) + currentTecnixThreadState.sourceDepsFrame = previousFrame; + currentTecnixThreadState.trackingContext = previousTrackingCtx; +} + +TrackedSourceDepsScope::TrackedSourceDepsScope(TrackingContext & trackingCtx) + : frame(trackingCtx, nullptr, currentTecnixThreadState.sourceDepsFrame) + , previousFrame(currentTecnixThreadState.sourceDepsFrame) +{ + currentTecnixThreadState.sourceDepsFrame = &frame; +} + +TrackedSourceDepsScope::~TrackedSourceDepsScope() +{ + if (currentTecnixThreadState.sourceDepsFrame == &frame) + currentTecnixThreadState.sourceDepsFrame = previousFrame; + mergeUnpublishedTrackedSourceDepsFrame(frame); +} + +EvalSourceAccessSetId TrackedSourceDepsScope::finish(Value * publishValue) +{ + if (frame.published) + return frame.accessSet; + + if (currentTecnixThreadState.sourceDepsFrame == &frame) + currentTecnixThreadState.sourceDepsFrame = previousFrame; + + frame.accessSet = internFrameAccessSet(frame); + if (frame.accessSet != emptyEvalSourceAccessSetId) { + if (frame.previous) + addToParentFrame(*frame.previous, frame.accessSet); + else + addToCurrentFrame(frame.trackingCtx, frame.accessSet); + } + + if (publishValue && frame.accessSet != emptyEvalSourceAccessSetId) + publishValue->setTrackedSourceAccessSet(frame.accessSet); + frame.published = true; + return frame.accessSet; +} + +static EvalState::TecnixEvalData * sourceDepsData(EvalState & state) +{ + return &state.tecnixEvalData(); +} + +static const EvalState::TecnixEvalData * sourceDepsData(const EvalState & state) +{ + return &state.tecnixEvalData(); +} + +void enableSourceAccessSetTracking(EvalState & state) +{ + sourceDepsData(state)->sourceAccessSetGraph->enable(); +} + +EvalSourceAccessSetStats trackedSourceAccessSetStats(const EvalState & state) +{ + return sourceDepsData(state)->sourceAccessSetGraph->stats(); +} + +ref trackedSourceAccessSetGraph(const EvalState & state) +{ + return sourceDepsData(state)->sourceAccessSetGraph; +} + +EvalSourceAccessSetId publishTrackedSourceAccessSetDependencies( + EvalSourceAccessSetGraph & graph, + Value & v, + std::span directAccesses, + std::span children) +{ + auto accessSet = graph.internAccessSet(directAccesses, children); + if (accessSet != emptyEvalSourceAccessSetId) + v.setTrackedSourceAccessSet(accessSet); + return accessSet; +} + +} // namespace nix diff --git a/src/libexpr/value-to-json.cc b/src/libexpr/value-to-json.cc index 58e3b3e539..da24ca2a2b 100644 --- a/src/libexpr/value-to-json.cc +++ b/src/libexpr/value-to-json.cc @@ -16,6 +16,12 @@ using json = nlohmann::json; static void parallelForceDeep(EvalState & state, Value & v, PosIdx pos) { + /* Tecnix: tracking contexts are thread-confined, so tracked evaluation + must not spawn detached work. Skip the prefetch; the caller forces + everything itself, producing identical results and closures. */ + if (currentTecnixThreadState.trackingContext) + return; + state.forceValue(v, pos); Executor::WorkItems work; diff --git a/src/libfetchers-tests/git-utils.cc b/src/libfetchers-tests/git-utils.cc index 4ab9eeb703..9c0174e162 100644 --- a/src/libfetchers-tests/git-utils.cc +++ b/src/libfetchers-tests/git-utils.cc @@ -251,6 +251,7 @@ TEST_F(GitUtilsTest, getSubtreeSha_missing_entry_throws) auto treeHash = Hash::parseNonSRIUnprefixed(sha, HashAlgorithm::SHA1); ASSERT_THROW(repo->getSubtreeSha(treeHash, "nonexistent"), Error); + ASSERT_THROW(repo->getSubtreeSha(treeHash, "existing"), Error); } // ============================================================================ diff --git a/src/libfetchers/fetch-to-store.cc b/src/libfetchers/fetch-to-store.cc index 3e932454dc..0321e6ca3c 100644 --- a/src/libfetchers/fetch-to-store.cc +++ b/src/libfetchers/fetch-to-store.cc @@ -38,10 +38,12 @@ std::pair fetchToStore2( { std::optional cacheKey; - auto [subpath, fingerprint] = filter ? std::pair>{path.path, std::nullopt} - : path.accessor->getFingerprint(path.path); + // Always try getFingerprint, even when a filter is present, so source + // accessors can record the access. Do not persistently cache filtered + // paths: the filter predicate is not part of the cache key. + auto [subpath, fingerprint] = path.accessor->getFingerprint(path.path); - if (fingerprint) { + if (fingerprint && !filter) { cacheKey = makeSourcePathToHashCacheKey(*fingerprint, method, subpath); if (auto res = settings.getCache()->lookup(*cacheKey)) { auto hash = Hash::parseSRI(fetchers::getStrAttr(*res, "hash")); @@ -63,10 +65,13 @@ std::pair fetchToStore2( } debug("source path '%s' not in store", path); } + } else if (filter) { + debug("source path '%s' has a filter; skipping persistent source-path cache", path); } else { static auto barf = getEnv("_NIX_TEST_BARF_ON_UNCACHEABLE").value_or("") == "1"; - if (barf && !filter && !(path.to_string().starts_with("/") || path.to_string().starts_with("«path:/"))) + if (barf && !(path.to_string().starts_with("/") || path.to_string().starts_with("«path:/"))) throw Error("source path '%s' is uncacheable (filter=%d)", path, (bool) filter); + // FIXME: could still provide in-memory caching keyed on `SourcePath`. debug("source path '%s' is uncacheable", path); } diff --git a/src/libfetchers/filtering-source-accessor.cc b/src/libfetchers/filtering-source-accessor.cc index ad038eb8a1..3c798f6272 100644 --- a/src/libfetchers/filtering-source-accessor.cc +++ b/src/libfetchers/filtering-source-accessor.cc @@ -62,6 +62,17 @@ std::pair> FilteringSourceAccessor::getFin return next->getFingerprint(prefix / path); } +bool FilteringSourceAccessor::tracksEvalAccesses(const CanonPath & path) +{ + return isAllowed(path) && next->tracksEvalAccesses(prefix / path); +} + +void FilteringSourceAccessor::recordEvalAccess(const CanonPath & path) +{ + checkAccess(path); + next->recordEvalAccess(prefix / path); +} + std::shared_ptr FilteringSourceAccessor::getProvenance(const CanonPath & path) { if (provenance) diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index 3d02f01208..795f3b373a 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -625,22 +625,43 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this return true; } - Hash getSubtreeSha(const Hash & treeSha, const std::string & entryName) override + std::optional getPathInfo(const Hash & treeSha, const std::string & relPath) override { - git_tree * tree = nullptr; - auto oid = hashToOID(treeSha); + if (relPath.empty()) + return GitPathInfo{.oid = treeSha, .mode = 0040000}; - if (git_tree_lookup(&tree, *this, &oid)) + auto oid = hashToOID(treeSha); + Tree tree; + if (git_tree_lookup(Setter(tree), *this, &oid)) throw Error("looking up tree %s: %s", treeSha.gitRev(), git_error_last()->message); - Finally freeTree([&]() { git_tree_free(tree); }); + git_tree_entry * entry = nullptr; + if (git_tree_entry_bypath(&entry, tree.get(), relPath.c_str()) != 0) + return std::nullopt; + Finally freeEntry([&]() { git_tree_entry_free(entry); }); + return GitPathInfo{ + .oid = toHash(*git_tree_entry_id(entry)), + .mode = static_cast(git_tree_entry_filemode(entry)), + }; + } - auto entry = git_tree_entry_byname(tree, entryName.c_str()); - if (!entry) + Hash getSubtreeSha(const Hash & treeSha, const std::string & entryName) override + { + if (entryName.empty()) + return treeSha; + + auto oid = hashToOID(treeSha); + Tree tree; + if (git_tree_lookup(Setter(tree), *this, &oid)) + throw Error("looking up tree %s: %s", treeSha.gitRev(), git_error_last()->message); + + git_tree_entry * entry = nullptr; + if (git_tree_entry_bypath(&entry, tree.get(), entryName.c_str()) != 0) throw Error("entry '%s' not found in tree %s", entryName, treeSha.gitRev()); + Finally freeEntry([&]() { git_tree_entry_free(entry); }); if (git_tree_entry_type(entry) != GIT_OBJECT_TREE) - throw Error("'%s' in tree %s is not a directory", entryName, treeSha.gitRev()); + throw Error("entry '%s' in tree %s is not a directory", entryName, treeSha.gitRev()); return toHash(*git_tree_entry_id(entry)); } diff --git a/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh b/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh index 859a1cf8e5..8bad0fe9a5 100644 --- a/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh +++ b/src/libfetchers/include/nix/fetchers/filtering-source-accessor.hh @@ -53,6 +53,10 @@ struct FilteringSourceAccessor : SourceAccessor std::pair> getFingerprint(const CanonPath & path) override; + bool tracksEvalAccesses(const CanonPath & path) override; + + void recordEvalAccess(const CanonPath & path) override; + std::shared_ptr getProvenance(const CanonPath & path) override; void invalidateCache(const CanonPath & path) override; diff --git a/src/libfetchers/include/nix/fetchers/git-utils.hh b/src/libfetchers/include/nix/fetchers/git-utils.hh index e9f63a5b9d..81e748d87f 100644 --- a/src/libfetchers/include/nix/fetchers/git-utils.hh +++ b/src/libfetchers/include/nix/fetchers/git-utils.hh @@ -50,6 +50,12 @@ struct GitAccessorOptions std::string makeFingerprint(const Hash & rev) const; }; +struct GitPathInfo +{ + Hash oid; + uint32_t mode; +}; + struct GitRepo { virtual ~GitRepo() {} @@ -126,6 +132,9 @@ struct GitRepo /** Get the SHA of a subtree entry within a tree object */ virtual Hash getSubtreeSha(const Hash & treeSha, const std::string & entryName) = 0; + /** Get the SHA and git file mode of any entry by full relative path within a tree. */ + virtual std::optional getPathInfo(const Hash & treeSha, const std::string & relPath) = 0; + /** Get the root tree SHA from a commit SHA */ virtual Hash getCommitTree(const Hash & commitSha) = 0; diff --git a/src/libstore/async-path-writer.cc b/src/libstore/async-path-writer.cc index ede52a146a..a14807b9dc 100644 --- a/src/libstore/async-path-writer.cc +++ b/src/libstore/async-path-writer.cc @@ -27,6 +27,7 @@ struct AsyncPathWriterImpl : AsyncPathWriter { std::vector items; std::unordered_map> futures; + StorePathSet addedPaths; bool quit = false; }; @@ -89,6 +90,7 @@ struct AsyncPathWriterImpl : AsyncPathWriter auto state(state_.lock()); std::promise promise; + state->addedPaths.insert(storePath); state->futures.insert_or_assign(storePath, promise.get_future()); state->items.push_back( Item{ @@ -118,6 +120,11 @@ struct AsyncPathWriterImpl : AsyncPathWriter future.get(); } + bool wasAdded(const StorePath & path) override + { + return state_.lock()->addedPaths.contains(path); + } + void waitForAllPaths() override { auto futures = ({ @@ -155,9 +162,20 @@ struct AsyncPathWriterImpl : AsyncPathWriter store->addMultipleToStore(std::move(sources), act, repair); #endif + StorePathSet pathsToCheck; for (auto & item : items) { - StringSource source(item.contents); store->addTempRoot(item.storePath); + if (item.repair == NoRepair) + pathsToCheck.insert(item.storePath); + } + + auto validPaths = pathsToCheck.empty() ? StorePathSet{} : store->queryValidPaths(pathsToCheck, NoSubstitute); + + for (auto & item : items) { + if (item.repair == NoRepair && validPaths.count(item.storePath)) + continue; + + StringSource source(item.contents); auto storePath = store->addToStoreFromDump( source, item.storePath.name(), @@ -168,6 +186,8 @@ struct AsyncPathWriterImpl : AsyncPathWriter item.repair, item.provenance); assert(storePath == item.storePath); + if (item.repair == NoRepair) + validPaths.insert(item.storePath); } } }; diff --git a/src/libstore/include/nix/store/async-path-writer.hh b/src/libstore/include/nix/store/async-path-writer.hh index 695321ccb1..303afd5761 100644 --- a/src/libstore/include/nix/store/async-path-writer.hh +++ b/src/libstore/include/nix/store/async-path-writer.hh @@ -19,6 +19,13 @@ struct AsyncPathWriter virtual void waitForAllPaths() = 0; + /** + * Whether `addPath` was called for `path` during this writer's lifetime. + * Unlike observing store writes, this includes paths whose write was + * elided because the store already had them. + */ + virtual bool wasAdded(const StorePath & path) = 0; + static ref make(ref store); }; diff --git a/src/libstore/include/nix/store/sqlite.hh b/src/libstore/include/nix/store/sqlite.hh index 789e821746..3e5dd8f386 100644 --- a/src/libstore/include/nix/store/sqlite.hh +++ b/src/libstore/include/nix/store/sqlite.hh @@ -4,6 +4,7 @@ #include #include #include +#include #include "nix/util/error.hh" @@ -140,6 +141,7 @@ struct SQLiteStmt bool next(); std::string getStr(int col); + std::string_view getBlob(int col); int64_t getInt(int col); bool isNull(int col); }; diff --git a/src/libstore/sqlite.cc b/src/libstore/sqlite.cc index 5f6119a427..cf288fa190 100644 --- a/src/libstore/sqlite.cc +++ b/src/libstore/sqlite.cc @@ -240,6 +240,17 @@ std::string SQLiteStmt::Use::getStr(int col) return s; } +std::string_view SQLiteStmt::Use::getBlob(int col) +{ + auto * data = sqlite3_column_blob(stmt, col); + auto len = sqlite3_column_bytes(stmt, col); + if (!data && len != 0) + SQLiteError::throw_(stmt.db, "reading SQLite blob column"); + if (!data) + return {}; + return {static_cast(data), static_cast(len)}; +} + int64_t SQLiteStmt::Use::getInt(int col) { // FIXME: detect nulls? diff --git a/src/libutil/archive.cc b/src/libutil/archive.cc index fc82758610..1f1f9d5e4a 100644 --- a/src/libutil/archive.cc +++ b/src/libutil/archive.cc @@ -11,6 +11,7 @@ #include "nix/util/posix-source-accessor.hh" #include "nix/util/source-path.hh" #include "nix/util/file-system.hh" +#include "nix/util/finally.hh" #include "nix/util/signals.hh" namespace nix { @@ -40,8 +41,13 @@ static constexpr size_t narMaxDepth = 64; PathFilter defaultPathFilter = [](const std::string &) { return true; }; +thread_local int SourceAccessor::dumpPathDepth = 0; + void SourceAccessor::dumpPath(const CanonPath & path, Sink & sink, PathFilter & filter) { + dumpPathDepth++; + Finally restoreDumpPathDepth([&]() { dumpPathDepth--; }); + auto dumpContents = [&](const CanonPath & path) { sink << "contents"; std::optional size; diff --git a/src/libutil/include/nix/util/forwarding-source-accessor.hh b/src/libutil/include/nix/util/forwarding-source-accessor.hh index c9693b9e5f..f999cc3130 100644 --- a/src/libutil/include/nix/util/forwarding-source-accessor.hh +++ b/src/libutil/include/nix/util/forwarding-source-accessor.hh @@ -43,6 +43,27 @@ struct ForwardingSourceAccessor : SourceAccessor return next->showPath(path); } + bool tracksEvalAccesses(const CanonPath & path) override + { + return next->tracksEvalAccesses(path); + } + + /** + * A forwarding accessor presents `next`'s content verbatim, so its path + * fingerprints (and any access recording done by computing them) forward + * too. Accessors that change the presented content must not derive from + * this class. + */ + std::pair> getFingerprint(const CanonPath & path) override + { + return next->getFingerprint(path); + } + + void recordEvalAccess(const CanonPath & path) override + { + next->recordEvalAccess(path); + } + std::optional getPhysicalPath(const CanonPath & path) override { return next->getPhysicalPath(path); diff --git a/src/libutil/include/nix/util/source-accessor.hh b/src/libutil/include/nix/util/source-accessor.hh index 41d24ca778..01ef2ce828 100644 --- a/src/libutil/include/nix/util/source-accessor.hh +++ b/src/libutil/include/nix/util/source-accessor.hh @@ -143,6 +143,31 @@ struct SourceAccessor : std::enable_shared_from_this virtual void dumpPath(const CanonPath & path, Sink & sink, PathFilter & filter = defaultPathFilter); + /** + * Whether this source path is backed by an accessor that records source + * accesses for evaluator dependency tracking. This must be side-effect-free: + * callers use it only to choose a safe evaluator cache domain. + */ + virtual bool tracksEvalAccesses(const CanonPath &) + { + return false; + } + + /** + * Replay an evaluator source access for cached work that did not call + * readFile/readDirectory/readLink on this accessor again. Accessors that + * return true from tracksEvalAccesses() should map the path the same way + * their read operations do and record it in the active tracking context. + */ + virtual void recordEvalAccess(const CanonPath &) {} + + /** + * Depth counter for dumpPath calls. Used by tracking infrastructure + * to suppress individual file tracking during NAR serialization + * (store copy), since the directory-level fingerprint is sufficient. + */ + static thread_local int dumpPathDepth; + Hash hashPath(const CanonPath & path, PathFilter & filter = defaultPathFilter, HashAlgorithm ha = HashAlgorithm::SHA256); diff --git a/src/libutil/mounted-source-accessor.cc b/src/libutil/mounted-source-accessor.cc index ca6d49275b..4a3b64ef71 100644 --- a/src/libutil/mounted-source-accessor.cc +++ b/src/libutil/mounted-source-accessor.cc @@ -79,6 +79,18 @@ struct MountedSourceAccessorImpl : MountedSourceAccessor return accessor->getPhysicalPath(subpath); } + bool tracksEvalAccesses(const CanonPath & path) override + { + auto [accessor, subpath] = resolve(path); + return accessor->tracksEvalAccesses(subpath); + } + + void recordEvalAccess(const CanonPath & path) override + { + auto [accessor, subpath] = resolve(path); + accessor->recordEvalAccess(subpath); + } + void mount(CanonPath mountPoint, ref accessor) override { mounts.emplace(std::move(mountPoint), std::move(accessor)); diff --git a/src/libutil/union-source-accessor.cc b/src/libutil/union-source-accessor.cc index 0a94acc63f..dc9cc7d9d9 100644 --- a/src/libutil/union-source-accessor.cc +++ b/src/libutil/union-source-accessor.cc @@ -79,6 +79,24 @@ struct UnionSourceAccessor : SourceAccessor return std::nullopt; } + bool tracksEvalAccesses(const CanonPath & path) override + { + for (auto & accessor : accessors) + if (accessor->tracksEvalAccesses(path)) + return true; + return false; + } + + void recordEvalAccess(const CanonPath & path) override + { + for (auto & accessor : accessors) { + if (accessor->tracksEvalAccesses(path)) { + accessor->recordEvalAccess(path); + return; + } + } + } + std::pair> getFingerprint(const CanonPath & path) override { if (fingerprint) diff --git a/src/nix/provenance.cc b/src/nix/provenance.cc index be937afb96..97e0510af1 100644 --- a/src/nix/provenance.cc +++ b/src/nix/provenance.cc @@ -1,4 +1,5 @@ #include "nix/cmd/command.hh" +#include "nix/store/async-path-writer.hh" #include "nix/store/store-api.hh" #include "nix/store/store-open.hh" #include "nix/expr/provenance.hh" @@ -452,7 +453,10 @@ struct CmdProvenanceVerify : StorePathsCommand logger->cout("✅ evaluated '%s#%s'", installable.flakeRef.to_string(true), flake->flakeOutput); if (path) { - if (!trackingStore->instantiatedPaths.contains(*path)) { + // Instantiated paths are observed either as store writes or, + // for paths the async path writer deduplicated against + // already-valid store contents, from the writer itself. + if (!trackingStore->instantiatedPaths.contains(*path) && !evalState->asyncPathWriter->wasAdded(*path)) { logger->cout( "❌ " ANSI_RED "evaluation did not re-instantiate path '%s'" ANSI_NORMAL, store.printStorePath(*path)); diff --git a/tests/functional/meson.build b/tests/functional/meson.build index 095434b888..488f7ad25f 100644 --- a/tests/functional/meson.build +++ b/tests/functional/meson.build @@ -227,6 +227,7 @@ subdir('git') subdir('git-hashing') subdir('local-overlay-store') subdir('tectonix') +subdir('tecnix') foreach suite : suites workdir = suite['workdir'] diff --git a/tests/functional/tecnix/builtins.sh b/tests/functional/tecnix/builtins.sh new file mode 100755 index 0000000000..740ba32c8b --- /dev/null +++ b/tests/functional/tecnix/builtins.sh @@ -0,0 +1,1081 @@ +#!/usr/bin/env bash +# Tests for the public Tecnix builtins: tecnixTargetNames and tecnixTargets. + +source "$(dirname "${BASH_SOURCE[0]}")/common.sh" + +TEST_WORLD="$TEST_ROOT/tecnix-world" +create_tecnix_builtin_test_world "$TEST_WORLD" +HEAD_SHA=$(get_head_sha "$TEST_WORLD") + +TEST_WORLD_OTHER="$TEST_ROOT/tecnix-world-other" +create_tecnix_builtin_test_world "$TEST_WORLD_OTHER" +HEAD_SHA_OTHER=$(get_head_sha "$TEST_WORLD_OTHER") + +tecnix_args() { + cat </dev/null <<< "$json"; then + echo "$json" >&2 + fail "$message" + fi +} + +assert_json_equal() { + local actual="$1" + local expected="$2" + local message="$3" + if ! diff -u <(jq -S . <<< "$expected") <(jq -S . <<< "$actual"); then + fail "$message" + fi +} + +assert_target_dependency_paths() { + local json="$1" + local target="$2" + local message="$3" + local expected actual + expected=$(cat) + actual=$(jq -r --arg target "$target" '.[$target] | keys[]' <<< "$json") + if ! diff -u <(printf '%s\n' "$expected") <(printf '%s\n' "$actual"); then + fail "$message" + fi +} + +assert_clean_dependency_paths() { + local deps="$1" + + assert_json_equal "$(jq -c 'keys' <<< "$deps")" '["//areas/app/web:alpha","//areas/app/web:beta","//areas/app/web:closureChainA","//areas/app/web:closureChainB","//areas/app/web:closureMiddleUser","//areas/app/web:existsCheck","//areas/app/web:fileTypeCheck","//areas/app/web:nestedOptional","//areas/app/web:optional","//areas/app/web:readDirCheck","//areas/app/web:readFileCheck","//areas/app/web:readFileSharedA","//areas/app/web:readFileSharedB","//areas/app/web:resolverModuleUser","//areas/app/web:sharedExistsA","//areas/app/web:sharedExistsB","//areas/app/web:sharedReadDirA","//areas/app/web:sharedReadDirB","//areas/app/web:srcdir","//areas/app/web:symlinked","//areas/app/web:treeShaCheck","//areas/lib/shared:gamma"]' \ + "clean dependency output should have the exact expected target keys" + + assert_target_dependency_paths "$deps" "//areas/app/web:alpha" "alpha dependency paths should be exact" <<'EOF' +areas/app/web/common.nix +areas/app/web/targets.nix +areas/app/web/targets/alpha.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:beta" "beta dependency paths should be exact" <<'EOF' +areas/app/web/common.nix +areas/app/web/targets.nix +areas/app/web/targets/beta.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:srcdir" "srcdir dependency paths should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/srcdir.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:symlinked" "symlinked dependency paths should be exact" <<'EOF' +areas/app/web/common.nix +areas/app/web/targets.nix +areas/app/web/targets/alpha.nix +areas/app/web/targets/current.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:optional" "optional dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/optional-marker.nix +areas/app/web/targets/optional.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:existsCheck" "existsCheck dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/exists-check.nix +areas/app/web/targets/exists-marker +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:fileTypeCheck" "fileTypeCheck dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/exists-marker +areas/app/web/targets/file-type-check.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:readDirCheck" "readDirCheck dependency paths should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/read-dir-check.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:readFileCheck" "readFileCheck dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/read-file-check.nix +areas/app/web/targets/read-file-marker.txt +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:readFileSharedA" "readFileSharedA dependency paths should be exact" <<'EOF' +areas/app/web/shared-read-file.txt +areas/app/web/targets.nix +areas/app/web/targets/read-file-shared.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:readFileSharedB" "readFileSharedB dependency paths should be exact" <<'EOF' +areas/app/web/shared-read-file.txt +areas/app/web/targets.nix +areas/app/web/targets/read-file-shared.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:sharedExistsA" "sharedExistsA dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/exists-marker +areas/app/web/targets/shared-exists.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:sharedExistsB" "sharedExistsB dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/exists-marker +areas/app/web/targets/shared-exists.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:sharedReadDirA" "sharedReadDirA dependency paths should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/shared-read-dir.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:sharedReadDirB" "sharedReadDirB dependency paths should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/shared-read-dir.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:closureMiddleUser" "closureMiddleUser dependency paths should be exact" <<'EOF' +areas/app/web/closure-leaf.txt +areas/app/web/targets.nix +areas/app/web/targets/closure-middle.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:closureChainA" "closureChainA dependency paths should be exact" <<'EOF' +areas/app/web/closure-leaf.txt +areas/app/web/targets.nix +areas/app/web/targets/closure-chain.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:closureChainB" "closureChainB dependency paths should be exact" <<'EOF' +areas/app/web/closure-leaf.txt +areas/app/web/targets.nix +areas/app/web/targets/closure-chain.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:resolverModuleUser" "resolverModuleUser dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/resolver-module-user.nix +system/tectonix/resolve.nix +system/tectonix/resolver-module.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:nestedOptional" "nestedOptional dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/nested-optional.nix +areas/app/web/untracked-dir/nested-marker.nix +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/app/web:treeShaCheck" "treeShaCheck dependency paths should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/tree-sha-check.nix +areas/lib/shared +system/tectonix/resolve.nix +EOF + + assert_target_dependency_paths "$deps" "//areas/lib/shared:gamma" "gamma dependency paths should be exact" <<'EOF' +areas/lib/shared/common.nix +areas/lib/shared/targets.nix +areas/lib/shared/targets/gamma.nix +system/tectonix/resolve.nix +EOF +} + +base_args=$(tecnix_args) +other_args=$(tecnix_other_args) + +echo "Testing tecnixTargetNames..." +target_names=$(tecnix_eval_json_no_cache "builtins.tecnixTargetNames ($base_args)") +assert_jq "$target_names" \ + '. == ["//areas/app/web:alpha", "//areas/app/web:beta", "//areas/app/web:srcdir", "//areas/app/web:symlinked", "//areas/app/web:optional", "//areas/app/web:existsCheck", "//areas/app/web:fileTypeCheck", "//areas/app/web:readDirCheck", "//areas/app/web:readFileCheck", "//areas/app/web:readFileSharedA", "//areas/app/web:readFileSharedB", "//areas/app/web:sharedExistsA", "//areas/app/web:sharedExistsB", "//areas/app/web:sharedReadDirA", "//areas/app/web:sharedReadDirB", "//areas/app/web:closureMiddleUser", "//areas/app/web:closureChainA", "//areas/app/web:closureChainB", "//areas/app/web:resolverModuleUser", "//areas/app/web:nestedOptional", "//areas/app/web:treeShaCheck", "//areas/lib/shared:gamma"]' \ + "tecnixTargetNames should return the expected flat target list" + +# Target discovery has its own dependency graph, useful for debugging why the +# discovered target list is or is not reusable. +echo "Testing tecnixTargetNames includeDependencies..." +target_name_deps=$(tecnix_eval_json_no_cache "tecnixTargetNameDependencyPathSet ($base_args)") +assert_json_equal "$(jq -c 'keys' <<< "$target_name_deps")" '[".meta/manifest.json","system/tectonix/resolve.nix"]' \ + "tecnixTargetNames includeDependencies should have the exact expected dependency paths" +assert_jq "$target_name_deps" '."system/tectonix/resolve.nix" | startswith("git:")' \ + "tecnixTargetNames includeDependencies should include the resolver fingerprint" +assert_jq "$target_name_deps" '.".meta/manifest.json" | startswith("git:")' \ + "tecnixTargetNames includeDependencies should include the manifest fingerprint" + +# The rebased resolver contract keeps zone.src as a string-valued store path +# while exposing zone.srcPath as a path value for path arithmetic. +echo "Testing zone source builtin result types..." +zone_source_types=$(tecnix_eval_json_with_settings '{ src = builtins.typeOf (builtins.unsafeTectonixInternalZoneSrc "//areas/app/web"); path = builtins.typeOf (builtins.unsafeTectonixInternalZonePath "//areas/app/web"); }') +assert_jq "$zone_source_types" '.src == "string" and .path == "path"' \ + "zone source builtins should preserve the resolver-facing string/path split" + +# Filtered source-path cache keys must not ignore the filter predicate. Use the +# same path name for both filtered paths so a stale persistent hit would return +# the first filter's store path for the second filter. +echo "Testing filtered zone source paths do not share a persistent cache key..." +filtered_zone_source=$(tecnix_eval_json_with_settings ' +let + src = builtins.unsafeTectonixInternalZoneSrc "//areas/app/web"; + onlyCommon = builtins.path { + name = "filtered-zone-source"; + path = src; + filter = path: type: type == "directory" || builtins.baseNameOf path == "common.nix"; + }; + onlyZone = builtins.path { + name = "filtered-zone-source"; + path = src; + filter = path: type: type == "directory" || builtins.baseNameOf path == "zone.nix"; + }; +in { + commonHasCommon = builtins.pathExists (onlyCommon + "/common.nix"); + commonHasZone = builtins.pathExists (onlyCommon + "/zone.nix"); + zoneHasCommon = builtins.pathExists (onlyZone + "/common.nix"); + zoneHasZone = builtins.pathExists (onlyZone + "/zone.nix"); +}') +assert_jq "$filtered_zone_source" '.commonHasCommon and (.commonHasZone | not) and (.zoneHasCommon | not) and .zoneHasZone' \ + "different filters over one fingerprinted source should produce different filtered paths" + +# A single EvalState has lazy repository accessors. Reusing it for a different +# repo context must fail rather than silently reusing the first context. +echo "Testing Tecnix EvalState rejects repo context changes..." +context_mismatch_err="$TEST_ROOT/tecnix-context-mismatch.err" +expect 1 nix eval --json \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tecnix-eval-cache false \ + --expr "let first = builtins.tecnixTargetNames ($base_args); in builtins.deepSeq first (builtins.tecnixTargetNames ($other_args))" \ + >/dev/null 2>"$context_mismatch_err" +grepQuiet "already configured" < "$context_mismatch_err" + +# Source-available eval must know the dirty overlay. If git status fails, do not +# continue as if the checkout were clean. +echo "Testing dirty checkout status failure is fatal..." +bad_checkout_err="$TEST_ROOT/tecnix-bad-checkout.err" +bad_checkout_expr=$(rewrite_tecnix_test_expr "tecnixTargetDependencyPathSet (($base_args) // { checkoutPath = \"$TEST_ROOT/not-a-checkout\"; targets = [ \"//areas/app/web:alpha\" ]; })") +expect 1 nix eval --json \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tecnix-eval-cache false \ + --expr "$bad_checkout_expr" \ + >/dev/null 2>"$bad_checkout_err" +grepQuiet "not-a-checkout" < "$bad_checkout_err" + +# `rev` may be omitted when `checkoutPath` names a git checkout: it defaults +# to the checkout's HEAD (resolveCheckoutHeadRev). Pin both sides of the +# fallback: HEAD resolution matches an explicit rev, and a checkout that +# cannot answer fails with guidance rather than proceeding. +echo "Testing omitted rev defaults to the checkout's HEAD..." +no_rev_args="{ gitDir = \"$TEST_WORLD/.git\"; resolver = \"system/tectonix/resolve.nix\"; args = { system = \"test-system\"; }; checkoutPath = \"$TEST_WORLD\"; }" +no_rev_names=$(tecnix_eval_json_no_cache "builtins.tecnixTargetNames ($no_rev_args)") +explicit_rev_names=$(tecnix_eval_json_no_cache "builtins.tecnixTargetNames ($base_args)") +assert_json_equal "$no_rev_names" "$explicit_rev_names" \ + "omitting rev should resolve the checkout's HEAD and match an explicit rev" + +echo "Testing omitted rev with an invalid checkout fails with guidance..." +no_rev_bad_checkout_err="$TEST_ROOT/tecnix-no-rev-bad-checkout.err" +expect 1 nix eval --json \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tecnix-eval-cache false \ + --expr "builtins.tecnixTargetNames { gitDir = \"$TEST_WORLD/.git\"; resolver = \"system/tectonix/resolve.nix\"; args = { }; checkoutPath = \"$TEST_ROOT/not-a-checkout\"; }" \ + >/dev/null 2>"$no_rev_bad_checkout_err" +grepQuiet "could not determine git SHA" < "$no_rev_bad_checkout_err" + +# Repo-root directory listings have no explicit source-path representation in +# the source closure yet. Fail closed instead of returning under-tracked deps. +echo "Testing repo-root source access fails closed..." +repo_root_access_err="$TEST_ROOT/tecnix-repo-root-access.err" +repo_root_access_expr=$(rewrite_tecnix_test_expr "tecnixTargetDependencyPathSet (($base_args) // { resolver = \"system/repo-root-read-dir/resolve.nix\"; targets = [ \"//repo:rootReadDir\" ]; })") +expect 1 nix eval --json \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tecnix-eval-cache false \ + --expr "$repo_root_access_expr" \ + >/dev/null 2>"$repo_root_access_err" +grepQuiet "repo-root source access" < "$repo_root_access_err" + +# These internal helpers are identity/lazy source-deps wrappers outside tracked +# Tecnix dependency evaluation. +echo "Testing internal source-deps wrapper builtins..." +source_deps_wrappers=$(tecnix_eval_json_no_cache '{ scope = builtins.tecnixInternalSourceDepsScope { x = 1; }; attrs = builtins.tecnixInternalSourceDepsAttrs { a = 2; b = 3; }; list = builtins.tecnixInternalSourceDepsList [ 4 5 ]; }') +assert_jq "$source_deps_wrappers" '.scope.x == 1 and .attrs.a == 2 and .attrs.b == 3 and .list == [4, 5]' \ + "internal source-deps wrapper builtins should preserve values" + +echo "Testing tecnixTargets on a clean worktree..." +clean_targets=$(tecnix_eval_json_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:beta\" \"//areas/app/web:alpha\" ]; })") +assert_jq "$clean_targets" 'length == 2' "tecnixTargets should return two results" +assert_jq "$clean_targets" '."//areas/app/web:beta".name == "beta" and ."//areas/app/web:beta".marker == "beta:clean-web-common:test-system" and ."//areas/app/web:beta".resolvedBy == "clean-resolver"' \ + "tecnixTargets should resolve beta and use the clean resolver" +assert_jq "$clean_targets" '."//areas/app/web:alpha".name == "alpha" and ."//areas/app/web:alpha".marker == "alpha:clean-web-common:test-system" and ."//areas/app/web:alpha".resolvedBy == "clean-resolver"' \ + "tecnixTargets should resolve alpha" + +# tecnixTargets can evaluate targets in parallel when eval cores are enabled. +echo "Testing parallel tecnixTargets evaluation..." +parallel_targets=$(tecnix_eval_json_parallel_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:beta\" \"//areas/app/web:alpha\" ]; })") +assert_jq "$parallel_targets" '."//areas/app/web:beta".name == "beta" and ."//areas/app/web:alpha".name == "alpha"' \ + "parallel tecnixTargets should resolve each requested target" + +echo "Testing tecnixTargets includeDependencies for overlapping clean targets..." +clean_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; })") +assert_jq "$clean_deps" 'has("//areas/app/web:alpha") and has("//areas/app/web:beta")' \ + "tecnixTargets includeDependencies should return dependencies keyed by target" +assert_jq "$clean_deps" '(."//areas/app/web:alpha" | has("system/tectonix/resolve.nix"))' \ + "alpha deps should include the resolver" +assert_jq "$clean_deps" '(."//areas/app/web:beta" | has("system/tectonix/resolve.nix"))' \ + "beta deps should include the resolver" +assert_jq "$clean_deps" '(."//areas/app/web:alpha"."system/tectonix/resolve.nix" | startswith("git:")) and (."//areas/app/web:alpha"."areas/app/web/common.nix" | startswith("git:"))' \ + "dependency entries should include source fingerprints" +assert_jq "$clean_deps" '(."//areas/app/web:alpha" | has("areas/app/web") | not) and (."//areas/app/web:beta" | has("areas/app/web") | not)' \ + "ordinary resolver path literals should not add the broad zone root as an eval input" +assert_jq "$clean_deps" '(."//areas/app/web:alpha" | has("areas/app/web/targets.nix") and has("areas/app/web/common.nix") and has("areas/app/web/targets/alpha.nix"))' \ + "alpha deps should include shared and alpha-specific files" +assert_jq "$clean_deps" '(."//areas/app/web:beta" | has("areas/app/web/targets.nix") and has("areas/app/web/common.nix") and has("areas/app/web/targets/beta.nix"))' \ + "beta deps should include shared and beta-specific files" +assert_jq "$clean_deps" '(."//areas/app/web:alpha" | has("areas/app/web/targets/beta.nix") | not)' \ + "alpha deps should not include beta-specific files" +assert_jq "$clean_deps" '(."//areas/app/web:beta" | has("areas/app/web/targets/alpha.nix") | not)' \ + "beta deps should not include alpha-specific files" + +# tecnixTargets includeDependencies can evaluate target misses in parallel when eval cores +# are enabled. It should still produce isolated per-target dependency graphs. +echo "Testing parallel tecnixTargets includeDependencies evaluation..." +parallel_deps=$(tecnix_eval_json_parallel_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; })") +assert_jq "$parallel_deps" '( ."//areas/app/web:alpha" | has("areas/app/web/common.nix") and has("areas/app/web/targets/alpha.nix") and (has("areas/app/web/targets/beta.nix") | not) )' \ + "parallel alpha deps should include alpha-specific files only" +assert_jq "$parallel_deps" '( ."//areas/app/web:beta" | has("areas/app/web/common.nix") and has("areas/app/web/targets/beta.nix") and (has("areas/app/web/targets/alpha.nix") | not) )' \ + "parallel beta deps should include beta-specific files only" + +# Source-deps scopes must be target-local. These two targets intentionally use +# the same scope key with different source files; mutable global replay would +# leak owner-collision-b.txt into A or vice versa. +echo "Testing source-deps scope isolation..." +owner_collision_deps=$(tecnix_eval_json_parallel_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:ownerCollisionA\" \"//areas/app/web:ownerCollisionB\" ]; })") +assert_target_dependency_paths "$owner_collision_deps" "//areas/app/web:ownerCollisionA" "ownerCollisionA deps should not include ownerCollisionB files" <<'EOF' +areas/app/web/common.nix +areas/app/web/owner-collision-a.txt +areas/app/web/targets.nix +areas/app/web/targets/owner-collision.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$owner_collision_deps" "//areas/app/web:ownerCollisionB" "ownerCollisionB deps should not include ownerCollisionA files" <<'EOF' +areas/app/web/common.nix +areas/app/web/owner-collision-b.txt +areas/app/web/targets.nix +areas/app/web/targets/owner-collision.nix +system/tectonix/resolve.nix +EOF + +# Reusing an already-forced source-deps-scoped value must publish its source +# provenance into the later target. This is the small form of the sequential +# all-target undertracking bug seen in large repos. +echo "Testing source-deps scoped value reuse for sequential target reuse..." +shared_owner_isolated=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:sharedOwnerB\" ]; })") +shared_owner_pair=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:sharedOwnerA\" \"//areas/app/web:sharedOwnerB\" ]; })") +if ! diff -u \ + <(jq -S '."//areas/app/web:sharedOwnerB"' <<< "$shared_owner_isolated") \ + <(jq -S '."//areas/app/web:sharedOwnerB"' <<< "$shared_owner_pair"); then + fail "sharedOwnerB deps should match isolated deps after sharedOwnerA preforces the source-deps-scoped value" +fi + +# Reusing an already-forced imported function result must also replay the +# imported file and downstream readFile provenance into the later target. +echo "Testing imported shared builder replay for sequential target reuse..." +shared_builder_isolated=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:sharedBuilderB\" ]; })") +shared_builder_pair=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:sharedBuilderA\" \"//areas/app/web:sharedBuilderB\" ]; })") +if ! diff -u \ + <(jq -S '."//areas/app/web:sharedBuilderB"' <<< "$shared_builder_isolated") \ + <(jq -S '."//areas/app/web:sharedBuilderB"' <<< "$shared_builder_pair"); then + fail "sharedBuilderB deps should match isolated deps after sharedBuilderA preforces the imported shared builder" +fi + +# The same replay must hold when the shared sourceful function file is imported +# independently by each target and the second import hits the tracked file eval cache. +echo "Testing tracked file cache replay for sequential imported builder reuse..." +indirect_builder_isolated=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:indirectBuilderB\" ]; })") +indirect_builder_pair=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:indirectBuilderA\" \"//areas/app/web:indirectBuilderB\" ]; })") +if ! diff -u \ + <(jq -S '."//areas/app/web:indirectBuilderB"' <<< "$indirect_builder_isolated") \ + <(jq -S '."//areas/app/web:indirectBuilderB"' <<< "$indirect_builder_pair"); then + fail "indirectBuilderB deps should match isolated deps after indirectBuilderA prewarms the shared imported builder file" +fi + +# The rust module shape uses callPackage (import ./builder.nix) in a shared let. +echo "Testing callPackage imported builder replay for sequential target reuse..." +callpackage_builder_isolated=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:callPackageBuilderB\" ]; })") +callpackage_builder_pair=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:callPackageBuilderA\" \"//areas/app/web:callPackageBuilderB\" ]; })") +if ! diff -u \ + <(jq -S '."//areas/app/web:callPackageBuilderB"' <<< "$callpackage_builder_isolated") \ + <(jq -S '."//areas/app/web:callPackageBuilderB"' <<< "$callpackage_builder_pair"); then + fail "callPackageBuilderB deps should match isolated deps after callPackageBuilderA prewarms the shared callPackage/import builder" +fi + +# This is a Nix-level shape for the ValueStorage::operator= provenance bug. +# copyOnlyPrewarm forces a shared resolver-level string. copyOnlyConsumer then +# uses builtins.break, whose implementation returns by copying its already-forced +# argument without forcing it again. Without copying Value* provenance, the +# consumer misses copy-only-marker.txt. +echo "Testing copy-only finished value provenance replay from Nix code..." +copy_only_pair=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//system/tectonix-copy-value:copyOnlyPrewarm\" \"//system/tectonix-copy-value:copyOnlyConsumer\" ]; })") +assert_jq "$copy_only_pair" '."//system/tectonix-copy-value:copyOnlyConsumer" | has("system/tectonix/copy-only-marker.txt")' \ + "copyOnlyConsumer deps should include the copied sourceful value" + +# The target builtin can return normal target values and dependency records in +# one public API shape. +echo "Testing tecnixTargets with dependency records..." +targets_with_deps=$(tecnix_eval_json_parallel_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; includeDependencies = true; })") +assert_jq "$targets_with_deps" 'length == 2 and .[0].target == "//areas/app/web:alpha" and .[1].target == "//areas/app/web:beta"' \ + "tecnixTargets includeDependencies should preserve input order" +assert_jq "$targets_with_deps" '.[0].value.drvPath == "/nix/store/00000000000000000000000000000000-alpha-clean-web-common.drv" and .[1].value.drvPath == "/nix/store/00000000000000000000000000000000-beta-clean-web-common.drv"' \ + "tecnixTargets includeDependencies should include normal target values" +assert_jq "$targets_with_deps" '((.[0].dependencies | has("areas/app/web/targets/alpha.nix") and (has("areas/app/web/targets/beta.nix") | not))) and ((.[1].dependencies | has("areas/app/web/targets/beta.nix") and (has("areas/app/web/targets/alpha.nix") | not)))' \ + "tecnixTargets dependencies should stay isolated per target" +assert_jq "$targets_with_deps" '.[0].dependencies."areas/app/web/targets/alpha.nix" | startswith("git:")' \ + "tecnixTargets dependencies should include source fingerprints" + +# A target that does src = ./. (represented here by forcing ../src-dir) should +# depend on the directory tree, not on every child file. This keeps dependency +# discovery cheap for large source directories. +echo "Testing directory source tracking..." +srcdir_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:srcdir\" ]; })") +assert_jq "$srcdir_deps" '(."//areas/app/web:srcdir" | has("areas/app/web/src-dir"))' \ + "srcdir deps should include the source directory" +assert_jq "$srcdir_deps" '(."//areas/app/web:srcdir" | has("areas/app/web/src-dir/file-001.txt") | not) and (."//areas/app/web:srcdir" | has("areas/app/web/src-dir/file-002.txt") | not)' \ + "srcdir deps should not include child files under the source directory" + +# Import resolution through symlinks should track both the symlink and the file +# it resolves to, otherwise a symlink retarget could leave stale dependencies. +echo "Testing symlink import tracking..." +symlink_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:symlinked\" ]; })") +assert_jq "$symlink_deps" '(."//areas/app/web:symlinked" | has("areas/app/web/targets/current.nix") and has("areas/app/web/targets/alpha.nix"))' \ + "symlinked deps should include both the symlink and resolved file" + +# Prewarming normal target evaluation in the same EvalState populates import +# resolution caches without tracking. Dependency discovery must still use a +# tracking-safe cache domain for Tecnix paths so symlink resolution is recorded. +echo "Testing dependency tracking after untracked import-resolution cache warmup..." +prewarmed_symlink_deps=$(tecnix_eval_json_no_cache "let args = (($base_args) // { targets = [ \"//areas/app/web:symlinked\" ]; }); prewarm = builtins.tecnixTargets args; deps = builtins.deepSeq prewarm (tecnixTargetDependencyPathSet args); in deps") +assert_jq "$prewarmed_symlink_deps" '(."//areas/app/web:symlinked" | has("areas/app/web/targets/current.nix") and has("areas/app/web/targets/alpha.nix"))' \ + "dependency tracking should not reuse untracked import-resolution cache entries for Tecnix paths" + +# Negative path existence checks are dependencies too. If a target branches on an +# absent file, adding that file later must invalidate the dependency cache. +echo "Testing absent path dependency tracking..." +missing_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:optional\" ]; })") +assert_jq "$missing_deps" '."//areas/app/web:optional"."areas/app/web/targets/optional-marker.nix" == "absent"' \ + "optional deps should include the absent optional-marker path with an absent fingerprint" + +# Positive existence checks are also dependencies, even when the target does not +# read or import the file after checking for it. +echo "Testing existing path dependency tracking..." +existing_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:existsCheck\" ]; })") +assert_jq "$existing_deps" '(."//areas/app/web:existsCheck"."areas/app/web/targets/exists-marker" | startswith("git:"))' \ + "existsCheck deps should include the existing marker path fingerprint" + +# File type checks are dependencies too, even when the target does not read the +# file contents. +echo "Testing readFileType dependency tracking..." +file_type_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:fileTypeCheck\" ]; })") +assert_jq "$file_type_deps" '(."//areas/app/web:fileTypeCheck"."areas/app/web/targets/exists-marker" | startswith("git:"))' \ + "fileTypeCheck deps should include the typed marker path fingerprint" + +# Directory enumeration is a dependency on the directory tree. Changes under the +# directory should invalidate via the directory fingerprint, without tracking +# every child as a separate dependency. +echo "Testing readDir dependency tracking..." +read_dir_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:readDirCheck\" ]; })") +assert_jq "$read_dir_deps" '(."//areas/app/web:readDirCheck"."areas/app/web/src-dir" | startswith("git:"))' \ + "readDirCheck deps should include the enumerated directory fingerprint" +assert_jq "$read_dir_deps" '(."//areas/app/web:readDirCheck" | has("areas/app/web/src-dir/file-001.txt") | not) and (."//areas/app/web:readDirCheck" | has("areas/app/web/src-dir/file-002.txt") | not)' \ + "readDirCheck deps should not include every enumerated child" + +# readFile dependencies are not imports, so they specifically exercise the +# SourceAccessor read path and replay of sourceful non-import thunks. +echo "Testing readFile dependency tracking..." +read_file_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:readFileCheck\" ]; })") +assert_jq "$read_file_deps" '(."//areas/app/web:readFileCheck"."areas/app/web/targets/read-file-marker.txt" | startswith("git:"))' \ + "readFileCheck deps should include the read file fingerprint" + +# Two targets share one lazy builtins.readFile thunk from targets.nix. If the +# first target forces it, the second target must still get the read file through +# Value* provenance replay rather than physical I/O. +echo "Testing shared readFile replay across targets..." +shared_read_file_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:readFileSharedA\" \"//areas/app/web:readFileSharedB\" ]; })") +assert_target_dependency_paths "$shared_read_file_deps" "//areas/app/web:readFileSharedA" "shared readFile A deps should be exact" <<'EOF' +areas/app/web/shared-read-file.txt +areas/app/web/targets.nix +areas/app/web/targets/read-file-shared.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$shared_read_file_deps" "//areas/app/web:readFileSharedB" "shared readFile B deps should be exact" <<'EOF' +areas/app/web/shared-read-file.txt +areas/app/web/targets.nix +areas/app/web/targets/read-file-shared.nix +system/tectonix/resolve.nix +EOF + +# Shared sourceful thunks can return any type. These catch regressions that only +# replay string-valued source thunks by also reusing a bool and a list. +echo "Testing shared pathExists/readDir replay across targets..." +shared_sourceful_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:sharedExistsA\" \"//areas/app/web:sharedExistsB\" \"//areas/app/web:sharedReadDirA\" \"//areas/app/web:sharedReadDirB\" ]; })") +assert_target_dependency_paths "$shared_sourceful_deps" "//areas/app/web:sharedExistsA" "sharedExistsA deps should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/exists-marker +areas/app/web/targets/shared-exists.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$shared_sourceful_deps" "//areas/app/web:sharedExistsB" "sharedExistsB deps should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/exists-marker +areas/app/web/targets/shared-exists.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$shared_sourceful_deps" "//areas/app/web:sharedReadDirA" "sharedReadDirA deps should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/shared-read-dir.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$shared_sourceful_deps" "//areas/app/web:sharedReadDirB" "sharedReadDirB deps should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/shared-read-dir.nix +system/tectonix/resolve.nix +EOF + +# Multi-hop replay must flatten the whole AccessSet closure, not just the direct +# reused value. Both targets share closureOuter -> closureMiddle -> closureLeaf. +echo "Testing chained source closure replay across targets..." +closure_chain_deps=$(tecnix_eval_json_access_set_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:closureMiddleUser\" \"//areas/app/web:closureChainA\" \"//areas/app/web:closureChainB\" ]; })") +assert_target_dependency_paths "$closure_chain_deps" "//areas/app/web:closureMiddleUser" "closureMiddleUser deps should include the leaf closure" <<'EOF' +areas/app/web/closure-leaf.txt +areas/app/web/targets.nix +areas/app/web/targets/closure-middle.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$closure_chain_deps" "//areas/app/web:closureChainA" "closureChainA deps should include the whole replayed closure" <<'EOF' +areas/app/web/closure-leaf.txt +areas/app/web/targets.nix +areas/app/web/targets/closure-chain.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$closure_chain_deps" "//areas/app/web:closureChainB" "closureChainB deps should include the whole already-published closure" <<'EOF' +areas/app/web/closure-leaf.txt +areas/app/web/targets.nix +areas/app/web/targets/closure-chain.nix +system/tectonix/resolve.nix +EOF + +# A resolver-side lazy module import should stay attached only to the target +# branch that forces it. This mirrors the real-world oracle failure where a +# resolver module file leaked into an unrelated target. +echo "Testing resolver-side lazy module dependency isolation..." +resolver_module_deps=$(tecnix_eval_json_access_set_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:resolverModuleUser\" \"//areas/lib/shared:gamma\" ]; })") +assert_target_dependency_paths "$resolver_module_deps" "//areas/app/web:resolverModuleUser" "resolverModuleUser deps should be exact" <<'EOF' +areas/app/web/targets.nix +areas/app/web/targets/resolver-module-user.nix +system/tectonix/resolve.nix +system/tectonix/resolver-module.nix +EOF +assert_target_dependency_paths "$resolver_module_deps" "//areas/lib/shared:gamma" "gamma deps should stay isolated from resolver module" <<'EOF' +areas/lib/shared/common.nix +areas/lib/shared/targets.nix +areas/lib/shared/targets/gamma.nix +system/tectonix/resolve.nix +EOF + +# Access-set output is now the canonical dependency engine. The default builtin +# path and the explicit compatibility env knob must stay byte-for-byte equivalent +# for every edge-case target in this fixture, including imports, readFile, +# pathExists, readFileType, readDir, symlinked imports, directory materialization, +# and cross-zone targets. +echo "Testing default dependency output matches explicit access-set output..." +all_target_deps_expr="let args = $base_args; targets = builtins.tecnixTargetNames args; in tecnixTargetDependencyPathSet (args // { inherit targets; })" +default_all_deps=$(tecnix_eval_json_no_cache "$all_target_deps_expr") +access_set_all_deps=$(tecnix_eval_json_access_set_no_cache "$all_target_deps_expr") +if ! diff -u <(jq -S . <<< "$default_all_deps") <(jq -S . <<< "$access_set_all_deps"); then + fail "default dependency output should match explicit access-set output for all fixture targets" +fi +assert_clean_dependency_paths "$default_all_deps" +assert_clean_dependency_paths "$access_set_all_deps" + +# Correctness oracle: multi-target dependency output must equal isolated +# single-target output for every target. Run isolated targets in separate evals so +# this mirrors the production oracle instead of comparing against a second +# dependency pass in an already-warmed EvalState. +echo "Testing access-set multi-target vs isolated-target oracle..." +isolated_access_set_deps='{}' +while IFS= read -r target; do + single_target_deps=$(tecnix_eval_json_access_set_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"$target\" ]; })") + isolated_access_set_deps=$(jq -S -s '.[0] * .[1]' <(printf '%s' "$isolated_access_set_deps") <(printf '%s' "$single_target_deps")) +done <<< "$(jq -r '.[]' <<< "$target_names")" +if ! diff -u <(jq -S . <<< "$access_set_all_deps") <(jq -S . <<< "$isolated_access_set_deps"); then + fail "access-set multi-target deps should match isolated single-target deps for all fixture targets" +fi + +# The same oracle must hold when target dependency evaluation is parallelized. +# This protects against shared evaluator/cache provenance leaking dependencies +# between concurrently evaluated targets. +echo "Testing parallel access-set multi-target vs isolated-target oracle..." +parallel_all_deps=$(tecnix_eval_json_parallel_no_cache "$all_target_deps_expr") +if ! diff -u <(jq -S . <<< "$parallel_all_deps") <(jq -S . <<< "$isolated_access_set_deps"); then + fail "parallel access-set multi-target deps should match isolated single-target deps for all fixture targets" +fi + +# The dependencies returned alongside target values should exactly match the +# dependency wrapper used by the rest of this test. +echo "Testing access-set target dependency-record equivalence..." +access_set_record_deps=$(tecnix_eval_json_access_set_no_cache "let args = $base_args; targets = builtins.tecnixTargetNames args; combined = builtins.tecnixTargets (args // { inherit targets; includeDependencies = true; }); combinedDeps = builtins.listToAttrs (map (record: { name = record.target; value = record.dependencies; }) combined); deps = tecnixTargetDependencyPathSet (args // { inherit targets; }); in builtins.deepSeq combined (builtins.deepSeq deps { equal = combinedDeps == deps; })") +assert_jq "$access_set_record_deps" '.equal == true' \ + "access-set target dependency records should match the dependency wrapper" + +# Detached parallel prefetch must not leak into tracked closures, and tracked +# closures must be identical with and without the parallel executor. +# builtins.parallel's prefetch list and toJSON's deep-force fanout are skipped +# under tracking: tracking contexts are thread-confined, so tracked evaluation +# must not spawn parallel work. +echo "Testing detached prefetch under tracked evaluation..." +prefetch_targets='[ "//areas/app/web:parallelPrefetch" "//areas/app/web:jsonCheck" ]' +prefetch_seq_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = $prefetch_targets; })") +prefetch_par_deps=$(tecnix_eval_json_parallel_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = $prefetch_targets; })") +if ! diff -u <(jq -S . <<< "$prefetch_seq_deps") <(jq -S . <<< "$prefetch_par_deps"); then + fail "prefetch-using targets should have identical closures with and without the parallel executor" +fi +assert_target_dependency_paths "$prefetch_par_deps" "//areas/app/web:parallelPrefetch" "parallelPrefetch deps should exclude the prefetch-only path" <<'EOF' +areas/app/web/common.nix +areas/app/web/targets.nix +areas/app/web/targets/parallel-prefetch.nix +system/tectonix/resolve.nix +EOF +assert_target_dependency_paths "$prefetch_par_deps" "//areas/app/web:jsonCheck" "jsonCheck deps should be exact" <<'EOF' +areas/app/web/common.nix +areas/app/web/shared-read-file.txt +areas/app/web/targets.nix +areas/app/web/targets/json-check.nix +system/tectonix/resolve.nix +EOF +# Path-filter functions run arbitrary Nix code from inside NAR serialization, +# where dump-internal tracking is suppressed. The filter's own source reads +# (here: an ignore list readFile'd for the first time inside the filter) are +# real dependencies and must appear in the closure, as must the filtered +# source tree itself. +echo "Testing path filter dependency tracking..." +filtered_src_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:filteredSrc\" ]; })") +assert_target_dependency_paths "$filtered_src_deps" "//areas/app/web:filteredSrc" "filteredSrc deps should include the ignore list read inside the filter" <<'EOF' +areas/app/web/common.nix +areas/app/web/filter-ignore.txt +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/filtered-src.nix +system/tectonix/resolve.nix +EOF + +# A single-target query evaluates on the coordinator thread with the executor +# enabled, which is the configuration where toJSON's prefetch used to spawn. +prefetch_json_single=$(tecnix_eval_json_parallel_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:jsonCheck\" ]; })") +assert_target_dependency_paths "$prefetch_json_single" "//areas/app/web:jsonCheck" "single-target jsonCheck deps should be exact on the coordinator thread" <<'EOF' +areas/app/web/common.nix +areas/app/web/shared-read-file.txt +areas/app/web/targets.nix +areas/app/web/targets/json-check.nix +system/tectonix/resolve.nix +EOF + +# Repeated path materialization in one EvalState must not lose the source path +# after the srcToStore cache has been populated. +echo "Testing repeated path materialization dependencies..." +repeated_srcdir_deps=$(tecnix_eval_json_access_set_no_cache "let args = (($base_args) // { targets = [ \"//areas/app/web:srcdir\" ]; }); first = tecnixTargetDependencyPathSet args; second = builtins.deepSeq first (tecnixTargetDependencyPathSet args); in { inherit first second; }") +for pass in first second; do + assert_target_dependency_paths "$(jq -c ".$pass" <<< "$repeated_srcdir_deps")" "//areas/app/web:srcdir" "srcdir $pass dependency paths should be exact" <<'EOF' +areas/app/web/src-dir +areas/app/web/targets.nix +areas/app/web/targets/srcdir.nix +system/tectonix/resolve.nix +EOF +done + +# The directory-level dependency still needs to see dirty child files through +# the checkout overlay. +echo "Testing dirty child affects directory source target..." +clean_srcdir_drv=$(tecnix_eval_json_no_cache "(builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:srcdir\" ]; })).\"//areas/app/web:srcdir\".drvPath") +cat > "$TEST_WORLD/areas/app/web/src-dir/file-001.txt" << 'DIRTY_SRC_EOF' +dirty source file 001 +DIRTY_SRC_EOF +dirty_srcdir_drv=$(tecnix_eval_json_no_cache "(builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:srcdir\" ]; })).\"//areas/app/web:srcdir\".drvPath") +if [[ "$clean_srcdir_drv" == "$dirty_srcdir_drv" ]]; then + echo "clean drvPath: $clean_srcdir_drv" >&2 + echo "dirty drvPath: $dirty_srcdir_drv" >&2 + fail "dirty child under source directory should change the source derivation" +fi + +# Repeated dependency calls in one EvalState should not lose imports to the +# evaluator-local file cache. +echo "Testing repeated tecnixTargets includeDependencies calls..." +repeated_deps=$(tecnix_eval_json_no_cache "let args = (($base_args) // { targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; }); first = tecnixTargetDependencyPathSet args; second = tecnixTargetDependencyPathSet args; in { inherit first second; }") +assert_jq "$repeated_deps" '(.first."//areas/app/web:alpha" | has("system/tectonix/resolve.nix")) and (.second."//areas/app/web:alpha" | has("system/tectonix/resolve.nix"))' \ + "repeated dependency calls should keep resolver deps" +assert_jq "$repeated_deps" '(.first."//areas/app/web:beta" | has("areas/app/web/common.nix")) and (.second."//areas/app/web:beta" | has("areas/app/web/common.nix"))' \ + "repeated dependency calls should keep shared zone deps" + +# Sequential dependency calls for different targets in one EvalState should not +# let evaluator-local caches hide imports forced by the first target. The fixture's +# targets.nix has a shared lazy `common = import ./common.nix ...`; alpha forces it +# first, and beta must still record common.nix when evaluated afterwards. +echo "Testing shared lazy import tracking across dependency calls..." +sequential_deps=$(tecnix_eval_json_no_cache "let alphaArgs = (($base_args) // { targets = [ \"//areas/app/web:alpha\" ]; }); betaArgs = (($base_args) // { targets = [ \"//areas/app/web:beta\" ]; }); alpha = tecnixTargetDependencyPathSet alphaArgs; beta = builtins.seq alpha (tecnixTargetDependencyPathSet betaArgs); in { inherit alpha beta; }") +assert_jq "$sequential_deps" '(.alpha."//areas/app/web:alpha" | has("areas/app/web/common.nix")) and (.beta."//areas/app/web:beta" | has("areas/app/web/common.nix"))' \ + "shared lazy imports forced by one target should still be tracked for later targets" +assert_jq "$sequential_deps" '(.alpha."//areas/app/web:alpha" | has("areas/app/web/targets/alpha.nix")) and (.beta."//areas/app/web:beta" | has("areas/app/web/targets/beta.nix"))' \ + "sequential dependency calls should keep target-specific deps" +assert_jq "$sequential_deps" '(.alpha."//areas/app/web:alpha" | has("areas/app/web/targets/beta.nix") | not) and (.beta."//areas/app/web:beta" | has("areas/app/web/targets/alpha.nix") | not)' \ + "sequential dependency calls should not leak target-specific deps" + +# Dependency cache behavior: first invocation records dependency fingerprints; +# the second invocation can reuse them without re-running target resolution. +echo "Testing dependency cache hit behavior through tecnixTargets includeDependencies..." +cache_test_system="cache-test-system-$$" +cache_expr=$(rewrite_tecnix_test_expr "tecnixTargetDependencyPathSet (($base_args) // { args = { system = \"$cache_test_system\"; }; targets = [ \"//areas/app/web:alpha\" ]; })") +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache miss, evaluating '//areas/app/web:alpha'" +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:alpha'" + +# A dependency-cache hit is not a target-value cache hit: with includeTargets +# (the default), a warm invocation must still resolve fresh target values while +# reusing the cached dependency closures, sequentially and in parallel. +echo "Testing target values alongside a dependency cache hit..." +warm_values_system="cache-values-system-$$" +warm_values_expr=$(rewrite_tecnix_test_expr "builtins.tecnixTargets (($base_args) // { args = { system = \"$warm_values_system\"; }; targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; includeDependencies = true; })") +nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$warm_values_expr" \ + > "$TEST_ROOT/cache-values-cold.json" 2> "$TEST_ROOT/cache-values-cold.err" +grepQuiet "tecnixTargets dependencies: dependency cache miss, evaluating '//areas/app/web:alpha'" "$TEST_ROOT/cache-values-cold.err" +nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$warm_values_expr" \ + > "$TEST_ROOT/cache-values-warm.json" 2> "$TEST_ROOT/cache-values-warm.err" +grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:alpha'" "$TEST_ROOT/cache-values-warm.err" +grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:beta'" "$TEST_ROOT/cache-values-warm.err" +if ! diff -u <(jq -S . "$TEST_ROOT/cache-values-cold.json") <(jq -S . "$TEST_ROOT/cache-values-warm.json"); then + fail "warm tecnixTargets output (values + dependencies) should match the cold output" +fi +assert_jq "$(cat "$TEST_ROOT/cache-values-warm.json")" '(.[0].value.drvPath | endswith("-alpha-clean-web-common.drv")) and (.[1].value.drvPath | endswith("-beta-clean-web-common.drv"))' \ + "warm tecnixTargets should resolve target values after a dependency cache hit" +nix eval --json --verbose \ + --extra-experimental-features 'nix-command parallel-eval' \ + --eval-cores 2 \ + --option lazy-trees true \ + --expr "$warm_values_expr" \ + > "$TEST_ROOT/cache-values-warm-parallel.json" 2> "$TEST_ROOT/cache-values-warm-parallel.err" +grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:alpha'" "$TEST_ROOT/cache-values-warm-parallel.err" +if ! diff -u <(jq -S . "$TEST_ROOT/cache-values-cold.json") <(jq -S . "$TEST_ROOT/cache-values-warm-parallel.json"); then + fail "parallel warm tecnixTargets output should match the cold output" +fi + +# Missing files have stable fingerprints in the dependency cache. The missing +# state can be cached, but creating the file must invalidate that cache row and +# re-evaluate the target. +echo "Testing dependency cache invalidation for newly-created files..." +optional_cache_expr=$(rewrite_tecnix_test_expr "tecnixTargetDependencyPathSet (($base_args) // { args = { system = \"$cache_test_system\"; }; targets = [ \"//areas/app/web:optional\" ]; })") +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$optional_cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache miss, evaluating '//areas/app/web:optional'" +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$optional_cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:optional'" +cat > "$TEST_WORLD/areas/app/web/targets/optional-marker.nix" << 'OPTIONAL_MARKER_EOF' +"present" +OPTIONAL_MARKER_EOF +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$optional_cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache miss, evaluating '//areas/app/web:optional'" +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$optional_cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:optional'" +rm "$TEST_WORLD/areas/app/web/targets/optional-marker.nix" +# The bounded history keeps both the absent-file and present-file closures, so +# switching back to the absent state can hit immediately instead of thrashing. +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$optional_cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:optional'" +expectStderr 0 nix eval --json --verbose \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --expr "$optional_cache_expr" \ + | grepQuiet "tecnixTargets dependencies: dependency cache hit for '//areas/app/web:optional'" +cat > "$TEST_WORLD/areas/app/web/targets/optional-marker.nix" << 'OPTIONAL_MARKER_EOF' +"present" +OPTIONAL_MARKER_EOF +dirty_optional_targets=$(tecnix_eval_json_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:optional\" ]; })") +assert_jq "$dirty_optional_targets" '."//areas/app/web:optional".marker == "optional:present:clean-web-common:test-system"' \ + "tecnixTargets should see newly-created optional files from checkoutPath" + +# Git may report untracked directories coarsely unless asked for all untracked +# files. The dirty overlay must still see newly-created nested files. +echo "Testing untracked nested file overlay..." +clean_nested_targets=$(tecnix_eval_json_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:nestedOptional\" ]; })") +assert_jq "$clean_nested_targets" '."//areas/app/web:nestedOptional".marker == "nested:missing:clean-web-common:test-system"' \ + "nestedOptional should start with the nested marker missing" +mkdir -p "$TEST_WORLD/areas/app/web/untracked-dir" +cat > "$TEST_WORLD/areas/app/web/untracked-dir/nested-marker.nix" << 'NESTED_MARKER_EOF' +"present" +NESTED_MARKER_EOF +dirty_nested_targets=$(tecnix_eval_json_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:nestedOptional\" ]; })") +assert_jq "$dirty_nested_targets" '."//areas/app/web:nestedOptional".marker == "nested:present:clean-web-common:test-system"' \ + "tecnixTargets should see newly-created nested files from checkoutPath" +dirty_nested_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:nestedOptional\" ]; })") +assert_jq "$dirty_nested_deps" '(."//areas/app/web:nestedOptional"."areas/app/web/untracked-dir/nested-marker.nix" | contains("dirty="))' \ + "nestedOptional deps should include the newly-created nested marker path fingerprint" + +# Dirty zone worktree: the committed rev stays the same, but checkoutPath should +# overlay dirty zone files. +echo "Testing dirty zone worktree overlay..." +cat > "$TEST_WORLD/areas/app/web/common.nix" << 'COMMON_EOF' +{ system }: +{ + marker = "dirty-web-common"; + inherit system; +} +COMMON_EOF + +dirty_zone_targets=$(tecnix_eval_json_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; })") +assert_jq "$dirty_zone_targets" '."//areas/app/web:alpha".marker == "alpha:dirty-web-common:test-system" and ."//areas/app/web:beta".marker == "beta:dirty-web-common:test-system"' \ + "tecnixTargets should see dirty zone source files from checkoutPath" + +dirty_zone_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:alpha\" \"//areas/app/web:beta\" ]; })") +assert_jq "$dirty_zone_deps" '(."//areas/app/web:alpha"."areas/app/web/common.nix" | contains("dirty=")) and (."//areas/app/web:beta"."areas/app/web/common.nix" | contains("dirty="))' \ + "tecnixTargets includeDependencies should track dirty zone files with repo-relative fingerprints" + +# Dirty resolver worktree: the resolver is outside a zone, so this exercises the +# repo-wide dirty overlay used for the resolver path. +echo "Testing dirty resolver overlay..." +sed -i "$TEST_WORLD/system/tectonix/resolve.nix" -e 's/resolvedBy = "clean-resolver";/resolvedBy = "dirty-resolver";/' + +dirty_resolver_targets=$(tecnix_eval_json_no_cache "builtins.tecnixTargets (($base_args) // { targets = [ \"//areas/app/web:alpha\" ]; })") +assert_jq "$dirty_resolver_targets" '."//areas/app/web:alpha".resolvedBy == "dirty-resolver" and ."//areas/app/web:alpha".marker == "alpha:dirty-web-common:test-system"' \ + "tecnixTargets should see dirty resolver files from checkoutPath" + +dirty_resolver_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($base_args) // { targets = [ \"//areas/app/web:alpha\" ]; })") +assert_jq "$dirty_resolver_deps" '(."//areas/app/web:alpha"."system/tectonix/resolve.nix" | contains("dirty="))' \ + "tecnixTargets includeDependencies should track the resolver path fingerprint when the resolver is dirty" + +dirty_target_name_deps=$(tecnix_eval_json_no_cache "tecnixTargetNameDependencyPathSet ($base_args)") +assert_jq "$dirty_target_name_deps" '(."system/tectonix/resolve.nix" | contains("dirty="))' \ + "tecnixTargetNames includeDependencies should track the resolver path fingerprint when the resolver is dirty" + +# ============================================================ +# Eval cache: enabled end-to-end roundtrip +# ============================================================ +# The persistent cache engages only under pure evaluation; these tests use an +# isolated XDG_CACHE_HOME and a committed-only world (no checkoutPath). + +echo "Testing eval cache roundtrip..." + +EVAL_CACHE_HOME="$TEST_ROOT/tecnix-eval-cache-home" + +tecnix_eval_json_cache() { + local expr + expr=$(rewrite_tecnix_test_expr "$1") + XDG_CACHE_HOME="$EVAL_CACHE_HOME" nix eval --json -v \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tecnix-eval-cache true \ + --pure-eval \ + --expr "$expr" +} + +CACHE_WORLD="$TEST_ROOT/tecnix-cache-world" +createGitRepo "$CACHE_WORLD" +( + cd "$CACHE_WORLD" + mkdir deps + echo "alpha dep" > deps/alpha.txt + echo "beta dep" > deps/beta.txt + cat > resolve.nix << 'RESOLVE_EOF' +args: { + allTargetNames = [ "alpha" "beta" ]; + resolve = id: { + drvPath = "/nix/store/00000000000000000000000000000000-${builtins.hashString "sha256" (builtins.readFile (./deps + "/${id}.txt"))}-${id}.drv"; + }; +} +RESOLVE_EOF + git add -A + git commit -m "cache world" +) +CACHE_HEAD=$(get_head_sha "$CACHE_WORLD") + +cache_args="{ gitDir = \"$CACHE_WORLD/.git\"; resolver = \"resolve.nix\"; args = { system = \"test-system\"; }; rev = \"$CACHE_HEAD\"; }" + +cold_deps=$(tecnix_eval_json_cache "tecnixTargetDependencyPathSet (($cache_args) // { targets = [ \"alpha\" \"beta\" ]; })" 2> "$TEST_ROOT/cache-cold.err") +warm_deps=$(tecnix_eval_json_cache "tecnixTargetDependencyPathSet (($cache_args) // { targets = [ \"alpha\" \"beta\" ]; })" 2> "$TEST_ROOT/cache-warm.err") +assert_json_equal "$warm_deps" "$cold_deps" "warm dependency query should equal cold" +assert_jq "$warm_deps" '.alpha | has("deps/alpha.txt")' "cached dependency sets should contain the per-target dep" +grepQuiet "dependency cache hit" "$TEST_ROOT/cache-warm.err" +test -f "$EVAL_CACHE_HOME/nix/tecnix-eval-cache-v1.sqlite" + +cold_names=$(tecnix_eval_json_cache "builtins.tecnixTargetNames ($cache_args)" 2> /dev/null) +warm_names=$(tecnix_eval_json_cache "builtins.tecnixTargetNames ($cache_args)" 2> "$TEST_ROOT/cache-warm-names.err") +assert_json_equal "$warm_names" "$cold_names" "warm discovery should equal cold" +grepQuiet "discovery cache hit" "$TEST_ROOT/cache-warm-names.err" + +# ============================================================ +# Raw-tree contract: git attributes do not filter the Tecnix view +# ============================================================ +# The clean backend serves the raw committed tree. An export-ignore rule must +# not hide a file from evaluation: fingerprints describe raw tree objects, so +# a view filtered by .gitattributes would make attribute state an input to +# evaluation that no closure entry certifies — an attribute change could hide +# a file whose blob oid still matches and validate a stale cached result. + +EXPORT_IGNORE_WORLD="$TEST_ROOT/tecnix-export-ignore-world" +createGitRepo "$EXPORT_IGNORE_WORLD" +( + cd "$EXPORT_IGNORE_WORLD" + echo "secret contents" > secret.txt + echo "secret.txt export-ignore" > .gitattributes + cat > resolve.nix << 'RESOLVE_EOF' +args: { + allTargetNames = [ "reader" ]; + resolve = id: { + drvPath = "/nix/store/00000000000000000000000000000000-${builtins.hashString "sha256" (builtins.readFile ./secret.txt)}-${id}.drv"; + }; +} +RESOLVE_EOF + git add -A + git commit -m "export-ignore world" +) +EXPORT_IGNORE_HEAD=$(get_head_sha "$EXPORT_IGNORE_WORLD") + +export_ignore_args="{ gitDir = \"$EXPORT_IGNORE_WORLD/.git\"; resolver = \"resolve.nix\"; args = { }; rev = \"$EXPORT_IGNORE_HEAD\"; }" + +export_ignore_deps=$(tecnix_eval_json_no_cache "tecnixTargetDependencyPathSet (($export_ignore_args) // { targets = [ \"reader\" ]; })") +assert_jq "$export_ignore_deps" '.reader | has("secret.txt")' \ + "a file with an export-ignore attribute must stay visible to Tecnix evaluation and appear in the closure" +assert_jq "$export_ignore_deps" '.reader."secret.txt" | startswith("git:")' \ + "an export-ignored file should carry an ordinary raw-tree git fingerprint" +assert_jq "$export_ignore_deps" '.reader | has(".gitattributes") | not' \ + "git attributes are inert in the raw-tree view and should not enter the closure" + +echo "Tecnix builtin tests passed!" diff --git a/tests/functional/tecnix/common.sh b/tests/functional/tecnix/common.sh new file mode 100644 index 0000000000..3e6edc6b4f --- /dev/null +++ b/tests/functional/tecnix/common.sh @@ -0,0 +1,620 @@ +# shellcheck shell=bash + +# Common setup for tecnix functional tests + +set -eu -o pipefail + +if [[ -z "${TECNIX_COMMON_SH_SOURCED-}" ]]; then + +TECNIX_COMMON_SH_SOURCED=1 + +# Source the main test framework +source "$(dirname "${BASH_SOURCE[0]}")/../common.sh" + +requireGit + +# Create a minimal repo for testing the public tecnix builtins. +create_tecnix_builtin_test_world() { + local dir="$1" + + git init "$dir" + cd "$dir" + + mkdir -p .meta + mkdir -p system/tectonix + mkdir -p areas/app/web/targets + mkdir -p areas/lib/shared/targets + + cat > .meta/manifest.json << 'MANIFEST_EOF' +{ + "//areas/app/web": { "id": "W-100001" }, + "//areas/lib/shared": { "id": "W-100002" } +} +MANIFEST_EOF + + cat > system/tectonix/resolve.nix << 'RESOLVE_EOF' +{ system ? builtins.currentSystem }: +let + parseTarget = target: + let match = builtins.match "(//.*):(.*)" target; + in if match == null then throw "invalid target: ${target}" else { + zone = builtins.elemAt match 0; + name = builtins.elemAt match 1; + }; + + repoRoot = ../..; + zoneSource = zonePath: + let + relPath = + if builtins.match "//.*" zonePath != null + then builtins.substring 2 (builtins.stringLength zonePath) zonePath + else throw "invalid zone path: ${zonePath}"; + in repoRoot + "/${relPath}"; + + loadTarget = target: + let + parsed = parseTarget target; + src = zoneSource parsed.zone; + targets = import (src + "/targets.nix") { inherit system; }; + in targets.${parsed.name}; + + resolverModule = import ./resolver-module.nix; + manifest = builtins.fromJSON (builtins.readFile ../../.meta/manifest.json); + + copyOnlyText = builtins.replaceStrings [ "\n" ] [ "" ] (builtins.readFile ./copy-only-marker.txt); + copyOnlyTargets = { + copyOnlyPrewarm = { + name = "copyOnlyPrewarm"; + marker = "copyOnlyPrewarm:${copyOnlyText}:${system}"; + drvPath = "/nix/store/00000000000000000000000000000000-copy-only-prewarm-${copyOnlyText}.drv"; + }; + copyOnlyConsumer = { + name = "copyOnlyConsumer"; + marker = "copyOnlyConsumer:${builtins.break copyOnlyText}:${system}"; + drvPath = "/nix/store/00000000000000000000000000000000-copy-only-consumer-${builtins.break copyOnlyText}.drv"; + }; + }; +in +{ + resolve = target: + let + parsed = parseTarget target; + base = if parsed.zone == "//system/tectonix-copy-value" then copyOnlyTargets.${parsed.name} else loadTarget target; + moduleAttrs = if parsed.name == "resolverModuleUser" then { + resolverModule = resolverModule.marker; + drvPath = "/nix/store/00000000000000000000000000000000-resolver-module-user-${resolverModule.marker}.drv"; + } else { }; + in base // moduleAttrs // { + target = target; + resolvedBy = "clean-resolver"; + }; + + allTargetNames = builtins.seq (builtins.attrNames manifest) [ + "//areas/app/web:alpha" + "//areas/app/web:beta" + "//areas/app/web:srcdir" + "//areas/app/web:symlinked" + "//areas/app/web:optional" + "//areas/app/web:existsCheck" + "//areas/app/web:fileTypeCheck" + "//areas/app/web:readDirCheck" + "//areas/app/web:readFileCheck" + "//areas/app/web:readFileSharedA" + "//areas/app/web:readFileSharedB" + "//areas/app/web:sharedExistsA" + "//areas/app/web:sharedExistsB" + "//areas/app/web:sharedReadDirA" + "//areas/app/web:sharedReadDirB" + "//areas/app/web:closureMiddleUser" + "//areas/app/web:closureChainA" + "//areas/app/web:closureChainB" + "//areas/app/web:resolverModuleUser" + "//areas/app/web:nestedOptional" + "//areas/app/web:treeShaCheck" + "//areas/lib/shared:gamma" + ]; +} +RESOLVE_EOF + + cat > system/tectonix/resolver-module.nix << 'RESOLVER_MODULE_EOF' +{ + marker = "clean-resolver-module"; +} +RESOLVER_MODULE_EOF + + cat > system/tectonix/copy-only-marker.txt << 'COPY_ONLY_MARKER_EOF' +clean-copy-only +COPY_ONLY_MARKER_EOF + + mkdir -p system/repo-root-read-dir + cat > system/repo-root-read-dir/resolve.nix << 'ROOT_READ_DIR_RESOLVE_EOF' +{ system ? builtins.currentSystem }: +let + rootEntries = builtins.attrNames (builtins.readDir ../..); + rootEntriesSlug = builtins.concatStringsSep "-" rootEntries; +in +{ + allTargetNames = [ "//repo:rootReadDir" ]; + resolve = target: { + name = "rootReadDir"; + inherit target system; + drvPath = "/nix/store/00000000000000000000000000000000-root-read-dir-${rootEntriesSlug}.drv"; + }; +} +ROOT_READ_DIR_RESOLVE_EOF + + cat > areas/app/web/zone.nix << 'ZONE_EOF' +{ } +ZONE_EOF + + cat > areas/app/web/common.nix << 'COMMON_EOF' +{ system }: +{ + marker = "clean-web-common"; + inherit system; +} +COMMON_EOF + + mkdir -p areas/app/web/src-dir + cat > areas/app/web/src-dir/file-001.txt << 'SRC_EOF' +clean source file 001 +SRC_EOF + cat > areas/app/web/src-dir/file-002.txt << 'SRC_EOF' +clean source file 002 +SRC_EOF + # Symlinks inside materialized trees are serialized via readLink during + # dumpPath; like file reads, they are covered by the directory-level + # fingerprint and must not add per-symlink closure entries. + ln -s file-001.txt areas/app/web/src-dir/link-001 + cat > areas/app/web/shared-read-file.txt << 'READ_FILE_EOF' +clean-shared-read-file-text +READ_FILE_EOF + cat > areas/app/web/closure-leaf.txt << 'CLOSURE_LEAF_EOF' +clean-closure-leaf-text +CLOSURE_LEAF_EOF + cat > areas/app/web/owner-collision-a.txt << 'OWNER_COLLISION_A_EOF' +clean-owner-collision-a +OWNER_COLLISION_A_EOF + cat > areas/app/web/owner-collision-b.txt << 'OWNER_COLLISION_B_EOF' +clean-owner-collision-b +OWNER_COLLISION_B_EOF + cat > areas/app/web/shared-owner.txt << 'SHARED_OWNER_EOF' +clean-shared-owner +SHARED_OWNER_EOF + cat > areas/app/web/shared-builder-marker.txt << 'SHARED_BUILDER_MARKER_EOF' +clean-shared-builder +SHARED_BUILDER_MARKER_EOF + + cat > areas/app/web/parallel-only-marker.txt << 'PARALLEL_ONLY_MARKER_EOF' +clean-parallel-only-marker +PARALLEL_ONLY_MARKER_EOF + + cat > areas/app/web/filter-ignore.txt << 'FILTER_IGNORE_EOF' +file-002.txt +FILTER_IGNORE_EOF + + cat > areas/app/web/shared-builder.nix << 'SHARED_BUILDER_EOF' +{ common }: +let + marker = builtins.replaceStrings [ "\n" ] [ "" ] (builtins.readFile ./shared-builder-marker.txt); +in +{ + value = "${marker}:${common.marker}"; +} +SHARED_BUILDER_EOF + + cat > areas/app/web/targets.nix << 'TARGETS_EOF' +{ system }: +let + common = import ./common.nix { inherit system; }; + sharedReadFile = builtins.readFile ./shared-read-file.txt; + sharedExists = builtins.pathExists ./targets/exists-marker; + sharedDirEntries = builtins.attrNames (builtins.readDir ./src-dir); + closureLeaf = builtins.readFile ./closure-leaf.txt; + closureMiddle = builtins.replaceStrings [ "\n" ] [ "" ] closureLeaf; + closureOuter = "${closureMiddle}"; + sharedOwnerText = builtins.tecnixInternalSourceDepsScope (builtins.readFile ./shared-owner.txt); + sharedBuilder = import ./shared-builder.nix { inherit common; }; + callPackage = f: args: f (args // { inherit common; }); + sharedCallPackageBuilder = callPackage (import ./shared-builder.nix) { }; +in +{ + alpha = import ./targets/alpha.nix { inherit common system; }; + beta = import ./targets/beta.nix { inherit common system; }; + srcdir = import ./targets/srcdir.nix { inherit system; }; + symlinked = import ./targets/current.nix { inherit common system; }; + optional = import ./targets/optional.nix { inherit common system; }; + existsCheck = import ./targets/exists-check.nix { inherit common system; }; + fileTypeCheck = import ./targets/file-type-check.nix { inherit common system; }; + readDirCheck = import ./targets/read-dir-check.nix { inherit common system; }; + readFileCheck = import ./targets/read-file-check.nix { inherit common system; }; + readFileSharedA = import ./targets/read-file-shared.nix { targetName = "readFileSharedA"; drvName = "read-file-shared-a"; inherit common sharedReadFile system; }; + readFileSharedB = import ./targets/read-file-shared.nix { targetName = "readFileSharedB"; drvName = "read-file-shared-b"; inherit common sharedReadFile system; }; + sharedExistsA = import ./targets/shared-exists.nix { targetName = "sharedExistsA"; drvName = "shared-exists-a"; inherit common sharedExists system; }; + sharedExistsB = import ./targets/shared-exists.nix { targetName = "sharedExistsB"; drvName = "shared-exists-b"; inherit common sharedExists system; }; + sharedReadDirA = import ./targets/shared-read-dir.nix { targetName = "sharedReadDirA"; drvName = "shared-read-dir-a"; inherit common sharedDirEntries system; }; + sharedReadDirB = import ./targets/shared-read-dir.nix { targetName = "sharedReadDirB"; drvName = "shared-read-dir-b"; inherit common sharedDirEntries system; }; + closureMiddleUser = import ./targets/closure-middle.nix { targetName = "closureMiddleUser"; drvName = "closure-middle"; inherit closureMiddle common system; }; + closureChainA = import ./targets/closure-chain.nix { targetName = "closureChainA"; drvName = "closure-chain-a"; inherit closureOuter common system; }; + closureChainB = import ./targets/closure-chain.nix { targetName = "closureChainB"; drvName = "closure-chain-b"; inherit closureOuter common system; }; + resolverModuleUser = import ./targets/resolver-module-user.nix { inherit common system; }; + nestedOptional = import ./targets/nested-optional.nix { inherit common system; }; + parallelPrefetch = import ./targets/parallel-prefetch.nix { inherit common system; }; + jsonCheck = import ./targets/json-check.nix { inherit common system; }; + filteredSrc = import ./targets/filtered-src.nix { inherit common system; }; + treeShaCheck = import ./targets/tree-sha-check.nix { inherit common system; treeSha = builtins.unsafeTectonixInternalTreeSha "//areas/lib/shared"; }; + ownerCollisionA = import ./targets/owner-collision.nix { + targetName = "ownerCollisionA"; + drvName = "owner-collision-a"; + ownerText = builtins.tecnixInternalSourceDepsScope (builtins.readFile ./owner-collision-a.txt); + inherit common system; + }; + ownerCollisionB = import ./targets/owner-collision.nix { + targetName = "ownerCollisionB"; + drvName = "owner-collision-b"; + ownerText = builtins.tecnixInternalSourceDepsScope (builtins.readFile ./owner-collision-b.txt); + inherit common system; + }; + sharedOwnerA = import ./targets/shared-owner.nix { targetName = "sharedOwnerA"; drvName = "shared-owner-a"; inherit common sharedOwnerText system; }; + sharedOwnerB = import ./targets/shared-owner.nix { targetName = "sharedOwnerB"; drvName = "shared-owner-b"; inherit common sharedOwnerText system; }; + sharedBuilderA = import ./targets/shared-builder-user.nix { targetName = "sharedBuilderA"; drvName = "shared-builder-a"; inherit common sharedBuilder system; }; + sharedBuilderB = import ./targets/shared-builder-user.nix { targetName = "sharedBuilderB"; drvName = "shared-builder-b"; inherit common sharedBuilder system; }; + indirectBuilderA = import ./targets/indirect-builder-user.nix { targetName = "indirectBuilderA"; drvName = "indirect-builder-a"; inherit common system; }; + indirectBuilderB = import ./targets/indirect-builder-user.nix { targetName = "indirectBuilderB"; drvName = "indirect-builder-b"; inherit common system; }; + callPackageBuilderA = import ./targets/shared-builder-user.nix { targetName = "callPackageBuilderA"; drvName = "callpackage-builder-a"; common = common; sharedBuilder = sharedCallPackageBuilder; inherit system; }; + callPackageBuilderB = import ./targets/shared-builder-user.nix { targetName = "callPackageBuilderB"; drvName = "callpackage-builder-b"; common = common; sharedBuilder = sharedCallPackageBuilder; inherit system; }; +} +TARGETS_EOF + + cat > areas/app/web/targets/alpha.nix << 'ALPHA_EOF' +{ common, system }: +{ + name = "alpha"; + marker = "alpha:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-alpha-${common.marker}.drv"; +} +ALPHA_EOF + + cat > areas/app/web/targets/beta.nix << 'BETA_EOF' +{ common, system }: +{ + name = "beta"; + marker = "beta:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-beta-${common.marker}.drv"; +} +BETA_EOF + + cat > areas/app/web/targets/srcdir.nix << 'SRCDIR_EOF' +{ system }: +derivation { + name = "srcdir"; + inherit system; + builder = "/bin/sh"; + args = [ "-c" "echo unused > $out" ]; + src = ../src-dir; +} +SRCDIR_EOF + + ln -s alpha.nix areas/app/web/targets/current.nix + + cat > areas/app/web/targets/optional.nix << 'OPTIONAL_EOF' +{ common, system }: +let + optional = if builtins.pathExists ./optional-marker.nix then import ./optional-marker.nix else "missing"; +in +{ + name = "optional"; + marker = "optional:${optional}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-optional-${optional}.drv"; +} +OPTIONAL_EOF + + cat > areas/app/web/targets/parallel-prefetch.nix << 'PARALLEL_PREFETCH_EOF' +{ common, system }: +let + # builtins.parallel only exists when the parallel-eval experimental feature + # is enabled. Its first argument is prefetch-only: the result never consumes + # it, so it must never appear in this target's source closure. + parallel = xs: x: if builtins ? parallel then builtins.parallel xs x else x; +in +{ + name = "parallelPrefetch"; + # The target's forced result (drvPath) goes through builtins.parallel so the + # prefetch list is offered during tracked evaluation. + drvPath = parallel [ (builtins.readFile ../parallel-only-marker.txt) ] + "/nix/store/00000000000000000000000000000000-parallel-prefetch-${common.marker}.drv"; +} +PARALLEL_PREFETCH_EOF + + cat > areas/app/web/targets/filtered-src.nix << 'FILTERED_SRC_EOF' +{ common, system }: +let + # The ignore list is read (readFile, not import: imports are also covered by + # the parse-cache replay hook) for the first time from inside the path + # filter, which runs during NAR serialization where dump-internal tracking + # is suppressed. It is a real dependency and must appear in the closure. + ignored = builtins.replaceStrings [ "\n" ] [ "" ] (builtins.readFile ../filter-ignore.txt); + src = builtins.path { + path = ../src-dir; + name = "filtered-src"; + filter = path: type: baseNameOf path != ignored; + }; +in +{ + name = "filteredSrc"; + drvPath = builtins.seq src "/nix/store/00000000000000000000000000000000-filtered-src-${common.marker}.drv"; +} +FILTERED_SRC_EOF + + cat > areas/app/web/targets/json-check.nix << 'JSON_CHECK_EOF' +{ common, system }: +let + # builtins.toJSON deep-forces its argument; with a parallel executor this + # prefetches attribute forcing, which must not spawn under tracking. + json = builtins.toJSON { + marker = common.marker; + shared = builtins.readFile ../shared-read-file.txt; + }; +in +{ + name = "jsonCheck"; + # Forced via drvPath so the toJSON deep-force runs under tracking. + drvPath = "/nix/store/00000000000000000000000000000000-json-check-${builtins.hashString "sha256" json}.drv"; +} +JSON_CHECK_EOF + + touch areas/app/web/targets/exists-marker + cat > areas/app/web/targets/exists-check.nix << 'EXISTS_CHECK_EOF' +{ common, system }: +let + exists = if builtins.pathExists ./exists-marker then "present" else "missing"; +in +{ + name = "existsCheck"; + marker = "exists:${exists}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-exists-${exists}.drv"; +} +EXISTS_CHECK_EOF + + cat > areas/app/web/targets/file-type-check.nix << 'FILE_TYPE_CHECK_EOF' +{ common, system }: +let + fileType = builtins.readFileType ./exists-marker; +in +{ + name = "fileTypeCheck"; + marker = "fileType:${fileType}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-file-type-${fileType}.drv"; +} +FILE_TYPE_CHECK_EOF + + cat > areas/app/web/targets/read-dir-check.nix << 'READ_DIR_CHECK_EOF' +{ common, system }: +let + entries = builtins.attrNames (builtins.readDir ../src-dir); +in +{ + name = "readDirCheck"; + marker = "readDir:${builtins.concatStringsSep "," entries}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-read-dir-${builtins.concatStringsSep "-" entries}.drv"; +} +READ_DIR_CHECK_EOF + + cat > areas/app/web/targets/read-file-marker.txt << 'READ_FILE_MARKER_EOF' +clean-read-file-text +READ_FILE_MARKER_EOF + + cat > areas/app/web/targets/read-file-check.nix << 'READ_FILE_CHECK_EOF' +{ common, system }: +let + text = builtins.replaceStrings [ "\n" ] [ "" ] (builtins.readFile ./read-file-marker.txt); +in +{ + name = "readFileCheck"; + marker = "readFile:${text}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-read-file-${text}.drv"; +} +READ_FILE_CHECK_EOF + + cat > areas/app/web/targets/read-file-shared.nix << 'READ_FILE_SHARED_EOF' +{ common, drvName, sharedReadFile, system, targetName }: +let + text = builtins.replaceStrings [ "\n" ] [ "" ] sharedReadFile; +in +{ + name = targetName; + marker = "${targetName}:${text}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${text}.drv"; +} +READ_FILE_SHARED_EOF + + cat > areas/app/web/targets/shared-exists.nix << 'SHARED_EXISTS_EOF' +{ common, drvName, sharedExists, system, targetName }: +let + exists = if sharedExists then "present" else "missing"; +in +{ + name = targetName; + marker = "${targetName}:${exists}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${exists}.drv"; +} +SHARED_EXISTS_EOF + + cat > areas/app/web/targets/shared-read-dir.nix << 'SHARED_READ_DIR_EOF' +{ common, drvName, sharedDirEntries, system, targetName }: +let + entries = builtins.concatStringsSep "-" sharedDirEntries; +in +{ + name = targetName; + marker = "${targetName}:${entries}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${entries}.drv"; +} +SHARED_READ_DIR_EOF + + cat > areas/app/web/targets/closure-middle.nix << 'CLOSURE_MIDDLE_EOF' +{ closureMiddle, common, drvName, system, targetName }: +{ + name = targetName; + marker = "${targetName}:${closureMiddle}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${closureMiddle}.drv"; +} +CLOSURE_MIDDLE_EOF + + cat > areas/app/web/targets/closure-chain.nix << 'CLOSURE_CHAIN_EOF' +{ closureOuter, common, drvName, system, targetName }: +{ + name = targetName; + marker = "${targetName}:${closureOuter}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${closureOuter}.drv"; +} +CLOSURE_CHAIN_EOF + + cat > areas/app/web/targets/resolver-module-user.nix << 'RESOLVER_MODULE_USER_EOF' +{ common, system }: +{ + name = "resolverModuleUser"; + marker = "resolverModuleUser:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-resolver-module-user-${common.marker}.drv"; +} +RESOLVER_MODULE_USER_EOF + + cat > areas/app/web/targets/owner-collision.nix << 'OWNER_COLLISION_EOF' +{ common, system, targetName, drvName, ownerText }: +let + cleanOwnerText = builtins.replaceStrings [ "\n" ] [ "" ] ownerText; +in +{ + name = targetName; + marker = "${targetName}:${cleanOwnerText}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${cleanOwnerText}-${common.marker}.drv"; +} +OWNER_COLLISION_EOF + + cat > areas/app/web/targets/shared-owner.nix << 'SHARED_OWNER_TARGET_EOF' +{ common, system, targetName, drvName, sharedOwnerText }: +let + cleanSharedOwnerText = builtins.replaceStrings [ "\n" ] [ "" ] sharedOwnerText; +in +{ + name = targetName; + marker = "${targetName}:${cleanSharedOwnerText}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${cleanSharedOwnerText}-${common.marker}.drv"; +} +SHARED_OWNER_TARGET_EOF + + cat > areas/app/web/targets/shared-builder-user.nix << 'SHARED_BUILDER_USER_EOF' +{ common, system, targetName, drvName, sharedBuilder }: +{ + name = targetName; + marker = "${targetName}:${sharedBuilder.value}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${sharedBuilder.value}-${common.marker}.drv"; +} +SHARED_BUILDER_USER_EOF + + cat > areas/app/web/targets/indirect-builder-user.nix << 'INDIRECT_BUILDER_USER_EOF' +{ common, system, targetName, drvName }: +let + sharedBuilder = import ../shared-builder.nix { inherit common; }; +in +{ + name = targetName; + marker = "${targetName}:${sharedBuilder.value}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-${drvName}-${sharedBuilder.value}-${common.marker}.drv"; +} +INDIRECT_BUILDER_USER_EOF + + cat > areas/app/web/targets/nested-optional.nix << 'NESTED_OPTIONAL_EOF' +{ common, system }: +let + optional = if builtins.pathExists ../untracked-dir/nested-marker.nix then "present" else "missing"; +in +{ + name = "nestedOptional"; + marker = "nested:${optional}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-nested-${optional}.drv"; +} +NESTED_OPTIONAL_EOF + + cat > areas/app/web/targets/tree-sha-check.nix << 'TREE_SHA_CHECK_EOF' +{ common, system, treeSha }: +{ + name = "treeShaCheck"; + marker = "treeSha:${treeSha}:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-tree-sha-${treeSha}.drv"; +} +TREE_SHA_CHECK_EOF + + cat > areas/lib/shared/zone.nix << 'ZONE_EOF' +{ } +ZONE_EOF + + cat > areas/lib/shared/common.nix << 'COMMON_EOF' +{ system }: +{ + marker = "clean-shared-common"; + inherit system; +} +COMMON_EOF + + cat > areas/lib/shared/targets.nix << 'TARGETS_EOF' +{ system }: +let + common = import ./common.nix { inherit system; }; +in +{ + gamma = import ./targets/gamma.nix { inherit common system; }; +} +TARGETS_EOF + + cat > areas/lib/shared/targets/gamma.nix << 'GAMMA_EOF' +{ common, system }: +{ + name = "gamma"; + marker = "gamma:${common.marker}:${system}"; + common = common.marker; + drvPath = "/nix/store/00000000000000000000000000000000-gamma-${common.marker}.drv"; +} +GAMMA_EOF + + git config user.email "test@example.com" + git config user.name "Test User" + + mkdir -p .git/info + cat > .git/info/sparse-checkout-roots << 'SPARSE_EOF' +W-100001 +W-100002 +SPARSE_EOF + + git add -A + git commit -m "Initial tecnix builtin test world" + + cd - > /dev/null +} + +# Get the HEAD SHA of a repo +get_head_sha() { + local dir="$1" + git -C "$dir" rev-parse HEAD +} + +fi # TECNIX_COMMON_SH_SOURCED diff --git a/tests/functional/tecnix/meson.build b/tests/functional/tecnix/meson.build new file mode 100644 index 0000000000..e3b79c0e47 --- /dev/null +++ b/tests/functional/tecnix/meson.build @@ -0,0 +1,8 @@ +suites += { + 'name' : 'tecnix', + 'deps' : [], + 'tests' : [ + 'builtins.sh', + ], + 'workdir' : meson.current_source_dir(), +}