feat(lockfile): record the feed set per target, and each package's origin - #244
feat(lockfile): record the feed set per target, and each package's origin#244mobileoverlord wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed functional issues in lockfile version migration (v7→v8 loading) and in save/merge semantics that can resurrect dropped feed records under stale-writer scenarios.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds per-target feed provenance recording to avocado.lock so builds can later show which configured feeds (and in what priority order) a target resolved against, addressing ambiguity when multiple feeds can satisfy the same name/version.
Changes:
- Bumps lockfile format to v8 and introduces
targets.<target>.feeds: Vec<LockedFeed>for per-target feed-set recording. - Extends feed resolution to retain a “configured locator” (pre container-rewrite) and exposes
ResolvedFeedSet::locked_feeds()to generate lockfile records. - Records/replaces the feed set in the lock once per target per invocation, and emits a user-visible message when the recorded set changes.
File summaries
| File | Description |
|---|---|
src/utils/lockfile.rs |
Adds v8 schema (LockedFeed, per-target feeds), merge semantics, and tests. |
src/utils/feeds.rs |
Tracks configured vs in-container locators and generates lockfile-ready feed records. |
src/utils/config.rs |
Records the resolved feed set into the lock during the per-invocation feed funnel and logs changes. |
Review details
Suppressed comments (2)
src/utils/lockfile.rs:1136
- The merge logic for TargetLocks.feeds can resurrect a feed that a later install removed: a stale in-memory LockFile with a non-empty feeds list will keep its old list and overwrite the newer on-disk list during save()'s merge. This violates the intended "replace, not merge" semantics for feed sets.
// Same rule as the snapshot: adopt the other side's record only when
// this side has none. An install replaces the set wholesale, so a
// merge must not resurrect feeds a later install dropped.
if self_target.feeds.is_empty() {
self_target.feeds = other_target.feeds;
}
src/utils/lockfile.rs:340
- This doc comment says on-disk feeds are recorded as a
file://URL, but the implementation intentionally records the configured locator (e.g.,./feed) rather than the in-container mount path. Updating the comment will prevent future confusion about what ends up in the lock.
/// The resolved URL, with `$releasever` and `$target` already expanded. A
/// `file://` path for an on-disk feed, and the `connect://<org>/...`
/// placeholder for a Connect feed — never the minted host, which changes per
/// build and is not an input.
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
c22b8f0 to
2ef4398
Compare
There was a problem hiding this comment.
🟡 Changes recommended
v7 lockfiles will not load/migrate correctly after the version bump due to missing v7 handling in the migration logic, breaking the stated v7→v8 compatibility.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/utils/lockfile.rs:49
- After bumping LOCKFILE_VERSION to 8, v7 lockfiles will parse successfully as the current struct but then fail migration:
parse_and_migrateonly treats versions 3/4/6 as “shape-compatible” and falls through to the legacy migration switch for v7, which returns "Unable to parse lock file format". This breaks the stated “v7 reads as v8” behavior and also prevents recording feeds into an existing v7 lock.
/// Version 7: Adds per-target `repo-snapshot`, the immutable channel snapshot
/// the target's packages were resolved against. Additive — v6
/// lockfiles read as v7 with `repo_snapshot: None` and behave
/// exactly as before (track the live channel head).
const LOCKFILE_VERSION: u32 = 8;
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
There is a confirmed v7→v8 lockfile load regression (migration fallthrough) plus merge/clearing semantics for per-target feeds that can resurrect stale records and prevent proper replacement when feeds are removed/disabled.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
src/utils/lockfile.rs:340
LockedFeed::url’s doc comment says on-disk feeds are recorded as afile://URL, butResolvedFeedSet::locked_feeds()records the configured locator (e.g., "./feed") specifically to avoid container mount paths. This mismatch can confuse consumers of the lock format.
/// The source as configured, not the container's view of it: a `url:` with
/// `$releasever` and `$target` expanded but WITHOUT the loopback rewrite, a
/// `path:` exactly as written in the config, and the `connect://<org>/...`
/// placeholder for a Connect feed — never the minted host, which changes per
src/utils/lockfile.rs:1894
- With
LOCKFILE_VERSION = 8, loading a v7 lockfile should succeed viaLockFile::load()/parse_and_migrate(). Right nowparse_and_migrate()only bumps versions 3/4/6 when the struct parse succeeds, so a v7 file parses but then falls through to the JSON migrationmatch(which doesn’t handle 7) and fails. This test currently bypasses that path by usingserde_json::from_strdirectly, so it won’t catch the regression.
.collect();
assert_eq!(names, vec!["a"], "dropped feed must not linger");
}
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Lite
86ae33e to
6a03764
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Important “recording failed / feeds changed” notices are currently emitted with output that can be suppressed under TUI/JSON modes, undermining the PR’s stated “reported rather than fatal” behavior.
Review details
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
src/utils/config.rs:4065
- These lock-recording failures are printed with
print_info, which is suppressed when the TUI or--jsonoutput is active, so the user may never see that the feed provenance was not recorded (contradicting the “reported rather than fatal” behavior described in the PR). Use a notice function that cannot be swallowed, e.g.print_warning_above(and emits a JSON warning event).
This issue also appears in the following locations of the same file:
- line 4083
- line 4093
src/utils/config.rs:4087
- The “feeds changed” notice is currently emitted via
print_info, which is suppressed under the TUI and in--jsonmode; this makes the reorder/change warning easy to miss even though it’s a behavior change the user is supposed to notice. Consider usingprint_warning_aboveso it is always surfaced (and becomes a JSON warning event).
crate::utils::output::print_info(
&format!(
"feeds for '{target}' changed since the lock was written: [{}] -> [{}]",
names(previous),
names(&locked)
src/utils/config.rs:4097
- Like the earlier read failure, a write failure here uses
print_info, which is suppressed when the TUI or--jsonoutput is active; users may not learn that the lock was not updated. Use a non-swallowable notice (e.g.print_warning_above).
if let Err(e) = lock.save(project_root) {
crate::utils::output::print_info(
&format!("could not record the feed set in the lock: {e}"),
crate::utils::output::OutputLevel::Normal,
);
- Files reviewed: 3/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
e460d1e to
6375163
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The new feed-recording logic in record_feed_set_in_lock will repeatedly re-save the lockfile for no-feeds projects and can miss emitting a “feeds changed” message for empty→non-empty transitions, which conflicts with the intended observable behavior and adds unnecessary churn.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 11/12 changed files
- Comments generated: 1
- Review effort level: Lite
6375163 to
79c4e5a
Compare
There was a problem hiding this comment.
🟡 Changes recommended
A new dnf ... command is constructed for bash -c with an unescaped installroot path, which can allow shell injection/breakout and should be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 11/12 changed files
- Comments generated: 1
- Review effort level: Lite
`repo-snapshot` pins the distro feed's content. Nothing equivalent exists for a third-party or on-disk feed, and packages are recorded as a name-to-version map with no provenance at all — so with more than one feed the lock says a package is at some version and nothing about where it came from. That matters because declaration order is a strict dnf priority override, which is the point of the feature: a locally built package shadows a released one at the same version. So the lock could read `hello-feed 1.0` while the bytes came from `local-build` on one machine and `avocado` on another, and reordering `distro.feeds` changes what installs without changing anything the lock records. Each target now carries its resolved set beside `repo-snapshot`: name, position, the source, a content digest for on-disk feeds, and the stages the feed applied to. **Per target, not global**, because a resolved URL contains `$target` and `targets:` can exclude a feed outright, so two targets in one project genuinely resolve different sets. A single list would record one target's sources against another's packages. **The source as configured, not the container's view of it.** A loopback URL is rewritten to `host.docker.internal` for reachability and a `path:` feed becomes a `file://` path inside the mount. Both are plumbing; recording them would put an ephemeral port and an internal mount path into a file people commit and diff. The canonical feed document still records the container view, and is unchanged — the new field is `#[serde(skip)]` precisely so no stamp hash moves. **Written from the resolution funnel, not from each install command.** The first attempt recorded it in `rootfs install` and therefore recorded nothing for an `sdk install`. This is the same shape as the snapshot pin, which records itself from `utils::snapshot` and saves immediately, for the same reason: it is the one place that always runs when feeds resolve. Once per target per invocation, and a failure to write is reported rather than fatal — a build should not stop because a provenance record could not be saved. The set is replaced rather than merged, so a feed dropped from the config leaves the lock instead of lingering. And when it changes, the install says so: reordering feeds silently changes which artifact a name and version resolve to, which is exactly the change nobody would otherwise notice. `LOCKFILE_VERSION` 7 to 8. Additive: a v7 lockfile reads as v8 with an empty list, which has a test. Built-ins are left out. They are served by the `.repo` files baked into the SDK image and have a repoid glob rather than a URL, so recording one as a source would describe something that is not one; their effect on resolution is already in the stamp projection. Verified on the integration rig: the lock records `../feed-a` with a digest, the distro feed with its releasever, and a stage-scoped vendor feed at its own position.
Raised in review, and it is worse than the review guessed: not a weak test but a
real regression this PR introduced.
`parse_and_migrate` has a fast path listing the versions that parse cleanly as the
current shape and only need their `version` field bumped — 3, 4 and 6. Version 7
was never in that list because it did not need to be: it equalled
`LOCKFILE_VERSION` and returned earlier. Bumping to 8 removed that early return
and sent every v7 lockfile — which is every real project's — to the
`_ => bail!("Unable to parse lock file format")` arm.
The test I wrote for v7 compatibility could not catch it, because it deserialized
with `serde_json` directly and so never went through `load`. Both tests now go
through the real entry point, and a second one exercises every carryable version,
since the list is easy to under-fill for exactly the reason above: a version is
silently covered while it is the current one and breaks the moment that changes.
Verified by reverting the fix — the test fails with the real error.
Also corrected a doc comment that still described `file://` for on-disk feeds,
which is what this PR deliberately stopped recording.
The synthetic v7 test covers the version list, which is only half of what a version bump can break. A real lock also carries `kernels`, `kernel-versions`, `repo-snapshot`, `runtimes` and `extensions`, and any one of those failing to parse as the current shape sends the whole file to the migration fallback and its `bail!` — with the same symptom and a different cause. This is a lock from a live project, 3.5 KB, checked for anything host- or customer-specific. It caught nothing new, because the fix was already right, but it is the test I should have written first: I diagnosed the original bug from a stale binary reporting a parse failure on a real lock, and a synthetic fixture would not have let me tell the two causes apart.
A lockfile that says `hello-feed: 1.0-r0` does not say which of a project's
feeds served it, and with `distro.feeds` ordering a name and version can resolve
to different bytes purely because the list was reordered. The feed *set* landed
in the previous commit; this is the per-package half.
`rpm` cannot answer the question — it records the package, not its origin. dnf
can, because it did the resolution and keeps the answer in the installroot's own
`history.sqlite`, so the query is `repoquery --installed --qf '%{name}|%{from_repo}'`
with `--disablerepo="*"`: entirely local, no metadata fetch, and therefore
unable to fail because a feed is unreachable.
**The format change is additive by construction.** `LockedPackage` serializes as
a bare version string when the origin is unknown, so a single-feed project's
lockfile is byte-identical to what it was before, and only packages whose origin
was actually resolved take the object form. It deserializes from either shape.
Three properties worth stating, each with a test:
- **Origins are applied only to packages the lock already tracks.** The query
returns everything installed in the sysroot, transitive dependencies included,
while the lock records what the config named. Adding the rest would change
what the lock is.
- **A version-only install does not erase an origin.** `update_sysroot_versions`
takes versions alone, and a second install must not undo what the first
established.
- **Provenance does not move any stamp hash.** `package_list_hash` folds name
and version and deliberately not the origin: a package's identity is its
NEVRA, and folding in provenance would rebuild every project the first time
origins are captured, for a record rather than an input. A real change of
origin already moves the hash through the feed projection.
Wired at every site that records versions — rootfs, runtime, the SDK's target
sysroot, and extension installs, which is the stage most likely to draw from a
second feed. The SDK's own sysroot has no installroot and is skipped by
construction: `build_origin_query_command` returns `None`, since those packages
come from the SDK image rather than from a feed. The query is best effort
throughout; an empty result means "unknown", never an error.
The local-feeds rig now asserts the end-to-end property: `hello-feed`, published
only to `local-build`, is recorded as coming from `local-build`.
…t moved Three review findings, one root cause and two smaller ones. **An empty feed set could not be recorded.** `merge_with` treated an empty `feeds` vec as "this writer did not touch feeds" and pulled the on-disk value back in — and since `save()` always merges with disk, removing the last named feed from a config left the lock describing a set the build no longer used, for every write from then on. `feeds` is now an `Option`: `None` is "untouched", `Some(vec![])` is "recorded as none", and the merge adopts the other side only for `None`. Regression test covers both directions. **The no-feeds path never wrote at all.** `materialize_feeds` returned early before reaching the recorder, so the clear above could never happen even with the merge fixed. It now records the empty set, once per target per invocation — guarded, because otherwise every container run of a project with no named feeds would load and re-check the lock for nothing. **The change message printed two identical lists** whenever a feed's locator, digest, stage scope or position moved without its name or order changing. It now names the feeds that moved and which fields moved on each. Also corrects the Version 8 doc comment: the lock records the locator *as configured*, not as used — no loopback rewrite, no minted host, and a `path:` feed exactly as written. Those are per-machine facts, not inputs, which is the whole reason `configured_locator` exists.
Two bugs in the recorder's early return, both from comparing the wrong things. The no-op test was `previous == locked && has_feed_record(target) == set.is_some()`. `set_feeds` always writes `Some(..)`, so on the no-feeds path the right-hand side was `true == false` and the guard never matched: every container run of a project whose feeds had once been recorded loaded, rewrote and saved the lock to say the same thing again. The test is now "a record exists and it equals what we would write", plus an explicit "nothing recorded and nothing to record" case so a project that never had named feeds does not gain an empty `feeds` key it will never use. Change reporting was gated on `!previous.is_empty()`, which stayed silent when a target that had resolved no feeds started resolving some. That is a change, and the message exists for exactly the changes a user would not otherwise notice. It is now gated on whether a record existed at all. Regression test covers all three: no lock appears for a project with no feeds, a recorded set is cleared and stays cleared, and recording "none" twice does not move the lock's mtime.
79c4e5a to
478c0f3
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
It changes lockfile format and recording paths across multiple install flows (feeds resolution, lock merging, dnf origin querying), so it warrants final human review despite strong test coverage.
Review details
- Files reviewed: 11/12 changed files
- Comments generated: 1
- Review effort level: Lite
`build_origin_query_command` was inserted between `/// Build the rpm -q command with proper environment and flags` and the function it described, so the rpm query lost its doc and the dnf query gained a wrong one. It compiles either way, which is why review caught it and the compiler did not. Swept the rest of the stack for the same shape — an inserted item whose immediately preceding line is a pre-existing doc comment or attribute — and this was the only one.
There was a problem hiding this comment.
🟡 Changes recommended
The new dnf-based origin query is missing the same RPM env-var unsetting used by installroot rpm queries, which can lead to incorrect/no provenance under the container entrypoint’s exported RPM config.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 11/12 changed files
- Comments generated: 1
- Review effort level: Lite
The container entrypoint exports `RPM_ETCCONFIGDIR="$AVOCADO_SDK_PREFIX"` on every run, and `build_query_command` already unsets it before an installroot query — its comment says why: pointing librpm at the SDK's rpmrc while asking about a target root is how that query reads the wrong database. dnf reaches the rpmdb through librpm too, so the origin query inherits the hazard, and its failure mode is silent: no origins, which reads exactly like a single-feed project. Nothing is re-exported afterwards because there is nothing to re-export: every sysroot with a `root_path` has both fields `None`, and the only one that sets them — the SDK — has no installroot and returns `None` above. The subshell closes before `|| true` so the unset cannot leak into the rest of the entrypoint; the test pins that boundary as well as the unset. The rig still records `hello-feed` as coming from `local-build`, so this removes a hazard without changing what resolves today.
There was a problem hiding this comment.
🟢 Approval recommended
The feature is implemented additively with strong migration and behavior tests; only a minor test robustness nit was identified.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/utils/config.rs:6649
- The test asserts the lock file is not rewritten by comparing
mtime, but the 20ms sleep can be shorter than the filesystem timestamp resolution (commonly 1s), which can hide a rewrite and let regressions slip through. Use a longer delay (>= 1s) so a rewrite would reliably changemodified()on coarse-resolution filesystems.
- Files reviewed: 11/12 changed files
- Comments generated: 0 new
- Review effort level: Lite
Records, per target, which feeds a build resolved against — and which feed each package came from.
Why
repo-snapshotpins the distro feed's content. Nothing equivalent exists for a third-party or on-disk feed, and packages are stored as a name-to-version map with no provenance — so with more than one feed the lock says a package is at some version and nothing about where it came from.That matters because declaration order is a strict dnf priority override, which is the point of the feature: a locally built package shadows a released one at the same version. The lock could read
hello-feed 1.0while the bytes came fromlocal-buildon one machine and the public feed on another, and reorderingdistro.feedschanges what installs without changing anything the lock records.What lands in the lock
Three decisions worth stating, because each has a wrong version that looks reasonable:
Per target, not global. A resolved URL contains
$target, andtargets:can exclude a feed outright, so two targets in one project genuinely resolve different sets. One global list would record one target's sources against another's packages.The source as configured, not the container's view. A loopback URL is rewritten to
host.docker.internalfor reachability, and apath:feed becomes afile://path inside the read-only mount. Both are plumbing. Recording them would put an ephemeral port and an internal mount path into a file people commit and diff. There is a test asserting the two views differ and that the lock takes the configured one.Written from the resolution funnel, not from each install command. My first attempt recorded it in
rootfs install, and therefore recorded nothing at all for ansdk install— the rig caught it. This follows the snapshot pin, which records itself fromutils::snapshotand saves immediately, for the same reason: it is the one place that always runs when feeds resolve. Once per target per invocation, and a failed write is reported rather than fatal, since a build should not stop because a provenance record could not be saved.Other behaviour
.repofiles baked into the SDK image and have a repoid glob rather than a URL, so recording one as a source would describe something that is not one. Their effect on resolution is already in the stamp projection.#[serde(skip)]specifically so no stamp hash moves and no cached sysroot is invalidated by this PR.Format version
LOCKFILE_VERSION7 → 8, additive. A v7 lockfile reads as v8 with an empty list, which has a test.Per-package provenance
The feed set says which feeds were available. It does not say which one served a given package, and with ordering as a strict override that is the difference that matters. So the version map became a struct:
Additive by construction.
LockedPackageserializes as a bare version string when the origin is unknown, so a single-feed project's lockfile is byte-identical to what it was before and only packages whose origin was actually resolved take the object form. It deserializes from either shape.rpmcannot answer this — it records the package, not where it came from. dnf can, because it did the resolution and keeps the answer in the installroot's ownhistory.sqlite, so the query isrepoquery --installed --qf '%{name}|%{from_repo}'with--disablerepo="*": entirely local, no metadata fetch, and unable to fail because a feed is unreachable.Three properties, each with a test:
package_list_hashfolds name and version and deliberately not the origin: a package's identity is its NEVRA, and folding provenance in would rebuild every project the first time origins are captured. A real change of origin already moves the hash through the feed projection.Wired at every site that records versions — rootfs, runtime, the SDK's target sysroot, and extension installs, the stage most likely to draw from a second feed. The SDK's own sysroot has no installroot and is skipped by construction.
What this does not do
It cannot make an unpinned feed reproducible. A third-party repository that serves no snapshots can still change under you; this records the source and, for on-disk feeds, a content digest.
Testing
Unit tests for per-target recording, replacement, v7 compatibility, the configured-source invariant, the additive serialization, and the recorder's no-op paths. Verified end to end on the integration rig: the lock records the on-disk feed with its digest, the distro feed with its release version, a stage-scoped vendor feed at its own position, and
hello-feed— published only tolocal-build— as coming fromlocal-build.Stacked on #243.