atelet: serve golden snapshot restores from the node-local file cache - #1551
Draft
Dmitry Berkovich (dberkov) wants to merge 6 commits into
Draft
atelet: serve golden snapshot restores from the node-local file cache#1551Dmitry Berkovich (dberkov) wants to merge 6 commits into
Dmitry Berkovich (dberkov) wants to merge 6 commits into
Conversation
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 9, 2026 01:25
d0bdb37 to
240c6ce
Compare
Dmitry Berkovich (dberkov)
marked this pull request as draft
September 10, 2026 03:59
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 10, 2026 04:04
240c6ce to
667cd76
Compare
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 11, 2026 22:15
667cd76 to
e4dd730
Compare
RestoreRequest borrowed ExternalCheckpointConfiguration — a write destination — for the snapshot it reads, leaving read-side attributes of a restore source no place to live. Split the restore arm into its own ExternalRestoreConfiguration; the oneof's type change is wire-compatible (snapshot_uri keeps field number 1, and old peers decode the message as before). base_config supersedes golden_snapshot_uri, typing the DATA_ON_GOLDEN base as a restore source too. The old field stays until both components roll — callers set both and atelet prefers base_config — and is removed (and reserved) in a follow-up. Pure restructuring: no caller sets base_config yet, nothing reads it, and the wire bytes of every current request are unchanged.
Prefer RestoreRequest.base_config and fall back to the superseded golden_snapshot_uri, so ateapi and atelet can roll in either order. A transitional caller sets both; validation requires them to agree, and a base snapshot of either spelling is rejected outside SNAPSHOT_SCOPE_DATA_ON_GOLDEN scope.
Set RestoreRequest.base_config on both DATA_ON_GOLDEN resume paths (an external data snapshot and a local pause checkpoint combining with the golden), alongside the superseded golden_snapshot_uri it duplicates: an atelet from before base_config reads only the old field, so the dual-write lets the two components roll in either order. The old field and this duplication go away together in a follow-up once both sides have rolled.
Golden snapshot files are immutable once published, yet every fresh-from-golden start and DATA_ON_GOLDEN resume re-downloads them into its own per-actor dir. Introduce the node-level cache that will hold them: a filecache.Store rooted under the ateom base path (cache hits will be served as hard links into per-actor restore dirs, which requires one mounted filesystem), opened at startup behind --golden-cache-dir with crash debris swept before atelet serves. No restore behavior changes yet: the store is wired into the herder but nothing reads through it. The cached download path and the eviction loop follow separately.
Golden snapshot files were downloaded into every actor's restore dir, even when a dozen actors resume from one golden on the same node. Route the golden legs of the restore download paths through the golden cache: the DATA_ON_GOLDEN golden half, the local-pause combine leg, and a fresh-from-golden start (an external FULL restore whose snapshot URI is owned by the reserved golden atespace — the URI is how atelet recognizes an immutable, cache-safe source). Actor-owned snapshot URIs stay uncached: each is read by one actor and deleted on its next suspend. Cache hits are hard links to a read-only shared copy, so serving is gated per sandbox class: ateom-gvisor consumes restore-state strictly read-only and gets links; ateom-microvm rewrites config.json in place at restore and merges checkpoint deltas into memory-ranges' inode at suspend, either of which would corrupt a shared inode, so its class downloads fresh until a private-copy mode serves it.
The golden cache served only gVisor restores: its hits are read-only hard links, and ateom-microvm mutates staged files in place (config.json is rewritten at restore, checkpoint deltas are merged into memory-ranges' inode at suspend), which would corrupt a shared inode. Add cacheModeCopy: hits are hole-preserving private copies (see filecache.GetFileCopyTo), so the micro-VM class keeps its in-place mutations — no ateom-microvm changes — while concurrent restores of one golden still share a single download and later restores read the node cache instead of object storage. A link-mode fetch that fails with EXDEV (cache configured on a different mount than the restore dirs) now degrades to a copy too, keeping the download savings where it previously fell back to a fresh download per restore.
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 11, 2026 22:33
e4dd730 to
03ae2e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements milestone M2 of the node-local artifact cache proposed in #690: atelet's golden snapshot downloads now go through the
filecachestore from #1517, so concurrent restores of one golden share a single download and later restores are served from the node instead of object storage. it reads well commit by commit:Stacked on #1602 (fork branches cannot be PR bases; #1517 is merged): please review only the top three commits — everything below
atelet: open a golden snapshot file cache at startupis #1602, the Restore snapshot-source refactor this work builds on.atelet: open a golden snapshot file cache at startup— lifecycle only:--golden-cache-dir(default<base-path>/golden-cache, empty disables; must share the actor dirs' mount because hits are hard links) and--golden-cache-min-age; the store opens at startup andSweepDebrisclears crash debris before atelet serves. No behavior change yet.atelet: serve golden snapshot downloads from the file cache— routes exactly the golden legs through the cache: the DATA_ON_GOLDEN golden half of the combined download, the local-pause combine leg, and fresh-from-golden starts. The latter arrive as plain external FULL restores, so the cache-safe source is recognized by URI ownership: golden snapshots belong to the reservedate-goldenatespace and are immutable once published. Actor-owned snapshot URIs stay uncached (single consumer, deleted on next suspend). Serving is per sandbox class (goldenCacheModeFor): gVisor consumes restore-state read-only and gets zero-cost hard links.atelet: serve micro-VM golden downloads as private copies— micro-VM cannot share inodes (ateom-microvm rewritesconfig.jsonin place at restore and merges checkpoint deltas intomemory-ranges' inode at suspend), so its hits are hole-preserving private copies (GetFileCopyTo): still one download per golden per node, mutations stay private, and no ateom-microvm changes were needed. A link-modeEXDEV(cache on a different mount) degrades to a copy too, keeping the download savings instead of falling back to a fresh download.The M2 exit criterion — N concurrent restores of one golden = one download — is pinned at the unit level by
TestDownloadExternalCheckpointSharesOneGoldenDownload(4 concurrent restores, 1 GCS get, shared inode) and its copy-mode counterpart.Still to come for M2 (will follow once this lands): the pressure-driven GC loop for the golden cache (
goldengcmirroringimagegc's watermark formula) — until then the cache grows unbounded, which matches today's behavior forstatic-filesand is bounded in practice by golden churn.Tested:
go test -race ./cmd/atelet/...; the cross-mount EXDEV fallback has a dedicated test that exercises two real filesystems where available (/dev/shmon Linux CI) and skips elsewhere;golangci-lint, gofmt, and boilerplate checks clean.🤖 Generated with Claude Code