Skip to content

Add a split durable-dir capture that hardlinks file contents - #1519

Open
Chenyi Wang (chw120) wants to merge 1 commit into
agent-substrate:mainfrom
chw120:durdir-split
Open

Add a split durable-dir capture that hardlinks file contents#1519
Chenyi Wang (chw120) wants to merge 1 commit into
agent-substrate:mainfrom
chw120:durdir-split

Conversation

@chw120

Copy link
Copy Markdown
Collaborator

Sealing a durable-dir volume as one tar reads and rewrites every byte the actor holds, on the paused critical path, so suspend latency scales with the actor's data rather than with what it changed. This adds a second arrangement that
hardlinks contents instead of copying them, at both ends.

Off by default. ATEOM_DURABLE_BACKEND unset — or set to anything but files — keeps the existing tar path, byte for byte. Only ateom-microvm reads the variable; ateom-gvisor is untouched.

What it does

ATEOM_DURABLE_BACKEND=files writes durable-dir.index.tar (metadata only, regular-file entries carry Size=0) plus one durable-dir.blob-NNNN per non-empty regular file, hardlinked out of the volume rather than copied.
Sealing costs one link(2) per file instead of a copy of the tree. Restore mirrors it: ExtractSplit adopts each blob with Linkat instead of writing it out again.

Both ends are guarded. CreateSplit falls back to copying on EXDEV/EMLINK/EPERM. ExtractSplit copies any blob whose link count shows a second owner still holds it, so the "this activation owns the blobs" contract is enforced by the code rather than by convention.

The blob name travels in a PAX record per entry, so dispatch is per entry, not per archive: ExtractSplit reads a plain Create archive too, and an archive can hold both kinds (empty files never get a blob). A restore dispatches on what
the snapshot contains rather than on the variable, so a node configured either way reads back an actor captured the other way.

atelet has to know the new names as well: carving durable data out of a FULL micro-VM capture for a DATA upload now selects every durable-dir snapshot file by predicate, and fails the upload when none are present instead of silently
uploading an empty set.

Measured

Old-new-old bracket, durdir_partial_512mb_pause_microvm, 8 min per arm, single-node micro-VM pool. The two tar arms agree within 2.5%, so the session did not drift.

p50 tar (before) split tar (after)
ateom durable_dir (seal) 5 921 ms 2.34 ms 6 073 ms
PauseActor, end to end 6 500 ms 190 ms 6 400 ms
cold-boot restore, durable half 362 ms 0.88 ms 362 ms
cold-boot restore, total 1 303 ms 951 ms 1 310 ms
atelet restore 1 302 ms 1 011 ms 1 256 ms
checkpoints completed in 8 min 43 94 41

Throughput +124%. Workload control (DurDirWrite) held at 848 / 842 / 851 ms p50 with no failures, so the guest was doing the same work in all three arms.

The cost is a snapshot of many objects rather than one, paid on upload — a directory of many small files would pay more for it than this workload's eight large ones does.

Testing

  • internal/tarutil/split_test.go covers round-trip, empty files getting no blob, hardlink trees, symlinks and device nodes, the nlink != 1 guard falling back to copy, the EXDEV fallback (linkat is a package variable the test stubs), and mixed archives.
  • cmd/ateom-microvm/durable_test.go covers arrangement selection and restore dispatching on snapshot contents in both directions.
  • cmd/atelet/main_test.go covers the widened DATA carve, and the existing "no durable files" case still returns FailedPrecondition.
  • make verify and the full unit suite pass on linux.

Caveat on the cluster numbers. They were produced by an equivalent build, not by this commit: the rebased tree needs a control plane carrying ateapi.WorkerService.SetWorkerCapacity, which the benchmark cluster does not
have yet, so the measured image is this branch's internal/tarutil and durable path grafted onto the pre-rebase base. The rebase touches the measured path in exactly three places — an import path, one comment, and upstream's tar buffering
— and the buffering was grafted in and measured separately: it is inert here by its own design, since copyBufSize (128 KiB) exceeds streamBufSize (64 KiB) specifically to route file contents around the stream buffer, and this snapshot
is eight large files. The atelet DATA carve was exercised on the cluster in its pre-rebase form (188 index files and 8×94 blobs uploaded); the gVisor widening is unit-test-only.

Potential Next Step (Not in scope)

Making the split arrangement the default, and whether the upload side should batch the blobs.

Sealing a durable-dir volume as one tar reads and rewrites every byte the
actor holds, on the paused critical path, so suspend latency scales with the
actor's data rather than with what it changed.

Add a second arrangement, selected by ATEOM_DURABLE_BACKEND=files: a
metadata-only index tar plus one blob per non-empty regular file, hardlinked
out of the directory instead of copied. Sealing costs one link per file, and
restore adopts the blobs the same way, so neither direction spends a full copy
of the tree. Both ends are guarded: CreateSplit falls back to copying on
EXDEV/EMLINK/EPERM, and ExtractSplit copies any blob whose link count shows a
second owner still holds it, which keeps the "this activation owns the blobs"
contract enforced by the code rather than by convention.

The blob name travels in a PAX record per entry, so a reader tells the two
arrangements apart entry by entry and ExtractSplit reads a plain archive too. A
restore dispatches on what the snapshot contains rather than on the variable,
so a node configured either way can read back an actor captured the other way.

atelet has to know the new names as well: carving durable data out of a FULL
micro-VM capture for a DATA upload now selects every durable-dir snapshot file,
not just the tar, and fails the upload when none are present instead of
silently uploading an empty set.

Also split the micro-VM cold-boot restore timing into its durable and boot
halves, which is what shows the durable side going to nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant