atelet: stream the durable-dir tar into its upload instead of staging it

[Chenyi Wang (chw120)]

Suspending an actor with a 512 MiB durable dir spends about six seconds in two serial halves: ateom writes the tar, fsyncs it, and returns, and only then does atelet open that file and upload it. The bytes cross the disk twice for no
reason — the archive exists solely to be uploaded, and nothing reads it in between.

atelet can now generate the archive itself, straight into the upload, so the walk overlaps compression and the network PUT and the archive never lands on disk. CheckpointWorkloadRequest gains skip_durable_dir_tar to tell ateom to
leave the directory alone.

Measured

512 MiB durable dir, durdir workload, DATA scope, 3 interleaved rounds per arm on one node, 39 checkpoints per arm, no failures in either:

	staged	stream
ateom durable_dir	3336 ms	0 ms	gone
pause	2.51 ms	2.49 ms	unchanged
teardown	138.4 ms	136.8 ms	unchanged
client SuspendActor p50	6167 ms	4067 ms	−34%
client SuspendActor p95	7467 ms	4567 ms	−39%

Per round, staged p50 was 6200 / 6100 / 6200 and stream 4300 / 4000 / 3900. The staged arm's 3336 ms matches the 3376 ms recorded for this size in the incremental-tar measurement, so the control is sound.

The remaining 4067 ms is close to what the arrangement can reach:max(tar 3336, upload ~2600) plus control-plane overhead, against tar + upload before. What is left is the archive itself, which is a different change.

What it gives up

The staged path hands an *os.File to sendZstd, which uploads through PutSparseFile — ranges compressed and PUT in parallel. A pipe cannot be seeked, so the streamed archive takes the single overlapped sendStreamingZstd
path instead. The sparse half costs nothing here (a tar is written densely and has no holes), but the parallel PUT is a real loss, and the numbers above are net of it.

Why it is safe to archive after the RPC

ateom calls terminateWorkload before returning, so by the time atelet walks the directory the VM is gone and nothing can write it. Even if that teardown fails — it is best-effort and only logs — the guest was paused at the top of the checkpoint and no path resumes it. The guest freeze window shrinks rather than grows.

Scope

Gated on ATELET_STREAM_DURABLE_TAR and off by default. Further limited to external checkpoints of micro-VM actors: a local checkpoint has no upload to overlap with, and moveLocalCheckpoint would find the file missing. ateom-gvisor rejects the flag with InvalidArgument rather than honoring it, because its archive drops the .gvisor.* files its runtime leaves in the durable dir and atelet does not know that rule.

The archive itself is unchanged. tarutil.CreateTo writes the same bytes to an arbitrary destination that Create writes to a file, which is what lets a snapshot taken either way restore the same tree; TestStreamDurableDirTarMatchesStagedArchive compares them byte for byte. The uploaded object does differ — plain zstd rather than the sparse-extent format — but decodeContent has always detected that from the leading magic, and the plain form is what every non-file upload already produces.