Golden snapshot state pollution - general problem

[git286]

Summary

While working on #853 (telemetry relay) we found that the metric counter problem is actually a special case of a much more general issue: any value computed before the snapshot is frozen into every actor restored from it. If that value is supposed to be fresh, unique, or current per actor, it is silently wrong in every actor, and nothing errors.

#802 already states this well: "Anything living in checkpointed process memory (env vars) or in the container image is frozen into snapshots... every actor restored from it would observe the source actor's values."

We have several issues today that are each one instance of this class (#761, #802, #1284, #853), but no issue for the class itself. I want to use this issue to name the general problem, collect the instances, and agree on the general solution strategies, so we stop solving it one case at a time in different shapes.

The general problem

The golden snapshot is taken after the template app finishes init. So everything the app computed during init is shared by all actors of the template, and everything it accumulated is inherited as a baseline. The pollution is not a one time event at actor creation: the polluted values sit in process memory, so a suspend snapshot carries them too, and the actor keeps them for its whole life.

I think the instances fall into 4 types:

1. Uniqueness broken. Values generated once at startup are identical in every clone: otel resource / instance id (Actor instrumented metrics might merge and inflate #761), hostname (hardcoded to runsc), app generated worker ids, session ids. The worst member here is RNG state: if the PRNG or crypto DRBG state is in frozen memory, all clones produce the same "random" sequence. Duplicate UUIDs, colliding tokens, repeated nonces. The snapshot/restore world outside (AWS Lambda SnapStart, CRIU, Java CRaC) treats this as a security problem and reseeds entropy on restore (VmGenID etc). I did not find any substrate issue about guest RNG reseed on restore, for either runtime. I think this is a real gap and deserves its own child issue.

2. Currency broken. Values that were true at build time but not now: cached credentials and certs (Add a SystemInfo volume source to ActorTemplate for delivering per-actor generated files #802 covers this, with short TTL + activation binding as mitigations), authorization decisions (No extension point at restore, so a restored actor cannot be reconciled against current policy #1284: "The actor comes back with the authority it had when the snapshot was taken"), cached config, feature flags, DNS caches, cached wall clock values and everything derived from them.

3. Zero point broken. Accumulated quantities that should start from the actor's birth but actually start from the golden build: metric counters (Actor instrumented metrics might merge and inflate #761, [Design] Actor Otel Telemetry Relay #853), uptime, rate limiter buckets, circuit breaker state. This is the case we hit in the telemetry design.

4. Phase alignment. All clones share the same timer offsets and the same jitter sequence (jitter from a frozen RNG is identical jitter). A thousand actors from one golden can retry and poll in lockstep. A thundering herd is built into the template. We never diagnosed this but I believe it exists.

Solution strategies

From the existing work I see 5 strategies. They differ on one important axis: does the fix need the workload to do something, or can the platform do it alone. This axis matters because actor code is untrusted and many authors will not read our docs. So the rule I propose: any guarantee the platform makes (trust, attribution, security) must only depend on strategies that need zero workload cooperation. Workload side strategies can reduce errors but never carry a guarantee.

Platform alone:

1. Correct in flight at platform boundaries. When polluted data transits a channel the platform owns, the platform fixes it without asking anyone. Telemetry is the flagship case: ateom stamps identity and clamps timestamps using activation time ([Design] Actor Otel Telemetry Relay #853). Note this does not generalize: most polluted state never leaves the sandbox, it only works because telemetry happens to pass through ateom.

2. Inject freshness at restore, platform side. Things the platform can reset before the actor becomes reachable: RNG reseed on restore (the unfiled gap above), No extension point at restore, so a restored actor cannot be reconciled against current policy #1284 policy reconciliation.

3. Make stale copies harmless. Design the value so a frozen copy is useless: Add a SystemInfo volume source to ActorTemplate for delivering per-actor generated files #802 does this for credentials with short TTLs and activation binding. The workload cannot opt out of this protection, so it counts as platform side.

Needs workload cooperation (optimizations and best practices, never guarantees):

4. Out of snapshot channels. Add a SystemInfo volume source to ActorTemplate for delivering per-actor generated files #802 SystemInfo volume: atelet regenerates per actor files on every Run/Restore. The platform provides the channel, but it only helps if the workload reads the files fresh instead of caching at startup. So this one is mixed: platform mechanism plus workload discipline.

5. Workload authoring rules. Documented rules for template authors: only compute constants at init (constant endpoints, constant addresses); keep golden build warmup off instrumented paths or reset accumulators (metrics SDK etc) as the last build step; defer init of anything per actor until after activation; when Actor lifecycle hooks #450 lands, register a restore hook that refreshes state (reseed app level RNG, recreate the otel MeterProvider). Following these makes a workload fully clean. Not following them means bounded, documented errors, never wrong attribution.