kagent WorkerPool spans AMD generations (m7a and c5ad): golden snapshots taken on gen 7 cannot restore on gen 5 workers, half of the default agent's cold turns fail after 60 s

[teemow]

Problem

On gazelle the kagent-default WorkerPool (4 workers, nodeSelector karpenter.k8s.aws/instance-cpu-manufacturer: amd) currently spans two AMD generations: two workers on an m7a.2xlarge (generation 7, AVX-512) and two on c5ad.xlarge (generation 5). The golden snapshots of sre-agent (Swarmgeist's default agent) and test-agent were taken on the m7a workers at 16:24Z on 2026-09-14. A new AgentInstance placed on a c5ad worker cannot restore them:

starting container: starting sub-container [/app]: failed to load kernel: incompatible FeatureSet: missing features: map[monitor:{} pcid:{} x2apic:{} invpcid:{} avx512f:{} avx512dq:{} avx512cd:{} avx512bw:{} avx512vl:{} …

Substrate retries the restore once on the same worker (30 s each), the controller answers SendStreamingMessage with Internal after 60 s, klaus-gateway logs actor "ai-…" request timed out and Slack shows "the turn failed; please try again". Observed live on 2026-09-14 19:29Z (first cold turn of the evening); the retry on an m7a worker answered in 9 s. Half of the pool cannot serve the default agent.

This is the caveat recorded when #429 pinned the manufacturer: a newer generation's gVisor checkpoint does not restore on an older one of the same vendor. Karpenter picks any AMD instance type for the pool, so the mix changes with every spot replacement.

Proposed solution

Pick one, in order of preference:

1. Pin the pool to one CPU generation/family through a chart value (karpenter.k8s.aws/instance-generation or karpenter.k8s.aws/instance-family in substrateWorkerPool.template.nodeSelector), chosen per installation from the generation the golden snapshots were taken on; document the capacity trade-off (a generation pin narrows the spot pool).
2. Make the golden snapshot the lowest common denominator: take goldens on the oldest generation admitted by the pool so they restore everywhere.
3. Longer term (Substrate proposal, fork-first): workers advertise their CPU feature set, snapshots record it, the scheduler only places a restore where the host's set is a superset.

Also: a restore that fails with incompatible FeatureSet should be retried on a different worker, not the same one twice.

Acceptance criteria

• All workers of a pool share a CPU feature set that covers the pool's golden snapshots (verified with kubectl get nodes -L karpenter.k8s.aws/instance-generation for the worker nodes).
• A cold turn of sre-agent on gazelle never fails with incompatible FeatureSet; 20 consecutive new threads succeed.
• The docs from docs(meta): the WorkerPool is one CPU feature set, not one architecture — vendor and family pins on CAPA, node-pool pin elsewhere, recognition recipe (#429) #430 describe the generation constraint and the value to pin it.