Karpenter installations: the stateful singletons (klaus-gateway, muster-valkey) run on spot capacity; #431's do-not-disrupt + PDB do not cover a spot reclaim

[teemow]

Problem

On installations where the workers are Karpenter spot capacity (gazelle: 15 spot workers, 0 on-demand; NodePool gazelle-karpenter allows spot,on-demand), the platform's stateful singletons are reclaimed with their nodes. On 2026-09-14 three spot reclaims in ten minutes (08:01, 08:03, 08:05Z) produced, for one Slack turn:

• muster-valkey (1 replica, Recreate, RWO gp3 PVC, no PDB, no do-not-disrupt) force-evicted at 08:05:34, its replacement Pending on Insufficient cpu / PV node affinity until a new node joined; back at ~08:07:24. muster's OAuth token store lives there, so every refresh grant blocked: a 27 s delay before the Slack bot reacted to one turn, another turn aborted on the client's 30 s timeout.
• klaus-gateway (1 replica, RWO PVC) on the same node: protected by karpenter.sh/do-not-disrupt and the minAvailable: 1 PDB from Karpenter consolidation evicted the single-replica platform data plane mid-turn (agentgateway, Backstage, Dex without PDB or do-not-disrupt) — fixed in agent-platform 4.10.1 + klaus-gateway 1.1.0: do-not-disrupt on the streaming pods, PodDisruptionBudgets on muster, kagent-controller, klaus-gateway, agent-manager #431, so the drain skipped it — and the instance was reclaimed anyway two minutes after the notice. The pod died mid-turn with the chart's 10 s grace; the replacement waited four minutes on a Multi-Attach error for the volume.
• kagent-default workers, an Envoy edge proxy, tunnel pods and MCP servers restarted in the same wave (those are designed for it: actors restore from checkpoints, the edge has two replicas).

The guards from #431 (do-not-disrupt, single-replica PDBs) address Karpenter's voluntary disruption. A spot reclaim is involuntary: Karpenter's CordonAndDrain on the interruption notice cannot evict a PDB-blocked pod, and the instance terminates regardless — so for these pods the guards turn a graceful two-minute move into a hard kill.

Status (2026-09-14)

The chart side is shipped — the placement decision is open. agent-platform#442 (meta chart 4.12.0, connectivity 4.12.0) adds scheduling.singletons.nodeSelector / tolerations — merged into the four singletons' own scheduling knobs (muster, valkey.valkey, kagent.controller, klausGateway) before their releases render, a component's own keys win, held back from the connectivity release — empty by default, so no installation changes behaviour until it is set. muster-valkey now carries karpenter.sh/do-not-disrupt (valkey.valkey.podAnnotations) and a PodDisruptionBudget muster-valkey (valkey.podDisruptionBudget, rendered by the connectivity chart; neither the wrapper nor the upstream subchart has a knob). UPGRADE.md carries the cost note and the roll on enabling. klaus-gateway takes the keys from chart 1.3.3 on (klaus-gateway#253: the earlier 1.x schemas refused every key under nodeSelector). Lab-proven with a labelled kind node (all four pods placed, platform-test green with the knob off and on, the valkey budget refuses an eviction).

Decision needed (option A recommended): enabling on gazelle is one values block on the installation's meta-chart values:

scheduling:
  singletons:
    nodeSelector:
      karpenter.sh/capacity-type: on-demand

Fleet check: gazelle 14 spot workers, NodePool gazelle-karpenter admits spot,on-demand, no taint (no toleration needed) — the target. graveler's 14 workers are already on-demand (harmless, gains nothing). glean (CAPZ) and spidertron (on-prem) run no Karpenter — never set it there (the four pods would stay Pending). Cost: one xlarge-class on-demand instance on gazelle; enabling rolls the four pods once, muster-valkey and klaus-gateway (Recreate) are down until the node is up (~2 min).

Proposed solution

Decision needed on the placement of the stateful singletons on Karpenter installations. Options:

• A. Place them on on-demand capacity (recommended now): a meta-chart knob, e.g. platform.statefulSingletons.nodeSelector: {karpenter.sh/capacity-type: on-demand} (name to settle), forwarded to klaus-gateway, muster-valkey, muster and kagent-controller. The existing NodePool already admits on-demand; Karpenter launches one small on-demand node for them. Cost: one or two xlarge-class on-demand instances per such installation.
• B. Make the components spot-tolerant (in parallel, in their repos): klaus-gateway without a node-bound volume and with a shutdown path that survives a restart mid-turn; mcp-oauth's token endpoint failing fast with temporarily_unavailable instead of hanging on Valkey; Valkey with a replica or a reconsidered role as the OAuth store.
• C. Accept the churn on gazelle as an internal installation. Customer installations do not run spot workers today, so this is a gazelle/graveler/glean/spidertron concern.

Also: muster-valkey currently carries none of #431's guards; whatever the placement decision, add the PDB and the annotation for consistency (they still help against consolidation).

Acceptance criteria

• Decision recorded (A, B, or both).
• The meta chart carries the knob (scheduling.singletons, feat(meta): the stateful singletons can be pinned to on-demand capacity (scheduling.singletons, opt-in); muster-valkey takes the #431 guards (#439) #442 → 4.12.0; make verify-disruption extended to it), documented in UPGRADE.md with the cost note.
• Rolled out on gazelle; over the following week Karpenter's spot_interrupted logs show no klaus-gateway, muster, muster-valkey or kagent-controller pod on an interrupted node, and the Envoy edge shows no multi-second POST /oauth/token.
• muster-valkey has a PodDisruptionBudget and the do-not-disrupt annotation like the other singletons (feat(meta): the stateful singletons can be pinned to on-demand capacity (scheduling.singletons, opt-in); muster-valkey takes the #431 guards (#439) #442 → 4.12.0; rolled out with the fleet's auto-track).

Context

Follow-up of #431 (voluntary disruption) — this is the involuntary half. Component-side issues are cross-linked below. Pau's #373 (two data-plane replicas behind a PDB) is untouched by this.

Related (same incident, 2026-09-14 on gazelle)

• Slack: a gateway pod that dies mid-turn leaves the thread silent — the ticker freezes at ⏳ … step N, the task runs on unobserved, and its finished answer never reaches Slack klaus-gateway#244
• Chart: the OBO link store on a ReadWriteOnce volume turns every node loss into a ~4 min gateway outage (Multi-Attach); move it off the node-bound volume klaus-gateway#245
• Token endpoint blocks for as long as the Valkey store is unreachable, and answers a transient storage error as invalid_grant mcp-oauth#578