feat(platform): the lab runs the 4.x line as an installation does — Substrate and the platform Postgres from the chart, JWT Strict on the controller route, a measured preflight budget, every image side-loaded

CLAUDE.md

@@ -242,9 +242,11 @@ Load-bearing invariants (details in docs/):
   `platform.chartPath` installs a local checkout instead; `platform.chartBranch`
   is the dev channel — the branch's newest dev build, resolved into
   `chartVersion` on every `configure`/`up`/`platform` (`chartbranch.go`,
-  `platform --pin` freezes it) — and implies Substrate, kagent main's actor
-  runtime, installed ahead of the platform (`substrate.go`,
-  `platform.substrate.enabled`). Never emit `gitops.namespace` with the
+  `platform --pin` freezes it). Agent Substrate and the platform Postgres
+  come with the chart (the 4.x line: the `substrate` and `cloudnative-pg`
+  component releases) — the lab installs neither; it reads the chart's
+  rendered roster (`platformRoster`) to know what to budget for, check the
+  apiserver gates for and preload. Never emit `gitops.namespace` with the
   engine on.
 - The lab's credentials are throwaway by design; plaintext passwords in
   `agentlab.yaml` are fine.

HACKS.md

@@ -514,27 +514,29 @@ Unblocks when kind's `load docker-image` logic (the re-tag of an image ID the
 node already has, the per-image save) survives the containerd image store and
 is worth reusing over the plain archive import.
 
-### U22. `substratepools.go`: Substrate's CA/JWT pool bootstrap is a Go port of `kubectl-ate` — BLOCKED UPSTREAM
+### U22. `substratepools.go`: Substrate's CA/JWT pool bootstrap is a Go port of `kubectl-ate` — FIXED upstream
 The substrate chart (0.0.26) mounts four pool Secrets, a trust-anchor Secret
-and an authentication ConfigMap it does not render. Upstream's install is
-`helm install`, then `kubectl-ate admin make-ca-pool` / `make-jwt-pool` plus a
-shell step (jq + openssl for the trust anchor, a heredoc for the
-authentication config), then a second `helm upgrade --wait` — the first
-install's pods restart on missing volumes until then. The lab downloads no
-binaries (kind, Helm and client-go are embedded; `kubectl-ate` is unsigned)
-and upstream publishes no image with the bootstrap in it (`ate-setup` is not
-published), so `substratepools.go` copies the generate + serialise subset of
-substrate's internal `localca` and `localjwtauthority` packages (Apache-2.0
-header kept, pinned to 0.0.26 in the comment) and `substrate.go` creates every
-bootstrap object BEFORE one waited install; the pre-created
-`podcertificate-controller-system` namespace the chart also renders is adopted
-with Helm's `--take-ownership`. Drift risk: a Substrate bump that changes the
-pool wire format shows up as ate-api-server never Ready. The issuer too:
-upstream's default authentication config names `https://kubernetes.default.svc`,
-which ate-api-server rejects against kind's tokens (their `iss` is
-`…svc.cluster.local`); the lab reads the issuer off the apiserver's discovery
-document, as upstream's own `ate-setup` does. Unblocks when the substrate
-chart renders the bootstrap (a hook Job) or the packages become importable.
+and an authentication ConfigMap it does not render, and upstream's install was
+`helm install` → `kubectl-ate admin make-ca-pool` / `make-jwt-pool` plus a
+shell step → a second `helm upgrade --wait`. While no chart shipped Substrate
+the lab installed it itself ahead of the platform, with `substratepools.go`
+copying the generate + serialise subset of substrate's `localca` and
+`localjwtauthority` packages and `substrate.go` creating every bootstrap
+object before one waited install (the POC channel's shape).
+**Fixed upstream (agent-platform 4.0.x):** the meta chart ships Agent
+Substrate as the `substrate-crds` and `substrate` components, and the
+connectivity release's `pre-install,pre-upgrade` hook Job
+`<release>-substrate-bootstrap` mints the same key material (openssl in an
+init container, the pools' wire format, a pool that exists is never touched)
+and publishes the trust anchors — the lab's port was that Job's prior art.
+The lab installs nothing of Substrate anymore: `substrate.go` keeps the one
+check the chart cannot make early — the apiserver serves
+`certificates.k8s.io/v1beta1`, else `agentlab down && agentlab up` — and the
+`platform.substrate` knob, the Go port, its tests and the Substrate values
+template are gone (agentlab#138). The chart's `substrate` component adopts a
+lab's earlier `substrate`/`substrate-crds` releases in `ate-system` by name
+(`storageNamespace` = `targetNamespace`), so a lab that ran the POC channel
+upgrades in place.
 
 ### U23. LM Studio has no delete over its API — `models-test` proves the refusal — BLOCKED UPSTREAM
 LM Studio's own API (`/api/v1`, 0.4.0+) serves the library, the download that

README.md

@@ -119,7 +119,7 @@ machine, is in [Getting started](docs/getting-started.md).
 | [Getting started](docs/getting-started.md) | Requirements, docker CPU and memory, install, what `configure` discovers, the first `up`, connecting Claude Code |
 | [Command reference](docs/cli.md) | Every `agentlab` command and flag, the environment variables, keeping the binary current |
 | [TLS](docs/tls.md) | The lab CA, `trust` and `untrust`, Node and browsers, bringing your own certificate |
-| [The agent platform](docs/platform.md) | muster + mcp-kubernetes: the request path, per-server sign-in, the fake fleet, toolsets, deviations from a real management cluster, the dev channel (a branch's dev builds, Substrate) |
+| [The agent platform](docs/platform.md) | muster + mcp-kubernetes: the request path, per-server sign-in, the fake fleet, toolsets, deviations from a real management cluster, the dev channel (a branch's dev builds), Agent Substrate and the platform Postgres from the chart |
 | [Agents](docs/agents.md) | The kagent runtime, the default ModelConfig and the API key Secret |
 | [Models](docs/models.md) | Extra model configs, model servers on the host, managed models through model-manager |
 | [Observability](docs/observability.md) | Prometheus + mcp-prometheus, and Backstage's metrics views |

docs/agents.md

@@ -49,21 +49,29 @@ there is the old port-forward:
 kubectl -n kagent port-forward svc/kagent-ui 8081:8080
 ```
 
-Lab deviations on the kagent side, same spirit as the [deviations
+The topology is the installation's (the 4.x line, kagent API v2): the
+controller's gRPC API is a `GRPCRoute` on the edge with the chart's JWT
+`Strict` policy in front — every bearer verified against the lab Dex,
+`x-user-id` set from the verified email claim — and the controller runs the
+fleet's `auth.mode: trusted-proxy`, re-deriving the caller from the same
+bearer; `agentlab platform-test` asserts both (a call without a token refused
+at the edge; a forged header attributed to the token's subject). Every agent
+is an `AgentTemplate` admitted by the platform `Harness kagent` (labelled
+`agent-platform.giantswarm.io/harness: kagent`) and runs as an actor on Agent
+Substrate — the `WorkerPool kagent-default`'s gVisor workers in the kagent
+namespace, the control plane in `ate-system` — which the chart ships and the
+lab installs nothing of; kagent's `kagent_v2` database lives on the platform's
+CNPG `Cluster kagent-pg` next to Substrate's (see [Agent Substrate and the
+platform Postgres](platform.md#agent-substrate-and-the-platform-postgres--from-the-chart)).
+The agent chart 1.x renders one `RemoteMCPServer` per agent pointing it at
+muster with its toolset header; kagent forwards the *caller's* token
+(`KAGENT_PROPAGATE_TOKEN`), so agent tool calls through muster are the
+person's. Lab deviations on the kagent side, same spirit as the [deviations
 table](platform.md#lab-specific-deviations-from-a-real-management-cluster):
-the controller runs `auth.mode: unsecure` (upstream's local-dev mode — the GS
-default `trusted-proxy` decodes bearer claims *without verification* and
-depends on a JWT-validating agentgateway this lab does not run), and the
-ServiceMonitor / OTel exporters are off (no Prometheus Operator, no OTLP
-gateway in kind). The chart also renders the shared `RemoteMCPServer`
-pointing agents at muster; note that kagent forwards the *caller's* token to
-muster, so agent tool calls through muster need a real Dex token on the way
-in — headless pokes at the unsecured controller API won't have one.
-
-On the [dev channel](platform.md#dev-channel) the runtime is kagent main
-(API v2: `Harness` + `AgentTemplate`, every agent an actor on Substrate,
-which the lab installs ahead of the platform); the Agent CRs and the heals
-above belong to the released line and are skipped there. Skills are the
-line's open question — an `AgentTemplate`'s git skill is fetched during the
-golden boot, which Substrate's egress gate refuses; `agentlab skills-test`
-is that proof, see [The skills proof](platform.md#the-skills-proof-the-golden-boot).
+the JWKS source is the lab Dex, the snapshot store the bundled RustFS, the
+ServiceMonitor / OTel exporters are off (the line serves no /metrics, no
+OTLP gateway in kind). A git skill of an `AgentTemplate` is fetched during
+the golden boot under Substrate's egress gate; `agentlab skills-test` is
+that proof, see [The skills proof](platform.md#the-skills-proof-the-golden-boot).
+On a chart that still serves `agents.kagent.dev` (the 0.10 product's 3.x
+line) the Agent CRs and the heals above apply instead.

docs/cli.md

@@ -47,7 +47,7 @@ kagent API v2's AgentTemplates on the [dev channel](platform.md#dev-channel)
 
 | Command | What it proves |
 |---|---|
-| `platform-test [email]` | Dex → muster → mcp-kubernetes → apiserver, the per-server OAuth sign-in challenge, the tool-group label on the fake fleet and, with observability on, mcp-prometheus and Backstage's metrics endpoint. See [The agent platform](platform.md). |
+| `platform-test [email]` | Dex → muster → mcp-kubernetes → apiserver (the identity proof: a viewer's kube-system Secrets *Forbidden* by the apiserver, a forged `x-user-id` changing nothing), with agents on the kagent controller route's JWT `Strict` policy (a call without a token refused at the edge; a valid token with a forged `x-user-id` attributed to the token's subject by the controller) and agent-manager writing as the caller (a viewer's create refused under the viewer's name), the per-server OAuth sign-in challenge, the tool-group label on the fake fleet and, with observability on, mcp-prometheus and Backstage's metrics endpoint. See [The agent platform](platform.md). |
 | `models-test [email]` | Managed models: 401 at the gateway without a token, then pull → ModelConfig → agent turn → MCP via muster → unload → delete, one backend per run. On `lmstudio` the teardown is the `501` refusal plus an unwire, since LM Studio serves no delete, and that run leaves its model downloaded. `--backend` picks one of `platform.modelManager.backends` (default: the first); `--model` a small, tool-calling capable model. See [Models](models.md). |
 | `agents-test [email]` | agent-manager through muster as the signed-in user: create → ready → update → delete; a viewer's create is Forbidden by the apiserver; the ServiceAccount holds no RBAC of its own. On kagent API v2 the AgentTemplate carries the user's field manager, is Ready on the Harness and its toolset rides on the per-agent muster carrier. |
 | `toolsets-test [email]` | Declared toolsets end to end: agent-manager requires one, the agent carries the header (the released kagent: on its Agent CR; kagent API v2: on the RemoteMCPServer its AgentTemplate binds), muster resolves and refuses per request, agents see their toolset through a turn on kagent, a per-server sign-in scopes a server's tools to the token, the portal's Tools step and apply path. `--model-config` picks the kagent ModelConfig the throwaway agents run on; `--skip-chat` skips the turns that need a model to answer; `--skip-portal` (kagent API v2) skips the portal's apply path. See [Toolsets](platform.md#toolsets-declared-tool-access). |
@@ -95,7 +95,6 @@ The flags pin a value regardless of the discovery, with or without
 | `--chart-version <x.y.z>` | The agent-platform chart release to install, an exact version (default: the release this agentlab was verified with, `config.DefaultChartVersion`). |
 | `--chart-path <dir>` | Install the agent-platform chart from a local checkout's `helm/agent-platform` directory instead of the pinned release; `--chart-path ""` clears it. See [Installing an unreleased chart](platform.md#installing-an-unreleased-chart). |
 | `--chart-branch <branch>` | The dev channel: follow this agent-platform branch's newest dev build — resolved now and on every `up`/`platform`, written to `chartVersion`; `--chart-branch ""` returns to the stable channel. Mutually exclusive with `--chart-path`; implies Substrate. See [Dev channel](platform.md#dev-channel). |
-| `--substrate[=false]` | Pin Substrate (kagent's actor runtime) on or off instead of following the chart channel (on with `--chart-branch`, off otherwise). |
 | `--model-manager[=false]` | Pin managed models on or off instead of following the host model servers the discovery finds (needs agents). |
 | `--model-manager-backends ollama,lmstudio` | Pin the host model servers, in order (`ollama`, `lemonade`, `lmstudio`); the first is model-manager's default backend. |
 

docs/getting-started.md

@@ -48,46 +48,57 @@ constraint, and it is a hard one**: the kube-scheduler refuses
 a pod whose CPU *request* does not fit, so a node that is 100m short simply
 leaves pods `Pending` forever. It does not degrade, it stalls.
 
-What a full default lab requests (measured from the chart renders at the
-pinned versions, plus kind's own control plane on a live node):
-
-| | CPU | Memory |
-|---|---|---|
-| kind's Kubernetes: apiserver, controller-manager, scheduler, etcd, CNI, CoreDNS | 950m | ~290 MiB |
-| the agent platform: muster + valkey, agentgateway + controller, mcp-kubernetes, agent-manager, model-manager, kagent + UI + postgres, Backstage | 1080m | ~1750 MiB |
-| Dex | 50m | 64 MiB |
-| the chart's Flux engine: the Flux Operator plus the `FluxInstance`'s source-controller and helm-controller (the lab shape brings it with the platform — it delivers every component and the agents) | 250m | 192 MiB |
-| observability: kube-state-metrics + mcp-prometheus | 300m | 328 MiB |
-| **total requests** | **≈ 2.6 CPU** | **≈ 2.5 GiB** |
-| the dev channel only (`--chart-branch`): Substrate's bundled PostgreSQL — its actor runtime, API server and data plane declare nothing | +1000m | +1 GiB |
-
-The memory column *understates* real use, and by a lot: the Prometheus server
-(its CR sets no `resources`), the prometheus-operator and node-exporter
-declare nothing at all, and Backstage requests 250 MiB while its Node process
-uses several times that. A node running the platform with Backstage and
-observability **off** was observed at 2.4 GiB actual — i.e. the whole
-requests budget — so a full lab wants roughly twice that.
+What a full default lab requests and uses (measured on a live lab on the 4.x
+line — agent-platform 4.7.11, kagent 0.11.0-gs.3, Substrate 0.0.27-gs.5 —
+2026-09-11; the first column is what the kube-scheduler is asked for, the
+second what the containers' memory working sets summed to):
+
+| | CPU requests | memory requests | memory in use |
+|---|---|---|---|
+| kind's Kubernetes: apiserver, controller-manager, scheduler, etcd, CNI, CoreDNS | 950m | 290 MiB | ~2.0 GiB (the apiserver 1.6 GiB after a day of platform churn) |
+| Dex | 50m | 64 MiB | 40 MiB |
+| the agent platform: muster + valkey, agentgateway + controller, mcp-kubernetes, agent-manager | 510m | 736 MiB | 245 MiB |
+| the agents runtime: kagent controller + UI | 200m | 384 MiB | 75 MiB |
+| Agent Substrate (from the chart): the WorkerPool's four gVisor workers at 250m/512Mi each; the control plane in `ate-system` (ate-api-server ×2, ate-controller, atelet, atenet router/egress/dns, RustFS) and the podcertificate-controller declare nothing | 1000m | 2048 MiB | 480 MiB (a worker idles at 9 MiB) |
+| the platform Postgres (from the chart): the CloudNativePG operator and the one-instance Cluster declare nothing | 0 | 0 | ~180 MiB (the operator 63, the instance 114 right after its bootstrap) |
+| model-manager | 55m | 80 MiB | 15 MiB |
+| Backstage | 20m | 250 MiB | 400 MiB |
+| the chart's Flux engine: the Flux Operator plus the `FluxInstance`'s source-controller and helm-controller (the lab shape brings it with the platform — it delivers every component and the agents) | 250m | 192 MiB | 320 MiB |
+| observability: kube-state-metrics + mcp-prometheus (the Prometheus server, its operator and node-exporter declare nothing) | 305m | 344 MiB | 710 MiB (the server 564 MiB) |
+| **total** | **≈ 3.3 CPU** | **≈ 4.3 GiB** | **≈ 4.4 GiB** |
+
+On a chart without Agent Substrate and the platform Postgres — the 0.10
+product's 3.x line — kagent's bundled PostgreSQL (250m / 256 MiB) takes the
+place of the two chart-shipped rows, and every agent is a pod of its own.
+
+The requests column and the use column disagree in both directions: Backstage,
+Prometheus and the kind apiserver use several times what they declare, while
+Substrate's WorkerPool reserves two GiB for workers that idle at 40 MiB — the
+reservation is capacity for the agents' sandboxes (one worker hosts one actor;
+its limits, 2 CPU / 2 GiB, bound that actor), not what runs idle. So the CPU
+floor is the requests (a request that does not fit never schedules), and the
+memory floor is the measured use with a quarter of headroom — for the turns
+(a Go ADK turn's working set on a worker is 55–70 MiB, about 0.3 core for a
+second or two) and for Backstage and Prometheus growing under load — never
+below the requests.
 
 Give docker at least:
 
 | | CPUs | Memory |
 |---|---|---|
-| the full default lab (platform + agents + observability + Backstage) | **4** | **6 GiB** (the floor is 5.1 GiB; whole GiB) |
-| platform + agents only (`configure --backstage=false --observability=false`) | 3 | 4 GiB (3.9 GiB) |
-| the full lab on the [dev channel](platform.md#dev-channel) (`--chart-branch`, Substrate on) | 5 | 8 GiB (7.1 GiB) |
-
-Those are the floors `agentlab up` enforces, and they already include room for
-the pods the platform creates at run time: every kagent agent is another pod,
-and `models-test` and `agents-test` each create one. `agentlab up` checks the
+| the full default lab (platform + agents + observability + Backstage) | **4** | **6 GiB** (the floor is 5.5 GiB; whole GiB) |
+| platform + agents only (`configure --backstage=false --observability=false`) | 4 (the WorkerPool is a CPU of requests by itself) | 5 GiB (4.2 GiB) |
+| the platform without agents (`configure --agents=false`) | 3 | 4 GiB (3.3 GiB) |
+
+Those are the floors `agentlab up` enforces — computed for what the chart
+about to be installed ships (its rendered roster, so a lab on the 3.x line is
+held to that line's smaller floor) — and they already include room for the
+pods the platform creates at run time: with Substrate a Go ADK turn inside a
+pre-provisioned worker, without it six agent pods. `agentlab up` checks the
 runtime before any cluster work — it prints the measured CPUs and memory next
-to this configuration's requests and floor on every boot, refuses below the
-CPU floor and warns below the memory one. Give it 6 CPUs and 8 GiB if you have
-them: the lab is then comfortable rather than exactly large enough.
-
-The second row is the one that has actually been measured on a live node: it
-requests ~2.3 CPU (no Backstage, no observability; the chart's Flux engine is
-always part of it) and sat at 2.4 GiB of real use — measured before the engine
-joined the platform, which adds 250m / 192 MiB of requests on top.
+to this configuration's requests, use and floor on every boot, refuses below
+the CPU floor and warns below the memory one. Give it 6 CPUs and 8 GiB if you
+have them: the lab is then comfortable rather than exactly large enough.
 
 Two CPUs — what a small Docker Desktop or Colima VM gives you — is not enough
 for any of it. The symptoms are specific, and worth recognising because