feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops

[teemow]

Problem

kagent API v2 has no runtime without Agent Substrate: a Harness requires spec.substrate.workerPoolRef and snapshotPolicy.location, and every agent runs as a Substrate actor in a gVisor worker pod. The plan (bumblebee-plans#51 D3) decided that Substrate ships inside meta chart 4.0 and goes to every installation — installed, bootstrapped and upgraded from the same chart and GitOps chain as kagent, with no imperative step per installation. Until now Substrate was cluster infrastructure installed outside the chart (agentlab installs the charts and runs a Go port of kubectl-ate admin during up; kagent.controller.substrate.* points at ate-system).

Solution

Substrate is a component of the 4.0 chart. Base main (4.0 is on main since #341; #299 and #345 landed before this — the branch is rebased on them). Three commits: Substrate in the chart, the Substrate pin as the line's release range, the release history of a component with its own targetNamespace stored there. Merge order on the connectivity 4.0.x train: this → #361 (#344) → #354 (#346).

Delivery route — roster components, not the kagent charts' substrate subchart switches. components.substrate-crds (the three ate.dev CRDs as templates) and components.substrate (the control plane in ate-system, the podcertificate-controller in its namespace) from the Giant Swarm Substrate line (oci://ghcr.io/giantswarm/substrate/helm). Both follow components.kagent when unset (the render refuses kagent on with either off — agent-platform.validateSubstrate), both land in ate-system through the new roster key components.<name>.targetNamespace, both carry one version pin — the same range in both versionRanges, its floor the tag of kagent.substrateWorkerPool.workerImage, held to one by make verify-components. Why: each release has its own Flux status on an installation, the order is explicit, the repository is a value (a mirror for a proxied installation, with substrate.image.registry), Substrate's namespace is not kagent's, and no fork stamping of SUBSTRATE_REPO into the kagent charts is needed. The pin is the line's release range >=0.0.27-gs.2 <0.0.28-0 (the kagent entry's shape), its floor 0.0.27-gs.2 the BOM pin and the worker image's tag. v0.0.27-gs.2 is upstream's anticipated 0.0.27 with the carried patches this chart forwards values for: the egress gateway authorizes a resuming actor at CONNECT time as a frontend policy of its agentgateway v1.5.1-gs.2 dataplane (giantswarm/substrate#7, #9), ate-api-server reads its connection string from a Secret and the atelet DaemonSet takes scheduling knobs (#8).

Install order (implements the program's decision on the cut-over sequencing): substrate-crds, kagent-crds → agent-platform-connectivity → substrate → kagent → the managers. The connectivity release dependsOn the CRD components and never kagent or substrate; those two dependsOn connectivity, whose hooks mint what their pods start against — so nothing waits on a pod that waits on a later release, and components.kagent.installDisableWait goes.

Bootstrap — the connectivity release's pre-install,pre-upgrade hook Job <release>-substrate-bootstrap (templates/substrate/bootstrap.yaml; the agent-platform.hooks.job include in templates/_hooks.tpl, identity <release>-hooks): mints the CA pools service-dns-ca-pool, pod-identity-ca-pool, the JWT authority pool actor-id-jwt-pool, the CA pool actor-id-ca-pool, the trust anchor actor-id-ca-certs and the ConfigMap ate-api-authentication with the apiserver's own issuer (from its OpenID discovery document — the FQDN on kubeadm, the IRSA issuer on CAPA) with openssl in an init container and kubectl, in the pools' wire format; creates the two namespaces bare when missing (the substrate release adopts them); never touches a pool that exists — every upgrade logs present. Neither line publishes a kubectl-ate image, and kubectl-ate admin make-*-pool is a plain Create (not idempotent), so the Job is a shell port of the same key material rather than a wrapper around it.

Database on CNPG — postgres.databases (a map: Helm merges maps, so the chart ships defaults and an installation adds entries; 1b's #346 adds kagent-v2 to the same map): one CNPG Database <clusterName>-<key> per entry plus a derived connection Secret <clusterName>-<key>-app written by the post-install,post-upgrade hook <release>-postgres-databases into postgres.namespace and every secretNamespaces entry (CNPG mints <clusterName>-app for the initdb database only). The shipped substrate entry is ate-api-server's database; substrate.postgres.enabled: auto resolves in the meta chart's shape pass (the Cluster's Database with postgres.enabled and a derived substrate.postgres.connectionStringSecretRef; the bundled StatefulSet without the Cluster — a lab). The substrate chart's postgres.connectionStringSecretRef is the carried patch giantswarm/substrate#8 (cherry-pick of the open upstream kagent-dev/substrate#32).

Snapshot store — kagent.harness.snapshotLocation (the contract name; #344 renders it into the Harness) is required whenever kagent is on; S3 via IRSA on CAPA (bucket provisioning documented, not rendered), an S3-compatible store through substrate.atelet.extraEnv, a lab's in-cluster store.

Scheduling — substrate.atelet.{nodeSelector,tolerations,affinity} (carried patch, giantswarm/substrate#8, in v0.0.27-gs.2; default: every node the DaemonSet controller admits — the worker nodes on a cluster with tainted control planes); the WorkerPool pinned to amd64 (kagent.substrateWorkerPool.template.nodeSelector).

Kyverno — one PolicyException per workload (substrate-atelet, substrate-workers on ate.dev/worker-pool, substrate-control-plane, substrate-podcertificate-controller), each naming exactly the rules its pod spec violates with their autogen- copies; kyvernoPolicies.rules (rule → ClusterPolicy) replaces the four seccomp*/volumeTypes* keys; the kagent-declarative-seccomp exception (app: kagent) is gone. New make verify-kyverno computes every workload's violations from the rendered pod specs (and the worker pod from ate-controller's template) and holds the exceptions to them.

Network policies (both flavours, templates/substrate/netpol.yaml) — Substrate's hops and the actors' destinations on the egress gateway atenet-egress, where an actor's outbound connections leave (muster, the kagent controller, the LLM listener/provider, DNS); the worker pods reach only the egress gateway, the dns and the cluster DNS; the app: kagent agent-pod rules go; the kagent controller opens ate-api and the router and admits the egress gateway.

Prerequisites — README "Agent Substrate", UPGRADE.md: Kubernetes 1.35, the three gates on the three components, giantswarm/cluster#1005 and the cluster chart's bridging knob, the gVisor asset for proxied installations, one architecture per pool. A live render refuses a cluster without certificates.k8s.io/v1beta1/PodCertificateRequest, naming the gates (kubelet's undetectable — the message says so); helm template/CI never refused. Security write-up: docs/substrate-security.md.

Verification

• Offline: all 19 make verify-* targets green, plus the new verify-kyverno; verify-kagent-netpol, verify-postgres, verify-components, verify-components-charts, verify-auto, verify-modes (golden diff filtered by GOLDEN_EXCLUDE for the two intentionally changed objects) extended; helm lint both charts; pre-commit run --all-files (schemas, helm-docs).
• make verify-components-charts and make verify-kyverno render the substrate chart at the pin with the forwarded values — red while the pin was the dev build that predates feat(helm): the Postgres connection string from a Secret (kagent-dev/substrate#32) and atelet scheduling knobs substrate#8's postgres.connectionStringSecretRef (the checks doing their job), green since the pin is v0.0.27-gs.2 (the pin commit: substrate and substrate-crds render the forwarded values at the range and at the BOM pin; every Substrate workload's violations match its exception).
• Lab, additive (agentlab-dev2, own namespaces, no platform roll): both hook Jobs run from the rendered chart objects — the databases hook against a CNPG-shaped Secret derived <cluster>-second-app into the Cluster namespace and a secretNamespaces entry with dbname/uri/jdbc-uri/pgpass rewritten, a re-run logs present and changes nothing (resourceVersion unchanged); the bootstrap hook (its two namespaces retargeted to scratch names) created the three CA pools, the JWT pool, the trust anchor and the authentication ConfigMap with the apiserver's FQDN issuer, a re-run logs present ×8. The minted pools were validated in Go against Substrate's wire format (x509.ParsePKCS8PrivateKey, ParseCertificate, key ↔ cert, self-signature, IsCA + keyCertSign, the trust anchor equals the active root, ES256 = ECDSA P-256) — which caught the JWT key coming out as SEC1 from openssl pkey -outform DER in alpine/openssl 3.5; the hook writes PKCS#8 with openssl pkcs8 -topk8 -outform DER now.
• Lab, the hand-over window (agentlab-dev2, under the lab lock, three windows 03:14–03:52Z; platform.chartPath on this branch, platform.substrate.enabled: false, the overlay lab-overlay-342.yaml: connectivity → this branch's dev build, siblings on the lab's pins, kagent.harness.snapshotLocation: s3://ate-snapshots/kagent, the bundled Postgres and RustFS, agent-manager at the released 1.x, kagent.serviceMonitor.enabled: false — Meta chart 4.0: components.kagent moves to the Bumblebee-owned kagent line (kagent + kagent-crds from giantswarm/kagent-upstream, one pinned build, the Go ADK image by digest), the ten bundled example agents go, the controller ServiceMonitor is gated off, the dev channel's semverFilter/range defaults return to release ranges (kagent API v2, bumblebee-plans#51 D2/D11/D12) #341's platform-test bridge). helm-controller adopted agentlab's substrate / substrate-crds releases in ate-system by name ("release not managed by controller: found existing release in storage" → upgrade: revision 33 → 34, chart 0.0.27-gs.2, one history, no second one in the platform namespace — the storageNamespace commit; the first window showed the default storage namespace produces a second release beside the hand-installed one instead). HelmReleases substrate-crds, substrate, kagent-crds, kagent, agent-platform-connectivity Ready with the dependsOn graph as designed; every ate-system pod Running on gs.2 (ate-api-server ×2, ate-controller, atelet on every node, atenet-egress 2/2 on agentgateway v1.5.1-gs.2, atenet-router 2/2, dns; the bundled postgres and rustfs kept); WorkerPool kagent-default on ateom-gvisor:0.0.27-gs.2 ready 4/4; the bootstrap hook Job completed and a second agentlab platform left every pool at the same resourceVersion (idempotent; the CA/JWT pools' uid/resourceVersion equal the BEFORE snapshot throughout); agentlab platform-test PASS; an AgentTemplate (kagent.dev/v1alpha3, label kagent.dev/harness: kagent) reached Ready on the platform Harness within the wait — an actor booted on the chart-installed WorkerPool through the chart-installed ate-api / atelet / atenet. Hand-back proven three times (suspend + finalizer drop + delete of the two HelmRelease CRs → no helm uninstall; agentlab's own upgrade continues the same history), the lab restored to its baseline after each window (snapshot diff: only revision numbers and pod names). agents-test / toolsets-test return agentlab's POC-shape expectation against agent-manager 1.x's composed release (Agent proofs assert the Generic chart 1.x contract: HelmRelease → AgentTemplate + per-agent RemoteMCPServer, commit-pinned skills, 1.x range; one agent helper; the v1alpha2 proofs, dispatch and heals retire with 4.0 agentlab#140; the same in the sibling windows) — recorded, not gating.
• Rebased onto main after feat(connectivity): the kagent catalog renders at kagent.dev/v1alpha3, the shared muster RemoteMCPServer is retired (the agent chart 1.x renders one per agent), the muster MCP URL reaches the composers from one helper, the Kyverno Agent mutations go #360 (Connectivity 4.0: ModelConfig and RemoteMCPServer templates at kagent.dev/v1alpha3; the shared cross-namespace muster RemoteMCPServer is retired — the Generic chart renders one per agent — and the Kyverno Agent/srt-settings mutations go (kagent API v2, bumblebee-plans#51 D7) #299: the v1alpha3 catalog, the branch publish job) and feat(connectivity): the kagent controller route is a GRPCRoute with the JWT policy on by default, the identity header set from the verified email claim, the UI path stripping it and the controller admitting agentgateway and the UI only #353 (Connectivity 4.0: the kagent controller route — JWT policy on by default in Strict mode with a request transformation that sets the identity header from the verified email claim and drops any inbound identity header, the UI path strips the identity header, a gRPC-capable route on the controller path for Swarmgeist and the Dev Portal (bumblebee-plans#51 D4, D13, D16) #345: the controller GRPCRoute — its rewrite of the controller-ingress policies and this PR's Substrate policies met in templates/kagent/netpol.yaml, _helpers.tpl and the CI verify list; the union is on this branch).

Not lab-verified, and why

Kyverno admission (the lab has no Kyverno), Cilium enforcement of the network policies (no CNI policy enforcement in kind), the CNPG database and derived Secret for Substrate (the lab runs kagent's bundled Postgres, no CNPG operator — render-proven by verify-postgres; the derived-Secret hook was proven against a CNPG-shaped Secret by #354's lab run), IRSA against S3 (the lab's store is the bundled in-cluster one). The runtime proof of the policies is the first Giant Swarm installation of the rollout. Two lab-tooling gaps met on the way, worked around in the window's overlay and left to giantswarm/agentlab#138 (the lab consuming Substrate from the chart as an installation will): agentlab platform hard-fails its post-install wait on the agent-manager MCPServer when components.agent-manager is legitimately off, and its chartPath-shaped values set kagent.serviceMonitor.enabled: true for a kagent line that serves no /metrics. agents-test / toolsets-test on agent-manager 1.x's composed release are giantswarm/agentlab#140.

Teammate alignment

• marians' fix(connectivity): the actors' egress opens the host model servers model-manager fronts #278 adds the host model servers' egress into <release>-kagent-agent-muster-egress, which this PR removes (the v1alpha2 agent pods are gone); the same rules belong on substrate-atenet-egress (the actors' egress gateway). Rebase-after: whoever merges second moves the block; his PR is untouched.
• Connectivity 4.0: the kagent controller route — JWT policy on by default in Strict mode with a request transformation that sets the identity header from the verified email claim and drops any inbound identity header, the UI path strips the identity header, a gRPC-capable route on the controller path for Swarmgeist and the Dev Portal (bumblebee-plans#51 D4, D13, D16) #345 owns the controller-ingress block of templates/kagent/netpol.yaml; this PR removes the app: kagent peer there (no such pod on API v2) and adds the egress gateway as a separate policy (substrate-actors-to-kagent-controller) — the later merger resolves.
• Connectivity 4.0: ModelConfig and RemoteMCPServer templates at kagent.dev/v1alpha3; the shared cross-namespace muster RemoteMCPServer is retired — the Generic chart renders one per agent — and the Kyverno Agent/srt-settings mutations go (kagent API v2, bumblebee-plans#51 D7) #299 deletes declarative-agent-{pod-security,srt-settings}.yaml; verify-kagent-netpol allows app: kagent in exactly those two files until then.
• Cut-over of an installation: a fresh kagent database on the existing CNPG cluster (the 0.10 database kept 30 days) and the migration Job that runs agent-manager's migrate once per installation as kagent-flux — rewrite portal-created Generic-chart releases to 1.x, emit the GitOps-owned diffs, then delete the v1alpha2 objects and CRDs (bumblebee-plans#51 D9, D11) #346 (1b) adopts postgres.databases as a map and adds kagent-v2; Connectivity 4.0: one platform Harness per managed namespace — kagent type on the line's Go ADK digest, KAGENT_PROPAGATE_TOKEN in its environment, the meta chart's default WorkerPool, the snapshot location from values, admission by the label agent-platform.giantswarm.io/harness: kagent; WorkerPool replicas and pod template as values with lab-measured defaults (bumblebee-plans#51 D5) #344 renders kagent.harness.snapshotLocation and adds capacity knobs to kagent.substrateWorkerPool.template.

Fixes #342

[circleci-architect]

Published Helm chart agent-platform

4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c

Chart	agent-platform
Version	4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c
OCI reference	oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c
Digest	sha256:307cad0cffa4fdd203a4c9efdbdcbabe2575c57c256d95358b04f128dac68218
Registry	public — gsoci.azurecr.io
Git catalog	giantswarm-test-catalog (index)

Pull this chart

helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c

_{Posted by architect-orb · build 5445 · commit a38b66c · updated in place on every push}

[circleci-architect]

Published Helm chart agent-platform-connectivity

4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c

Chart	agent-platform-connectivity
Version	4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c
OCI reference	oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c
Digest	sha256:79fdcad2f559a83d9f7a903f5b926cf063130b04837650829b01c27aec95385e
Registry	public — gsoci.azurecr.io
Git catalog	giantswarm-test-catalog (index)

Pull this chart

helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.2.1-dev.substrate-in-chart.2026-09-11.03-54-03.ha38b66c

_{Posted by architect-orb · build 5444 · commit a38b66c · updated in place on every push}