Meta chart 4.0: Agent Substrate ships inside the chart — the `substrate`/`substrate-crds` charts at the version the kagent pin requires, an idempotent CA/JWT-pool bootstrap Job, Substrate's Postgres on CNPG, the snapshot location as a per-installation value, a node selector for `atelet`, PolicyExceptions and network policies for the worker pods; Kubernetes 1.35 and the three feature gates as documented prerequisites (bumblebee-plans#51 D3)

[teemow]

Problem

kagent API v2 has no runtime without Agent Substrate: a Harness requires spec.substrate.workerPoolRef and snapshotPolicy.location, and every agent runs as a Substrate actor in a gVisor worker pod. The plan decided (bumblebee-plans#51 D3, grill of 2026-09-10) that Substrate ships inside meta chart 4.0 and goes to every installation — the Giant Swarm installations in the plan's order, then the customer installations — installed, bootstrapped and upgraded from the same chart and GitOps chain as kagent, with no imperative step per installation. The dev line poc/kagent-main (#330) treats Substrate as cluster infrastructure installed outside the chart (kagent-crds.substrate.enabled: false, the kagent chart's substrate subchart off, controller.substrate.* pointing at ate-system, substrateWorkerPool kagent-default with four gVisor workers on the line's ateom-gvisor build, ghcr.io/giantswarm/substrate/ateom-gvisor:0.0.27-dev…); agentlab installs the charts and runs a Go port of the bootstrap during up. That was a POC-phase arrangement.

What Substrate needs on a Giant Swarm management cluster (the plan's research notes 2026-09-09-substrate.md and 2026-09-09-dev-channel.md):

• The charts substrate-crds and substrate come from the Giant Swarm Substrate line (giantswarm/substrate, giantswarm/giantswarm#37757): oci://ghcr.io/giantswarm/substrate/helm, images under ghcr.io/giantswarm/substrate, at the line's build of the version the pinned kagent commit requires (kagent's go.mod replace directive → v0.0.26 today; the line publishes 0.0.27-dev.giantswarm.* builds and vX.Y.Z-gs.N releases, its FORK.md is the ledger). The kagent and kagent-crds charts carry both as switchable subcharts. Whichever delivery route is chosen (the kagent charts' substrate subchart switches — which then must be stamped with the line's repository and version by the fork, whose export still points at upstream's SUBSTRATE_REPO — or roster components), the installed Substrate is the line's.
• The bootstrap objects the charts mount but do not create: the CA pools service-dns-ca-pool and pod-identity-ca-pool (namespace podcertificate-controller-system), actor-id-jwt-pool, actor-id-ca-pool and the derived actor-id-ca-certs Secret (namespace ate-system), and the ate-api-authentication ConfigMap naming the cluster's service-account issuer — the apiserver's FQDN issuer, not a short form. Upstream does this with four kubectl-ate admin invocations (kagent's scripts/setup-cluster/setup-cluster.sh); the pool commands are Go in kubectl-ate's admin packages, and agentlab carries an in-cluster Go port of them (substratepools.go).
• Postgres for Substrate's control plane (a bundled StatefulSet by default; postgres.connectionString for an external database) and a snapshot object store (bundled RustFS with static credentials by default — a demo value; atelet uses the standard AWS credential chain, so S3 via IRSA works on CAPA; other providers' stores are open).
• Privileges restricted PSS refuses: atelet is a privileged DaemonSet with hostPorts 8085/9090 and hostPath mounts and has no node selector in the chart (it lands on control-plane nodes too); worker pods run as root with SYS_ADMIN, SYS_PTRACE, NET_ADMIN and more, AppArmor and seccomp Unconfined, a hostPath with HostToContainer propagation; the other Substrate Deployments declare no securityContext at all. Kyverno enforces restricted on the fleet, so each needs a PolicyException — the platform already ships exceptions in the exception namespace (kyvernoPolicies.policyExceptionNamespace; templates/kagent/policy-exception.yaml, templates/postgres/policy-exception.yaml).
• Cluster-scoped objects the chart brings: the SandboxConfig (per-architecture gVisor asset URL and sha256, fetched by atelet from gs://gvisor/releases/... at run time — not an OCI image, so a proxied installation needs it mirrored and re-pointed) and a ValidatingAdmissionPolicy.
• Mixed-architecture WorkerPools do not work (the golden snapshot is architecture-specific); the fleet's management clusters are single-architecture, so pools are pinned to one.
• Network policy: the connectivity chart's agent-pod CiliumNetworkPolicy and the kagent-declarative-seccomp PolicyException select app: kagent, the label of the v1alpha2 per-agent Deployments — nothing carries it on API v2. The worker pods (kagent's WorkerPool selector label kagent.dev/worker-pool) need egress to muster, the kagent controller, the LLM provider (or the agentgateway LLM listener) and DNS, and Substrate's own hops need rules: atelet ↔ ate-api, the internal router → a worker's tunnel on :443, the bootstrap API → the node agent (an egress rule upstream lacks), atenet-egress. Found on a Cilium always-enforce cluster in the dev channel: an actor's DNS lookup from the nested network namespace is forwarded rather than translated by Cilium's socket load balancer unless socketLB.hostNamespaceOnly is on (the Giant Swarm Cilium app's default); the internal router's connect to a healthy actor timing out, mutual authentication suspected, is still open there; two hops fail silently (nothing is logged when the bootstrap tunnel refuses an unactivated connection or the router's connect fails).
• Cluster prerequisites: Kubernetes 1.35 (Substrate uses certificates.k8s.io/v1beta1 PodCertificateRequest, which 1.34 serves only as v1alpha1) and the feature gates ClusterTrustBundle, ClusterTrustBundleProjection, PodCertificateRequest on kube-apiserver, kube-controller-manager and kubelet. feat: enable ClusterTrustBundle, ClusterTrustBundleProjection and PodCertificateRequest feature gates by default cluster#1005 turns them on by default; until that release reaches an installation the staff-only internal.advancedConfiguration.*.featureGates knob bridges (repeating the chart's default entries — Helm replaces lists). --runtime-config=api/all=true already serves the API group. Enabling them rolls the control plane and every node. The per-installation values are the fleet's work (giantswarm/shared-configs#732, extras/agent-platform under meta chart 4.0 (kagent API v2): per-installation cut-over pattern, the base README contract, and the final bound advance management-cluster-bases#738); this chart states the prerequisites.

Proposed solution

Substrate is a component of the 4.0 chart:

• Installed, bootstrapped and upgraded by the chart's GitOps chain. Whether through the kagent and kagent-crds charts' substrate subchart switches or as roster components from the OCI charts (mirrored for the fleet, images included) is the implementer's call under these constraints: one version pin next to the kagent pin, moved together on a re-pin; CRDs before CRs (the kagent-crds shape); the render refuses kagent on without Substrate; installable where ghcr.io is not reachable.
• A bootstrap Job reusing kubectl-ate's admin packages (or agentlab's substratepools.go port of them), run with the chart's hook identity (the hooks/_job.tpl pattern): mints the CA/JWT pools only when they are missing, never rotates existing material, creates ate-api-authentication with the FQDN issuer, is a no-op on every later upgrade, and reports what it did.
• Postgres on CNPG: Substrate's control-plane database on the platform's CNPG cluster (a Database on the existing cluster or its own — a value), the bundled StatefulSet off; the CNPG PolicyException pattern reused where needed. Substrate's database on the platform's CNPG cluster and the kagent database of Cut-over of an installation: a fresh kagent database on the existing CNPG cluster (the 0.10 database kept 30 days) and the migration Job that runs agent-manager's migrate once per installation as kagent-flux — rewrite portal-created Generic-chart releases to 1.x, emit the GitOps-owned diffs, then delete the v1alpha2 objects and CRDs (bumblebee-plans#51 D9, D11) #346 share one values shape (e.g. postgres.databases[]) — decided together with Cut-over of an installation: a fresh kagent database on the existing CNPG cluster (the 0.10 database kept 30 days) and the migration Job that runs agent-manager's migrate once per installation as kagent-flux — rewrite portal-created Generic-chart releases to 1.x, emit the GitOps-owned diffs, then delete the v1alpha2 objects and CRDs (bumblebee-plans#51 D9, D11) #346.
• The snapshot location is a per-installation value, required whenever kagent is on: S3 via IRSA on CAPA (bucket provisioning is the installation's infrastructure — documented, not rendered), the in-cluster store only for the lab (the bundled RustFS off unless the lab asks for it). The platform Harness's snapshotPolicy.location derives from the same value (Connectivity 4.0: one platform Harness per managed namespace — kagent type on the line's Go ADK digest, KAGENT_PROPAGATE_TOKEN in its environment, the meta chart's default WorkerPool, the snapshot location from values, admission by the label agent-platform.giantswarm.io/harness: kagent; WorkerPool replicas and pod template as values with lab-measured defaults (bumblebee-plans#51 D5) #344).
• Scheduling: atelet gets nodeSelector/tolerations values (default: worker nodes, never control-plane nodes); the WorkerPool is pinned to the installation's architecture; its replicas and pod template are Connectivity 4.0: one platform Harness per managed namespace — kagent type on the line's Go ADK digest, KAGENT_PROPAGATE_TOKEN in its environment, the meta chart's default WorkerPool, the snapshot location from values, admission by the label agent-platform.giantswarm.io/harness: kagent; WorkerPool replicas and pod template as values with lab-measured defaults (bumblebee-plans#51 D5) #344's values.
• Policy: PolicyExceptions for atelet, the worker pods and the other Substrate workloads in the platform's exception style and namespace, each naming only the rules that workload violates (precedent: the eBPF DaemonSet exception of observability-operator); the kagent-declarative-seccomp exception is re-targeted from app: kagent to the worker pods, or folded into theirs.
• Network policies in both flavours: the worker pods' egress (muster, the controller, the LLM path, DNS) and Substrate's own hops; the app: kagent agent-pod rules go. The lab renders no network policies, so the runtime proof is the first Giant Swarm installation in the rollout order (an actor's turn completes with Cilium enforcing; extras/agent-platform under meta chart 4.0 (kagent API v2): per-installation cut-over pattern, the base README contract, and the final bound advance management-cluster-bases#738), the render proof is the chart's assertions (make verify-kagent-netpol).
• Prerequisites documented in README and UPGRADE.md: Kubernetes 1.35, the three gates on the three components, feat: enable ClusterTrustBundle, ClusterTrustBundleProjection and PodCertificateRequest feature gates by default cluster#1005 and the bridging knob, the gVisor asset for proxied installations, the single-architecture pool. Where helm-controller's live Capabilities can see it, the render refuses kagent on a cluster that does not serve certificates.k8s.io/v1beta1/PodCertificateRequest with a message naming the gates (kubelet gates are not detectable at render time — the message says so).
• A security write-up in docs/ for installation owners and security reviewers (PRD user stories 25, 27): the node-level and cluster-level changes, the worker pods' capability set, the DaemonSet's privileges, the compensating controls (gVisor, network policy, namespace scoping), the open upstream items (worker-pod host escape is an open Critical in Substrate's threat model).

Acceptance criteria

• On a kind cluster with the gates (agentlab, on a lab with platform.chartPath — and platform.substrate.enabled: false if a branch is followed — so the chart is the only Helm owner; The lab runs the meta chart 4.0 topology as an installation will: Substrate and the platform Harness from the chart (up stops installing Substrate), JWT Strict on the controller route, a measured preflight budget, every image side-loaded agentlab#138 makes that the default on a 4.0 line), helm install of the 4.0 chart with kagent on brings up Substrate — ate-system pods Ready, atelet on every eligible node, the WorkerPool's workers Ready — with the bootstrap done by the chart and no kubectl-ate step; helm upgrade re-runs the bootstrap as a no-op; helm uninstall tears it down in order. — proven 2026-09-11 03:14–03:52Z (agentlab, PR feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359 → v4.3.0, d11dfce3): the chart installed substrate/substrate-crds at 0.0.27-gs.2 into ate-system (adopting the lab's releases by name), every ate-system pod Running, atelet on every node, WorkerPool kagent-default ready 4/4, the bootstrap hook Job completed, a second agentlab platform left every pool untouched (no kubectl-ate step); the helm uninstall order is render-asserted only — the windows handed Substrate back to agentlab instead of uninstalling (see "Lab proof" below).
• Values and schema: the Substrate version pin, the snapshot location (required with kagent on), atelet nodeSelector/tolerations, the Postgres wiring, the architecture pin; the README documents each. — done 2026-09-11 (PR feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359): components.substrate/substrate-crds versionRange >=0.0.27-gs.2 <0.0.28-0 held to one with the worker image by make verify-components, kagent.harness.snapshotLocation required with kagent on, substrate.atelet.{nodeSelector,tolerations,affinity}, substrate.postgres.enabled: auto + the postgres.databases map, the WorkerPool amd64 pin; both READMEs document them.
• Fleet render (make verify-auto, and a new verify-kyverno target or its assertions inside verify-auto): every Substrate workload is covered by a PolicyException naming only the rules it violates; no app: kagent selector remains in the chart. — done 2026-09-11 (PR feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359): new make verify-kyverno (tests/verify-kyverno.py, in CI's verify job) renders the pinned substrate chart with the forwarded values and holds substrate-atelet, substrate-workers, substrate-control-plane and substrate-podcertificate-controller to exactly the rules their workloads violate; verify-auto passes; the last app: kagent selector left with feat(connectivity): the kagent controller route is a GRPCRoute with the JWT policy on by default, the identity header set from the verified email claim, the UI path stripping it and the controller admitting agentgateway and the UI only #353 — none remains on main.
• make verify-kagent-netpol covers the worker-pod and Substrate policies in both flavours — done 2026-09-11 (PR feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359, templates/substrate/netpol.yaml; render-asserted in CI — the lab runs networkPolicy.enabled: false).
• On the first Giant Swarm installation in the rollout order a turn completes through the Dev Portal with Cilium enforcing — open: the runtime proof of the glean cut-over, giantswarm/giantswarm#37744.
• make verify-components-charts renders the Substrate charts at the pin with the forwarded values. — done 2026-09-11 (PR feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359): tests/verify-components-charts.py resolves substrate and substrate-crds from the release range the way Flux does and renders them at 0.0.27-gs.2 with the forwarded values.
• README/UPGRADE.md prerequisites section; the render-time detection where possible; the security write-up in docs/. — done 2026-09-11 (PR feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359): README and UPGRADE.md name Kubernetes 1.35 and the ClusterTrustBundle, ClusterTrustBundleProjection and PodCertificateRequest gates; a live render refuses a cluster without certificates.k8s.io/v1beta1/PodCertificateRequest; docs/substrate-security.md.
• agentlab's up consumes Substrate from the chart instead of installing it itself — open: The lab runs the meta chart 4.0 topology as an installation will: Substrate and the platform Harness from the chart (up stops installing Substrate), JWT Strict on the controller route, a measured preflight budget, every image side-loaded agentlab#138 (the lab windows used platform.substrate.enabled: false with platform.chartPath, so the chart adopted the lab's Substrate releases by name).
• Every agentlab proof passes on the 4.0 render before merge — proven 2026-09-11 03:46–03:52Z on the chart-installed Substrate at 0.0.27-gs.2: agentlab platform-test PASS and an AgentTemplate Ready on the platform Harness; agents-test/toolsets-test still assert the POC composer shape and are rewritten by Agent proofs assert the Generic chart 1.x contract: HelmRelease → AgentTemplate + per-agent RemoteMCPServer, commit-pinned skills, 1.x range; one agent helper; the v1alpha2 proofs, dispatch and heals retire with 4.0 agentlab#140 (recorded, not gating).

Dependencies

• The kagent line's pin (which Substrate version) and its carried patches for the upstream gaps found on real infrastructure — the FQDN issuer (giantswarm/giantswarm#37010; upstream engagement giantswarm/giantswarm#37742).
• The Giant Swarm Substrate line (giantswarm/giantswarm#37757): the substrate/substrate-crds charts and images this chart consumes, its carried patches on the giantswarm branch (the sandbox-config RBAC — fix: Helm-installed atelet cannot prewarm sandbox assets kagent-dev/substrate#33; the bootstrap-API egress rule and the architecture-aware sandbox-runtime reference — giantswarm/giantswarm#37742 rows 11 and 12; ledger FORK.md) and its release tags (vX.Y.Z-gs.N).
• feat: enable ClusterTrustBundle, ClusterTrustBundleProjection and PodCertificateRequest feature gates by default cluster#1005 for the fleet's default; the per-installation gates and the S3 bucket (giantswarm/shared-configs#732, extras/agent-platform under meta chart 4.0 (kagent API v2): per-installation cut-over pattern, the base README contract, and the final bound advance management-cluster-bases#738).
• The snapshot store on the non-AWS installations — three CAPZ, one Cloud Director — and the values it needs from this chart (backend name, endpoint, path style, credential Secret): giantswarm/giantswarm#37756.
• Connectivity 4.0: one platform Harness per managed namespace — kagent type on the line's Go ADK digest, KAGENT_PROPAGATE_TOKEN in its environment, the meta chart's default WorkerPool, the snapshot location from values, admission by the label agent-platform.giantswarm.io/harness: kagent; WorkerPool replicas and pod template as values with lab-measured defaults (bumblebee-plans#51 D5) #344 (the WorkerPool values, the snapshot location's second consumer).

Plan: bumblebee-plans#51 — PRD decisions D3, D5, D15; sections "The kagent line the platform runs" (the Substrate bullet), "Connectivity and meta chart", "Further Notes"; research 2026-09-09-substrate.md, 2026-09-10-cluster-chart-feature-gates.md. Epic: giantswarm/giantswarm#37705. Dev line: #330 (not modified by this issue). Related: #317 — the runtime slice for workload clusters includes Substrate under 4.0.

Implementation (PR #359, on main after #350, #360 and #353)

Decisions made while implementing, folded in here so the description stays the source of truth:

• Delivery route: roster components — components.substrate-crds and components.substrate from oci://ghcr.io/giantswarm/substrate/helm (the Giant Swarm Substrate line), both following components.kagent when unset (the render refuses kagent on with either off), both in ate-system through the new roster key components.<name>.targetNamespace, one version pin = the same range in the two versionRanges (>=0.0.27-gs.2 <0.0.28-0, the kagent entry's shape), its floor the tag of kagent.substrateWorkerPool.workerImage (make verify-components holds them to one). v0.0.27-gs.2 (giantswarm/substrate @ ef304330) = the egress gateway's CONNECT-time authorization as a frontend policy of the agentgateway v1.5.1-gs.2 dataplane (fork(publish): pin the agentgateway line's v1.5.1-gs.2 — CONNECT-time authorization that admits a resuming actor substrate#7, feat: bundle agentgateway-crds as a sub-chart dependency #9) + the connection string from a Secret and the atelet scheduling knobs (feat(helm): the Postgres connection string from a Secret (kagent-dev/substrate#32) and atelet scheduling knobs substrate#8). The release history of a component with its own targetNamespace is stored there too (storageNamespace; where helm -n <ns> keeps it), so a release installed by hand in that namespace is adopted by name — without it helm-controller installed a second substrate beside the lab's (found in the first lab window). Not the kagent charts' substrate subchart switches: own Flux status per release, explicit order, a mirrorable repository, Substrate's namespace is not kagent's, no fork stamping of SUBSTRATE_REPO. The pin is the dev build the kagent line was proven with until the Substrate line's first release tag; then the release range >=0.0.27-gs.1 <0.0.28-0.
• Install order (the cut-over sequencing decision): substrate-crds, kagent-crds → agent-platform-connectivity → substrate → kagent → the managers. Connectivity dependsOn the CRD components, never kagent/substrate — those depend on connectivity, whose hooks mint what their pods start against; components.kagent.installDisableWait goes.
• Bootstrap: neither line publishes a kubectl-ate image and kubectl-ate admin make-*-pool is a plain Create (not idempotent), so the connectivity release's pre-install,pre-upgrade hook Job <release>-substrate-bootstrap mints the same key material with openssl (init container) + kubectl, in the pools' wire format (PKCS#8 keys — validated in Go against x509.ParsePKCS8PrivateKey/ParseCertificate, which caught an SEC1 key from openssl pkey -outform DER), creates the two namespaces bare when missing, and never touches an existing pool (every upgrade logs present); the trust anchor follows the pool, the authentication ConfigMap is kept and its issuer compared. Identity <release>-hooks (a ClusterRole on secrets/configmaps/namespaces for the hook's lifetime).
• Database on CNPG: postgres.databases is a map (Helm merges maps; Cut-over of an installation: a fresh kagent database on the existing CNPG cluster (the 0.10 database kept 30 days) and the migration Job that runs agent-manager's migrate once per installation as kagent-flux — rewrite portal-created Generic-chart releases to 1.x, emit the GitOps-owned diffs, then delete the v1alpha2 objects and CRDs (bumblebee-plans#51 D9, D11) #346 adds kagent-v2 to it) — one CNPG Database and one derived connection Secret <clusterName>-<key>-app per entry, written by the post-install,post-upgrade hook <release>-postgres-databases into the Cluster namespace and every secretNamespaces entry. substrate.postgres.enabled: auto resolves in the meta chart's shape pass: the Cluster's Database with postgres.enabled (and a derived substrate.postgres.connectionStringSecretRef), the bundled StatefulSet without it. The substrate chart's postgres.connectionStringSecretRef and atelet.{nodeSelector,tolerations,affinity} are carried patches of the line: feat(helm): the Postgres connection string from a Secret (kagent-dev/substrate#32) and atelet scheduling knobs substrate#8 (a cherry-pick of the open upstream Support PostgreSQL connection secrets kagent-dev/substrate#32, and ours with the upstream branch upstream/atelet-scheduling; giantswarm/giantswarm#37742 rows 23–24).
• Snapshot store: kagent.harness.snapshotLocation (the contract name; Connectivity 4.0: one platform Harness per managed namespace — kagent type on the line's Go ADK digest, KAGENT_PROPAGATE_TOKEN in its environment, the meta chart's default WorkerPool, the snapshot location from values, admission by the label agent-platform.giantswarm.io/harness: kagent; WorkerPool replicas and pod template as values with lab-measured defaults (bumblebee-plans#51 D5) #344 renders it) is required whenever kagent is on. A non-AWS S3-compatible store takes substrate.atelet.extraEnv; ate-api-server has no such knob upstream yet — a gap for giantswarm/giantswarm#37756.
• Kyverno: one PolicyException per workload (substrate-atelet, substrate-workers on ate.dev/worker-pool — the label ate-controller puts on the worker pods; kagent.dev/worker-pool is the WorkerPool CR's — substrate-control-plane, substrate-podcertificate-controller), kyvernoPolicies.rules (rule → ClusterPolicy) replaces the four seccomp*/volumeTypes* keys, kagent-declarative-seccomp is gone; make verify-kyverno computes every workload's violations from the rendered pod specs and holds the exceptions to them.
• Network policies: the actors' outbound connections leave through atenet-egress (the tunnel), so the actors' allow-list (muster, the controller, the LLM path, DNS) lives on the egress gateway and the worker pods reach only the egress gateway, the dns and the cluster DNS — a correction of the Problem's "worker pods need egress to muster…" wording; ate-api → atelet :8085 (the rule upstream lacks) and the other hops in both flavours.
• Prerequisites: a live render refuses a cluster without certificates.k8s.io/v1beta1/PodCertificateRequest, naming the gates (kubelet's undetectable); helm template/CI never refused. docs/substrate-security.md is the write-up.
• On the connectivity 4.0.x train: rebased onto main after feat(connectivity): the kagent catalog renders at kagent.dev/v1alpha3, the shared muster RemoteMCPServer is retired (the agent chart 1.x renders one per agent), the muster MCP URL reaches the composers from one helper, the Kyverno Agent mutations go #360 (Connectivity 4.0: ModelConfig and RemoteMCPServer templates at kagent.dev/v1alpha3; the shared cross-namespace muster RemoteMCPServer is retired — the Generic chart renders one per agent — and the Kyverno Agent/srt-settings mutations go (kagent API v2, bumblebee-plans#51 D7) #299: the v1alpha3 catalog, the branch publish job) and feat(connectivity): the kagent controller route is a GRPCRoute with the JWT policy on by default, the identity header set from the verified email claim, the UI path stripping it and the controller admitting agentgateway and the UI only #353 (Connectivity 4.0: the kagent controller route — JWT policy on by default in Strict mode with a request transformation that sets the identity header from the verified email claim and drops any inbound identity header, the UI path strips the identity header, a gRPC-capable route on the controller path for Swarmgeist and the Dev Portal (bumblebee-plans#51 D4, D13, D16) #345: the controller GRPCRoute; the two rewrites of the controller-ingress policies met in templates/kagent/netpol.yaml, _helpers.tpl and the CI verify list — the union is on the branch). Merge order: feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops #359 → feat(connectivity): one platform Harness per managed namespace #361 (Connectivity 4.0: one platform Harness per managed namespace — kagent type on the line's Go ADK digest, KAGENT_PROPAGATE_TOKEN in its environment, the meta chart's default WorkerPool, the snapshot location from values, admission by the label agent-platform.giantswarm.io/harness: kagent; WorkerPool replicas and pod template as values with lab-measured defaults (bumblebee-plans#51 D5) #344) → feat(connectivity): cut-over — the kagent_v2 database with a derived connection Secret, and the agent-manager migrate Job per installation #354 (Cut-over of an installation: a fresh kagent database on the existing CNPG cluster (the 0.10 database kept 30 days) and the migration Job that runs agent-manager's migrate once per installation as kagent-flux — rewrite portal-created Generic-chart releases to 1.x, emit the GitOps-owned diffs, then delete the v1alpha2 objects and CRDs (bumblebee-plans#51 D9, D11) #346); the orchestrator merges.

State: PR #359 lab-proven and CI-green, ready to merge. Lab proof (agentlab-dev2, under the lab lock, 03:14–03:52Z, three windows): with platform.chartPath on the branch and platform.substrate.enabled: false, helm-controller adopted agentlab's substrate/substrate-crds releases in ate-system by name ("found existing release in storage" → upgrade, revision 33 → 34 at 0.0.27-gs.2, one history); HelmReleases substrate-crds, substrate, kagent-crds, kagent, agent-platform-connectivity Ready with the dependsOn graph as designed; every ate-system pod Running on gs.2 (ate-api-server ×2, ate-controller, atelet on every node, atenet-egress 2/2 on agentgateway v1.5.1-gs.2, atenet-router 2/2, dns); WorkerPool kagent-default on ateom-gvisor:0.0.27-gs.2 ready 4/4; the bootstrap hook Job completed and a second agentlab platform left every pool at the same resourceVersion (the CA/JWT pools' uid/resourceVersion equal the BEFORE snapshot throughout — no kubectl-ate step); agentlab platform-test PASS; an AgentTemplate reached Ready on the platform Harness — an actor booted on the chart-installed WorkerPool through the chart-installed ate-api/atelet/atenet. The hand-back to agentlab (the lab's own Substrate install) worked without a helm uninstall; the lab was restored after each window. Two lab-tooling gaps for giantswarm/agentlab#138 (worked around in the window's overlay): agentlab platform hard-fails its wait on the agent-manager MCPServer when components.agent-manager is legitimately off; its chartPath-shaped values set kagent.serviceMonitor.enabled: true for a kagent line without /metrics. agents-test/toolsets-test against agent-manager 1.x's composed release are giantswarm/agentlab#140. helm uninstall tear-down order is not lab-proven (the window hands Substrate back instead of uninstalling it). Not lab-verifiable, as the acceptance criteria say: Kyverno admission, Cilium enforcement, CNPG for Substrate (render-proven; the derived-Secret hook proven against a CNPG-shaped Secret), IRSA — the first Giant Swarm installation of the rollout is the runtime proof.

Working rule: kagent, Substrate and related upstream projects

General rule of epic giantswarm/giantswarm#37705, carried on every sub-issue.

Implementing this epic will surface bugs and gaps in kagent, Substrate and the projects around them (kmcp, agentgateway, the Go ADK, the harnesses). For each of them:

1. Check upstream first. Search the upstream repository's issues and pull requests, open and closed (kagent-dev/kagent, kagent-dev/substrate, kagent-dev/kmcp, …), and the rows already collected in giantswarm/giantswarm#37742. If upstream tracks it or a pull request is in flight, we align with that work — test it, review it, add our evidence — rather than open a competing change.
2. Change it in our fork. A change we need lands as a carried patch on the consumed branch of our line — giantswarm/kagent-upstream for kagent, giantswarm/substrate for Substrate, the giantswarm/ fork of any other upstream project (created on the same pattern when there is none). One commit per patch, written upstream-ready (upstream's conventions and tests, DCO sign-off, nothing Giant Swarm-specific in it), recorded in the fork's ledger (FORK.md).
3. Prepare the upstream pull request in the fork; do not open it yet. The upstream-facing branch and pull-request text are prepared in the fork and listed as a row in giantswarm/giantswarm#37742 — that list is the review queue. Nothing is opened against upstream before the team has reviewed it there. We send in considered batches and only what is useful to upstream, never a stream of Giant Swarm-specific changes.
4. Giant Swarm-specific behaviour never goes upstream. It lives in the meta chart, the connectivity chart, or in a patch the ledger marks as ours to keep.

Upstream issues we want to file follow the same path: collected in giantswarm/giantswarm#37742, reviewed by the team, then filed.