feat(meta): Agent Substrate ships inside the chart — the Substrate line's substrate and substrate-crds components on one pin, the bootstrap hook, Substrate's database on the platform CNPG Cluster, the snapshot location, atelet scheduling, PolicyExceptions and network policies for the worker pods and Substrate's hops

.circleci/custom.yml

@@ -13,7 +13,7 @@
 #     `make verify-engine` + `make verify-self` + `make verify-labels` + `make verify-postgres` +
 #     `make verify-secrets` + `make verify-login-connector` +
 #     `make verify-components` + `make verify-auto` + `make verify-identity` +
-#     `make verify-wiring` + `make verify-llm-routing` + `make verify-kagent-crds` checks on branches (ingress-mode guards, the global.*
+#     `make verify-wiring` + `make verify-llm-routing` + `make verify-kagent-crds` + `make verify-kyverno` checks on branches (ingress-mode guards, the global.*
 #     contract, the model-manager / agent-manager wiring, the kagent tool-server
 #     egress, app-of-apps meta-package render, the bundled Flux engine's two
 #     shapes, the self-management shapes, the postgres.backup wiring and guards,
@@ -46,7 +46,8 @@ jobs:
         name: install helm + make + PyYAML
         command: |
           # python3 is in cimg/base, PyYAML is not: tests/verify-kagent-crds.py
-          # parses the rendered objects and the kagent line's CRDs with it.
+          # and tests/verify-kyverno.py parse rendered objects (and the kagent
+          # line's CRDs) with it.
           (command -v make >/dev/null 2>&1 && python3 -c 'import yaml' 2>/dev/null) || (sudo apt-get update -qq && sudo apt-get install -y -qq make python3-yaml)
           # Pinned (not `main`) so a mid-queue Helm minor can't change behaviour;
           # 3.17.x is well past OCI dependency GA (3.8). Avoids the curl|bash pattern.
@@ -100,7 +101,7 @@ jobs:
         # any render) are the single source of truth, also run locally. The meta chart's one
         # dependency is the flux-engine subchart, a directory in its charts/,
         # which needs no `helm dependency build`.
-        command: make verify-modes verify-global verify-meta verify-engine verify-self verify-labels verify-insecure verify-managers verify-kagent-netpol verify-kagent-route verify-kagent-discovery verify-kagent-crds verify-postgres verify-secrets verify-login-connector verify-presets verify-components verify-components-charts verify-auto verify-identity verify-wiring verify-llm-routing
+        command: make verify-modes verify-global verify-meta verify-engine verify-self verify-labels verify-insecure verify-managers verify-kagent-netpol verify-kagent-route verify-kagent-discovery verify-kagent-crds verify-kyverno verify-postgres verify-secrets verify-login-connector verify-presets verify-components verify-components-charts verify-auto verify-identity verify-wiring verify-llm-routing
 
 workflows:
   build:

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **Agent Substrate ships inside the chart** (meta chart 4.0, giantswarm/agent-platform#342). kagent API v2 has no runtime without it — every agent is a Substrate actor in a gVisor worker pod — so two roster entries deliver it from the Giant Swarm Substrate line ([giantswarm/substrate](https://github.com/giantswarm/substrate), branch `giantswarm`: the upstream release kagent's `go.mod` pins plus the carried patches; `FORK.md` there is the ledger), `oci://ghcr.io/giantswarm/substrate/helm`: `components.substrate-crds` (the three `ate.dev` CRDs as templates) and `components.substrate` (the control plane — `ate-api-server`, `ate-controller`, the `atenet` router, egress gateway and dns, the per-node `atelet` DaemonSet — in `ate-system`, the `podcertificate-controller` in its own namespace). Both follow `components.kagent` when left unset (the render refuses kagent on with either off, and Substrate on without its CRDs — `agent-platform.validateSubstrate`), both land in `ate-system` through the new roster key `components.<name>.targetNamespace` (the chart's Roles and Service names are fixed to it), both carry **one version pin** — the same string in the two `versionRange`s and in `kagent.substrateWorkerPool.workerImage`, held to one by `make verify-components`: the range `>=0.0.27-gs.2 <0.0.28-0` (the line's releases of upstream 0.0.27 from `v0.0.27-gs.2` on — the egress gateway's frontend policy for its agentgateway dataplane, the connection string from a Secret, atelet scheduling), its floor the worker image's tag. Roster components rather than the kagent charts' `substrate` subchart switches: each release has its own Flux status, the order is explicit, the repository is a value (a mirror for a proxied installation, with `substrate.image.registry`), Substrate's namespace is not kagent's. The `substrate:` block is forwarded verbatim (the chart has no schema): `postgres.enabled: auto` (below), `rustfs.enabled: false` (the bundled in-cluster store is a lab's), `atelet.storageBackend`, `atelet.{nodeSelector,tolerations,affinity}` (carried patch of the line; empty = every node the DaemonSet controller admits, the worker nodes on a cluster whose control planes are tainted), `atelet.extraEnv` (an S3-compatible store's endpoint and credentials). **Bootstrap**: the connectivity release's `pre-install,pre-upgrade` hook Job `<release>-substrate-bootstrap` mints what the substrate chart mounts but does not render — the CA pools `service-dns-ca-pool`, `pod-identity-ca-pool` (`podcertificate-controller-system`), the JWT authority pool `actor-id-jwt-pool`, the CA pool `actor-id-ca-pool` and the derived trust anchor `actor-id-ca-certs` (`ate-system`), and the ConfigMap `ate-api-authentication` with the apiserver's own issuer read from its OpenID discovery document — with `openssl` in an init container (an ED25519 root per CA pool, an ECDSA P-256 ES256 authority, in the pools' wire format) and `kubectl`, creates the two namespaces bare when missing, and **never touches a pool that exists** (a re-run, every upgrade, logs `present`; a regenerated root would orphan every certificate issued from it). Hook include `agent-platform.hooks.job` (`templates/_hooks.tpl`) and the hook identity `<release>-hooks` (a ClusterRole on secrets, configmaps and namespaces for the hook's lifetime; `hooks.kubectlImage`, `hooks.opensslImage`). **Install order**: `substrate-crds`, `kagent-crds` → `agent-platform-connectivity` → `substrate` → `kagent` → the managers — the connectivity release `dependsOn` the CRD components, never kagent or substrate, which depend on it (their pods start against what its hooks mint), so nothing waits on a pod that waits on a later release and `components.kagent.installDisableWait` is gone. **Database**: `postgres.databases` — a map of further CNPG `Database`s on the platform Cluster (`<clusterName>-<key>`, owned by the application role, `retain`) each with a derived connection Secret `<clusterName>-<key>-app` copied into `postgres.namespace` and every `secretNamespaces` entry by the `post-install,post-upgrade` hook `<release>-postgres-databases` (CNPG mints `<clusterName>-app` for the initdb database only, a `Database` mints none; the hook waits for it and rewrites `dbname`, `uri`, `jdbc-uri`, `pgpass`; server-side applied, idempotent); shipped entry `substrate` (spec.name `substrate`, `secretNamespaces: [ate-system]`, renders while the component is on), an installation adds its own. `substrate.postgres.enabled: auto` resolves in the meta chart's shape pass: the Cluster's Database with `postgres.enabled` (the meta chart derives `substrate.postgres.connectionStringSecretRef` → `<clusterName>-substrate-app`/`uri`; a disagreeing explicit value fails the render), the chart's bundled single-instance StatefulSet without it (a lab); `true`/`false` force, `connectionString` names an external one, none of the three fails the render. **Snapshot store**: `kagent.harness.snapshotLocation` (the platform Harness's `snapshotPolicy.location`, `s3://<bucket>/<prefix>`) is required whenever kagent is on — S3 via IRSA on CAPA (bucket provisioning documented, not rendered), an S3-compatible store through `substrate.atelet.extraEnv`, a lab's in-cluster store; the render fails without it. **Scheduling**: the WorkerPool pinned to one CPU architecture (`kagent.substrateWorkerPool.template.nodeSelector` `kubernetes.io/arch: amd64`; the golden snapshot is architecture-specific). **Kyverno**: one `PolicyException` per Substrate workload in `kyvernoPolicies.policyExceptionNamespace` — `substrate-atelet`, `substrate-workers` (label `ate.dev/worker-pool`, every WorkerPool in the kagent namespace), `substrate-control-plane`, `substrate-podcertificate-controller` — each naming exactly the restricted-PSS rules its pod spec violates with their `autogen-` copies, looked up in the new `kyvernoPolicies.rules` map; `make verify-kyverno` (`tests/verify-kyverno.py`) renders the substrate chart at the pin with the forwarded values, computes every workload's violations (and the worker pod's, from ate-controller's pod template) and holds the exceptions to them. **Network policies** (both flavours, `templates/substrate/netpol.yaml`): Substrate's hops — `atelet` ↔ `ate-api`, the kagent controller → `ate-api` and the `atenet` router, the router → a worker's tunnel :443, `ate-api` → `atelet`'s hostPort 8085 (the egress rule upstream lacks), the workers → the egress gateway and the dns — and the actors' destinations on the egress gateway `atenet-egress` (muster, the kagent controller, the LLM provider or the agentgateway LLM listener, the built-in tool server, world:443, cluster 443/10443), which the kagent controller admits (`substrate-actors-to-kagent-controller`); the worker pods reach only the egress gateway, the dns and the cluster DNS; `make verify-kagent-netpol` asserts it. **Prerequisites**: Kubernetes 1.35 with the `ClusterTrustBundle`, `ClusterTrustBundleProjection` and `PodCertificateRequest` gates on apiserver, controller-manager and kubelet (README "Agent Substrate", UPGRADE.md); a live render refuses a cluster that does not serve `certificates.k8s.io/v1beta1/PodCertificateRequest`, naming the gates and that the kubelet's cannot be seen — never `helm template`. Security write-up for installation owners and reviewers: [docs/substrate-security.md](./docs/substrate-security.md). Not lab-verified: Kyverno admission, Cilium enforcement of the policies and IRSA against S3 (the lab has none of the three); their render is asserted, the runtime proof is the first Giant Swarm installation of the rollout.
+
 ### Fixed
 
 - `agent-manager.agentChart.semver` is `>=0.2.1 <1.0.0` (was `x.x.x`) in both charts. Generic `agent` chart 1.0.0 (published 2026-09-11) renders the kagent API v2 shape (`kagent.dev/v1alpha3 AgentTemplate`), which no 3.x installation can install; agent-manager composes this range into the per-namespace `agent` `OCIRepository` it writes on every `create_agent`, and the meta chart's explicit forward wins over agent-manager 0.4.5's own bounded default — so an open range here pulled 1.0.0 into every new managed namespace (the ATS round trip on this branch reproduced it: `HelmRelease ats-managed-agent` InstallFailed `no matches for kind AgentTemplate`). The three pre-existing per-namespace `OCIRepository` objects were bounded by hand already; this makes every future one bounded too. **Every installation with agent-manager on rolls the agent-manager pod once** (`--agent-chart-semver` changes); nothing else in the render differs. The 4.0 line moves the range to `1.x`.
@@ -27,8 +31,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **The shared `RemoteMCPServer agent-platform/muster`** (`allowedNamespaces.from: All`, the cross-namespace contract the Generic `agent` chart 0.x's default `serverRef` depended on). On kagent API v2 an `AgentTemplate` binds a `RemoteMCPServer` of its own namespace and the binding carries no headers, so the Generic agent chart 1.x renders one per agent — muster's URL, `STREAMABLE_HTTP`, the `X-Muster-Toolset` header, the `kagent.dev/discovery: disabled` label — and binds it (giantswarm/agent#25); Helm deletes the shared object with the connectivity release on the upgrade (#299). `make verify-kagent-discovery` is re-scoped: no `RemoteMCPServer` of the platform's own targets muster, nothing the platform renders for muster carries a static header, the operator extras render `v1alpha3` in the kagent namespace with `tokenSecret` as a Secret-sourced `Authorization` header and no opt-out label. docs/authentication.md describes the per-agent carrier and the opt-out label the agent chart sets.
 - **The two Kyverno ClusterPolicies that mutated the kagent controller's input** — `<release>-kagent-declarative-pod-security` (`spec.declarative.deployment.*SecurityContext` on every `Agent`) and `<release>-kagent-srt-settings` (`enableWeakerNestedSandbox` into the per-agent `srt-settings.json` Secret) — and their README text: no `Agent` CR, no per-agent Deployment and no config Secret exists on kagent API v2 (#299). `kyvernoPolicies` keeps its keys — the seccomp `PolicyException` (re-targeted at the Substrate worker pods in #342), the CNPG exception and `agentSandbox.podSecurity` read them. `make verify-modes` asserts the two `kyverno.io` objects of the kagent + agent-sandbox shape and the absence of any `Agent` mutation; `make verify-auto` asserts it on every cluster shape.
 
+### Changed
+
+- `kyvernoPolicies.rules` (rule → ClusterPolicy of the cluster's Pod Security Standard policies, upstream kyverno-policies' names by default) replaces `kyvernoPolicies.seccompPolicyName`, `seccompRuleNames`, `volumeTypesPolicyName` and `volumeTypesRuleNames`; the CNPG ImageVolume exception reads `restricted-volumes` from it. The helper `agent-platform.kyverno.exceptions` renders an exception's `exceptions:` from a rule list. An installation that overrode the four keys moves the names into the map (UPGRADE.md); the schema refuses the old keys.
+- The kagent controller's network policies open Substrate: egress to `ate-api-server` :443 and `atenet-router` :8080 in `ate-system` (replacing the egress to the v1alpha2 agent pods), and the CNPG Cluster's policy admits `ate-api-server` on 5432 while Substrate's database rides the Cluster. The model-serving predictor policies admit the actors through Substrate's egress gateway instead of `app: kagent` pods.
+
 ### Removed
 
+- The `kagent-declarative-seccomp` PolicyException and the `<release>-kagent-agent-muster-egress` CiliumNetworkPolicy, both selecting `app: kagent` — the label of the v1alpha2 per-agent Deployments, which nothing carries on kagent API v2 (`templates/kagent/policy-exception.yaml` is gone). Their successors are `substrate-workers` and the actors' rules on `substrate-atenet-egress`; the kagent controller's ingress policies no longer admit `app: kagent` pods. `components.kagent.installDisableWait`: the kagent release depends on the connectivity release now, so the CNPG connection Secret exists before its first install and every component install waits for its workload.
 - **The ten bundled example agents' value blocks** (`kagent.k8s-agent` … `kagent.cilium-debug-agent`, each `enabled: false` with `namespaceOverride: kagent`), `kagent.controller.skillsInitImage` and the `METRICS_BIND_ADDRESS` / `METRICS_SECURE` entries of `kagent.controller.env`: the kagent line ships none of the agents and no skills-init image, and its controller serves no Prometheus `/metrics`. With that `kagent.serviceMonitor.enabled` defaults to `false` in both charts and the forwarded `kagent.controller.metrics.enabled` is `false` explicitly — the connectivity chart's controller metrics `Service` and `ServiceMonitor` and upstream's metrics Service would point at a port nothing listens on and sit at `up=0`; nothing reads those metrics (the fleet's `KagentControllerDown` reads kube-state-metrics). `make verify-global` asserts the default renders no kagent monitor and that `kagent.serviceMonitor.enabled: true` still renders it under the global gate; both go back on when upstream serves metrics. The ATS kind smoke keeps kagent and agent-manager off and skips the agent round trips with the reason while the cluster has neither the feature gates nor Substrate (`tests/ats/values-kagent.yaml`, `KAGENT_ON`); agentlab proves them, giantswarm/agent-platform#343 turns them back on.
 
 ### Changed