Skip to content

Provision the Substrate snapshot store on CAPA through the chart: Crossplane S3 bucket + IRSA role like the kagent-pg backup bucket, the snapshot location derived, the role annotation forwarded to atelet and ate-api-server #411

Description

@teemow

Problem

Meta chart 4.x requires kagent.harness.snapshotLocation whenever kagent is on: the object store Substrate writes every actor's snapshots to. On CAPA that is an S3 bucket reached through IRSA by Substrate's atelet DaemonSet and ate-api-server. Today the chart says the bucket "is the installation's infrastructure, not rendered here" (README "Agent Substrate: the snapshot store"), and the substrate chart offers no ServiceAccount annotation for the IRSA role — so an installation has no way to give Substrate an S3 store through values at all. The fleet template (giantswarm/shared-configs#732) has to render the store per installation from the Konfiguration variables (.codename, .provider.region, .provider.controlPlane.account), the way it renders the kagent-pg backup bucket — and the kagent-pg backup bucket is rendered by this chart: postgres.backup.crossplane.aws renders the Crossplane Bucket, its lifecycle rule, public-access block and TLS-only policy, and an IAM Role with an inline S3 policy trusted by the Cluster's ServiceAccount through the installation's IRSA OIDC provider, then derives serviceAccount.annotations and inheritFromIAMRole (helm/agent-platform-connectivity/templates/postgres/crossplane-aws.yaml). The fleet provisions object storage for its own components exactly one way; Substrate's store should not be the exception (giantswarm/giantswarm#37756 "CAPA stays as #342 defines it: S3 + IRSA; Crossplane bucket + role like the kagent-pg backup bucket"; epic giantswarm/giantswarm#37705, wave 1e).

The Substrate line side — the ServiceAccount annotation knobs on the substrate chart (atelet, ate-api-server) — is a carried patch on giantswarm/substrate (line issue giantswarm/giantswarm#37757); this issue consumes those keys.

Proposed solution

  • A values block on the meta chart, next to the existing snapshot location, that renders the CAPA store through Crossplane the way postgres.backup.crossplane does — same helpers (agent-platform.postgres.crossplane*, the partition helper, the management policies without Delete, helm.sh/resource-policy: keep), same inputs (providerConfigRef, region, tags, aws.bucketName, aws.accountId, aws.oidcProvider, aws.roleName defaulting to the bucket name, aws.lifecycleDays), rendered by the connectivity chart. Off by default; nothing changes for an installation that sets its location by hand (a lab's rustfs, an S3-compatible store through substrate.atelet.extraEnv).
  • When the block renders the bucket: kagent.harness.snapshotLocation is derived as s3://<bucketName>/<prefix> (an explicit value that differs fails the render, the componentDerivedValues pattern), and the IRSA role annotation eks.amazonaws.com/role-arn is forwarded to the substrate release's atelet and ate-api-server ServiceAccounts through the substrate chart's new annotation values. The role's trust policy admits both ServiceAccounts (system:serviceaccount:<substrate namespace>:<release>-atelet, …-ate-api-server) through the OIDC provider; the inline S3 policy covers list/get/put/delete on the bucket and its objects — what a snapshot store needs, no more.
  • The snapshot lifecycle: an expiration rule with a documented default (snapshots are rebuilt from the golden boot; a lost snapshot costs one cold start).
  • The substrate line's minimum version in components.substrate.versionRange moves to the release carrying the annotation keys; tests/verify-components.py follows.
  • README "Agent Substrate: the snapshot store" and UPGRADE.md describe the block; the .nancy/CI unit tests (helm-unittest on the connectivity chart, a ci/test-substrate-store-aws-values.yaml like ci/test-postgres-backup-aws-values.yaml) assert the rendered Bucket, Role, derived location and the forwarded annotations. Azure and the S3-compatible endpoint stay out of this issue (giantswarm/giantswarm#37756 decides them).

Acceptance criteria

  • With the block on (provider: aws), the connectivity chart renders the Crossplane Bucket (+ lifecycle, public-access block, TLS-only policy) and the IAM Role trusted by Substrate's two ServiceAccounts through the IRSA OIDC provider, mirroring postgres.backup.crossplane.aws (helpers reused, not copied).
  • kagent.harness.snapshotLocation is derived from the bucket (s3://<bucket>/<prefix>); an explicit disagreeing value fails the render naming both keys.
  • The role annotation reaches the substrate release's atelet and ate-api-server ServiceAccounts (helm template of the meta chart shows it in the substrate HelmRelease values; the substrate range floor is the release that carries the keys).
  • Off by default: a render with the block off is byte-identical to today's for every existing CI values file.
  • helm-unittest / a CI values file covers the on shape; make verify-components passes with the new substrate floor.
  • README and UPGRADE.md updated; released as a 4.x minor.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions