Skip to content

fix(connectivity): the s3proxy façade declares ephemeral-storage requests and limit and bounds its emptyDirs (#438) - #440

Merged
teemow merged 2 commits into
mainfrom
fix/s3proxy-ephemeral-storage
Sep 14, 2026
Merged

teemow merged 2 commits into
mainfrom
fix/s3proxy-ephemeral-storage

Conversation

@teemow

@teemow teemow commented Sep 14, 2026

Copy link
Copy Markdown
Member

Fixes #438.

Problem

The substrate-s3proxy container (kagent.harness.snapshotStore.s3proxy, on with provider capz or s3proxy.enabled) mounts two emptyDirs — /tmp for the JVM and the image's /data — without ephemeral-storage requests or limits. Kyverno's require-emptydir-requests-and-limits (Audit on the fleet) reports it on every rollout:

Containers mounting emptyDir volumes must specify requests and limits for ephemeral-storage.
rule autogen-check-emptydir-requests-limits failed at path /resources/limits/ephemeral-storage/

Seen on glean with connectivity 4.10.3 and 4.10.4 (2026-09-13) — the one Kyverno finding the platform leaves on a fresh 4.x installation; a cluster that enforces the policy would refuse the façade.

Change

  • kagent.harness.snapshotStore.s3proxy.resources defaults requests.ephemeral-storage: 256Mi and limits.ephemeral-storage: 1Gi in both charts (the meta chart forwards the block; the façade streams, nothing of a snapshot lands on its disk, so a small bound is enough).
  • Both emptyDirs take the limit as their sizeLimit (--set …resources.limits.ephemeral-storage=2Gi moves both).
  • The render refuses a resources block without either field, naming the key (…s3proxy.resources.<requests|limits>.ephemeral-storage is required).
  • make verify-substrate-store: the fields, the bound (no emptyDir: {} left), the knob, the forwarded default in the meta chart's connectivity release, and the guard — on the capz and the façade-alone shapes. tests/verify-components.py: the two charts' s3proxy.resources defaults are one, with ephemeral-storage in requests and limits.
  • CHANGELOG (Fixed), UPGRADE.md (the façade's two pods roll once; own overrides merge over the defaults), README prose, helm-docs tables regenerated.

Adjacent, kept: #437's JCLOUDS_CREDENTIAL: "" on the capz branch; #435's re-pins.

Verification

  • make verify-substrate-store and make verify-components pass locally and in execute-chart-tests.
  • agentlab (a throwaway lab, the façade alone against Azurite via hack/agentlab/, meta chart installed from this branch's dev build 4.10.5-dev.fix-s3prox--ral-storage.2026-09-14.09-43-01.h864b91f, the connectivity release pinned to the same tag):
    • On connectivity 4.10.4 with the branch's meta chart, the forwarded default already put ephemeral-storage requests/limits on the pods (the 4.10.4 template passes resources through); the emptyDirs stayed unbounded. A probe AgentTemplate went Ready in ~15 s, its golden snapshot (5 objects, 1.3 MiB) landed in Azurite through the façade. platform-test all PASS.
    • Upgrading the connectivity release to the dev build rolled the façade in 6 s (new ReplicaSet 2/2 Ready, old at 0, no restarts): live pods carry requests.ephemeral-storage: 256Mi, limits.ephemeral-storage: 1Gi, both emptyDirs sizeLimit: 1Gi. A second probe template created right after went Ready in 5 s; its golden snapshot (probe-438-b-kagent-1e299b5e8488, 5 objects) was written through the rolled façade at 10:19:55Z. platform-test all PASS again.
    • Not in the lab: Kyverno (agentlab renders no policies) — the acceptance criterion is glean, below.
  • glean: after the auto-release rolls, no new PolicyViolation event on substrate-s3proxy (baseline before: one require-emptydir-requests-and-limits event, the Deployment without ephemeral-storage).

@teemow
teemow requested a review from a team as a code owner September 14, 2026 09:43
@circleci-architect

circleci-architect Bot commented Sep 14, 2026

Copy link
Copy Markdown

Published Helm chart agent-platform

4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18
Chart agent-platform
Version 4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18
Digest sha256:2c6a098bf0041e9e93fe802716f00bdc6a73e230e5decbc2b127359974933ee1
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18

Posted by architect-orb · build 7382 · commit b2bfe18 · updated in place on every push

@circleci-architect

circleci-architect Bot commented Sep 14, 2026

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18
Chart agent-platform-connectivity
Version 4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18
Digest sha256:2a07ac3841426dd60918f401a3a49d2bee41bc592b36b49c1a5eda727d9b8a05
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.10.6-dev.fix-s3prox--ral-storage.2026-09-14.10-23-40.hb2bfe18

Posted by architect-orb · build 7383 · commit b2bfe18 · updated in place on every push

…ests and limit and bounds its emptyDirs — Kyverno require-emptydir-requests-and-limits (#438)

The substrate-s3proxy container mounts two emptyDirs (/tmp for the JVM, the
image's /data) and carried no ephemeral-storage requests or limits, so Kyverno's
require-emptydir-requests-and-limits (Audit on the fleet) reported the
Deployment on every rollout (glean, connectivity 4.10.3 and 4.10.4).

kagent.harness.snapshotStore.s3proxy.resources now defaults
requests.ephemeral-storage 256Mi and limits.ephemeral-storage 1Gi in both charts
(the meta chart forwards the block), both emptyDirs take the limit as their
sizeLimit, and the render refuses a resources block without either field.
verify-substrate-store asserts the fields, the bound, the knob, the forwarded
default and the guard on the capz and the façade-alone shapes;
verify-components holds the two charts' resources defaults together.
UPGRADE.md: the façade's two pods roll once.
@teemow
teemow force-pushed the fix/s3proxy-ephemeral-storage branch from 9dcab8a to b2bfe18 Compare September 14, 2026 10:24
@teemow
teemow merged commit c12d6f6 into main Sep 14, 2026
13 checks passed
@teemow
teemow deleted the fix/s3proxy-ephemeral-storage branch September 14, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3proxy façade: the container mounts an emptyDir without ephemeral-storage requests/limits (Kyverno require-emptydir-requests-and-limits)

1 participant