Skip to content

fix(connectivity): the s3proxy façade declares ephemeral-storage requests and limit and bounds its emptyDirs (#438) #440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
teemow merged 2 commits into from
Sep 14, 2026
Merged

fix(connectivity): the s3proxy façade declares ephemeral-storage requests and limit and bounds its emptyDirs (#438) #440

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb4b2e5
fix(connectivity): the s3proxy façade declares ephemeral-storage requ…
teemow Sep 14, 2026
b2bfe18
changelog: the lab proof for the s3proxy ephemeral-storage fix (#438)
teemow Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **The s3proxy façade declares its ephemeral storage and bounds its emptyDirs** (giantswarm/agent-platform#438; glean on connectivity 4.10.3 and 4.10.4, 2026-09-13). The `substrate-s3proxy` container mounts two emptyDirs (`/tmp` for the JVM, the image's `/data`) and carried no `ephemeral-storage` requests or limits, so Kyverno's `require-emptydir-requests-and-limits` (Audit on the fleet) reported `Containers mounting emptyDir volumes must specify requests and limits for ephemeral-storage` on every rollout — the one Kyverno finding the platform left on a fresh 4.x installation, and a refusal on a cluster that enforces the policy. `kagent.harness.snapshotStore.s3proxy.resources` now defaults `requests.ephemeral-storage: 256Mi` and `limits.ephemeral-storage: 1Gi` in both charts (the meta chart forwards the block; the façade streams, nothing of a snapshot lands on its disk), both emptyDirs take the limit as their `sizeLimit`, and the render refuses a resources block without either field, naming the key. `make verify-substrate-store` asserts the fields and the bound on the capz and the façade-alone shapes, the knob, the forwarded default and the guard; `tests/verify-components.py` holds the two charts' resources defaults together. **An installation with the façade rolls its two `substrate-s3proxy` pods once** (a pod template change; the rolling update surges the new pod first) — UPGRADE.md. Lab proof (agentlab, the façade alone against Azurite, `hack/agentlab/`): a lab on connectivity 4.10.4 upgraded to this branch's dev build rolled the façade in 6 s (new ReplicaSet, 2/2 Ready, old at 0) with `ephemeral-storage` 256Mi/1Gi and both emptyDirs `sizeLimit: 1Gi` on the live pods; an `AgentTemplate` created right after went Ready on the platform Harness in 5 s with its golden snapshot (5 objects, 1.3 MiB) written through the rolled façade into Azurite; `agentlab platform-test` all PASS before and after. The Kyverno half is fleet-only (agentlab runs no Kyverno): the acceptance is no `require-emptydir-requests-and-limits` event on glean once the release rolls.
- **The s3proxy façade authenticates to Azure Blob with Workload Identity** (giantswarm/agent-platform#436; found on glean's cut-over, 2026-09-13). With `kagent.harness.snapshotStore.crossplane.provider: capz` the connectivity chart left `JCLOUDS_CREDENTIAL` unset so that s3proxy would defer to `DefaultAzureCredential` — but the image (`gaul/s3proxy` 4.1.1) carries `JCLOUDS_CREDENTIAL="remote-credential"` as a Dockerfile default, and s3proxy takes the `DefaultAzureCredential` path only while the credential is *empty*: the façade signed every request as the account with that literal as the shared key and Azure's SDK refused it on the base64 decode (`'base64Key' was not a valid Base64 scheme … Illegal base64 character 2d` on every `PUT …/snapshots/…`), so every golden boot crashed before its snapshot was taken. The `capz` branch now sets `JCLOUDS_CREDENTIAL: ""` explicitly next to `AZURE_CLIENT_ID` / `AZURE_TENANT_ID`; the account-key branch is unchanged. `make verify-substrate-store` asserts both shapes. Live proof on glean: with the variable empty the golden snapshot of `sre-agent` landed in `giantswarm-glean-substrate` through the façade on the next boot (Workload Identity end to end, the half agentlab cannot exercise — its façade proof runs the account-key branch against Azurite).

### Changed
Expand Down
16 changes: 15 additions & 1 deletion Makefile.custom.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1865,7 +1865,7 @@ SUBSTRATE_STORE_CAPZ_SET := --set kagent.harness.snapshotStore.crossplane.provid
SUBSTRATE_STORE_META_CAPZ := helm template t $(CHART_DIR) -f $(CHART_DIR)/ci/ci-values.yaml $(ENGINE_OFF) --set kagent.harness.snapshotLocation= --set kagent.harness.snapshotStore.crossplane.enabled=true --set kagent.harness.snapshotStore.crossplane.providerConfigRef=ci $(SUBSTRATE_STORE_CAPZ_SET)
# The façade alone: no Crossplane block, an account provisioned by hand (or a lab's Azurite) named in s3proxy.azure.*.
SUBSTRATE_STORE_S3PROXY := --set kagent.harness.snapshotLocation= --set kagent.harness.snapshotStore.s3proxy.enabled=true --set kagent.harness.snapshotStore.s3proxy.azure.endpoint=http://azurite.agent-platform.svc:10000/devstoreaccount1 --set kagent.harness.snapshotStore.s3proxy.azure.account=devstoreaccount1 --set kagent.harness.snapshotStore.s3proxy.azure.container=ate-snapshots --set kagent.harness.snapshotStore.s3proxy.azure.accountKeySecretRef.name=azurite --set kagent.harness.snapshotStore.s3proxy.azure.accountKeySecretRef.key=key
verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harness.snapshotStore, #411): off by default nothing renders; on, the connectivity chart renders the Crossplane Bucket (+ lifecycle, public-access block, TLS-only policy, never deleted, kept) and the IAM Role trusted by the atelet and ate-api-server ServiceAccounts in ate-system with the S3 policy on the bucket; the meta chart derives kagent.harness.snapshotLocation (s3://<bucket>/<prefix>) for the kagent release and holds the block back from it, forwards the role annotation to both ServiceAccounts of the substrate release next to an installation's own annotations; provider capz (#417) renders the Azure Account (TLS-only, private, soft delete, never deleted, kept), Container, lifecycle ManagementPolicy, the UserAssignedIdentity + FederatedIdentityCredential for the s3proxy ServiceAccount, the bridged RoleAssignment and client-id Secret, and the s3proxy façade (Deployment from gsoci, Service, the key pair in both namespaces, PDB, network policies, Substrate's egress to it); the façade alone renders no Crossplane object and reads an account key; the meta chart derives s3://<container>/<prefix> and the façade's S3 environment on both substrate components next to an installation's own; the guards (a disagreeing explicit location, role or s3proxy.azure.*, an unknown provider, missing inputs, a numeric account id, a bad account name, the bundled store on, an own S3 variable). Off and aws render as before.
verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harness.snapshotStore, #411): off by default nothing renders; on, the connectivity chart renders the Crossplane Bucket (+ lifecycle, public-access block, TLS-only policy, never deleted, kept) and the IAM Role trusted by the atelet and ate-api-server ServiceAccounts in ate-system with the S3 policy on the bucket; the meta chart derives kagent.harness.snapshotLocation (s3://<bucket>/<prefix>) for the kagent release and holds the block back from it, forwards the role annotation to both ServiceAccounts of the substrate release next to an installation's own annotations; provider capz (#417) renders the Azure Account (TLS-only, private, soft delete, never deleted, kept), Container, lifecycle ManagementPolicy, the UserAssignedIdentity + FederatedIdentityCredential for the s3proxy ServiceAccount, the bridged RoleAssignment and client-id Secret, and the s3proxy façade (Deployment from gsoci with ephemeral-storage requests and limit and its two emptyDirs bounded by the limit (#438), Service, the key pair in both namespaces, PDB, network policies, Substrate's egress to it); the façade alone renders no Crossplane object and reads an account key; the meta chart derives s3://<container>/<prefix> and the façade's S3 environment on both substrate components next to an installation's own; the guards (a disagreeing explicit location, role or s3proxy.azure.*, an unknown provider, missing inputs, a numeric account id, a bad account name, the bundled store on, an own S3 variable, the façade without an ephemeral-storage request or limit). Off and aws render as before.
@echo "====> $@"
@echo "--> off by default: no Crossplane object of the store in the substrate render, nothing derived"
@helm template t $(CONNECTIVITY_DIR) -f $(CONNECTIVITY_DIR)/ci/test-substrate-values.yaml >/tmp/vss-off.out 2>&1 || { cat /tmp/vss-off.out; exit 1; }
Expand Down Expand Up @@ -1944,6 +1944,11 @@ verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harne
@grep -q 'runAsNonRoot: true' /tmp/vss-capz-deploy.out || { echo "FAIL: the façade runs as root"; exit 1; }
@grep -q 'automountServiceAccountToken: false' /tmp/vss-capz-deploy.out || { echo "FAIL: the façade mounts the default ServiceAccount token"; exit 1; }
@grep -q 'name: S3PROXY_JAVA_OPTS' /tmp/vss-capz-deploy.out || { echo "FAIL: the façade's JVM options are not set"; exit 1; }
@grep -A2 '^ *limits:$$' /tmp/vss-capz-deploy.out | grep -q 'ephemeral-storage: 1Gi' || { echo "FAIL: the façade has no ephemeral-storage limit (Kyverno require-emptydir-requests-and-limits, #438)"; exit 1; }
@grep -A3 '^ *requests:$$' /tmp/vss-capz-deploy.out | grep -q 'ephemeral-storage: 256Mi' || { echo "FAIL: the façade has no ephemeral-storage request (Kyverno require-emptydir-requests-and-limits, #438)"; exit 1; }
@[ "$$(grep -c 'sizeLimit: "1Gi"' /tmp/vss-capz-deploy.out)" = "2" ] || { echo "FAIL: the façade's two emptyDirs (/tmp, /data) are not both bounded by the ephemeral-storage limit"; exit 1; }
@if grep -q 'emptyDir: {}' /tmp/vss-capz-deploy.out; then echo "FAIL: the façade mounts an unbounded emptyDir"; exit 1; fi
@[ "$$(helm template t $(CONNECTIVITY_DIR) -f $(SUBSTRATE_STORE_CAPZ_CI) --set kagent.harness.snapshotStore.s3proxy.resources.limits.ephemeral-storage=2Gi 2>&1 | awk '/^kind: Deployment$$/,/^---/' | grep -c 'sizeLimit: "2Gi"')" = "2" ] || { echo "FAIL: the emptyDirs' sizeLimit does not follow s3proxy.resources.limits.ephemeral-storage"; exit 1; }
@grep -A3 '^kind: Service$$' /tmp/vss-capz.out | grep -q '^ name: substrate-s3proxy$$' || { echo "FAIL: the s3proxy Service is missing"; exit 1; }
@grep -q '^kind: PodDisruptionBudget$$' /tmp/vss-capz.out || { echo "FAIL: the façade has no PodDisruptionBudget"; exit 1; }
@awk '/^ name: substrate-s3proxy-ingress$$/,/^---/' /tmp/vss-capz.out >/tmp/vss-capz-ingress.out
Expand All @@ -1962,6 +1967,10 @@ verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harne
@if grep -q -E '^kind: (Account|Container|UserAssignedIdentity|FederatedIdentityCredential|Object)$$' /tmp/vss-s3p.out; then echo "FAIL: the façade alone renders Crossplane objects"; exit 1; fi
@awk '/^kind: Deployment$$/,/^---/' /tmp/vss-s3p.out | grep -A3 'name: JCLOUDS_CREDENTIAL$$' | grep -q 'secretKeyRef' || { echo "FAIL: the façade alone does not read the account key from the Secret"; exit 1; }
@if awk '/^kind: Deployment$$/,/^---/' /tmp/vss-s3p.out | grep -q 'azure.workload.identity'; then echo "FAIL: the façade alone claims Workload Identity"; exit 1; fi
@awk '/^kind: Deployment$$/,/^---/' /tmp/vss-s3p.out >/tmp/vss-s3p-deploy.out
@grep -A2 '^ *limits:$$' /tmp/vss-s3p-deploy.out | grep -q 'ephemeral-storage: 1Gi' || { echo "FAIL: the façade alone has no ephemeral-storage limit (#438)"; exit 1; }
@grep -A3 '^ *requests:$$' /tmp/vss-s3p-deploy.out | grep -q 'ephemeral-storage: 256Mi' || { echo "FAIL: the façade alone has no ephemeral-storage request (#438)"; exit 1; }
@[ "$$(grep -c 'sizeLimit: "1Gi"' /tmp/vss-s3p-deploy.out)" = "2" ] || { echo "FAIL: the façade alone's two emptyDirs are not both bounded by the ephemeral-storage limit"; exit 1; }
@echo "ok: the façade alone"
@echo "--> meta chart, capz: the derived location, the façade's S3 environment on both substrate components"
@$(SUBSTRATE_STORE_META_CAPZ) >/tmp/vss-meta-capz.out 2>&1 || { cat /tmp/vss-meta-capz.out; exit 1; }
Expand All @@ -1971,6 +1980,7 @@ verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harne
@[ "$$(grep -c 'value: http://substrate-s3proxy.default.svc:80' /tmp/vss-meta-capz-substrate.out)" = "2" ] || { echo "FAIL: AWS_ENDPOINT_URL does not reach both substrate components"; cat /tmp/vss-meta-capz-substrate.out; exit 1; }
@[ "$$(grep -c 'name: substrate-s3proxy' /tmp/vss-meta-capz-substrate.out)" = "4" ] || { echo "FAIL: the key pair Secret is not read by both components"; exit 1; }
@if grep -q 'eks.amazonaws.com/role-arn' /tmp/vss-meta-capz-substrate.out; then echo "FAIL: capz derives an IRSA annotation"; exit 1; fi
@awk '/^ name: agent-platform-connectivity$$/,/^---/' /tmp/vss-meta-capz.out | grep -q 'ephemeral-storage: 1Gi' || { echo "FAIL: the meta chart's default s3proxy.resources (ephemeral-storage) does not reach the connectivity release"; exit 1; }
@$(SUBSTRATE_STORE_META_CAPZ) --set 'substrate.atelet.extraEnv[0].name=FOO' --set 'substrate.atelet.extraEnv[0].value=bar' 2>&1 | awk '/^ name: substrate$$/,/^---/' | grep -q 'name: FOO' || { echo "FAIL: an installation's own atelet extraEnv entry is dropped by the derivation"; exit 1; }
@helm template t $(CHART_DIR) -f $(CHART_DIR)/ci/ci-values.yaml $(ENGINE_OFF) $(SUBSTRATE_STORE_S3PROXY) 2>&1 | awk '/^ name: kagent$$/,/^---/' | grep -q 'snapshotLocation: s3://ate-snapshots/kagent' || { echo "FAIL: the façade alone does not derive the location at the meta chart"; exit 1; }
@echo "ok: the meta chart wires the façade"
Expand Down Expand Up @@ -2017,6 +2027,10 @@ verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harne
@grep -q 'must be an http(s) URL' /tmp/vss-g15.out || { echo "FAIL: the endpoint guard is silent"; exit 1; }
@if helm template t $(CONNECTIVITY_DIR) -f $(CONNECTIVITY_DIR)/ci/test-substrate-values.yaml $(SUBSTRATE_STORE_S3PROXY) --set kagent.harness.snapshotStore.s3proxy.azure.accountKeySecretRef.name= >/tmp/vss-g16.out 2>&1; then echo "FAIL: the façade alone rendered without an account key"; exit 1; fi
@grep -q 'accountKeySecretRef.name and .key are required' /tmp/vss-g16.out || { echo "FAIL: the account-key guard is silent"; exit 1; }
@for k in requests limits; do \
if helm template t $(CONNECTIVITY_DIR) -f $(SUBSTRATE_STORE_CAPZ_CI) --set kagent.harness.snapshotStore.s3proxy.resources.$$k.ephemeral-storage=null >/tmp/vss-g17.out 2>&1; then echo "FAIL: the façade rendered without resources.$$k.ephemeral-storage (Kyverno require-emptydir-requests-and-limits, #438)"; exit 1; fi; \
grep -q "s3proxy.resources.$$k.ephemeral-storage is required" /tmp/vss-g17.out || { echo "FAIL: the ephemeral-storage guard for $$k is silent"; cat /tmp/vss-g17.out; exit 1; }; \
done
@if helm template t $(CONNECTIVITY_DIR) -f $(SUBSTRATE_STORE_CAPZ_CI) --set kagent.harness.snapshotStore.s3proxy.azure.accountKeySecretRef.name=x --set kagent.harness.snapshotStore.s3proxy.azure.accountKeySecretRef.key=k >/tmp/vss-g17.out 2>&1; then echo "FAIL: capz rendered with an account key"; exit 1; fi
@grep -q 'accountKeySecretRef is set next to crossplane.provider capz' /tmp/vss-g17.out || { echo "FAIL: the capz account-key guard is silent"; exit 1; }
@if helm template t $(CONNECTIVITY_DIR) -f $(SUBSTRATE_STORE_CI) --set kagent.harness.snapshotStore.s3proxy.enabled=true >/tmp/vss-g18.out 2>&1; then echo "FAIL: the façade rendered next to an aws bucket"; exit 1; fi
Expand Down
Loading
Loading