fix(connectivity): the s3proxy façade on CAPZ authenticates with Workload Identity — JCLOUDS_CREDENTIAL set empty so the image default cannot fill it

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **The s3proxy façade authenticates to Azure Blob with Workload Identity** (giantswarm/agent-platform#436; found on glean's cut-over, 2026-09-13). With `kagent.harness.snapshotStore.crossplane.provider: capz` the connectivity chart left `JCLOUDS_CREDENTIAL` unset so that s3proxy would defer to `DefaultAzureCredential` — but the image (`gaul/s3proxy` 4.1.1) carries `JCLOUDS_CREDENTIAL="remote-credential"` as a Dockerfile default, and s3proxy takes the `DefaultAzureCredential` path only while the credential is *empty*: the façade signed every request as the account with that literal as the shared key and Azure's SDK refused it on the base64 decode (`'base64Key' was not a valid Base64 scheme … Illegal base64 character 2d` on every `PUT …/snapshots/…`), so every golden boot crashed before its snapshot was taken. The `capz` branch now sets `JCLOUDS_CREDENTIAL: ""` explicitly next to `AZURE_CLIENT_ID` / `AZURE_TENANT_ID`; the account-key branch is unchanged. `make verify-substrate-store` asserts both shapes. Live proof on glean: with the variable empty the golden snapshot of `sre-agent` landed in `giantswarm-glean-substrate` through the façade on the next boot (Workload Identity end to end, the half agentlab cannot exercise — its façade proof runs the account-key branch against Azurite).
+
 ### Changed
 
 - **The Substrate line re-pins to `v0.0.27-gs.9`, the kagent line to `v0.11.0-gs.12`** (giantswarm/giantswarm#37773): `components.substrate` / `components.substrate-crds` move to `>=0.0.27-gs.9 <0.0.28-0`, `components.kagent` / `components.kagent-crds` to `>=0.11.0-gs.12 <0.11.1-0` — the kagent release whose chart stamps the `0.0.27-gs.9` worker image, where the gVisor worker's terminate collects a workload whose runsc containers are already gone (giantswarm/substrate#30). Before it, an `AgentTemplate` changed while its previous golden boot was crash-looping left the superseded revision uncollectable — `failed to collect runtime revision … runsc state: exit status 128` every minute per template — and its golden actor pinned one worker of the pool for good (gazelle 2026-09-13: two of four). Proven in agentlab on the fork's dev build: on `0.0.27-gs.7` the superseded revision was never collected in 56 minutes; on the patch the sweep after the template change collected it and freed the worker. The `WorkerPool` rolls once with the kagent release, which by itself frees the workers an installation on 4.10.x has pinned. UPGRADE.md carries the recognition and the by-hand recipe (delete the hosting worker pod) for a Substrate release before `gs.9`. BOM, `tests/verify-components.py` and `tests/verify-kagent-crds.py` follow the pins.

Makefile.custom.mk

@@ -1926,7 +1926,8 @@ verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harne
 	@grep -q 'image: gsoci.azurecr.io/giantswarm/s3proxy:' /tmp/vss-capz-deploy.out || { echo "FAIL: the s3proxy image is not pulled from gsoci"; exit 1; }
 	@grep -q 'azure.workload.identity/use: "true"' /tmp/vss-capz-deploy.out || { echo "FAIL: the s3proxy pods do not use Workload Identity"; exit 1; }
 	@grep -q 'value: "https://giantswarmcisubstrate.blob.core.windows.net"' /tmp/vss-capz-deploy.out || { echo "FAIL: JCLOUDS_ENDPOINT is not the account's blob endpoint"; exit 1; }
-	@if grep -q 'JCLOUDS_CREDENTIAL' /tmp/vss-capz-deploy.out; then echo "FAIL: capz hands s3proxy an account key (DefaultAzureCredential expected)"; exit 1; fi
+	@grep -A1 'name: JCLOUDS_CREDENTIAL$$' /tmp/vss-capz-deploy.out | grep -q 'value: ""' || { echo "FAIL: capz does not set JCLOUDS_CREDENTIAL to the empty string (the image's default remote-credential would make s3proxy sign with a shared key instead of DefaultAzureCredential, #436)"; exit 1; }
+	@if grep -A3 'name: JCLOUDS_CREDENTIAL$$' /tmp/vss-capz-deploy.out | grep -q 'secretKeyRef'; then echo "FAIL: capz hands s3proxy an account key (DefaultAzureCredential expected)"; exit 1; fi
 	@grep -q 'name: AZURE_CLIENT_ID' /tmp/vss-capz-deploy.out || { echo "FAIL: AZURE_CLIENT_ID is not read from the bridged Secret"; exit 1; }
 	@[ "$$(grep -c '^kind: Secret$$' /tmp/vss-capz.out)" = "2" ] || { echo "FAIL: the key pair is not rendered in both the release namespace and ate-system"; exit 1; }
 	@[ "$$(awk '/^kind: Secret$$/,/^---/' /tmp/vss-capz.out | grep -c 'helm.sh/resource-policy: keep')" = "2" ] || { echo "FAIL: a key-pair Secret is not kept on uninstall"; exit 1; }
@@ -1950,7 +1951,7 @@ verify-substrate-store: ## Assert Agent Substrate's snapshot store (kagent.harne
 	@echo "--> the façade alone (an account provisioned by hand, a lab's Azurite): no Crossplane object, an account key"
 	@helm template t $(CONNECTIVITY_DIR) -f $(CONNECTIVITY_DIR)/ci/test-substrate-values.yaml $(SUBSTRATE_STORE_S3PROXY) >/tmp/vss-s3p.out 2>&1 || { cat /tmp/vss-s3p.out; exit 1; }
 	@if grep -q -E '^kind: (Account|Container|UserAssignedIdentity|FederatedIdentityCredential|Object)$$' /tmp/vss-s3p.out; then echo "FAIL: the façade alone renders Crossplane objects"; exit 1; fi
-	@awk '/^kind: Deployment$$/,/^---/' /tmp/vss-s3p.out | grep -q 'JCLOUDS_CREDENTIAL' || { echo "FAIL: the façade alone does not read the account key"; exit 1; }
+	@awk '/^kind: Deployment$$/,/^---/' /tmp/vss-s3p.out | grep -A3 'name: JCLOUDS_CREDENTIAL$$' | grep -q 'secretKeyRef' || { echo "FAIL: the façade alone does not read the account key from the Secret"; exit 1; }
 	@if awk '/^kind: Deployment$$/,/^---/' /tmp/vss-s3p.out | grep -q 'azure.workload.identity'; then echo "FAIL: the façade alone claims Workload Identity"; exit 1; fi
 	@echo "ok: the façade alone"
 	@echo "--> meta chart, capz: the derived location, the façade's S3 environment on both substrate components"

UPGRADE.md

@@ -20,6 +20,15 @@ Operator action required between releases. CHANGELOG.md captures the diff; UPGRA
   A healthy superseded golden is `ACTOR_STATE_SUSPENDED` with no worker; the kagent controller collects it on its next sweep.
 - **Removing a stuck template by hand** (`v0.0.27-gs.8` and earlier): the delete cannot succeed while the worker pod that hosted the crashed golden is alive — the same `runsc state` failure answers `kubectl ate delete actor <name> -a ate-golden`. Delete that worker pod (`kubectl -n kagent delete pod <ATEOM POD>`; the `WorkerPool` replaces it): the next controller sweep, within a minute, deletes the golden actor (worker gone = terminated), the `ActorTemplate` and the runtime revision, the log line stops and the replacement worker registers `FREE`. Nothing is edited in Substrate's store. With `0.0.27-gs.9` workers the sweep succeeds on the live worker and this recipe is not needed.
 
+## \<current\> → \<next\> (the s3proxy façade on CAPZ authenticates with Workload Identity)
+
+The `capz` branch of the façade (`kagent.harness.snapshotStore.crossplane.provider: capz`) sets `JCLOUDS_CREDENTIAL` to the empty string explicitly (giantswarm/agent-platform#436). Before, the variable was left unset and the image's own default (`remote-credential`, a Dockerfile `ENV` of `gaul/s3proxy`) filled it: s3proxy then signed as the account with that literal as the shared key instead of deferring to `DefaultAzureCredential`, and every snapshot request failed on the base64 decode — on glean every golden boot crashed before its snapshot was taken. The account-key branch (`s3proxy.azure.accountKeySecretRef`) is unchanged.
+
+### Operator action
+
+- None: the connectivity release re-renders the `substrate-s3proxy` Deployment on the upgrade and the façade authenticates with the identity the chart federated. An installation that bridged the defect by patching the Deployment by hand (`JCLOUDS_CREDENTIAL=""`) is overwritten with the same value.
+- An installation on 4.10.0–4.10.3 with provider `capz` whose `AgentTemplate`s read `ActorTemplateRetrying: golden boot N of 6 failed (GoldenActorCrashed …)` while the façade logs `'base64Key' was not a valid Base64 scheme`: this is the cause; once the release is on the cluster the next boot succeeds (a template that exhausted its six boots — `ActorTemplateFailed … golden boots crashed` — is re-tried by a change to the template, an annotation suffices).
+
 ## \<current\> → \<next\> (the chart provisions Agent Substrate's snapshot store on CAPZ: Crossplane storage account + Workload Identity behind an s3proxy façade)
 
 `kagent.harness.snapshotStore.crossplane.provider: capz` renders the Azure half of the store the CAPA one got in the previous entry: the connectivity chart renders the storage `Account` (TLS-only, no public blobs, soft delete; never deleted by Crossplane, kept by Helm), the blob `Container` with a lifecycle `ManagementPolicy` (`capz.lifecycleDays`, 30), a `UserAssignedIdentity` federated for the s3proxy ServiceAccount with Storage Blob Data Contributor on the container (provider-kubernetes bridges the generated ids), and the **s3proxy façade** — Substrate speaks S3 only, so a stateless `substrate-s3proxy` Deployment in the release namespace translates to Azure Blob as that identity. The meta chart derives `kagent.harness.snapshotLocation` = `s3://<capz.containerName>/<prefix>` and hands the substrate release the façade's S3 environment on `atelet.extraEnv` and `ateApiServer.extraEnv` (README "Agent Substrate: the snapshot store").

helm/agent-platform-connectivity/templates/substrate/s3proxy.yaml

@@ -135,12 +135,20 @@ spec:
             - name: JCLOUDS_IDENTITY
               value: {{ $az.account | quote }}
             {{- if $capz }}
-            # No jclouds credential: the azureblob backend defers to
-            # DefaultAzureCredential, which takes the Workload Identity path from
-            # these two and what the azure-workload-identity webhook injects for
-            # the azure.workload.identity/use label (AZURE_FEDERATED_TOKEN_FILE,
+            # An empty jclouds credential, set explicitly: the azureblob backend
+            # defers to DefaultAzureCredential only while the credential is
+            # empty, and the image's Dockerfile carries
+            # JCLOUDS_CREDENTIAL="remote-credential" as its default — left unset
+            # here, the façade would sign as the account with that literal as
+            # the shared key and every request would fail on its base64 decode
+            # (giantswarm/agent-platform#436, glean 2026-09-13).
+            # DefaultAzureCredential takes the Workload Identity path from the
+            # two ids below and what the azure-workload-identity webhook injects
+            # for the azure.workload.identity/use label (AZURE_FEDERATED_TOKEN_FILE,
             # AZURE_AUTHORITY_HOST, the projected token) — a hard prerequisite
             # of provider capz (UPGRADE.md).
+            - name: JCLOUDS_CREDENTIAL
+              value: ""
             - name: AZURE_CLIENT_ID
               valueFrom:
                 secretKeyRef: