Skip to content

fix(connectivity): the s3proxy façade on CAPZ authenticates with Workload Identity — JCLOUDS_CREDENTIAL set empty so the image default cannot fill it - #437

Merged
teemow merged 1 commit into
mainfrom
s3proxy-wi-empty-credential
Sep 13, 2026
Merged

teemow merged 1 commit into
mainfrom
s3proxy-wi-empty-credential

Conversation

@teemow

@teemow teemow commented Sep 13, 2026

Copy link
Copy Markdown
Member

Closes #436. Found on glean's cut-over to kagent API v2 (giantswarm/giantswarm#37744), the first CAPZ installation on the 4.x line.

Problem

With kagent.harness.snapshotStore.crossplane.provider: capz the connectivity chart left JCLOUDS_CREDENTIAL unset so that s3proxy would defer to DefaultAzureCredential. s3proxy 4.1.1 does that only while the credential is empty (AzureBlobStore: identity and credential both non-empty → AzureNamedKeyCredential, else DefaultAzureCredentialBuilder), and the image carries JCLOUDS_CREDENTIAL="remote-credential" as a Dockerfile ENV default that the entrypoint copies into jclouds.credential verbatim. The façade therefore signed every request as the account with the literal remote-credential as the shared key:

java.lang.RuntimeException: 'base64Key' was not a valid Base64 scheme. Ensure the Storage account key or SAS key is properly formatted.
    at com.azure.storage.common.StorageSharedKeyCredential.generateAuthorizationHeader
    at org.gaul.s3proxy.azureblob.AzureBlobStore.putBlob(AzureBlobStore.java:804)
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 2d

on every PUT …/snapshots/<id>/pages, and every golden boot crashed before its snapshot was taken (ActorTemplateRetrying: golden boot N of 6 failed (GoldenActorCrashed …)). The Workload Identity half itself was complete: webhook injection, the bridged ids, identity, federated credential and role assignment all Ready.

Change

  • templates/substrate/s3proxy.yaml, the capz branch: JCLOUDS_CREDENTIAL: "" set explicitly next to AZURE_CLIENT_ID / AZURE_TENANT_ID, with the reason in the comment. The account-key branch is unchanged.
  • make verify-substrate-store: the capz Deployment must carry JCLOUDS_CREDENTIAL with value: "" and no secretKeyRef for it (the old assertion required the variable to be absent — the shape that fails); the façade-alone shape must read the key from the Secret.
  • CHANGELOG (Fixed) and UPGRADE.md (no operator action; the recognition for an installation on 4.10.0–4.10.3).

Verification

  • make verify-substrate-store passes (all seven groups).
  • Live on glean (connectivity 4.10.3 + this change applied to the Deployment by hand, 2026-09-13 22:42Z): with JCLOUDS_CREDENTIAL="" the façade authenticated with Workload Identity and the sre-agent golden snapshot landed in giantswarm-glean-substrate on the next boot (template Ready 22:44:12Z after four crashed boots on the unpatched Deployment; no façade error since). That is the end-to-end proof of the Workload Identity path agentlab cannot run — its façade proof (agent-platform#423) exercises the account-key branch against Azurite, which this change does not touch.
  • Not lab-verified: the change is confined to the capz branch of the façade's environment.

…load Identity — JCLOUDS_CREDENTIAL set empty so the image default cannot fill it (#436)
@teemow
teemow requested a review from a team as a code owner September 13, 2026 22:47
@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd
Chart agent-platform-connectivity
Version 4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd
Digest sha256:cbc88bd3e968d281d2adb598519b060b051c6bf5d6fc887ab21d6b2e67bcd6d2
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd

Posted by architect-orb · build 7257 · commit eb0a6cd · updated in place on every push

@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform

4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd
Chart agent-platform
Version 4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd
Digest sha256:2b0c59272988f5329beadcb14791cffeecac6e3edd854f877fa7ccfd9c4e54f3
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.10.4-dev.s3proxy-wi--credential.2026-09-13.22-47-01.heb0a6cd

Posted by architect-orb · build 7258 · commit eb0a6cd · updated in place on every push

@teemow
teemow merged commit eb46088 into main Sep 13, 2026
13 checks passed
@teemow
teemow deleted the s3proxy-wi-empty-credential branch September 13, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3proxy façade on CAPZ signs with the image's default remote-credential instead of Workload Identity — JCLOUDS_CREDENTIAL must be set empty

1 participant