GitHub action v1.0.6 against AWS fails to succeed.

[jonmat]

I'm trying ZeroFS for the first time. The s3 bucket, its policy grants seem all correct. Using us-east-2, and OIDC connector to AWS (which succeeds).

Here's the bootup output with error at end:

thanks in advance for taking a look!

Setting up ZeroFS
Downloading ZeroFS archive from: https://github.com/Barre/zerofs/releases/download/v1.0.6/zerofs-pgo-multiplatform.tar.gz
Will extract binary: zerofs-linux-amd64-pgo
Using temp directory: /tmp/tmp.y8CsAFDedU
ZeroFS installed at: /usr/local/bin/zerofs
Installing NFS utilities...
Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
Hit:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
Get:3 http://azure.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:5 http://azure.archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:6 https://packages.microsoft.com/repos/azure-cli noble InRelease [3564 B]
Get:7 https://packages.microsoft.com/ubuntu/24.04/prod noble InRelease [3600 B]
Get:8 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1807 kB]
Get:9 http://azure.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [332 kB]
Get:10 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [178 kB]
Get:11 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [16.7 kB]
Get:12 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1565 kB]
Get:13 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe Translation-en [319 kB]
Get:14 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [386 kB]
Get:15 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [32.9 kB]
Get:16 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2748 kB]
Get:17 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [632 kB]
Get:18 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:19 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 c-n-f Metadata [556 B]
Get:20 http://azure.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:21 http://azure.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7300 B]
Get:22 http://azure.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:23 http://azure.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [212 B]
Get:24 http://azure.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:25 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 Packages [1507 kB]
Get:38 https://packages.microsoft.com/repos/azure-cli noble/main amd64 Packages [2072 B]
Get:26 http://azure.archive.ubuntu.com/ubuntu noble-security/main Translation-en [241 kB]
Get:27 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:28 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 c-n-f Metadata [10.1 kB]
Get:29 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 Packages [976 kB]
Get:30 http://azure.archive.ubuntu.com/ubuntu noble-security/universe Translation-en [218 kB]
Get:31 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 Components [74.2 kB]
Get:32 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [20.6 kB]
Get:33 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2612 kB]
Get:39 https://packages.microsoft.com/ubuntu/24.04/prod noble/main arm64 Packages [79.5 kB]
Get:40 https://packages.microsoft.com/ubuntu/24.04/prod noble/main amd64 Packages [101 kB]
Get:34 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted Translation-en [603 kB]
Get:35 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Components [212 B]
Get:36 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted amd64 c-n-f Metadata [544 B]
Get:37 http://azure.archive.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [212 B]
Fetched 14.9 MB in 2s (7717 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libnfsidmap1 rpcbind
Suggested packages:
watchdog
The following NEW packages will be installed:
libnfsidmap1 nfs-common rpcbind
0 upgraded, 3 newly installed, 0 to remove and 50 not upgraded.
Need to get 343 kB of archives.
After this operation, 1243 kB of additional disk space will be used.
Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
Get:2 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 libnfsidmap1 amd64 1:2.6.4-3ubuntu5.1 [48.3 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 rpcbind amd64 1.2.6-7ubuntu2 [46.5 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 nfs-common amd64 1:2.6.4-3ubuntu5.1 [248 kB]
Fetched 343 kB in 0s (2998 kB/s)
Selecting previously unselected package libnfsidmap1:amd64.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 219118 files and directories currently installed.)
Preparing to unpack .../libnfsidmap1_1%3a2.6.4-3ubuntu5.1_amd64.deb ...
Unpacking libnfsidmap1:amd64 (1:2.6.4-3ubuntu5.1) ...
Selecting previously unselected package rpcbind.
Preparing to unpack .../rpcbind_1.2.6-7ubuntu2_amd64.deb ...
Unpacking rpcbind (1.2.6-7ubuntu2) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../nfs-common_1%3a2.6.4-3ubuntu5.1_amd64.deb ...
Unpacking nfs-common (1:2.6.4-3ubuntu5.1) ...
Setting up libnfsidmap1:amd64 (1:2.6.4-3ubuntu5.1) ...
Setting up rpcbind (1.2.6-7ubuntu2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rpcbind.service → /usr/lib/systemd/system/rpcbind.service.
Created symlink /etc/systemd/system/sockets.target.wants/rpcbind.socket → /usr/lib/systemd/system/rpcbind.socket.
Setting up nfs-common (1:2.6.4-3ubuntu5.1) ...
Creating config file /etc/idmapd.conf with new version
Creating config file /etc/nfs.conf with new version
info: Selecting UID from range 100 to 999 ...
info: Adding system user statd' (UID 114) ... info: Adding new user statd' (UID 114) with group nogroup' ... info: Not creating home directory /var/lib/nfs'.
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-client.target → /usr/lib/systemd/system/nfs-client.target.
Created symlink /etc/systemd/system/remote-fs.target.wants/nfs-client.target → /usr/lib/systemd/system/nfs-client.target.
auth-rpcgss-module.service is a disabled or a static unit, not starting it.
nfs-idmapd.service is a disabled or a static unit, not starting it.
nfs-utils.service is a disabled or a static unit, not starting it.
proc-fs-nfsd.mount is a disabled or a static unit, not starting it.
rpc-gssd.service is a disabled or a static unit, not starting it.
rpc-statd-notify.service is a disabled or a static unit, not starting it.
rpc-statd.service is a disabled or a static unit, not starting it.
rpc-svcgssd.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.12.0-4build2) ...
Not building database; man-db/auto-update is not 'true'.
Processing triggers for libc-bin (2.39-0ubuntu8.7) ...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
Waiting for ZeroFS to start...
Error: ZeroFS failed to start within 30 seconds
2026-03-11T21:09:05.124002Z INFO zerofs::cli::server: Starting ZeroFS server with AmazonS3(qual-ci-e76a4f1c-9d3b-4c7a-b8f2-6e5d1a0c3f9e) backend
2026-03-11T21:09:05.124028Z INFO zerofs::cli::server: DB Path: 99-merge
2026-03-11T21:09:05.124030Z INFO zerofs::cli::server: Base Cache Directory: /tmp/zerofs-cache
2026-03-11T21:09:05.124032Z INFO zerofs::cli::server: Cache Size: 5 GB
2026-03-11T21:09:05.124035Z INFO zerofs::cli::server: Checking bucket identity...
Error: Failed to read bucket ID marker: Generic {
store: "S3",
source: RetryError(
RetryErrorImpl {
method: PUT,
uri: Some(
http://169.254.169.254/latest/api/token,
),
retries: 0,
max_retries: 10,
elapsed: 2.023276ms,
retry_timeout: 180s,
inner: Status {
status: 411,
body: Some(
"\r\n<TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii">\r\n

Length Required

\r\n

---

HTTP Error 411. The request must be chunked or have a content length.

\r\n\r\n",
),
},
},
),
}
Error: Process completed with exit code 1.

[Barre]

The SDK is falling back to the EC2 metadata service for credentials, which means it's not finding them through your config. Could you share your ZeroFS configuration and the relevant part of your GitHub Actions workflow?

[jonmat]

just to be clear, the oidc connector that I'm using to auth against aws is described here: https://github.com/aws-actions/configure-aws-credentials

after running the action to connect, I verified from aws cli that I have connectivity, with tests against S3.

The connection creates a 3rd param needed to be plumbed through, for the session. I believe this 3rd param is what is missing from ZeroFS's implementation.