feat: s3 module — stdlib-only S3-compatible client (Phases 1 & 2)#114
Merged
Conversation
Implements AWS Signature Version 4 and a minimal S3 REST client with four operations: bucket_exists, make_bucket, get_object, put_object. - Zero external dependencies (hashlib, hmac, http.client, urllib, xml.etree) - Compatible with AWS S3, Cloudflare R2, Oracle OCI, MinIO - Path-style and virtual-host-style URL support - S3NoSuchBucket / S3NoSuchKey typed exceptions - minio-compatible S3Response (read / close / release_conn / context manager) - 46 unit tests: SigV4 signing, XML error parsing, URL construction, mocked HTTP operations, local mock server round-trips - Integration test skeleton (skipped without S3_TEST_* env vars) Closes #110 Part of #109
Remove unused imports (io, hmac, xml.etree.ElementTree), shorten long comments, replace lambda with def for log_message, and apply ruff format. Also add s3/ to ty exclude list (xml.etree.ElementTree unresolved-import, same as xml/ module).
Owner
|
Pushed a lint fix (351a3ae):
All 46 tests still pass. One thing — make sure to install the pre-commit hooks locally ( |
The AWS SigV4 spec requires canonical header names to be lowercased, but _sign_request() was using original-case keys (e.g. Content-Length instead of content-length). This caused SignatureDoesNotMatch errors against real S3 backends (MinIO, AWS) for any request with mixed-case headers (PUT, POST). HEAD requests worked because they had no additional headers beyond the already-lowercase ones.
Three-way benchmark against a real MinIO container (auto-started via conftest.py). Covers put_object, get_object, bucket_exists, and make_bucket at 1KB/100KB/1MB payload sizes. Includes cross-validation tests to verify all three clients produce identical results. Also adds s3 to testpaths, ty extra-paths, and bench-s3 extra in pyproject.toml, plus benchmark-s3/test-s3 Makefile targets.
Add 6 new S3Client methods: stat_object, remove_object, copy_object, list_objects, list_buckets, remove_bucket. Add ObjectInfo and BucketInfo dataclasses. Includes mocked + local mock server correctness tests (58 passed) and three-way benchmarks (40 passed). Also addresses elena's review feedback: add docstring note on bucket_exists 301 behavior, TODO for streaming PUT, comment on canonical header case normalization, atexit vs finally explanation in conftest, and cleaner make_bucket header construction. Closes #111
Oaklight
changed the title
Replace the single 10-object list benchmark with three tiers: small (10), medium (100), large (1500). The large tier forces pagination (2 pages at 1000/page). Objects use deterministic varying sizes (64B–4KB, seeded RNG) so no files need to be stored. Results show zerodep scales well: ~6x faster than boto3 and ~4x faster than minio at 1500 objects, where XML parsing and pagination overhead dominate.
Owner
|
CI green, all checks pass. Merging. Phase 3 (async client via httpclient module + multipart upload) will be a separate PR — it involves swapping the HTTP backend from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #110, closes #111
Full-featured stdlib-only S3-compatible storage client with AWS SigV4 signing. Zero dependencies.
Phase 1 — Core operations
bucket_exists,make_bucket,get_object,put_objectS3Responsewrapper (minio-compatible interface)Phase 2 — Object/bucket management
stat_object— HEAD object metadata →ObjectInfodataclassremove_object— DELETE objectcopy_object— server-side copy viax-amz-copy-sourcelist_objects— ListObjectsV2 with pagination, prefix/delimiter supportlist_buckets— list all accessible buckets →BucketInfodataclassremove_bucket— delete empty bucketTests & benchmarks
Compatibility
Tested against MinIO. Compatible with AWS S3, Cloudflare R2, Oracle OCI, and any S3-compatible backend.
Test plan
make test-s3— 58 passed, 3 skipped (integration)make benchmark-s3— 40 passedpre-commit run --all-files— lint clean