MANTA-5482 S3 compatibility layer for manta-buckets-api #68

cneira · 2025-09-11T19:31:53Z

Add an S3 compatibility layer to manta-buckets-api that maps S3 bucket and object requests to manta-buckets-api requests. This layer will perform the following:

Support S3 requests for buckets: HEAD, GET, PUT, DELETE
Support S3 request for objects: HEAD, GET, PUT, DELETE
Support S3 multipart uploads
Support third-party S3 clients (s3cmd)
Basic ACL (public/private)
Multipart uploads

Tests included in the ticket

Co-Authored-By: Claude [email protected]

danmcd

Checkpointing the incremental changes. Will need to set myself up for another full pass on the code files, probably the doc files too, and maybe the teset files.

lib/server.js

danmcd · 2025-10-28T23:57:04Z

lib/server.js

+
+                                    res.write(s3XmlError, 'utf8');
+                                    res.end();
                                    next(false); // Stop route processing


This next(false); // Stop route processing and the one in the else case... can't they just be pulled out into one line after the if, and the else{} on 990-991 just goes away?

lib/server.js

danmcd

Incremental check.

danmcd · 2025-11-03T14:09:36Z

docs/quickstart.md

+npm install -g json
+```
+
+#### SmartOS (pkgsrc)


pkgsrc also works for MacOS.

docs/quickstart.md

danmcd · 2025-11-03T14:14:06Z

lib/common.js

        // Response headers
        var sh = shark.headers;
+        var isAwsCli = req.headers['user-agent'] &&
+                      req.headers['user-agent'].indexOf('aws-cli') !== -1;


Re-ping. I expect this to be, "actually more than there should be, but you know how lazy other implementers are, Dan." :)

danmcd · 2025-11-03T14:14:53Z

lib/s3-mako-v2-commit.js


    // Get target sharks for final object placement
    var targetSharks = getTargetSharks(req, uploadRecord, partPaths);
+    if (!targetSharks) {


Good catch, sorry I missed this earlier.

Regarding the 'aws-cli' user-agent check, I removed those from common.js, as those were not needed as we are just processing header ranges, other clients as well could send/require those.

Implement complete Cross-Origin Resource Sharing functionality: - AWS S3 CORS API compatibility (PutBucketCors, GetBucketCors, DeleteBucketCors) - Object-level CORS via metadata headers with m-access-control-* prefix - CORS preflight OPTIONS request handling for browser compliance - Presigned URL CORS support for direct browser access - Comprehensive testing with HTML pages for real-world scenarios - Documentation with AWS S3 feature comparison matrix Technical implementation: - XML and JSON CORS configuration parsing with validation - Bucket-level CORS stored as .cors-configuration metadata objects - Refactor XML utilities to eliminate code duplication Generated with https://claude.ai/code Co-Authored-By: Claude mailto:[email protected]

Enhanced lib/auth.js to detect MPU operations via uploadId and partNumber query parameters in convertS3PresignedToManta(). Added operation detection for UploadPart requests and preserved MPU-specific parameters in S3 request object for proper authentication pipeline integration. Generated with https://claude.ai/code Co-Authored-By: Claude <[email protected]>

Add CORS headers support for all MPU operations to enable browser-based multipart uploads using AWS SDK for JavaScript. - Add tryBucketLevelCors to MPU handlers (UploadPart, CompleteMultipartUpload, ListParts, ListMultipartUploads) - Add CORS support to ListBucketObjects handler - Fix SigV4 signature verification for browser AWS SDK by detecting x-amz-user-agent header - Update MPU handlers to load bucket object for CORS processing - Update test CORS config to include required exposed headers All MPU operations now apply bucket-level CORS configuration set via put-bucket-cors, enabling browser compatibility while maintaining existing authentication for non-browser clients. Generated with https://claude.ai/code Co-Authored-By: Claude <[email protected]>

danmcd

Incremental review, and not yet done becuase of large diffs or new files: s3-compat-awscli-test.sh, boto3-tests.py, cors-middleware.js, cors.js.

This is a big incremental, even not including the aforementioned not-yet-files. I'm going to assume CORS buys us a lot here, not just more green checkmarks? :)

danmcd · 2025-11-18T20:47:27Z

docs/divergences.md

+| **GET BUCKET CORS** | ✅ Supported | ✅ Supported | **IMPLEMENTED** | Returns S3 compatible XML response |
+| **DELETE BUCKET CORS** | ✅ Supported | ✅ Supported | **IMPLEMENTED** | Removes bucket-level CORS configuration |
+| **CORS Preflight (OPTIONS)** | ✅ Supported | ✅ Supported | **IMPLEMENTED** | Full CORS specification compliance |
+| **Object-level CORS** | ❌ Not Supported | ✅ Supported | **ENHANCED** | CORS via object metadata (m-access-control-*) |


Wow, this is something we have above&beyond S3?

danmcd · 2025-11-18T20:47:42Z

docs/divergences.md

+|---------|--------|------------------|----------------------|
+| **Configuration Scope** |  |  |  |
+| Bucket-level CORS rules | ✅ Supported | ✅ Supported | Identical AWS S3 API compatibility |
+| Object-level CORS rules | ❌ Not Available | ✅ Supported | **Manta Enhancement**: via `m-access-control-*` metadata |


danmcd · 2025-11-18T21:05:37Z

lib/s3-multipart.js

                    res.setHeader('ETag', '"' + finalETag + '"');
-                    res.send(200);
-                    next(false);
+


Is it just me or are all replies in s3UploadPartHandler() now going to perform CORS ? Or am I missing (maybe from new files or large-changes I've not yet read yet) something?

danmcd · 2025-11-18T21:05:54Z

lib/s3-multipart.js

+                        req.metadata = { headers: {} };
+                    }
+
+                    // Add CORS headers from bucket CORS configuration


Same question here.

danmcd · 2025-11-18T21:06:46Z

lib/s3-multipart.js

-                res.send(xml);
-                next(false); // Stop processing
+
+                // Add CORS headers from bucket CORS configuration


danmcd · 2025-11-18T21:08:10Z

lib/server.js


+    // CORS processing is handled in addCustomHeaders function in common.js
+    // for both object-level metadata and bucket-level CORS configuration
+


This may answer my confused-about-mandatory-CORS above.

danmcd · 2025-11-18T21:13:04Z

lib/common.js

+
+        // Log each metadata key individually to see exact names
+        mdKeys.forEach(function (k) {
+            if (k.toLowerCase().includes('control') || k.toLowerCase().


None of these checks will yield false-positives for CORS?

danmcd · 2025-11-18T21:14:00Z

lib/common.js

+            headerName = k.substring(2); // Remove 'm-' prefix
+        }
+
+        var isCorsHeader = CORS_RES_HDRS.indexOf(headerName) !== -1;


Is this why we always do cors processing elsewhere? We can catch/avoid it here if there are none?

cneira added 30 commits August 1, 2025 17:27

update documentation about roles

1dc258f

Fix --acl-public and --acl-private

83d9e30

jslint + updated document on testing

5dc0d87

delete obsolete doc

7a14be4

Add tests for s3cmd

7ac8563

remove friendly redirects

d3eaf65

Add anonymous access to nested objects.

cfa5297

Bulk deletes now work, preserve content-md5 to pass to mahi.

a6bd62f

bulk delete actually delete objects now

f2eb5f5

Fix awscli and s3cmd specific canonical url quirks.

6b71664

bulk delete now can delete files with spaces

e5b09dd

jslint

1a0229e

mock mpu

c2b95b1

mpu through mdapi

7ff5230

mpu works wip

7a98a5f

fix download of objects from multipart objects

92a38d9

js lint

380aa30

remove not used vars

f4e640f

remove mako references, calculate part size

215c9f4

jslint

e39e776

remove mako message

225483a

update README, fix tests

c25e56b

Doc for mpu

7f34080

Fix MPU bug used the wrong object id when storing

81ef0d9

remove unused var

21959ec

Respect durability-level header in mpu uploads

5053dff

Also respect the durability per part in MPU

5436ffc

Store durability-level in a cache for MPU

62fbd2a

js lint

20046e7

Store durability value in tmp obj metadata

076df4c

cneira added 5 commits October 24, 2025 22:20

mpu use storinfo.choose

e00d8f0

fix mpu small entity

9f94654

Remove dead code for mpu

7bea1f6

no more s3-multipart-v2

7362a77

js lint mpu, remove sections from quickstart.md

afe07a3

cneira requested a review from danmcd October 28, 2025 20:46

danmcd reviewed Oct 28, 2025

View reviewed changes

cneira added 2 commits October 29, 2025 21:14

move info to debug, fix audit logging.

259360b

Modify documentation, add presigned doc.

388c2af

cneira requested a review from danmcd November 3, 2025 11:58

Remove redundant next(false)

387b3cb

danmcd reviewed Nov 3, 2025

View reviewed changes

cneira added 8 commits November 3, 2025 14:51

Add aws-sdk useragent to patch MANTA-331

95c5e81

Remove redundant awscli condition, add pkgsrc macOS

cf806bb

remove comments for awscli

12c5f19

move logging from warn to debug

c8815bf

sanitize remaining logs, change warn to debug.

f8f510d

Only log errors.

f43d6e6

jslint

d5a6328

Cap server side copying to 5GiB per AWS docs.

63e9d03

cneira requested a review from danmcd November 5, 2025 17:35

cneira and others added 7 commits November 8, 2025 03:19

fix CORS not such configuration Error handling

c9a977e

NoSuchCORSConfiguration when config does not exists

95225d0

jslint

894da3f

jslint

2c44338

danmcd reviewed Nov 18, 2025

View reviewed changes

cneira mentioned this pull request Dec 5, 2025

MANTA-5485 Add STS/IAM support to manta-buckets-api #69

Open


		// CORS processing is handled in addCustomHeaders function in common.js
		// for both object-level metadata and bucket-level CORS configuration

MANTA-5482 S3 compatibility layer for manta-buckets-api #68

Are you sure you want to change the base?

MANTA-5482 S3 compatibility layer for manta-buckets-api #68

Uh oh!

Conversation

cneira commented Sep 11, 2025

Uh oh!

danmcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danmcd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danmcd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants