…s-project#370) This commit resolves issue s3s-project#370 by implementing defense-in-depth protections against unbounded memory allocation in POST object multipart/form-data handling. POST object requests could cause unbounded memory allocation at two points: 1. Multipart form parsing buffered all field data without limits 2. File content was aggregated entirely into memory before processing This created a potential DoS vector where malicious clients could exhaust server memory by sending oversized form fields or files. Added three safety constants following MinIO's approach: - MAX_FORM_FIELD_SIZE = 1 MB per field - MAX_FORM_FIELDS_SIZE = 20 MB total for all fields - MAX_FORM_PARTS = 1000 maximum parts Implemented enforcement at multiple layers: - Buffer size check during accumulation - Per-field size validation during parsing - Total fields size tracking - Parts count validation Removed the aggregate_unlimited() call that loaded entire files into memory. File streams are now passed directly to backend implementations for true streaming, allowing backends to handle files of arbitrary size. Extended request handling to support both: - Known-length streams (vec_stream) - existing path - Unknown-length streams (file_stream) - new path for POST object Modified codegen to gracefully handle POST object with unknown content-length, as multipart/form-data doesn't provide file size upfront. Future size test adjustments: The service.rs future size limits were increased by ~100 bytes (~3%) to accommodate the new file_stream field in S3Extensions. This minimal increase in future size (from 2900→3000 and 3200→3300 bytes) is necessary for the streaming implementation and represents an excellent trade-off: 100 extra bytes in the future to prevent potentially unlimited memory consumption. - crates/s3s/src/http/multipart.rs: Add limits and budget tracking - crates/s3s/src/ops/mod.rs: Remove aggregate_unlimited(), pass FileStream - crates/s3s/src/http/request.rs: Add file_stream field to S3Extensions - codegen/src/v1/ops.rs: Support both vec_stream and file_stream - crates/s3s/src/service.rs: Adjust future size limits for streaming support - crates/s3s/src/ops/generated*.rs: Regenerated from codegen - crates/s3s-aws/src/**/generated*.rs: Regenerated from codegen - Added 4 new unit tests for limit enforcement - All 80 existing tests pass - E2E tests validated against live s3s-fs server - Zero regressions in existing functionality Implementation follows MinIO's defense-in-depth strategy: - Multiple independent size limits at different layers - Early rejection of malicious requests before resource exhaustion - Graceful degradation with clear error messages Related upstream issue: s3s-project#155 (Unbounded Memory Allocation in http::body) Fixes s3s-project#370 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Unbounded Memory Allocation in POST object #370

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions