Skip to content

Automatic inclusion of Content-MD5 header breaks S3.upload_part_copy #695

@ararslan

Description

@ararslan

Signing requests with signature v4 will automatically, unconditionally insert a Content-MD5 header:

"Content-MD5" => base64encode(digest(MD_MD5, request.content)),

AFAICT, AWS does not require the Content-MD5 header for anything but multi-object delete requests, and it's otherwise just noted as being recommended for verifying integrity. There appears to be at least one case where the Content-MD5 header cannot be included, which is in an UploadPartCopy request, exposed by AWS.jl as S3.upload_part_copy. Actually using that function results in the following error response:

<Error>
  <Code>InvalidArgument</Code>
  <Message>The specified header is not valid in this context</Message>
  <ArgumentName>Content-MD5</ArgumentName>
  <ArgumentValue>1B2M2Y8AsgTpgAmY7PhCfg==</ArgumentValue>
  <RequestId>...</RequestId>
  <HostId>...</HostId>
</Error>

Amending sign_aws4! to not add Content-MD5 allows S3.upload_part_copy to succeed. It's not actually surprising that the MD5 checksum of the request contents is invalid in this case, since the body of an UploadPartCopy request is empty; all required information for this kind of request is specified in x-amz-* headers.

Many years ago, before AWS.jl as we know it today came into existence, I briefly worked on moving AWS SigV4 functionality out of HTTP.jl into a separate package, the ill-fated AWSAuth.jl, the contents of which now lives here. However, one feature that was included in AWSAuth.jl but not carried over to AWS.jl is JuliaCloud/AWSAuth.jl@b467cb4, which made inserting the Content-MD5 header optional. We may want to consider something like that, though I'm not sure how exactly we'd want to track internally when to include or exclude it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions