Use Beamlab ExAWS.SQS - #89
Conversation
|
I'll update this to use |
ca72c0b to
d5e74b2
Compare
This PR updates Broadway to use https://github.com/BeamLabEU/ex_aws_sqs instead of the apparently abandoned ex_aws_sqs. Beamlab ExAWS.SQS 5.0 has three breaking changes that would impact Broadway.SQS: - Only the JSON APIs are used - Elixir 1.18 is required - Hackney 4.0 is required for tests, not Hackney 1.0 The JSON API changes introduce their own potential breakage because the message _shape_ has changed. The XML parsers for ExAws.SQS previously normalized fields as `:message_id` instead of `"MessageId"`, but the Beamlabs implementation makes no such changes. This wasn't automatic atom creation, but deliberate transformation, see [`lib/ex_aws/sqs/sweet_xml_parser.ex:196-218`][lines]. There are values (such as the request ID and the action) which are no longer in the _body_ of the request or the response (as appropriate), but are in headers (`x-amz-requestid` and `x-amz-target`, respectively). Req is supported via ExAws configuration. [lines]: https://github.com/ex-aws/ex_aws_sqs/blob/main/lib/ex_aws/sqs/sweet_xml_parser.ex#L196-L218
d5e74b2 to
4a24ad7
Compare
|
IMO we should get support with merging this PR instead of making changes to |
|
Even if the decision is made to not switch to the Beamlab implementation, this is useful for a future point where ExAws.SQS switches from the XML API to the JSON API or if Broadway.SQS switches to the JSON API (#88) through Finch or Req (#73). The work is done. It could be that I could do some work that would work with either the XML or JSON implementation regardless of which ExAws.SQS implementation you use (the Beamlab implementation is now installed with |
It would be nice to get some sort of fix moved through wherever we can since this is blocking various other commonly used libraries from being upgraded. |
|
We do support a pluggable client as BroawdaySQS.Client, so that could be used alongside the override to use the latest version. However, I really think we should find ways to be independent of ExAWS.SQS:
Is anyone interested in spiking a proof of concept for consuming SQS through JSON? @wojtekmach already shared a starting point here: #73 (comment), it should hopefully be a matter of triggering the right endpoint. |
This PR updates Broadway to use https://github.com/BeamLabEU/ex_aws_sqs instead of the apparently abandoned ex_aws_sqs.
Beamlab ExAWS.SQS
4.05.0 introduces three breaking changes that would impact Broadway.SQS:The JSON API changes introduce their own potential breakage because the message shape has changed. The XML parsers for ExAws.SQS previously normalized fields as
:message_idinstead of"MessageId", but the Beamlabs implementation makes no such changes. This wasn't automatic atom creation, but deliberate transformation, seelib/ex_aws/sqs/sweet_xml_parser.ex:196-218.There are values (such as the request ID and the action) which are no longer in the body of the request or the response (as appropriate), but are in headers (
x-amz-requestidandx-amz-target, respectively).Req is supported via ExAws configuration.