Skip to content

Use Beamlab ExAWS.SQS - #89

Open
halostatue wants to merge 1 commit into
elixir-broadway:mainfrom
halostatue:use-beamlab-ex_aws_sqs
Open

Use Beamlab ExAWS.SQS#89
halostatue wants to merge 1 commit into
elixir-broadway:mainfrom
halostatue:use-beamlab-ex_aws_sqs

Conversation

@halostatue

@halostatue halostatue commented Jul 17, 2026

Copy link
Copy Markdown

This PR updates Broadway to use https://github.com/BeamLabEU/ex_aws_sqs instead of the apparently abandoned ex_aws_sqs.

Beamlab ExAWS.SQS 4.05.0 introduces three breaking changes that would impact Broadway.SQS:

  • Only the JSON APIs are used
  • Elixir 1.18 is required
  • Hackney 4.0 is required for tests, not Hackney 1.0

The JSON API changes introduce their own potential breakage because the message shape has changed. The XML parsers for ExAws.SQS previously normalized fields as :message_id instead of "MessageId", but the Beamlabs implementation makes no such changes. This wasn't automatic atom creation, but deliberate transformation, see lib/ex_aws/sqs/sweet_xml_parser.ex:196-218.

There are values (such as the request ID and the action) which are no longer in the body of the request or the response (as appropriate), but are in headers (x-amz-requestid and x-amz-target, respectively).

Req is supported via ExAws configuration.

@halostatue

halostatue commented Jul 17, 2026

Copy link
Copy Markdown
Author

Indirectly related to #73 and implements #88 by proxy.

Note that the workflow will fail because it uses 1.14 and beamlabs_ex_aws_sqs requires 1.18; I didn't update that because I wasn't sure if this was a direction that would be preferred.

@halostatue

Copy link
Copy Markdown
Author

I'll update this to use beamlab_ex_aws_sqs 5.0 with the hex override tomorrow.

@halostatue
halostatue force-pushed the use-beamlab-ex_aws_sqs branch from ca72c0b to d5e74b2 Compare July 20, 2026 13:58
This PR updates Broadway to use https://github.com/BeamLabEU/ex_aws_sqs
instead of the apparently abandoned ex_aws_sqs.

Beamlab ExAWS.SQS 5.0 has three breaking changes that would impact
Broadway.SQS:

- Only the JSON APIs are used
- Elixir 1.18 is required
- Hackney 4.0 is required for tests, not Hackney 1.0

The JSON API changes introduce their own potential breakage because the
message _shape_ has changed. The XML parsers for ExAws.SQS previously
normalized fields as `:message_id` instead of `"MessageId"`, but the
Beamlabs implementation makes no such changes. This wasn't automatic
atom creation, but deliberate transformation, see
[`lib/ex_aws/sqs/sweet_xml_parser.ex:196-218`][lines].

There are values (such as the request ID and the action) which are no
longer in the _body_ of the request or the response (as appropriate),
but are in headers (`x-amz-requestid` and `x-amz-target`, respectively).

Req is supported via ExAws configuration.

[lines]: https://github.com/ex-aws/ex_aws_sqs/blob/main/lib/ex_aws/sqs/sweet_xml_parser.ex#L196-L218
@halostatue
halostatue force-pushed the use-beamlab-ex_aws_sqs branch from d5e74b2 to 4a24ad7 Compare July 20, 2026 14:25
@slashmili

Copy link
Copy Markdown
Contributor

IMO we should get support with merging this PR instead of making changes to broadway_sqs.

@halostatue

Copy link
Copy Markdown
Author

Even if the decision is made to not switch to the Beamlab implementation, this is useful for a future point where ExAws.SQS switches from the XML API to the JSON API or if Broadway.SQS switches to the JSON API (#88) through Finch or Req (#73). The work is done. It could be that I could do some work that would work with either the XML or JSON implementation regardless of which ExAws.SQS implementation you use (the Beamlab implementation is now installed with {:ex_aws_sqs, "~> 5.0", hex: :beamlab_ex_aws_sqs} which means that if ExAws.SQS adopts the changes present in the Beamlab implementation, it's all good.

@mhooverdev

Copy link
Copy Markdown

IMO we should get support with merging this PR instead of making changes to broadway_sqs.

It would be nice to get some sort of fix moved through wherever we can since this is blocking various other commonly used libraries from being upgraded.

@josevalim

Copy link
Copy Markdown
Contributor

We do support a pluggable client as BroawdaySQS.Client, so that could be used alongside the override to use the latest version.

However, I really think we should find ways to be independent of ExAWS.SQS:

  • the API that we need is minimal
  • SQS supports JSON, which it did not when we created this project. JSON is part of OTP, so fewer dependencies
  • there is a separate aws_credentials project that handles the annoying credential side of AWS

Is anyone interested in spiking a proof of concept for consuming SQS through JSON? @wojtekmach already shared a starting point here: #73 (comment), it should hopefully be a matter of triggering the right endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants