occasional 401 exceptions during routine operations due to credential checking on EC2 instances

[ericphanson]

    nested task error: AWS.AWSExceptions.AWSException: 401 -- AWSException                                                                                                                    
                                                                                                                                                                                              
    HTTP.Exceptions.StatusError(401, "GET", "/latest/meta-data/iam/info", HTTP.Messages.Response:                                                                                             
    """                                                                                                                                                                                       
    HTTP/1.1 401 Unauthorized                                                                                                                                                                 
    Content-Length: 0                                                                                                                                                                         
    Date: Thu, 01 May 2025 19:03:16 GMT                                                        
    Server: EC2ws
    Connection: close
    Content-Type: text/plain
     
    """)
     
    Stacktrace:
      [1] (::HTTP.ConnectionRequest.var"#connections#4"{…})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int6
4, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{…})
        @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/MIUdD/src/clientlayers/ConnectionRequest.jl:144
      [2] connections
        @ ~/.julia/packages/HTTP/MIUdD/src/clientlayers/ConnectionRequest.jl:60 [inlined]
      [3] (::HTTP.RetryRequest.var"#manageretries#3"{…})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempoten
t::Bool, kw::@Kwargs{…})
        @ HTTP.RetryRequest ~/.julia/packages/HTTP/MIUdD/src/clientlayers/RetryRequest.jl:35
      [4] manageretries
        @ ~/.julia/packages/HTTP/MIUdD/src/clientlayers/RetryRequest.jl:30 [inlined]
      [5] (::HTTP.CookieRequest.var"#managecookies#4"{…})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{…})
        @ HTTP.CookieRequest ~/.julia/packages/HTTP/MIUdD/src/clientlayers/CookieRequest.jl:42
      [6] managecookies
        @ ~/.julia/packages/HTTP/MIUdD/src/clientlayers/CookieRequest.jl:19 [inlined]
      [7] (::HTTP.HeadersRequest.var"#defaultheaders#2"{…})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_hea
ders::Bool, kw::@Kwargs{…})
        @ HTTP.HeadersRequest ~/.julia/packages/HTTP/MIUdD/src/clientlayers/HeadersRequest.jl:71
      [8] defaultheaders
        @ ~/.julia/packages/HTTP/MIUdD/src/clientlayers/HeadersRequest.jl:14 [inlined]
      [9] (::HTTP.RedirectRequest.var"#redirects#3"{…})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::No
thing, kw::@Kwargs{…})
        @ HTTP.RedirectRequest ~/.julia/packages/HTTP/MIUdD/src/clientlayers/RedirectRequest.jl:25
     [10] redirects
        @ ~/.julia/packages/HTTP/MIUdD/src/clientlayers/RedirectRequest.jl:14 [inlined]
     [11] (::HTTP.MessageRequest.var"#makerequest#3"{…})(method::String, url::URIs.URI, headers::Vector{…}, body::Vector{…}; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.S
trings.HTTPVersion, verbose::Int64, kw::@Kwargs{…})
        @ HTTP.MessageRequest ~/.julia/packages/HTTP/MIUdD/src/clientlayers/MessageRequest.jl:35
     [12] makerequest
        @ ~/.julia/packages/HTTP/MIUdD/src/clientlayers/MessageRequest.jl:24 [inlined]
     [13] request(stack::HTTP.MessageRequest.var"#makerequest#3"{…}, method::String, url::URIs.URI, h::Vector{…}, b::Vector{…}, q::Nothing; headers::Vector{…}, body::Vector{…}, query::Nothin
g, kw::@Kwargs{…})
        @ HTTP ~/.julia/packages/HTTP/MIUdD/src/HTTP.jl:457
     [14] #request#20
        @ ~/.julia/packages/HTTP/MIUdD/src/HTTP.jl:315 [inlined]
     [15] request
        @ ~/.julia/packages/HTTP/MIUdD/src/HTTP.jl:313 [inlined]
     [16] macro expansion
        @ ~/.julia/packages/Mocking/rzcpa/src/mock.jl:50 [inlined]
     [17] _http_request(::String, ::Vararg{Any}; status_exception::Bool, kwargs::@Kwargs{})
        @ AWS.IMDS ~/.julia/packages/AWS/SqWtW/src/IMDS.jl:118
     [18] _http_request
        @ ~/.julia/packages/AWS/SqWtW/src/IMDS.jl:115 [inlined]
     [19] request(session::AWS.IMDS.Session, method::String, path::String; kwargs::@Kwargs{})
        @ AWS.IMDS ~/.julia/packages/AWS/SqWtW/src/IMDS.jl:112
     [20] request
        @ ~/.julia/packages/AWS/SqWtW/src/IMDS.jl:100 [inlined]
     [21] get(session::AWS.IMDS.Session, path::String)
        @ AWS.IMDS ~/.julia/packages/AWS/SqWtW/src/IMDS.jl:168
     [22] get
        @ ~/.julia/packages/AWS/SqWtW/src/IMDS.jl:180 [inlined]
     [23] ec2_instance_credentials(profile::String)
        @ AWS ~/.julia/packages/AWS/SqWtW/src/AWSCredentials.jl:233
     [24] (::AWS.var"#12#18"{String})()
        @ AWS ~/.julia/packages/AWS/SqWtW/src/AWSCredentials.jl:132
     [25] check_credentials(aws_creds::AWSCredentials; force_refresh::Bool)
        @ AWS ~/.julia/packages/AWS/SqWtW/src/AWSCredentials.jl:201
     [26] check_credentials
        @ ~/.julia/packages/AWS/SqWtW/src/AWSCredentials.jl:196 [inlined]
     [27] sign_aws4!(aws::AWSConfig, request::Request, time::DateTime)
        @ AWS ~/.julia/packages/AWS/SqWtW/src/utilities/sign.jl:56

Several hours into doing the same operations in a big loop, I get a 401 like this. I believe there is likely a missing retry or something like that.

[ericphanson]

I believe this is a separate issue from #698 (not thread safety) because I'm now provisioning separate AWS configs with task local storage, but it is occurring under high load, so the /latest/meta-data/iam/info endpoint may be strained or something. Nonetheless we should not crash here if we can help it.

oops, actually this call is using the global config, so it could be related to thread-safety as well