Lock convoy between Http2Writer monitor and Okio's global AsyncTimeout lock causes stuck requests and unbounded heap growth in `recoveredFailures`

[hgh1472]

OkHttp version

4.12.0

Okio version

3.6.0

Description

In a production Spring Boot service (JDK 21, using OkHttp as the transport for an HTTP/2 API client), we experienced an incident where two Tomcat request threads never returned, eventually consuming ~945MB (~72%) of the JVM heap and leaving hundreds of other threads BLOCKED/WAITING for over 2 hours until the instance was manually restarted.

We reconstructed the following causal chain from heap dumps (Eclipse MAT) and full thread dumps (including virtual threads):

1. RetryAndFollowUpInterceptor.recoveredFailures grew unbounded on a single call

Two request threads accumulated recovered connection-failure exceptions without limit:

okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept()
  -> recoveredFailures: ArrayList

Heap contents for the two stuck threads:

Thread	Retained heap	recoveredFailures size
exec-A	781,957,496 bytes (59.4%)	~40,000
exec-B	162,947,616 bytes (12.4%)	~8,310

Dominant object types across the heap: [java.net](http://java.net/).SocketException (~48,319), [java.io](http://java.io/).IOException (~48,310), java.lang.StackTraceElement (~9.67M).

Because the request threads never returned, these local variables stayed reachable from the thread stack GC root and were never collected.

2. Threads holding Http2Writer were blocked on the same global AsyncTimeout.lock

At the same time, both stuck threads held a different Http2Writer monitor each, while both were waiting to acquire the same Okio companion-object lock:

exec-A -> holds Http2Writer@0x...590 -> waiting on AsyncTimeout.lock
exec-B -> holds Http2Writer@0x...bd0 -> waiting on AsyncTimeout.lock

Other request threads then queued up BLOCKED behind whichever Http2Writer monitor they needed next. Standard JVM deadlock detection reported nothing (no classic cyclic wait), but functionally this was a persistent lock convoy: the same lock relationship was observed in two thread dumps taken ~2h36m apart.

3. The call-level timeout mechanism itself could not fire

  -> AsyncTimeout.timedOut() -> RealCall.cancel()
  -> internally needs AsyncTimeout.lock to update the timeout queue
  -> lock already contended by the writer threads above
  -> cancellation itself stalls

So the safety mechanism that should have terminated the call was blocked by the same global lock it needed to update its own scheduling state.

4. Amplification factors on our side (application-level, not OkHttp's fault, listed for context)

• An HTTP client SDK layered on top of OkHttp had its own retry (2 attempts) enabled, in addition to retryOnConnectionFailure(true).
• Our application-level timeouts (Future.get(timeout)) only stopped the caller's wait; they never called Call.cancel() on the underlying OkHttp call, so nothing forced the retry loop to end from our side either.
• JDK 21 virtual thread pinning was also observed (VirtualThread.parkOnCarrierThread() while holding a synchronized monitor), which we believe worsened propagation but was not the primary driver of the
  heap growth.