Skip to content

reverse_tunnels: intercept RPING on upstream reverse connections#43666

Open
aakugan wants to merge 1 commit intoenvoyproxy:mainfrom
aakugan:fix/upstream-rc-races
Open

reverse_tunnels: intercept RPING on upstream reverse connections#43666
aakugan wants to merge 1 commit intoenvoyproxy:mainfrom
aakugan:fix/upstream-rc-races

Conversation

@aakugan
Copy link
Contributor

@aakugan aakugan commented Feb 26, 2026

Commit Message

Intercept RPING on upstream reverse connections

Additional Description

If a RPING is sent and then the connection is upgraded to http2 via the conn pool the downstream reverse connection io handle's rping response causes a 502 protocol error.

More specifically: Remote peer returned unexpected data while we expected SETTINGS frame

Additional logs:

2026-02-26 17:01:22.937 trace connection[thread=41 func=onReadReady file=source/common/network/connection_impl.cc:721] [Tags: "ConnectionId":"10614"] read ready. dispatch_buffered_data=0
2026-02-26 17:01:22.937 info misc[thread=41 func=read file=./source/extensions/bootstrap/reverse_tunnel/upstream_socket_interface/reverse_connection_io_handle.h:89] ALERT GOT SOME RPING Bytes: RPING
2026-02-26 17:01:22.937 trace connection[thread=41 func=doRead file=source/common/network/raw_buffer_socket.cc:25] [Tags: "ConnectionId":"10614"] read returns: 20745
2026-02-26 17:01:22.937 info misc[thread=41 func=read file=./source/extensions/bootstrap/reverse_tunnel/upstream_socket_interface/reverse_connection_io_handle.h:89] ALERT GOT SOME RPING Bytes: RPING
2026-02-26 17:01:22.937 trace connection[thread=41 func=doRead file=source/common/network/raw_buffer_socket.cc:39] [Tags: "ConnectionId":"10614"] read error: Resource temporarily unavailable, code: 0
2026-02-26 17:01:22.937 trace http2[thread=41 func=dispatch file=source/common/http/http2/codec_impl.cc:1077] [Tags: "ConnectionId":"10614"] dispatching 20745 bytes
2026-02-26 17:01:22.937 debug http2[thread=41 func=onError file=source/common/http/http2/codec_impl.cc:1395] [Tags: "ConnectionId":"10614"] invalid http2: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.

The Alert was a custom log line added by me for easier debugging.
Its better for both the downstream and upstream rc io handles to share a common read implementation.

Testing

Unit tests.
Additionally, manually validated the example in the docs which also works after the change.

Signed-off-by: aakugan <aakashganapathy2@gmail.com>
@aakugan aakugan force-pushed the fix/upstream-rc-races branch from e5dd925 to f248e1e Compare February 26, 2026 21:16
@basundhara-c
Copy link
Contributor

@aakugan can you detail a little more on the steps leading to this issue? Is this a race between upstream receiving a RPONG in plaintext from downstream, after having sent a data request on a HTTP/2 stream?

@aakugan
Copy link
Contributor Author

aakugan commented Feb 27, 2026

Yes @basundhara-c that is correct under heavy load / latency we observed that the connection may get upgraded ie PRI... was sent and the codec is waiting for the settings frame before the downstream RPONG is received which is then directly passed to the client_codec_ causing this issue.

@RyanTheOptimist
Copy link
Contributor

/assign @botengyao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants