Repository navigation
reverse_tunnel: reject new streams during listener drain - #49361
AntonKanug wants to merge 7 commits into
Conversation
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
d853940 to
7f733eb
Compare
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
7f733eb to
0963cb6
Compare
|
/assign basundhara-c |
|
/wait |
|
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to |
|
/retest |
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
a27b51c to
18a3a34
Compare
agrawroh
left a comment
There was a problem hiding this comment.
🚫 Suggested Fixes. The layering is right, and the two blockers are the conflict with main's node selection and idle tunnels on in-place updates.
⚠️ This conflicts with #49367 on main, which made this round-robin prefer a node with an idle socket, and each side's new test fails under the other's rule.⚠️ Withenable_drain_with_goaway, an in-place filter-chain update under an immediate drain closes every idle tunnel at once, and on a worker with no busy tunnel they wait up to 11.5 seconds for the next maintenance tick, while the HCM alone spread these closes over the window's last third.- ❓ After an admin drain with
skip_exitthe listeners keep accepting, but the initiator's stop latch never clears, so that listener never dials again until a full LDS update replaces it. - 💡 The idle-tunnel poll runs every 100 ms although nothing can close a tunnel before a drain event, and under a gradual drain each tick redraws the ramp, so most idle tunnels close in the first 20 seconds of a 600 second drain.
- 💡 A first request on an idle tunnel can be dropped unread once its drain starts, since
onData()runs the drain check again, which closes at once under an immediate drain and redraws the ramp under a gradual one. - 💡 The idle-close filter is added to every connection, so on a plain listener a drain closes clients that haven't sent their first request, and under an immediate drain every client accepted during it, which the HCM alone would serve.
- 💡 Handshakes cut short by the stop keep their connecting state and gauge, and queued tunnels log a successful handshake with no close.
- 💡 Nothing logs when the handle stops dialing, while each busy tunnel that drains afterwards still logs at info that it is dialing a replacement.
- 💡 A request to a node whose tunnels all received GOAWAY now fails as no healthy upstream, counted only in
upstream_cx_none_healthyand logged at debug, where it used to count asfallback_no_reverse_socket. - 💡 Responders exclude these initiators only when the reverse connection cluster sets the upstream codec's
enable_drain_with_goaway, and an excluded node now fails as a 503 with no healthy upstream. - 💡 The listener-drain stop applies to every reverse listener whatever
enable_drain_with_goawaysays, while the field comment still describes only the peer GOAWAY redial and omits that the field now closes unused tunnels on a drain. - 💡 Only stream listeners still running at drain start call
onDrain(), so UDP and QUIC listeners, an LDS removal and a hot restart never see it, and their handles get onlyresetFileEvents(). - 💡 Could the description add the Commit Message, Risk Level, Testing, Docs Changes and Release Notes fields from the PR template, and the title name the initiator's dial stop, since both become the squash commit?
- 💡 With no drain event delivered, the MODIFY_ONLY health-check test passes whatever the drain type.
- 💡 No unit test runs the filter factory callback, and the integration tests run it only with the option on and the default guard, so the filter's order and legacy drain scope are unpinned.
- 💡 A request for a known node now hashes the key twice in the new eligibility check, opted in or not.
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
agrawroh
left a comment
There was a problem hiding this comment.
💚 Approved. Both blockers from the last review are fixed for the cases they named, and what's left is one question and some suggestions.
- ❓ Under the immediate strategy, a full LDS update or a hot restart closes idle tunnels before the successor's are up, while the merge-base kept them for most of the window, so requests to an initiator with only idle tunnels fail meanwhile.
- 💡 Under the default gradual drain, the HCM still closes about one idle tunnel in six in the window's last third, so on an in-place update those wait for the maintenance tick.
- 💡 The filter's idle closes count in no drain stat, while the HCM's close of the same tunnel counts
downstream_cx_drain_close, so withenable_drain_with_goawaythat stat misses most idle closes. - 💡 The new stop log doesn't say which listener's initiator stopped, and it also logs at worker shutdown, where nothing drains.
- 💡 No test pins that the stop keeps an accepted tunnel's Connected state and gauge, so clearing every state here would still pass.
- 💡 The
listener_stoppedcheck passes with or withoutskip_exit, since a graceful drain stops listeners only when its 600 s drain time ends, which the test never reaches. - 💡 This release's connection-metadata drain note still says a draining tunnel redials a replacement at drain start, which a whole-listener drain no longer does.
- 💡 Could the codec option's field comment say this too, that it now skips initiators whose tunnels are all draining and can fail a request as no healthy upstream?
- 💡 Could the note name
upstream_rq_all_tunnels_draininghere, since these 503s now count there rather than infallback_no_reverse_socket? - 💡 This comment no longer says a reverse tunnel's delayed soft GOAWAY replaces the one at drain start, and it states the rotation dial without
enable_drain_with_goaway, which alone wires it.
This approval covers the core listener hooks too, while the proto changes still need an API shepherd.
| return; | ||
| } | ||
| drain_scheduled_ = true; | ||
| std::chrono::milliseconds delay{0}; |
There was a problem hiding this comment.
❓ Under the immediate strategy, a full LDS update or a hot restart closes idle tunnels before the successor's are up, while the merge-base kept them for most of the window, so requests to an initiator with only idle tunnels fail meanwhile. Is that gap intended on those two paths?
There was a problem hiding this comment.
Restored bounded idle grace for full LDS updates and hot-restart parent draining, with tests for both.
| drain_time.count() > 0) { | ||
| // Sample once over the original drain window. Repeated probabilistic polling would close | ||
| // almost every idle tunnel near the start of a gradual drain. | ||
| const std::chrono::milliseconds close_after{server_context_.api().randomGenerator().random() % |
There was a problem hiding this comment.
💡 Under the default gradual drain, the HCM still closes about one idle tunnel in six in the window's last third, so on an in-place update those wait for the maintenance tick. Could the filter draw its close from the first two thirds, guarding a window that rounds to zero, and test a gradual in-place update under the default guard?
There was a problem hiding this comment.
Close times now fall before the two-thirds boundary, with zero-window handling and a regression test.
| // In-place filter-chain updates must refill the idle pool before the maintenance tick. A | ||
| // whole-listener drain has already stopped the parent, so the callback cannot replenish it. | ||
| tunnel_iohandle_->markTunnelDrainingAndDialReplacement(); | ||
| connection_->close(Network::ConnectionCloseType::NoFlush, "reverse_tunnel_listener_draining"); |
There was a problem hiding this comment.
💡 The filter's idle closes count in no drain stat, while the HCM's close of the same tunnel counts downstream_cx_drain_close, so with enable_drain_with_goaway that stat misses most idle closes. Could the filter bump it through the HCM config's stats?
There was a problem hiding this comment.
Idle closes now increment HCM’s downstream_cx_drain_close, with test coverage.
| if (connections_stopped_) { | ||
| return; | ||
| } | ||
| ENVOY_LOG(info, "reverse_tunnel: listener draining; stopping pending and replacement dials"); |
There was a problem hiding this comment.
💡 The new stop log doesn't say which listener's initiator stopped, and it also logs at worker shutdown, where nothing drains. Could it name the source node and cluster, and say it stopped tunnel initiation rather than that the listener is draining?
There was a problem hiding this comment.
Added the source node and cluster and changed the wording to “stopping tunnel initiation.”
| for (auto& [host, host_info] : host_to_conn_info_map_) { | ||
| for (auto state = host_info.connection_states.begin(); | ||
| state != host_info.connection_states.end();) { | ||
| if (state->second == ReverseConnectionState::Connecting) { |
There was a problem hiding this comment.
💡 No test pins that the stop keeps an accepted tunnel's Connected state and gauge, so clearing every state here would still pass. Could the accepted-child drain test set a Connected state and check both survive onListenerDrain(), before it marks the child draining?
There was a problem hiding this comment.
Added assertions that Connected state and gauges survive listener drain until the child drains.
| EXPECT_EQ(1, test_server_->counter("cluster.cluster_1.upstream_cx_total")->value()); | ||
|
|
||
| // Neither closing an idle tunnel nor sending GOAWAY may replenish a draining listener's pool. | ||
| EXPECT_EQ(0, test_server_->counter("listener_manager.listener_stopped")->value()); |
There was a problem hiding this comment.
💡 The listener_stopped check passes with or without skip_exit, since a graceful drain stops listeners only when its 600 s drain time ends, which the test never reaches. Could it let the drain finish before this check?
There was a problem hiding this comment.
The test now checks that the listener stays running without redials beyond the 600-second deadline.
| @@ -0,0 +1,8 @@ | |||
| reverse_tunnel: fixed listener draining so initiators stop pending handshakes and replacement dials when | |||
There was a problem hiding this comment.
💡 This release's connection-metadata drain note still says a draining tunnel redials a replacement at drain start, which a whole-listener drain no longer does. Could this PR amend that note too?
There was a problem hiding this comment.
Updated the note to distinguish rotation redials from whole-listener draining.
| reverse_tunnel: fixed listener draining so initiators stop pending handshakes and replacement dials when | ||
| the listener begins draining, without stopping replacement tunnels during in-place filter-chain updates. | ||
| With ``enable_drain_with_goaway``, unused tunnels close while accepted tunnels follow the configured drain schedule. | ||
| When the responder's reverse connection cluster enables ``ReverseTunnelUpstreamCodecOptions.enable_drain_with_goaway``, |
There was a problem hiding this comment.
💡 Could the codec option's field comment say this too, that it now skips initiators whose tunnels are all draining and can fail a request as no healthy upstream?
There was a problem hiding this comment.
Documented exclusion of all-draining initiators and the resulting no-healthy-upstream / HTTP 503 response.
| responders exclude initiators whose remaining tunnels have received GOAWAY when selecting a node for new streams, | ||
| while retaining those tunnels for existing streams. Node selection prefers idle sockets; when none are available, | ||
| it falls back to round-robin among nodes with a tunnel that has not received GOAWAY. A request with no eligible | ||
| initiator fails with HTTP 503 (no healthy upstream). Early drain METADATA still only notifies the reporter. |
There was a problem hiding this comment.
💡 Could the note name upstream_rq_all_tunnels_draining here, since these 503s now count there rather than in fallback_no_reverse_socket?
There was a problem hiding this comment.
Added upstream_rq_all_tunnels_draining to the note.
| // ``enable_drain_with_goaway``, a reverse tunnel sends the soft GOAWAY and an optional | ||
| // METADATA frame at the configured delays before the final GOAWAY. Without | ||
| // ``shutdown_notice_delay``, such a tunnel receives no GOAWAY until the final one. | ||
| // Per-connection rotation dials a replacement before the old tunnel closes. Whole-listener draining | ||
| // instead stops pending handshakes and replacement dials regardless of ``enable_drain_with_goaway``. |
There was a problem hiding this comment.
💡 This comment no longer says a reverse tunnel's delayed soft GOAWAY replaces the one at drain start, and it states the rotation dial without enable_drain_with_goaway, which alone wires it. Could it say both?
| // ``enable_drain_with_goaway``, a reverse tunnel sends the soft GOAWAY and an optional | |
| // METADATA frame at the configured delays before the final GOAWAY. Without | |
| // ``shutdown_notice_delay``, such a tunnel receives no GOAWAY until the final one. | |
| // Per-connection rotation dials a replacement before the old tunnel closes. Whole-listener draining | |
| // instead stops pending handshakes and replacement dials regardless of ``enable_drain_with_goaway``. | |
| // ``enable_drain_with_goaway``, a reverse tunnel instead sends the soft GOAWAY and an optional | |
| // METADATA frame at the configured delays before the final GOAWAY. Without | |
| // ``shutdown_notice_delay``, such a tunnel receives no GOAWAY until the final one. On | |
| // per-connection rotation, such a tunnel also dials a replacement before the old one closes. | |
| // Whole-listener draining instead stops pending handshakes and replacement dials regardless of | |
| // ``enable_drain_with_goaway``. |
There was a problem hiding this comment.
Clarified that delayed soft GOAWAY replaces the immediate notice and rotation redials require enable_drain_with_goaway.
basundhara-c
left a comment
There was a problem hiding this comment.
This changes drain behavior on three paths: the initiator's dial stop on any listener drain, the idle-tunnel close on a filter-chain drain, and the responder's node exclusion after GOAWAY. None of them is runtime-guarded, and two ride on the existing enable_drain_with_goaway, so every deployment that already sets it on either side gets the new behavior on upgrade, with no way back short of a binary rollback.
Operators drain reverse tunnels in different ways. One example: a responder-driven /reverse_tunnel/drain_clusters GOAWAY followed by drain_listeners?inboundonly, where the initiator already redials on the peer GOAWAY. Could we put the new behavior behind a runtime guard (for example envoy.reloadable_features.reverse_tunnel_listener_drain) or a new opt-in field, rather than widening what enable_drain_with_goaway means? That keeps today's semantics for existing users and lets the change roll out deliberately.
Missing integration coverage for the configurations this changes by default:
- An in-place update with the
Immediatestrategy and default runtime. The existing test forcesuse_connection_event_drain=false. It should send a request during the update and require it to succeed. - A responder-initiated drain:
/reverse_tunnel/drain_clusters, then the initiator redials on the peer GOAWAY, then new requests land on the replacement and draining nodes aren't selected. - Node-ID keys with tenant isolation through the cluster, asserting the 503 and
upstream_rq_all_tunnels_draining. Only cluster-ID routing is covered end to end. enable_drain_with_goaway=falseon both sides: a whole-listener drain, and a node whose tunnels got GOAWAY still being selected as before.
| } | ||
| // In-place filter-chain updates must refill the idle pool before the maintenance tick. A | ||
| // whole-listener drain has already stopped the parent, so the callback cannot replenish it. | ||
| tunnel_iohandle_->markTunnelDrainingAndDialReplacement(); |
There was a problem hiding this comment.
On an in-place filter-chain update this closes the idle tunnel right away. markTunnelDrainingAndDialReplacement() only arms a 0 ms timer (reverse_connection_io_handle.cc:892), so the close always runs first. Under --drain-strategy immediate, every worker's idle tunnel closes in the same pass, and the initiator has no tunnel until the new handshakes complete. Before this PR, HCM closed idle tunnels staggered between 2/3 of the window and drain_time - drain_timeout, and the old tunnels kept serving in the meantime.
It also races the responder: a request already being written onto this socket is reset mid-flight, and a reset after send isn't safely retryable.
Could the in-place path request the replacement and leave the old idle tunnel to HCM's existing proactive close? An early close only seems needed once the parent has stopped initiating.
There was a problem hiding this comment.
In-place updates now request replacements without closing the old idle tunnels. HCM retains its existing close schedule.
| // strictly before it ensures this filter requests a replacement before HCM closes an idle | ||
| // tunnel without a codec. | ||
| const auto close_window = | ||
| std::chrono::duration_cast<std::chrono::milliseconds>(drain_time * (2.0 / 3.0)); |
There was a problem hiding this comment.
This duplicates HCM's private ProactiveDrainWindowStart (conn_manager_impl.cc:113), and the filter is only correct while it stays strictly earlier than HCM. If HCM's constant changes, this silently loses the race. If the in-place close above is left to HCM, this branch goes away. Otherwise, could the constant move into drain_close_util.h so both share it?
There was a problem hiding this comment.
Removed the duplicated constant by leaving in-place idle closure to HCM.
| if (strategy == Server::DrainStrategy::Immediate) { | ||
| // Keep coverage of connection events when HCM uses the legacy drain decision. | ||
| config_helper_.addRuntimeOverride("envoy.reloadable_features.use_connection_event_drain", | ||
| "false"); |
There was a problem hiding this comment.
This forces the legacy drain decision for the Immediate case, so no test covers in-place replacement under Immediate with default runtime. Could we add that case with a request sent during the update that must succeed? It would catch the gap where all idle tunnels close before their replacements are up.
There was a problem hiding this comment.
Added the Immediate case using HCM's default connection-event drain path, with the new guard enabled. A request succeeds while replacement handshakes are still pending.
| bool shouldDrainClose() { | ||
| if (!use_connection_event_drain_) { | ||
| return drain_decision_.drainClose(Network::DrainDirection::All); | ||
| return drain_decision_.drainClose(drain_direction_); |
There was a problem hiding this comment.
On the legacy drain path, an INBOUND reverse listener now drains on drain_listeners?inboundonly, where drainClose(All) used to return false (drain_manager_impl.cc, direction > current_drain.second). That's a separate behavior change from listener-drain handling and isn't in the changelog. Could it be split out, or at least guarded and noted?
There was a problem hiding this comment.
Guarded the legacy direction change and documented it. With the guard off, the legacy path still uses drainClose(All).
| std::string node_id = socket_manager->getNodeWithSocket(std::string(host_id)); | ||
| ENVOY_LOG(debug, "reverse_connection: resolved key '{}' to node: '{}'", host_id, node_id); | ||
| if (node_id.empty()) { | ||
| upstream_rq_all_tunnels_draining_->inc(); |
There was a problem hiding this comment.
Requests to an all-draining node now fail as no healthy upstream (UH) instead of going through the fallback socket and failing as UF. That's fine functionally, but it shifts the response flags that callers and dashboards classify on. Could the changelog call this out as a response-flag change?
There was a problem hiding this comment.
Updated the changelog to call out the UF to UH change and the new counter.
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
|
Addressed the rollout and coverage review in 97a6437. Added |
Stop reverse-tunnel initiators from creating or replacing tunnels when their listener starts draining, including pending handshakes. Close unused tunnels with drain-aware HCM enabled, while preserving replacement behavior for in-place filter-chain updates and normal connection rotation.
Preserve the current METADATA and GOAWAY schedule. Once GOAWAY arrives, responders exclude initiators whose remaining tunnels are all draining from new-stream selection while existing streams finish within the listener drain window.
Continues #47446, preserving its commit and review history.
Updated with AI assistance.