Skip to content

reverse_tunnel: reject new streams during listener drain - #49361

Open
AntonKanug wants to merge 7 commits into
envoyproxy:mainfrom
AntonKanug:antonk/reverse-tunnel-strict-drain
Open

AntonKanug wants to merge 7 commits into
envoyproxy:mainfrom
AntonKanug:antonk/reverse-tunnel-strict-drain

Conversation

@AntonKanug

Copy link
Copy Markdown
Contributor

Stop reverse-tunnel initiators from creating or replacing tunnels when their listener starts draining, including pending handshakes. Close unused tunnels with drain-aware HCM enabled, while preserving replacement behavior for in-place filter-chain updates and normal connection rotation.

Preserve the current METADATA and GOAWAY schedule. Once GOAWAY arrives, responders exclude initiators whose remaining tunnels are all draining from new-stream selection while existing streams finish within the listener drain window.

Continues #47446, preserving its commit and review history.

Updated with AI assistance.

Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
@repokitteh-read-only

Copy link
Copy Markdown

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #49361 was opened by AntonKanug.

see: more, trace.

@AntonKanug
AntonKanug force-pushed the antonk/reverse-tunnel-strict-drain branch 2 times, most recently from d853940 to 7f733eb Compare November 17, 2026 18:35
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
@AntonKanug
AntonKanug force-pushed the antonk/reverse-tunnel-strict-drain branch from 7f733eb to 0963cb6 Compare November 17, 2026 18:39
@basundhara-c

Copy link
Copy Markdown
Contributor

/assign basundhara-c

@basundhara-c

Copy link
Copy Markdown
Contributor

/wait

@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @paul-r-gall
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #49361 was ready_for_review by AntonKanug.

see: more, trace.

@AntonKanug

Copy link
Copy Markdown
Contributor Author

/retest

Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
@AntonKanug
AntonKanug force-pushed the antonk/reverse-tunnel-strict-drain branch from a27b51c to 18a3a34 Compare November 17, 2026 20:41

@agrawroh agrawroh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 Suggested Fixes. The layering is right, and the two blockers are the conflict with main's node selection and idle tunnels on in-place updates.

  • ⚠️ This conflicts with #49367 on main, which made this round-robin prefer a node with an idle socket, and each side's new test fails under the other's rule.
  • ⚠️ With enable_drain_with_goaway, an in-place filter-chain update under an immediate drain closes every idle tunnel at once, and on a worker with no busy tunnel they wait up to 11.5 seconds for the next maintenance tick, while the HCM alone spread these closes over the window's last third.
  • ❓ After an admin drain with skip_exit the listeners keep accepting, but the initiator's stop latch never clears, so that listener never dials again until a full LDS update replaces it.
  • 💡 The idle-tunnel poll runs every 100 ms although nothing can close a tunnel before a drain event, and under a gradual drain each tick redraws the ramp, so most idle tunnels close in the first 20 seconds of a 600 second drain.
  • 💡 A first request on an idle tunnel can be dropped unread once its drain starts, since onData() runs the drain check again, which closes at once under an immediate drain and redraws the ramp under a gradual one.
  • 💡 The idle-close filter is added to every connection, so on a plain listener a drain closes clients that haven't sent their first request, and under an immediate drain every client accepted during it, which the HCM alone would serve.
  • 💡 Handshakes cut short by the stop keep their connecting state and gauge, and queued tunnels log a successful handshake with no close.
  • 💡 Nothing logs when the handle stops dialing, while each busy tunnel that drains afterwards still logs at info that it is dialing a replacement.
  • 💡 A request to a node whose tunnels all received GOAWAY now fails as no healthy upstream, counted only in upstream_cx_none_healthy and logged at debug, where it used to count as fallback_no_reverse_socket.
  • 💡 Responders exclude these initiators only when the reverse connection cluster sets the upstream codec's enable_drain_with_goaway, and an excluded node now fails as a 503 with no healthy upstream.
  • 💡 The listener-drain stop applies to every reverse listener whatever enable_drain_with_goaway says, while the field comment still describes only the peer GOAWAY redial and omits that the field now closes unused tunnels on a drain.
  • 💡 Only stream listeners still running at drain start call onDrain(), so UDP and QUIC listeners, an LDS removal and a hot restart never see it, and their handles get only resetFileEvents().
  • 💡 Could the description add the Commit Message, Risk Level, Testing, Docs Changes and Release Notes fields from the PR template, and the title name the initiator's dial stop, since both become the squash commit?
  • 💡 With no drain event delivered, the MODIFY_ONLY health-check test passes whatever the drain type.
  • 💡 No unit test runs the filter factory callback, and the integration tests run it only with the option on and the default guard, so the filter's order and legacy drain scope are unpinned.
  • 💡 A request for a known node now hashes the key twice in the new eligibility check, opted in or not.

Comment thread envoy/network/listener.h Resolved
Anton Kanugalawattage added 2 commits November 20, 2026 05:12
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
agrawroh
agrawroh previously approved these changes Nov 21, 2026

@agrawroh agrawroh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💚 Approved. Both blockers from the last review are fixed for the cases they named, and what's left is one question and some suggestions.

  • ❓ Under the immediate strategy, a full LDS update or a hot restart closes idle tunnels before the successor's are up, while the merge-base kept them for most of the window, so requests to an initiator with only idle tunnels fail meanwhile.
  • 💡 Under the default gradual drain, the HCM still closes about one idle tunnel in six in the window's last third, so on an in-place update those wait for the maintenance tick.
  • 💡 The filter's idle closes count in no drain stat, while the HCM's close of the same tunnel counts downstream_cx_drain_close, so with enable_drain_with_goaway that stat misses most idle closes.
  • 💡 The new stop log doesn't say which listener's initiator stopped, and it also logs at worker shutdown, where nothing drains.
  • 💡 No test pins that the stop keeps an accepted tunnel's Connected state and gauge, so clearing every state here would still pass.
  • 💡 The listener_stopped check passes with or without skip_exit, since a graceful drain stops listeners only when its 600 s drain time ends, which the test never reaches.
  • 💡 This release's connection-metadata drain note still says a draining tunnel redials a replacement at drain start, which a whole-listener drain no longer does.
  • 💡 Could the codec option's field comment say this too, that it now skips initiators whose tunnels are all draining and can fail a request as no healthy upstream?
  • 💡 Could the note name upstream_rq_all_tunnels_draining here, since these 503s now count there rather than in fallback_no_reverse_socket?
  • 💡 This comment no longer says a reverse tunnel's delayed soft GOAWAY replaces the one at drain start, and it states the rotation dial without enable_drain_with_goaway, which alone wires it.

This approval covers the core listener hooks too, while the proto changes still need an API shepherd.

return;
}
drain_scheduled_ = true;
std::chrono::milliseconds delay{0};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Under the immediate strategy, a full LDS update or a hot restart closes idle tunnels before the successor's are up, while the merge-base kept them for most of the window, so requests to an initiator with only idle tunnels fail meanwhile. Is that gap intended on those two paths?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored bounded idle grace for full LDS updates and hot-restart parent draining, with tests for both.

drain_time.count() > 0) {
// Sample once over the original drain window. Repeated probabilistic polling would close
// almost every idle tunnel near the start of a gradual drain.
const std::chrono::milliseconds close_after{server_context_.api().randomGenerator().random() %

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Under the default gradual drain, the HCM still closes about one idle tunnel in six in the window's last third, so on an in-place update those wait for the maintenance tick. Could the filter draw its close from the first two thirds, guarding a window that rounds to zero, and test a gradual in-place update under the default guard?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close times now fall before the two-thirds boundary, with zero-window handling and a regression test.

// In-place filter-chain updates must refill the idle pool before the maintenance tick. A
// whole-listener drain has already stopped the parent, so the callback cannot replenish it.
tunnel_iohandle_->markTunnelDrainingAndDialReplacement();
connection_->close(Network::ConnectionCloseType::NoFlush, "reverse_tunnel_listener_draining");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The filter's idle closes count in no drain stat, while the HCM's close of the same tunnel counts downstream_cx_drain_close, so with enable_drain_with_goaway that stat misses most idle closes. Could the filter bump it through the HCM config's stats?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idle closes now increment HCM’s downstream_cx_drain_close, with test coverage.

if (connections_stopped_) {
return;
}
ENVOY_LOG(info, "reverse_tunnel: listener draining; stopping pending and replacement dials");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The new stop log doesn't say which listener's initiator stopped, and it also logs at worker shutdown, where nothing drains. Could it name the source node and cluster, and say it stopped tunnel initiation rather than that the listener is draining?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the source node and cluster and changed the wording to “stopping tunnel initiation.”

for (auto& [host, host_info] : host_to_conn_info_map_) {
for (auto state = host_info.connection_states.begin();
state != host_info.connection_states.end();) {
if (state->second == ReverseConnectionState::Connecting) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 No test pins that the stop keeps an accepted tunnel's Connected state and gauge, so clearing every state here would still pass. Could the accepted-child drain test set a Connected state and check both survive onListenerDrain(), before it marks the child draining?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added assertions that Connected state and gauges survive listener drain until the child drains.

EXPECT_EQ(1, test_server_->counter("cluster.cluster_1.upstream_cx_total")->value());

// Neither closing an idle tunnel nor sending GOAWAY may replenish a draining listener's pool.
EXPECT_EQ(0, test_server_->counter("listener_manager.listener_stopped")->value());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The listener_stopped check passes with or without skip_exit, since a graceful drain stops listeners only when its 600 s drain time ends, which the test never reaches. Could it let the drain finish before this check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test now checks that the listener stays running without redials beyond the 600-second deadline.

@@ -0,0 +1,8 @@
reverse_tunnel: fixed listener draining so initiators stop pending handshakes and replacement dials when

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 This release's connection-metadata drain note still says a draining tunnel redials a replacement at drain start, which a whole-listener drain no longer does. Could this PR amend that note too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the note to distinguish rotation redials from whole-listener draining.

reverse_tunnel: fixed listener draining so initiators stop pending handshakes and replacement dials when
the listener begins draining, without stopping replacement tunnels during in-place filter-chain updates.
With ``enable_drain_with_goaway``, unused tunnels close while accepted tunnels follow the configured drain schedule.
When the responder's reverse connection cluster enables ``ReverseTunnelUpstreamCodecOptions.enable_drain_with_goaway``,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Could the codec option's field comment say this too, that it now skips initiators whose tunnels are all draining and can fail a request as no healthy upstream?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented exclusion of all-draining initiators and the resulting no-healthy-upstream / HTTP 503 response.

responders exclude initiators whose remaining tunnels have received GOAWAY when selecting a node for new streams,
while retaining those tunnels for existing streams. Node selection prefers idle sockets; when none are available,
it falls back to round-robin among nodes with a tunnel that has not received GOAWAY. A request with no eligible
initiator fails with HTTP 503 (no healthy upstream). Early drain METADATA still only notifies the reporter.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Could the note name upstream_rq_all_tunnels_draining here, since these 503s now count there rather than in fallback_no_reverse_socket?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added upstream_rq_all_tunnels_draining to the note.

Comment on lines +31 to +35
// ``enable_drain_with_goaway``, a reverse tunnel sends the soft GOAWAY and an optional
// METADATA frame at the configured delays before the final GOAWAY. Without
// ``shutdown_notice_delay``, such a tunnel receives no GOAWAY until the final one.
// Per-connection rotation dials a replacement before the old tunnel closes. Whole-listener draining
// instead stops pending handshakes and replacement dials regardless of ``enable_drain_with_goaway``.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 This comment no longer says a reverse tunnel's delayed soft GOAWAY replaces the one at drain start, and it states the rotation dial without enable_drain_with_goaway, which alone wires it. Could it say both?

Suggested change
// ``enable_drain_with_goaway``, a reverse tunnel sends the soft GOAWAY and an optional
// METADATA frame at the configured delays before the final GOAWAY. Without
// ``shutdown_notice_delay``, such a tunnel receives no GOAWAY until the final one.
// Per-connection rotation dials a replacement before the old tunnel closes. Whole-listener draining
// instead stops pending handshakes and replacement dials regardless of ``enable_drain_with_goaway``.
// ``enable_drain_with_goaway``, a reverse tunnel instead sends the soft GOAWAY and an optional
// METADATA frame at the configured delays before the final GOAWAY. Without
// ``shutdown_notice_delay``, such a tunnel receives no GOAWAY until the final one. On
// per-connection rotation, such a tunnel also dials a replacement before the old one closes.
// Whole-listener draining instead stops pending handshakes and replacement dials regardless of
// ``enable_drain_with_goaway``.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified that delayed soft GOAWAY replaces the immediate notice and rotation redials require enable_drain_with_goaway.

Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>

@basundhara-c basundhara-c left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes drain behavior on three paths: the initiator's dial stop on any listener drain, the idle-tunnel close on a filter-chain drain, and the responder's node exclusion after GOAWAY. None of them is runtime-guarded, and two ride on the existing enable_drain_with_goaway, so every deployment that already sets it on either side gets the new behavior on upgrade, with no way back short of a binary rollback.

Operators drain reverse tunnels in different ways. One example: a responder-driven /reverse_tunnel/drain_clusters GOAWAY followed by drain_listeners?inboundonly, where the initiator already redials on the peer GOAWAY. Could we put the new behavior behind a runtime guard (for example envoy.reloadable_features.reverse_tunnel_listener_drain) or a new opt-in field, rather than widening what enable_drain_with_goaway means? That keeps today's semantics for existing users and lets the change roll out deliberately.

Missing integration coverage for the configurations this changes by default:

  • An in-place update with the Immediate strategy and default runtime. The existing test forces use_connection_event_drain=false. It should send a request during the update and require it to succeed.
  • A responder-initiated drain: /reverse_tunnel/drain_clusters, then the initiator redials on the peer GOAWAY, then new requests land on the replacement and draining nodes aren't selected.
  • Node-ID keys with tenant isolation through the cluster, asserting the 503 and upstream_rq_all_tunnels_draining. Only cluster-ID routing is covered end to end.
  • enable_drain_with_goaway=false on both sides: a whole-listener drain, and a node whose tunnels got GOAWAY still being selected as before.

}
// In-place filter-chain updates must refill the idle pool before the maintenance tick. A
// whole-listener drain has already stopped the parent, so the callback cannot replenish it.
tunnel_iohandle_->markTunnelDrainingAndDialReplacement();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On an in-place filter-chain update this closes the idle tunnel right away. markTunnelDrainingAndDialReplacement() only arms a 0 ms timer (reverse_connection_io_handle.cc:892), so the close always runs first. Under --drain-strategy immediate, every worker's idle tunnel closes in the same pass, and the initiator has no tunnel until the new handshakes complete. Before this PR, HCM closed idle tunnels staggered between 2/3 of the window and drain_time - drain_timeout, and the old tunnels kept serving in the meantime.

It also races the responder: a request already being written onto this socket is reset mid-flight, and a reset after send isn't safely retryable.

Could the in-place path request the replacement and leave the old idle tunnel to HCM's existing proactive close? An early close only seems needed once the parent has stopped initiating.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In-place updates now request replacements without closing the old idle tunnels. HCM retains its existing close schedule.

// strictly before it ensures this filter requests a replacement before HCM closes an idle
// tunnel without a codec.
const auto close_window =
std::chrono::duration_cast<std::chrono::milliseconds>(drain_time * (2.0 / 3.0));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates HCM's private ProactiveDrainWindowStart (conn_manager_impl.cc:113), and the filter is only correct while it stays strictly earlier than HCM. If HCM's constant changes, this silently loses the race. If the in-place close above is left to HCM, this branch goes away. Otherwise, could the constant move into drain_close_util.h so both share it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the duplicated constant by leaving in-place idle closure to HCM.

if (strategy == Server::DrainStrategy::Immediate) {
// Keep coverage of connection events when HCM uses the legacy drain decision.
config_helper_.addRuntimeOverride("envoy.reloadable_features.use_connection_event_drain",
"false");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This forces the legacy drain decision for the Immediate case, so no test covers in-place replacement under Immediate with default runtime. Could we add that case with a request sent during the update that must succeed? It would catch the gap where all idle tunnels close before their replacements are up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the Immediate case using HCM's default connection-event drain path, with the new guard enabled. A request succeeds while replacement handshakes are still pending.

bool shouldDrainClose() {
if (!use_connection_event_drain_) {
return drain_decision_.drainClose(Network::DrainDirection::All);
return drain_decision_.drainClose(drain_direction_);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the legacy drain path, an INBOUND reverse listener now drains on drain_listeners?inboundonly, where drainClose(All) used to return false (drain_manager_impl.cc, direction > current_drain.second). That's a separate behavior change from listener-drain handling and isn't in the changelog. Could it be split out, or at least guarded and noted?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guarded the legacy direction change and documented it. With the guard off, the legacy path still uses drainClose(All).

std::string node_id = socket_manager->getNodeWithSocket(std::string(host_id));
ENVOY_LOG(debug, "reverse_connection: resolved key '{}' to node: '{}'", host_id, node_id);
if (node_id.empty()) {
upstream_rq_all_tunnels_draining_->inc();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requests to an all-draining node now fail as no healthy upstream (UH) instead of going through the fallback socket and failing as UF. That's fine functionally, but it shifts the response flags that callers and dashboards classify on. Could the changelog call this out as a response-flag change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the changelog to call out the UF to UH change and the new counter.

Signed-off-by: Anton Kanugalawattage <antonk@palantir.com>
@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/runtime-guard-changes: FYI only for changes made to (source/common/runtime/runtime_features.cc).

🐱

Caused by: #49361 was synchronize by AntonKanug.

see: more, trace.

@AntonKanug

Copy link
Copy Markdown
Contributor Author

Addressed the rollout and coverage review in 97a6437.

Added envoy.reloadable_features.reverse_tunnel_listener_drain, defaulting off so existing deployments retain their behavior. Added integration coverage for in-place requests, responder-triggered draining, tenant-scoped node routing, and disabled-feature behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants