Skip to content

Changes to let the upstream drain gracefully - #49458

Open
aakugan wants to merge 2 commits into
envoyproxy:mainfrom
aakugan:upstream-drain-gracefully
Open

aakugan wants to merge 2 commits into
envoyproxy:mainfrom
aakugan:upstream-drain-gracefully

Conversation

@aakugan

@aakugan aakugan commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Commit Message

reverse_tunnel: changes to let the upstream drain gracefully

Additional Description

As described in: #47717 this PR makes the changes to let the upstream side drain gracefully.
We hook in the changes for the upstream drain and the downstream to open a new tunnel via the connection metadata hooks added.

Risk level

Low

Testing

Added unit, integ tests.

Signed-off-by: aakugan <aakashganapathy2@gmail.com>
@repokitteh-read-only

Copy link
Copy Markdown

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #49458 was opened by aakugan.

see: more, trace.

@aakugan

aakugan commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @mattklein123
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #49458 was ready_for_review by aakugan.

see: more, trace.

@agrawroh agrawroh assigned agrawroh and unassigned mattklein123 Nov 21, 2026

@agrawroh agrawroh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔄 Follow-ups. Moving the drain start to METADATA looks right.

  • ❓ Is there a config that wants the drain-aware codec without this marker?
  • 💡 The old drain-start GOAWAY never reached the wire, since nghttp2 rejects a client GOAWAY whose last stream id is odd and oghttp2 drops client GOAWAYs, yet goaway_sent counted it.
  • 💡 The enable_drain_with_goaway comment still says the codec sends a GOAWAY when the drain begins, while it now sends one only after the drain time.
  • 💡 In phase one this test checks only the timer and goaway_sent, so it still passes with sendDrainMetadata() emptied or allow_metadata off.
  • 💡 Contrib tests run only in the release jobs, so no sanitizer job runs the initiator's redial on this METADATA.
  • 💡 With goaway_sent no longer moving at drain start, no stat shows a drain began until the final GOAWAY.
  • 💡 Could we rename on_peer_goaway_ and peer_goaway_handled_ to on_peer_drain_ and peer_drain_handled_, and fix the callbacks_wrapper_ comment, since METADATA now fires them too?
  • 🔄 This and #49361 both rewrite the drain-aware HCM's enable_drain_with_goaway comment, and #49361 also rewrites the createCodec() comment, so whichever lands second should merge main and keep the METADATA trigger and the rotation in both.

// Requires ``enable_drain_with_goaway`` to be true (otherwise config is rejected) and the
// cluster's ``http2_protocol_options.allow_metadata`` to be true (otherwise the HTTP/2 codec
// drops the METADATA frame).
bool send_drain_metadata = 2;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Is there a config that wants the drain-aware codec without this marker? Unset, /reverse_tunnel/drain_clusters sends nothing until the final GOAWAY, and an initiator that can't read the marker ignores it, so enable_drain_with_goaway could send it whenever the cluster sets allow_metadata.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Met, the marker now rides on enable_drain_with_goaway with no separate field.

<envoy_v3_api_field_extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions.send_drain_metadata>`
to the reverse-tunnel upstream HTTP/2 codec so upstream-initiated graceful drains (such as via
``/reverse_tunnel/drain_clusters``) emit a connection-level (stream 0) METADATA frame carrying
``envoy.reverse_tunnel.drain: true`` before the final GOAWAY, replacing the client-side graceful

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The old drain-start GOAWAY never reached the wire, since nghttp2 rejects a client GOAWAY whose last stream id is odd and oghttp2 drops client GOAWAYs, yet goaway_sent counted it. Could this note move to minor_behavior_changes and say so, and that goaway_sent now counts only the final GOAWAY, with the same and a Fixes commit #45874 line in the description?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fragment now sits under minor behavior changes and says why. Could the description also carry a Fixes commit #45874 line, since that commit added the drain-start GOAWAY?

// ``http2_protocol_options.allow_metadata``.
bool enable_drain_with_goaway = 1;

// When true, starting a graceful drain emits a connection-level (stream 0) METADATA frame

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The enable_drain_with_goaway comment still says the codec sends a GOAWAY when the drain begins, while it now sends one only after the drain time. Could we reword it, and if this field stays, name http2.metadata_dropped as the sign that a cluster without allow_metadata dropped the marker?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Met, the field doc now describes the METADATA start and names http2.metadata_dropped, and the file header above the message could mention the METADATA start too.

/*registry=*/nullptr, /*cluster=*/"", h2_raw);

// Phase 1: graceful GOAWAY emitted now via the HTTP/2 subclass (counted as sent).
// Phase 1: stream-0 METADATA frame emitted now (no GOAWAY yet).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 In phase one this test checks only the timer and goaway_sent, so it still passes with sendDrainMetadata() emptied or allow_metadata off. Could it capture what the codec writes before the timer fires and assert one stream-0 METADATA frame, with a twin that sees none when the field or allow_metadata is off?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Met, the write capture and the allow_metadata twin pin the frame.

// (3) after drain_time -> upstream sends final GOAWAY (goaway_sent==1), the pool drains the
// old connection, the old tunnel closes (reverse_tunnel_closed==1), traffic flows on the
// replacement.
TEST_P(GrpcClientIntegrationTest, UpstreamAdminDrainMetadata) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Contrib tests run only in the release jobs, so no sanitizer job runs the initiator's redial on this METADATA. Could we add a case beside DrainingAwareHcmSendsMetadataThenGoAway where the fake responder writes only a stream-0 drain METADATA and the test sees the replacement connection while the old tunnel stays open?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Met, InitiatorRedialsOnPeerDrainMetadata covers the redial outside contrib.

} else {
inner_->shutdownNotice();
}
sendDrainMetadata();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 With goaway_sent no longer moving at drain start, no stat shows a drain began until the final GOAWAY. Could we count each drain start beside goaway_sent, whether or not the marker goes out, since a count of the marker alone would include frames a cluster without allow_metadata drops?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Met, drain_started counts each drain start.

private:
// Invokes the peer-drain callback at most once per connection.
void triggerPeerDrain() {
if (peer_goaway_handled_ || !on_peer_goaway_) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Could we rename on_peer_goaway_ and peer_goaway_handled_ to on_peer_drain_ and peer_drain_handled_, and fix the callbacks_wrapper_ comment, since METADATA now fires them too?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header reads right now, while the config that builds the wrapper still names the callback on_peer_goaway and its comments describe a GOAWAY only. Could we rename it and fix both comments there? #49361 also rewrites the one that opens createCodec(), so whichever lands second keeps both triggers in it.

Signed-off-by: aakugan <aakashganapathy2@gmail.com>
@aakugan
aakugan force-pushed the upstream-drain-gracefully branch from ac15e5e to bd90645 Compare November 21, 2026 17:11

@agrawroh agrawroh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 Suggested Fixes. One new test flakes, and the other asks from the earlier review are in apart from the notes left in three threads.

  • ⚠️ The admin handler returns once it posts the drain to the workers, so this drain_started read can run before any worker bumps it, and it failed 4 of 1,500 local runs.
  • 💡 Phase 2 checks the final GOAWAY only through http2.goaway_sent, which counts on submit, the kind of check that missed the old drain-start GOAWAY.
  • 💡 The replacement now overlaps the draining tunnel for the whole drain window, so under enable_connection_limit a worker already at max_connections_per_node for the node answers it with a 429, and each refusal lengthens the initiator's backoff.
  • 💡 The absl/status/status.h include and the absl/status Bazel dependency came with the absl::InvalidArgumentError check, which is gone, and the base built without them.

ASSERT_TRUE(admin_response != nullptr);
ASSERT_TRUE(admin_response->complete());
EXPECT_EQ("200", admin_response->headers().Status()->value().getStringView());
EXPECT_EQ(1, test_server_->counter(drain_started)->value());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ The admin handler returns once it posts the drain to the workers, so this drain_started read can run before any worker bumps it, and it failed 4 of 1,500 local runs. Could it wait for the counter?

Suggested change
EXPECT_EQ(1, test_server_->counter(drain_started)->value());
test_server_->waitForCounter(drain_started, testing::Eq(1));

drain_timer->invokeCallback();
EXPECT_EQ(2, stats_.goaway_sent_.value());
EXPECT_EQ(1, stats_.drain_started_.value());
EXPECT_EQ(1, cluster_.http2CodecStats().goaway_sent_.value());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Phase 2 checks the final GOAWAY only through http2.goaway_sent, which counts on submit, the kind of check that missed the old drain-start GOAWAY. Could both real-codec tests also count one stream-0 GOAWAY in captured_wire?

Suggested change
EXPECT_EQ(1, cluster_.http2CodecStats().goaway_sent_.value());
EXPECT_EQ(1, cluster_.http2CodecStats().goaway_sent_.value());
EXPECT_EQ(1, countHttp2Frames(captured_wire, Envoy::Http::Http2::Http2Frame::Type::GoAway, 0));

// drains the pool connection. Emitting and observing drain METADATA frames requires the
// cluster's ``http2_protocol_options.allow_metadata`` to be true (otherwise the HTTP/2 codec
// drops the outgoing METADATA frame and increments ``http2.metadata_dropped``). Defaults to
// false, in which case the stock HTTP/2 client codec is used unchanged.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The replacement now overlaps the draining tunnel for the whole drain window, so under enable_connection_limit a worker already at max_connections_per_node for the node answers it with a 429, and each refusal lengthens the initiator's backoff. Could the doc say the limit needs room for it?

#include "source/common/protobuf/utility.h"
#include "source/extensions/upstreams/http/reverse_tunnel/drain_aware_client_connection.h"

#include "absl/status/status.h"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The absl/status/status.h include and the absl/status Bazel dependency came with the absl::InvalidArgumentError check, which is gone, and the base built without them. Could we drop both?

Suggested change
#include "absl/status/status.h"

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants