Repository navigation
Conversation
Signed-off-by: aakugan <aakashganapathy2@gmail.com>
|
/retest |
|
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to |
agrawroh
left a comment
There was a problem hiding this comment.
🔄 Follow-ups. Moving the drain start to METADATA looks right.
- ❓ Is there a config that wants the drain-aware codec without this marker?
- 💡 The old drain-start GOAWAY never reached the wire, since nghttp2 rejects a client GOAWAY whose last stream id is odd and oghttp2 drops client GOAWAYs, yet
goaway_sentcounted it. - 💡 The
enable_drain_with_goawaycomment still says the codec sends a GOAWAY when the drain begins, while it now sends one only after the drain time. - 💡 In phase one this test checks only the timer and
goaway_sent, so it still passes withsendDrainMetadata()emptied orallow_metadataoff. - 💡 Contrib tests run only in the release jobs, so no sanitizer job runs the initiator's redial on this METADATA.
- 💡 With
goaway_sentno longer moving at drain start, no stat shows a drain began until the final GOAWAY. - 💡 Could we rename
on_peer_goaway_andpeer_goaway_handled_toon_peer_drain_andpeer_drain_handled_, and fix thecallbacks_wrapper_comment, since METADATA now fires them too? - 🔄 This and #49361 both rewrite the drain-aware HCM's
enable_drain_with_goawaycomment, and #49361 also rewrites thecreateCodec()comment, so whichever lands second should merge main and keep the METADATA trigger and the rotation in both.
| // Requires ``enable_drain_with_goaway`` to be true (otherwise config is rejected) and the | ||
| // cluster's ``http2_protocol_options.allow_metadata`` to be true (otherwise the HTTP/2 codec | ||
| // drops the METADATA frame). | ||
| bool send_drain_metadata = 2; |
There was a problem hiding this comment.
❓ Is there a config that wants the drain-aware codec without this marker? Unset, /reverse_tunnel/drain_clusters sends nothing until the final GOAWAY, and an initiator that can't read the marker ignores it, so enable_drain_with_goaway could send it whenever the cluster sets allow_metadata.
There was a problem hiding this comment.
Met, the marker now rides on enable_drain_with_goaway with no separate field.
| <envoy_v3_api_field_extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions.send_drain_metadata>` | ||
| to the reverse-tunnel upstream HTTP/2 codec so upstream-initiated graceful drains (such as via | ||
| ``/reverse_tunnel/drain_clusters``) emit a connection-level (stream 0) METADATA frame carrying | ||
| ``envoy.reverse_tunnel.drain: true`` before the final GOAWAY, replacing the client-side graceful |
There was a problem hiding this comment.
💡 The old drain-start GOAWAY never reached the wire, since nghttp2 rejects a client GOAWAY whose last stream id is odd and oghttp2 drops client GOAWAYs, yet goaway_sent counted it. Could this note move to minor_behavior_changes and say so, and that goaway_sent now counts only the final GOAWAY, with the same and a Fixes commit #45874 line in the description?
There was a problem hiding this comment.
The fragment now sits under minor behavior changes and says why. Could the description also carry a Fixes commit #45874 line, since that commit added the drain-start GOAWAY?
| // ``http2_protocol_options.allow_metadata``. | ||
| bool enable_drain_with_goaway = 1; | ||
|
|
||
| // When true, starting a graceful drain emits a connection-level (stream 0) METADATA frame |
There was a problem hiding this comment.
💡 The enable_drain_with_goaway comment still says the codec sends a GOAWAY when the drain begins, while it now sends one only after the drain time. Could we reword it, and if this field stays, name http2.metadata_dropped as the sign that a cluster without allow_metadata dropped the marker?
There was a problem hiding this comment.
Met, the field doc now describes the METADATA start and names http2.metadata_dropped, and the file header above the message could mention the METADATA start too.
| /*registry=*/nullptr, /*cluster=*/"", h2_raw); | ||
|
|
||
| // Phase 1: graceful GOAWAY emitted now via the HTTP/2 subclass (counted as sent). | ||
| // Phase 1: stream-0 METADATA frame emitted now (no GOAWAY yet). |
There was a problem hiding this comment.
💡 In phase one this test checks only the timer and goaway_sent, so it still passes with sendDrainMetadata() emptied or allow_metadata off. Could it capture what the codec writes before the timer fires and assert one stream-0 METADATA frame, with a twin that sees none when the field or allow_metadata is off?
There was a problem hiding this comment.
Met, the write capture and the allow_metadata twin pin the frame.
| // (3) after drain_time -> upstream sends final GOAWAY (goaway_sent==1), the pool drains the | ||
| // old connection, the old tunnel closes (reverse_tunnel_closed==1), traffic flows on the | ||
| // replacement. | ||
| TEST_P(GrpcClientIntegrationTest, UpstreamAdminDrainMetadata) { |
There was a problem hiding this comment.
💡 Contrib tests run only in the release jobs, so no sanitizer job runs the initiator's redial on this METADATA. Could we add a case beside DrainingAwareHcmSendsMetadataThenGoAway where the fake responder writes only a stream-0 drain METADATA and the test sees the replacement connection while the old tunnel stays open?
There was a problem hiding this comment.
Met, InitiatorRedialsOnPeerDrainMetadata covers the redial outside contrib.
| } else { | ||
| inner_->shutdownNotice(); | ||
| } | ||
| sendDrainMetadata(); |
There was a problem hiding this comment.
💡 With goaway_sent no longer moving at drain start, no stat shows a drain began until the final GOAWAY. Could we count each drain start beside goaway_sent, whether or not the marker goes out, since a count of the marker alone would include frames a cluster without allow_metadata drops?
There was a problem hiding this comment.
Met, drain_started counts each drain start.
| private: | ||
| // Invokes the peer-drain callback at most once per connection. | ||
| void triggerPeerDrain() { | ||
| if (peer_goaway_handled_ || !on_peer_goaway_) { |
There was a problem hiding this comment.
💡 Could we rename on_peer_goaway_ and peer_goaway_handled_ to on_peer_drain_ and peer_drain_handled_, and fix the callbacks_wrapper_ comment, since METADATA now fires them too?
There was a problem hiding this comment.
The header reads right now, while the config that builds the wrapper still names the callback on_peer_goaway and its comments describe a GOAWAY only. Could we rename it and fix both comments there? #49361 also rewrites the one that opens createCodec(), so whichever lands second keeps both triggers in it.
Signed-off-by: aakugan <aakashganapathy2@gmail.com>
ac15e5e to
bd90645
Compare
agrawroh
left a comment
There was a problem hiding this comment.
🚫 Suggested Fixes. One new test flakes, and the other asks from the earlier review are in apart from the notes left in three threads.
⚠️ The admin handler returns once it posts the drain to the workers, so thisdrain_startedread can run before any worker bumps it, and it failed 4 of 1,500 local runs.- 💡 Phase 2 checks the final GOAWAY only through
http2.goaway_sent, which counts on submit, the kind of check that missed the old drain-start GOAWAY. - 💡 The replacement now overlaps the draining tunnel for the whole drain window, so under
enable_connection_limita worker already atmax_connections_per_nodefor the node answers it with a 429, and each refusal lengthens the initiator's backoff. - 💡 The
absl/status/status.hinclude and theabsl/statusBazel dependency came with theabsl::InvalidArgumentErrorcheck, which is gone, and the base built without them.
| ASSERT_TRUE(admin_response != nullptr); | ||
| ASSERT_TRUE(admin_response->complete()); | ||
| EXPECT_EQ("200", admin_response->headers().Status()->value().getStringView()); | ||
| EXPECT_EQ(1, test_server_->counter(drain_started)->value()); |
There was a problem hiding this comment.
drain_started read can run before any worker bumps it, and it failed 4 of 1,500 local runs. Could it wait for the counter?
| EXPECT_EQ(1, test_server_->counter(drain_started)->value()); | |
| test_server_->waitForCounter(drain_started, testing::Eq(1)); |
| drain_timer->invokeCallback(); | ||
| EXPECT_EQ(2, stats_.goaway_sent_.value()); | ||
| EXPECT_EQ(1, stats_.drain_started_.value()); | ||
| EXPECT_EQ(1, cluster_.http2CodecStats().goaway_sent_.value()); |
There was a problem hiding this comment.
💡 Phase 2 checks the final GOAWAY only through http2.goaway_sent, which counts on submit, the kind of check that missed the old drain-start GOAWAY. Could both real-codec tests also count one stream-0 GOAWAY in captured_wire?
| EXPECT_EQ(1, cluster_.http2CodecStats().goaway_sent_.value()); | |
| EXPECT_EQ(1, cluster_.http2CodecStats().goaway_sent_.value()); | |
| EXPECT_EQ(1, countHttp2Frames(captured_wire, Envoy::Http::Http2::Http2Frame::Type::GoAway, 0)); |
| // drains the pool connection. Emitting and observing drain METADATA frames requires the | ||
| // cluster's ``http2_protocol_options.allow_metadata`` to be true (otherwise the HTTP/2 codec | ||
| // drops the outgoing METADATA frame and increments ``http2.metadata_dropped``). Defaults to | ||
| // false, in which case the stock HTTP/2 client codec is used unchanged. |
There was a problem hiding this comment.
💡 The replacement now overlaps the draining tunnel for the whole drain window, so under enable_connection_limit a worker already at max_connections_per_node for the node answers it with a 429, and each refusal lengthens the initiator's backoff. Could the doc say the limit needs room for it?
| #include "source/common/protobuf/utility.h" | ||
| #include "source/extensions/upstreams/http/reverse_tunnel/drain_aware_client_connection.h" | ||
|
|
||
| #include "absl/status/status.h" |
There was a problem hiding this comment.
💡 The absl/status/status.h include and the absl/status Bazel dependency came with the absl::InvalidArgumentError check, which is gone, and the base built without them. Could we drop both?
| #include "absl/status/status.h" |
Commit Message
reverse_tunnel: changes to let the upstream drain gracefully
Additional Description
As described in: #47717 this PR makes the changes to let the upstream side drain gracefully.
We hook in the changes for the upstream drain and the downstream to open a new tunnel via the connection metadata hooks added.
Risk level
Low
Testing
Added unit, integ tests.