Skip to content

tls: add max_send_fragment to cap the TLS record size - #49503

Open
devnarula wants to merge 5 commits into
envoyproxy:mainfrom
devnarula:narula/network-filter-tls-chunk-send-limit
Open

devnarula wants to merge 5 commits into
envoyproxy:mainfrom
devnarula:narula/network-filter-tls-chunk-send-limit

Conversation

@devnarula

@devnarula devnarula commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Commit Message: dynamic_modules: let a network filter cap the downstream TLS record size

The goal of this PR is to let a dynamic module network filter choose a smaller downstream TLS record size than the 16384-byte maximum.

Additional Description:

SslSocket::doWrite passes at most 16384 plaintext bytes to each SSL_write, so Envoy always sends full-size TLS records, and nothing can ask for smaller ones. Some clients do not cope with 16 KiB records. For example, asynchronous libpq clients can hang on them, so a PostgreSQL proxy written as a dynamic module network filter needs to keep downstream records at 8192 bytes.

This adds:

  • Network::TransportSocket::setTlsDataChunkSendLimit() and Network::Connection::setTransportSocketDataChunkSendLimit(). SslSocket uses the limit in place of the hard-coded 16384, so the limit caps the plaintext of each record. The StartTLS socket passes the limit to both its clear-text and TLS sockets until it switches, so a limit set while the connection is still clear-text is not lost. Passthrough-based sockets forward it, and the happy eyeballs connection replays it on later attempts. The raw buffer, ALTS and dynamic module transport sockets, QUIC, and the API listener's synthetic connection ignore it, since none of them sends TLS records.
  • envoy_dynamic_module_callback_network_filter_set_transport_socket_data_chunk_send_limit, which accepts 1 through 16384 and returns false for anything else. The Rust SDK exposes this as EnvoyNetworkFilter::set_transport_socket_data_chunk_send_limit.

Risk Level: Low. The default stays 16384, and nothing changes unless a module calls the new callback. The two interface methods are new pure virtuals, so out-of-tree Connection and TransportSocket implementations need to add them.
Testing: Unit and integration tests.
Docs Changes: N/A (ABI header and Rust SDK docs updated)
Release Notes: Added changelogs/current/new_features/dynamic_modules__network-filter-tls-data-chunk-send-limit.rst.
Platform Specific Features: N/A

SslSocket passes at most 16384 plaintext bytes to each SSL_write, so Envoy
always sends full-size TLS records. Add
Network::TransportSocket::setTlsDataChunkSendLimit() and
Network::Connection::setTransportSocketDataChunkSendLimit() so the per-write
plaintext, and with it the record size, can be lowered. SslSocket uses the
limit in place of the hard-coded 16384. StartTLS passes it to both of its
sockets until it switches to TLS, passthrough sockets forward it, the happy
eyeballs connection replays it on later attempts, and sockets that do not send
TLS records ignore it.

Expose it to dynamic module network filters as
envoy_dynamic_module_callback_network_filter_set_transport_socket_data_chunk_send_limit,
which accepts 1 through 16384 and returns false otherwise, and in the Rust SDK
as EnvoyNetworkFilter::set_transport_socket_data_chunk_send_limit.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Dev Narula <devnarula2003@gmail.com>
@repokitteh-read-only

Copy link
Copy Markdown

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #49503 was opened by devnarula.

see: more, trace.

…imit test

With TLS 1.3 the client's encrypted Certificate, CertificateVerify and
Finished flight is sent as an application data record, and the data chunk
send limit does not apply to it. Start recording record lengths once the
handshake completes, so the test only measures data writes. Also apply the
formatter's line wrapping to two of the new tests.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Dev Narula <devnarula2003@gmail.com>
@devnarula
devnarula marked this pull request as ready for review November 20, 2026 19:35

@ggreenway ggreenway left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds niche-feature-specific methods to common interfaces, so it's pretty messy.

It also doesn't restrict the length of handshake records, such as if there's a long cert chain.

I think it would be better to just add a config knob on the TLS transport socket config, and not make this configurable from a filter at all. That makes most of the diff go away.

/wait

Comment thread source/common/tls/ssl_socket.cc Outdated
bytes_to_retry_ = 0;
} else {
bytes_to_write = std::min(write_buffer.length(), static_cast<uint64_t>(16384));
bytes_to_write = std::min(write_buffer.length(), data_chunk_send_limit_);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this, you should call SSL_CTX_set_max_send_fragment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to use SSL_CTX_set_max_send_fragment and added the field to the proto

…parameter

Per review, drop the Connection and TransportSocket setters, the dynamic module
network filter callback and the Rust SDK method, and add an optional
max_send_fragment to TlsParameters instead. When it is set, the TLS context
calls SSL_CTX_set_max_send_fragment, so BoringSSL caps the plaintext of every
record it sends, handshake messages included. The value must be 512 through
16384, the range BoringSSL accepts. It is rejected in certificate-level
tls_params, where it would not be applied, and has no effect on QUIC. The
OpenSSL compatibility layer now exposes SSL_CTX_set_max_send_fragment.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Dev Narula <devnarula2003@gmail.com>
@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @wbpcode
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #49503 was synchronize by devnarula.

see: more, trace.

@devnarula devnarula changed the title dynamic_modules: let a network filter cap the downstream TLS record size tls: add max_send_fragment to cap the TLS record size Nov 20, 2026
@devnarula

devnarula commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

This adds niche-feature-specific methods to common interfaces, so it's pretty messy.

It also doesn't restrict the length of handshake records, such as if there's a long cert chain.

I think it would be better to just add a config knob on the TLS transport socket config, and not make this configurable from a filter at all. That makes most of the diff go away.

/wait

Thanks for the feedback, instead of adding the byte limit directly, I have added the parameter max_send_fragment to TlsParameters and the tls record should default to 16kb (as the original functionality) if this field is missing.

@devnarula
devnarula requested a review from ggreenway November 20, 2026 21:37
@devnarula

Copy link
Copy Markdown
Contributor Author

/retest

@agrawroh agrawroh self-assigned this Nov 21, 2026
agrawroh
agrawroh previously approved these changes Nov 21, 2026

@agrawroh agrawroh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💚 Approved. It still needs an API shepherd's approval, one from another company and a re-review from the reviewer who asked for this redesign.

  • ❓ Should max_send_fragment live on CommonTlsContext, beside alpn_protocols and key_log?
  • 💡 The rendered docs never show the 512 floor, "not supported" reads as ignored while the code rejects it, and the doc does not say what a small value costs.
  • 💡 The record test runs only at 512, BoringSSL's floor, so a change that passed 512 for every configured value still passes every new test.
  • 💡 The description still describes the removed connection and dynamic module setters, names a release note file the diff does not add and says integration tests were added. Could we rewrite it for the TLS parameter, with the API considerations and AI-use note the pull request and contribution guides ask for, since it can become the squash commit?


// The maximum plaintext size, in bytes, of each TLS record sent, handshake records included.
// Defaults to 16384. Not supported in certificate-level ``tls_params`` and ignored by QUIC.
google.protobuf.UInt32Value max_send_fragment = 7

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Should max_send_fragment live on CommonTlsContext, beside alpn_protocols and key_log? It is the first TlsParameters field a server certificate cannot override, so here it needs per-certificate support through SSL_set_max_send_fragment or this reject and a carve-out in the certificate tls_params doc, and once it ships it can only be deprecated, never moved. Either placement works for me.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Should we put it in the ComonTlsContext directly rather than put it at there and reject certificates level API?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, yeah it makes sense to move the field to CommonTlsContext since it applies to the entire context. I have moved it and updated the pr.

Comment on lines +217 to +218
// The maximum plaintext size, in bytes, of each TLS record sent, handshake records included.
// Defaults to 16384. Not supported in certificate-level ``tls_params`` and ignored by QUIC.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The rendered docs never show the 512 floor, "not supported" reads as ignored while the code rejects it, and the doc does not say what a small value costs. Could we state all three?

Suggested change
// The maximum plaintext size, in bytes, of each TLS record sent, handshake records included.
// Defaults to 16384. Not supported in certificate-level ``tls_params`` and ignored by QUIC.
// The maximum plaintext size, in bytes, of each TLS record sent, handshake records included.
// Must be 512 to 16384 inclusive, and defaults to 16384. Smaller values send more records, which
// costs CPU, socket writes and bytes on the wire. QUIC ignores it, and setting it in a
// certificate's ``tls_params`` is rejected.

// max_send_fragment caps every TLS record both sides send, handshake records included, and the data
// still arrives intact.
BORINGSSL_TEST_P(SslReadBufferLimitTest, MaxSendFragment) {
constexpr uint32_t max_send_fragment = 512;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 The record test runs only at 512, BoringSSL's floor, so a change that passed 512 for every configured value still passes every new test. Could it also run at 8192 and check that its largest record is exactly 8192 plus the record overhead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, i added a test with 8192 byte cap

@wbpcode

wbpcode commented Nov 22, 2026

Copy link
Copy Markdown
Member

/wait

max_send_fragment applies to the whole TLS context: connections are created
from the first certificate's SSL_CTX and SSL_new snapshots the fragment size,
so a certificate cannot override it. Every TlsParameters field can be
overridden per certificate, so the field moves to CommonTlsContext beside
alpn_protocols and key_log, and the certificate-level rejection goes away.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Dev Narula <devnarula2003@gmail.com>
The record test only ran at 512, BoringSSL's floor, so a change that applied
512 for every configured value would still pass. Run the exchange at 8192 and
check that the largest record fills to 8192 bytes of plaintext without
exceeding it.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Dev Narula <devnarula2003@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants