Skip to content

xds: skip unchanged listeners, clusters and route configs by serialized bytes - #49279

Merged
agrawroh merged 10 commits into
envoyproxy:mainfrom
agrawroh:agrawroh/xds-unchanged-config-bytes-hash
Nov 22, 2026
Merged

agrawroh merged 10 commits into
envoyproxy:mainfrom
agrawroh:agrawroh/xds-unchanged-config-bytes-hash

Conversation

@agrawroh

@agrawroh agrawroh commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Description

This PR makes LDS, CDS and RDS updates cheaper when they resend unchanged listeners, clusters and route configurations, which state-of-the-world xDS does on every push. Each resource is first compared by the hash of its deterministic serialization, and the reflection-based hash runs only when the bytes differ or the resource is new, so on a full-proto build every update is blocked, applied or reverted as before. A resource that serializes differently but is unchanged is still skipped, and the hash of its new bytes is kept for the next update.

The reflection-based hash stays because it finds some resources unchanged whose bytes differ, such as one whose Any payload the sender re-encoded, since the bytes keep each Any payload as the sender encoded it. A bytes-only check would rebuild such a resource, which is why #8231 moved MessageUtil::hash() off these bytes. A sender that re-encodes an unchanged resource on every push pays for the byte check on top of the reflection-based hash, about 1.6 µs for the listener below.

A new or changed listener, cluster or route configuration is now serialized once more, which costs well under 1% of building it, and the listener, cluster or route configuration it builds keeps that hash. Secrets are left alone, since hashing the bytes of a large inline certificate is slower than the reflection-based hash, and a route configuration made mostly of large inline response bodies is slower to check than before for the same reason. Checking an unchanged listener with ten filter chains goes from about 23 µs to about 1.6 µs, and an EDS cluster from about 3.9 µs to about 0.5 µs.


Commit Message: xds: skip unchanged listeners, clusters and route configs by serialized bytes
Risk Level: Low
Testing: Added unit tests that a re-encoded Any leaves a listener, a cluster and a route configuration unchanged, a warming listener and a warming cluster included, that each keeps the hash of the bytes it was built from or blocked, that the listener checks those bytes before MessageUtil::hash(), that a rejected cluster or route configuration is rejected again when it is resent, and that a changed one still applies.
Docs Changes: N/A
Release Notes: N/A

This pull request and its description were written by Isaac.

…ed bytes

Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
@wbpcode

wbpcode commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

One question: could we replace the previous hash with a new hash directly? if we cannot? why and which scenarios that new hash cannot cover?

if (!listener.blockLdsUpdate(*hash)) {
return false;
}
listener.setSerializedConfigHash(serialized_hash);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't get why we need this? Seems this may happens when the configuration's bytes are changed but the configuration self is not changed based on the MessageUtil::hash(config). A deterministic control plane never reaches here. And if the control plane cannot it send the deterministic bytes for same configuration, the hash only fluctuates.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you explain a little more about the necessity. and add a comment there if you do think it make sense.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's for a sender whose bytes change once and then repeat, such as after a restart or an upgrade, since without it every later push would pay for both hashes, more than main does, while a sender that re-encodes every push pays for both either way, and the comment there now says so.

@wbpcode wbpcode left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with only one comment. :) Thanks for the contribution.

agrawroh and others added 9 commits November 21, 2026 17:35
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
…on resend

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
…ed hash

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
@agrawroh

Copy link
Copy Markdown
Member Author

We can't, since Envoy hashed these same bytes until #8231 moved to an Any-aware hash for #5744, because an Any payload keeps the bytes its sender encoded, so an unchanged resource whose Any payload came back with its map entries in another order would rebuild its listener, cluster or route table.

@wbpcode wbpcode left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@agrawroh
agrawroh merged commit 9216235 into envoyproxy:main Nov 22, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants