Repository navigation
Conversation
…e child's workers A hot restart parent forwards the UDP/QUIC packets it cannot dispatch tagged with its own worker index, and the child passed that index straight to UdpListenerWorkerRouterImpl::deliver(), which only ASSERTed it was in range. A child running with a lower --concurrency than its parent read past the end of its worker table in release builds. deliver() now maps an index it does not have onto its workers modulo the worker count. For the deterministic QUIC connection ID generator this is the worker the child selects itself when the parent's worker count is a multiple of the child's. Signed-off-by: Bruno Palermo <b@palermo.dev>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit Message: hot restart: map a forwarded UDP packet's parent worker index onto the child's workers
Additional Description:
During a hot restart the parent forwards the UDP/QUIC packets it cannot dispatch to the child, tagged with the index of the parent worker they were meant for, and
HotRestartingChild::onForwardedUdpPacket()hands that index straight toUdpListenerWorkerRouterImpl::deliver().deliver()onlyASSERTs that the index is in range before indexing its worker table, so when the child runs with a lower--concurrencythan its parent (say 4 -> 2), a packet tagged with parent worker 2 or 3 reads past the end of the table in a release build and the child crashes.deliver()now maps an index it does not have onto its own workers modulo the worker count (logged withENVOY_LOG_EVERY_POW_2_MISC), and drops the packet if no worker is registered. In-process callers are unaffected:ActiveUdpListenerBase::onData()always passes an in-range index.Modulo rather than dropping, because the deterministic QUIC connection ID generator selects the worker as
snippet % concurrency: when the parent's worker count is a multiple of the child's,(x % 4) % 2 == x % 2, so a forwarded packet lands on the worker the child's own routing picks for that connection, and a handshake that began in the parent can complete in the child. In the remaining cases the packet reaches a live worker instead of crashing the process, which is no worse than the existing mismatch the comment inonForwardedUdpPacket()already describes.Possible follow-up: raising concurrency across a hot restart does not crash (the parent's indices are all in range in the child), but
x % 4andx % 8disagree, so some forwarded packets reach a different worker than the child's connection ID routing would pick for later packets of the same connection. Fixing that properly needs the receiving worker to re-dispatch forwarded packets by their computed destination, the redesign the comment inonForwardedUdpPacket()sketches. I'm happy to do that as a separate change if it's wanted.Risk Level: Low (only changes behavior for an index that was previously read out of bounds)
Testing: New unit tests
UdpListenerWorkerRouterImplTest.DeliverMapsAWorkerIndexItDoesNotHaveOntoItsOwnWorkersandHotRestartingChildTest.ForwardsPacketFromAParentWorkerTheChildDoesNotHave; both fail without the fix.Docs Changes:
UdpListenerWorkerRouter::deliver()documents the mapping.Release Notes: Added under bug fixes.
Platform Specific Features: N/A
Runtime guard: None. The previous behavior is an out-of-bounds read, so there is nothing to revert to.