Skip to content

hot restart: map a forwarded UDP packet's parent worker index onto the child's workers - #49523

Open
bpalermo wants to merge 1 commit into
envoyproxy:mainfrom
bpalermo:hot-restart-forwarded-udp-worker-index
Open

bpalermo wants to merge 1 commit into
envoyproxy:mainfrom
bpalermo:hot-restart-forwarded-udp-worker-index

Conversation

@bpalermo

Copy link
Copy Markdown
Contributor

Commit Message: hot restart: map a forwarded UDP packet's parent worker index onto the child's workers
Additional Description:

During a hot restart the parent forwards the UDP/QUIC packets it cannot dispatch to the child, tagged with the index of the parent worker they were meant for, and HotRestartingChild::onForwardedUdpPacket() hands that index straight to UdpListenerWorkerRouterImpl::deliver(). deliver() only ASSERTs that the index is in range before indexing its worker table, so when the child runs with a lower --concurrency than its parent (say 4 -> 2), a packet tagged with parent worker 2 or 3 reads past the end of the table in a release build and the child crashes.

deliver() now maps an index it does not have onto its own workers modulo the worker count (logged with ENVOY_LOG_EVERY_POW_2_MISC), and drops the packet if no worker is registered. In-process callers are unaffected: ActiveUdpListenerBase::onData() always passes an in-range index.

Modulo rather than dropping, because the deterministic QUIC connection ID generator selects the worker as snippet % concurrency: when the parent's worker count is a multiple of the child's, (x % 4) % 2 == x % 2, so a forwarded packet lands on the worker the child's own routing picks for that connection, and a handshake that began in the parent can complete in the child. In the remaining cases the packet reaches a live worker instead of crashing the process, which is no worse than the existing mismatch the comment in onForwardedUdpPacket() already describes.

Possible follow-up: raising concurrency across a hot restart does not crash (the parent's indices are all in range in the child), but x % 4 and x % 8 disagree, so some forwarded packets reach a different worker than the child's connection ID routing would pick for later packets of the same connection. Fixing that properly needs the receiving worker to re-dispatch forwarded packets by their computed destination, the redesign the comment in onForwardedUdpPacket() sketches. I'm happy to do that as a separate change if it's wanted.

Risk Level: Low (only changes behavior for an index that was previously read out of bounds)
Testing: New unit tests UdpListenerWorkerRouterImplTest.DeliverMapsAWorkerIndexItDoesNotHaveOntoItsOwnWorkers and HotRestartingChildTest.ForwardsPacketFromAParentWorkerTheChildDoesNotHave; both fail without the fix.
Docs Changes: UdpListenerWorkerRouter::deliver() documents the mapping.
Release Notes: Added under bug fixes.
Platform Specific Features: N/A
Runtime guard: None. The previous behavior is an out-of-bounds read, so there is nothing to revert to.

…e child's workers

A hot restart parent forwards the UDP/QUIC packets it cannot dispatch tagged
with its own worker index, and the child passed that index straight to
UdpListenerWorkerRouterImpl::deliver(), which only ASSERTed it was in range.
A child running with a lower --concurrency than its parent read past the end
of its worker table in release builds.

deliver() now maps an index it does not have onto its workers modulo the
worker count. For the deterministic QUIC connection ID generator this is the
worker the child selects itself when the parent's worker count is a multiple
of the child's.

Signed-off-by: Bruno Palermo <b@palermo.dev>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant