Skip to content

Fix stacked latency toxics and document jitter - #793

Open
officialasishkumar wants to merge 1 commit into
Shopify:mainfrom
officialasishkumar:fix/stacked-latency-toxics
Open

officialasishkumar wants to merge 1 commit into
Shopify:mainfrom
officialasishkumar:fix/stacked-latency-toxics

Conversation

@officialasishkumar

Copy link
Copy Markdown

Multiple latency toxics on the same stream are meant to add up (#94). Each toxic advances the chunk's Timestamp, and the next one sleeps for its own delay measured from that timestamp. The problem is that it advanced the timestamp by the remaining sleep rather than by the toxic's delay. Those are only equal when the chunk didn't wait at all.

Under sustained traffic every chunk after the first queues behind the previous one, so its timestamp ended up too early and the next toxic deducted the queueing time from its own delay. With a 100ms and a 50ms downstream latency toxic, streaming a timestamped line every 5ms through a real toxiproxy-server:

min median max
main 100ms 101ms 151ms
this branch 150ms 151ms 157ms

Only the first chunk got the full 150ms on main.

Fix

  • Advance the timestamp by the full delay, so stacked toxics behave like a single toxic with the summed latency. Queueing time still counts towards a chunk's delay, as it does with one toxic.
  • Clamp the sampled delay at zero. With jitter > latency the delay could be negative. The sleep already treated that as "no delay", but the timestamp moved backwards and took time off any following toxic.

Docs

#556 asks how jitter is distributed, what happens when jitter > latency, and whether stacked latency toxics add up. The latency section of the README now answers all three:

  • Delays are drawn per chunk, uniformly from [latency - jitter, latency + jitter) in whole milliseconds.
  • Negative delays mean no delay.
  • Data is never reordered.
  • Stacked toxics add up.
  • toxicity applies per connection, not per chunk.

Tests

Two stub-level tests that assert the output timestamp exactly, so they don't depend on scheduling:

  • TestTwoLatencyToxicsWithQueuedChunk: a chunk read 60ms ago goes through two 100ms toxics. It gets 140ms on main and 200ms here.
  • TestLatencyToxicJitterBelowZero: {latency: 0, jitter: 1} (−1ms or 0ms) followed by a 100ms toxic. Without the clamp, chunks come out 1ms early.

Both fail without the change. The existing latency tests and golangci-lint pass.

Fixes #556

Multiple latency toxics on the same stream are meant to add up (Shopify#94):
each toxic advances the chunk's timestamp before passing it on, and the
next toxic sleeps for its own delay measured from that timestamp.

The timestamp was advanced by the remaining sleep rather than by the
toxic's delay. That only matches when the chunk did not wait at all.
A chunk that queued behind earlier ones, which is every chunk but the
first under sustained traffic, got a timestamp that was too early, so
the next toxic deducted the queueing time from its own delay. With a
100ms and a 50ms latency toxic, a steady stream saw ~100ms of latency
instead of 150ms.

Advance the timestamp by the full delay instead, so stacked toxics
behave like a single toxic with the summed latency.

Also clamp the sampled delay at zero. With `jitter` greater than
`latency` the delay could be negative; the sleep already treated that
as no delay, but the timestamp moved backwards and took time off any
following toxic.

Document the jitter distribution in the README: delays are drawn per
chunk, uniformly from [latency - jitter, latency + jitter), negative
values mean no delay, data is never reordered, stacked toxics add up,
and toxicity applies per connection.

Fixes Shopify#556
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation for distribution of latency with different jitter values

1 participant