Repository navigation
Fix stacked latency toxics and document jitter - #793
Open
officialasishkumar wants to merge 1 commit into
Open
officialasishkumar wants to merge 1 commit into
officialasishkumar wants to merge 1 commit into
Conversation
Multiple latency toxics on the same stream are meant to add up (Shopify#94): each toxic advances the chunk's timestamp before passing it on, and the next toxic sleeps for its own delay measured from that timestamp. The timestamp was advanced by the remaining sleep rather than by the toxic's delay. That only matches when the chunk did not wait at all. A chunk that queued behind earlier ones, which is every chunk but the first under sustained traffic, got a timestamp that was too early, so the next toxic deducted the queueing time from its own delay. With a 100ms and a 50ms latency toxic, a steady stream saw ~100ms of latency instead of 150ms. Advance the timestamp by the full delay instead, so stacked toxics behave like a single toxic with the summed latency. Also clamp the sampled delay at zero. With `jitter` greater than `latency` the delay could be negative; the sleep already treated that as no delay, but the timestamp moved backwards and took time off any following toxic. Document the jitter distribution in the README: delays are drawn per chunk, uniformly from [latency - jitter, latency + jitter), negative values mean no delay, data is never reordered, stacked toxics add up, and toxicity applies per connection. Fixes Shopify#556
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multiple latency toxics on the same stream are meant to add up (#94). Each toxic advances the chunk's
Timestamp, and the next one sleeps for its own delay measured from that timestamp. The problem is that it advanced the timestamp by the remaining sleep rather than by the toxic's delay. Those are only equal when the chunk didn't wait at all.Under sustained traffic every chunk after the first queues behind the previous one, so its timestamp ended up too early and the next toxic deducted the queueing time from its own delay. With a 100ms and a 50ms downstream latency toxic, streaming a timestamped line every 5ms through a real
toxiproxy-server:mainOnly the first chunk got the full 150ms on
main.Fix
jitter>latencythe delay could be negative. The sleep already treated that as "no delay", but the timestamp moved backwards and took time off any following toxic.Docs
#556 asks how jitter is distributed, what happens when
jitter>latency, and whether stacked latency toxics add up. The latency section of the README now answers all three:[latency - jitter, latency + jitter)in whole milliseconds.toxicityapplies per connection, not per chunk.Tests
Two stub-level tests that assert the output timestamp exactly, so they don't depend on scheduling:
TestTwoLatencyToxicsWithQueuedChunk: a chunk read 60ms ago goes through two 100ms toxics. It gets 140ms onmainand 200ms here.TestLatencyToxicJitterBelowZero:{latency: 0, jitter: 1}(−1ms or 0ms) followed by a 100ms toxic. Without the clamp, chunks come out 1ms early.Both fail without the change. The existing latency tests and golangci-lint pass.
Fixes #556