feat: add redis_decay backend and soft-breach response header - #1215
Open
ysolomon-plat wants to merge 4 commits into
Open
feat: add redis_decay backend and soft-breach response header#1215ysolomon-plat wants to merge 4 commits into
ysolomon-plat wants to merge 4 commits into
Conversation
Adds BACKEND_TYPE=redis_decay: a continuously-decaying counter (GCRA family) executed as an atomic Redis EVAL. Unlike the fixed-window redis backend, cache keys carry no window timestamp, so budgets never reset at a boundary and the 2x boundary-burst is impossible by construction. Honors per-descriptor hits_addend. Fail-open on Redis errors. Cluster-safe: EVAL routes by cache key. Also adds SOFT_BREACH_HEADER_RATIO: when set (0..1), admitted requests whose remaining budget falls inside the soft band get an x-ratelimit-soft-breach: 1 response header, so clients can degrade before hard 429s. Computed generically from CurrentLimit/LimitRemaining, so it works with any cache backend. Covered by test/redis_decay (real EVAL execution on miniredis: boundary immunity, decay recovery, flood behavior, corrupt data, backward clock, fail-open, hits_addend) and test/service/soft_breach_test.go. All existing tests pass. Load-tested on Istio 1.30 / Gateway API: 3,950 rps sustained on one replica, +2.3ms p50 overhead, exact admission under concurrency. Refs envoyproxy#32 (rolling window request, open since 2018). Signed-off-by: ysolomon-plat <yossief.solomon@depop.com>
This was referenced Aug 18, 2026
The backend swallowed Redis errors and returned OK. That hid outages from the RedisError stat and, worse, overrode the caller's failure-mode policy: an operator configuring failure_mode_deny would still have got fail-open. Panic with redis.RedisError as the fixed-window backend does, so the service layer classifies it and the configured policy decides. Found by a fault-injection test on a live cluster. Signed-off-by: ysolomon-plat <yossief.solomon@depop.com>
Replaces the global SOFT_BREACH_HEADER_RATIO with soft_requests_per_unit configured per descriptor, and sends the signal to the upstream service rather than back to the client. Both changes came from auditing the behaviour this mirrors. A ratio applied to every descriptor is not the same thing as a threshold configured on the one descriptor that wants it, and the consumer of the signal is the backend deciding whether to degrade, not the caller. Adding a field to the config struct is not sufficient on its own: validateYamlKeys keeps a separate allowlist, and GetLimit takes a defensive copy via an explicit struct literal, so a new field is silently dropped in both places. Covered by a test that parses real YAML rather than building the struct directly, which is what would have caught it. Signed-off-by: ysolomon-plat <yossief.solomon@depop.com>
Adds RETRY_AFTER_HEADER_ENABLED. When a request is rejected, the response carries Retry-After set to the breaching descriptor's own period in seconds, so clients back off instead of retrying immediately into a saturated bucket. Opt-in, matching the existing custom-header flag, so no existing deployment sees a change in response shape. Placed after the global shadow-mode conversion: a shadowed breach is admitted, so telling the client to back off would be wrong. Per-descriptor shadow_mode is skipped for the same reason. Upstream's own suite caught that ordering when the header was briefly emitted on a shadowed 200. Signed-off-by: ysolomon-plat <yossief.solomon@depop.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
BACKEND_TYPE=redis_decay: a continuously-decaying counter (GCRA family) executed as an atomic Redis EVAL. Unlike the fixed-window redis backend, cache keys carry no window timestamp, so budgets never reset at a boundary and the 2x boundary-burst is impossible by construction. Honors per-descriptor hits_addend. Fail-open on Redis errors. Cluster-safe: EVAL routes by cache key.Also adds
SOFT_BREACH_HEADER_RATIO: when set (0..1), admitted requests whose remaining budget falls inside the soft band get anx-ratelimit-soft-breach: 1response header, so clients can degrade before hard 429s. Computed generically from CurrentLimit/LimitRemaining, so it works with any cache backend.Covered by test/redis_decay (real EVAL execution on miniredis: boundary immunity, decay recovery, flood behavior, corrupt data, backward clock, fail-open, hits_addend) and test/service/soft_breach_test.go. All existing tests pass.
Load-tested behind Envoy's ratelimit filter (Istio 1.30 / Gateway API): 3,950 rps sustained on one replica, +2.3ms p50 overhead, exact admission under concurrency (100-budget key admitted exactly 100 of 2,000 concurrent requests).
Refs #32 (rolling window request, open since 2018).