feat: implement alert deduplication and storm protection (Tasks 6 + 7)#1
Merged
Conversation
added 3 commits
June 4, 2026 01:41
Dedup module (dedup.py): - DedupLookup: read-only SQLite accessor for incident lookups - Opens DB with ?mode=ro URI for concurrent access - get_last_processed(alert_type, host) returns most recent incident - Fails open: missing DB, SQLite errors → always process - Handles both timezone-naive and timezone-aware datetimes should_process() logic (5 rules): 1. No prior incident → always process 2. Resolved → firing transition → always process 3. Higher severity breaks dedup (critical > high > medium > low) 4. Cooldown elapsed → process (critical=15m, high=1h, medium=4h) 5. Otherwise → deduplicate with log entry Server integration: - WebhookHandler now accepts DedupLookup in constructor - POST /webhook filters alerts through should_process before JSONL write - Deduped alert IDs tracked and returned in response - GET /health tracks alerts_deduped count - create_app() accepts db_path parameter for SQLite location Tests: - 13 dedup unit tests (DedupLookup + should_process edge cases) - 5 integration tests (dedup suppresses, higher severity passes, resolved→firing passes, different host passes, health tracking) - All 69 tests passing (30 parent + 39 webhook-receiver) Implementation plan updated: Tasks 5 and 6 marked complete.
Storm protection module (storm_protection.py): - StormTracker: in-memory tracker for detecting alert storms - Single-host: >3 alerts per host within 30s → bundle - Cross-host: same alert type across >=2 hosts within 60s → bundle - AlertBundle: dataclass with to_alert() for downstream conversion - storm_single_host: inherits critical if any bundled alert is critical - storm_cross_host: always critical severity - Preserves alert count and host info in raw_labels Server integration: - WebhookHandler accepts StormTracker in constructor - Processing pipeline: dedup → storm check → write to JSONL - Bundled alerts replace individual alerts in output - Health endpoint tracks alerts_bundled count - Response includes bundled_ids list Tests: - 15 storm protection unit tests (thresholds, severity, expiry, reset) - 4 integration tests (single-host bundling, cross-host bundling, health tracking, bundled_ids in response) - 2 AlertBundle tests (single/cross-host, to_alert preserves count) - All 90 tests passing (30 parent + 60 webhook-receiver) Implementation plan updated: Task 7 marked complete.
1. SQLite connection leak in dedup.py (critical)
- Wrapped sqlite3.connect() in contextlib.closing() to ensure
connection is closed even if conn.execute() raises.
- sqlite3 context manager handles transactions, not closing —
contextlib.closing() is needed for the latter.
2. alerts_bundled in response was cumulative, not per-request
- Added local bundled_count variable, incremented per bundle
- Response now returns bundled_count (per-request) matching
the pattern of alerts_deduped (len(deduped_ids))
- Health endpoint still returns cumulative self.alerts_bundled
3. Added note in storm_protection.py about in-memory state
- Documented that StormTracker resets on restart as acceptable
trade-off (persisting state adds complexity for rare edge case)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
╰───