Skip to content

feat: implement alert deduplication and storm protection (Tasks 6 + 7)#1

Merged
itenev merged 3 commits into
mainfrom
feat/alert-storm-protection
Jun 3, 2026
Merged

feat: implement alert deduplication and storm protection (Tasks 6 + 7)#1
itenev merged 3 commits into
mainfrom
feat/alert-storm-protection

Conversation

@itenev

@itenev itenev commented Jun 3, 2026

Copy link
Copy Markdown
Owner
What this PR does

Adds two critical alert management features to the webhook receiver:

Task 6 — Deduplication Logic
- Prevents duplicate alerts from flooding the pipeline when the same issue re-fires within a cooldown window
- Cooldown windows by severity: critical=15m, high=1h, medium=4h
- Read-only SQLite lookup against the incidents table
- Five rules:
  1. No prior incident → always process
  2. Resolved → firing transition → always process
  3. Higher severity breaks dedup (critical > high > medium > low)
  4. Cooldown elapsed → process
  5. Otherwise → deduplicate (skip)
- Fails open: if the SQLite DB is unavailable or unreadable, alerts always process (no alerts lost)
- Handles both timezone-naive and timezone-aware datetimes

Task 7 — Alert Storm Protection
- Bundles alerts during storm events to prevent operator fatigue
- Single-host storm: >3 alerts for same host within 30s → bundled as storm_single_host
- Cross-host storm: same alert type across >=2 hosts within 60s → bundled as storm_cross_host (always critical severity)
- Bundled alerts are written as a single JSONL entry with severity inheritance (critical if any bundled alert is critical)
- Automatic counter reset after bundle creation to allow subsequent storms
- Expired entry cleanup to prevent memory leaks

Files changed

| File                                                      | Description                                           |
|-----------------------------------------------------------|-------------------------------------------------------|
| webhook-receiver/src/webhook_receiver/dedup.py            | DedupLookup class + should_process() logic            |
| webhook-receiver/src/webhook_receiver/storm_protection.py | StormTracker + AlertBundle                            |
| webhook-receiver/src/webhook_receiver/server.py           | Integrated both features into WebhookHandler pipeline |
| webhook-receiver/tests/test_dedup.py                      | 13 dedup unit tests                                   |
| webhook-receiver/tests/test_storm_protection.py           | 15 storm protection + 2 AlertBundle unit tests        |
| webhook-receiver/tests/test_server.py                     | 5 dedup integration + 4 storm integration tests       |
| docs/hermes-implementation-plan.md                        | Tasks 5, 6, 7 marked complete                         |

Processing pipeline

The webhook handler now processes alerts in three phases:


validate → dedup → storm protection → write to JSONL


- Deduped alerts are skipped (tracked in alerts_deduped)
- Storm-bundled alerts replace individual entries (tracked in alerts_bundled)
- Health endpoint reports all three counters: alerts_received, alerts_deduped, alerts_bundled
- Response includes deduped_ids and bundled_ids lists

Test results

- 90 tests passing (30 parent project + 60 webhook-receiver)
- Covers: schema validation, dedup rules, storm thresholds, severity inheritance, expiry, integration with HTTP endpoints

╰───

loki-hermes-agent[bot] added 3 commits June 4, 2026 01:41
Dedup module (dedup.py):
- DedupLookup: read-only SQLite accessor for incident lookups
  - Opens DB with ?mode=ro URI for concurrent access
  - get_last_processed(alert_type, host) returns most recent incident
  - Fails open: missing DB, SQLite errors → always process
  - Handles both timezone-naive and timezone-aware datetimes

should_process() logic (5 rules):
  1. No prior incident → always process
  2. Resolved → firing transition → always process
  3. Higher severity breaks dedup (critical > high > medium > low)
  4. Cooldown elapsed → process (critical=15m, high=1h, medium=4h)
  5. Otherwise → deduplicate with log entry

Server integration:
- WebhookHandler now accepts DedupLookup in constructor
- POST /webhook filters alerts through should_process before JSONL write
- Deduped alert IDs tracked and returned in response
- GET /health tracks alerts_deduped count
- create_app() accepts db_path parameter for SQLite location

Tests:
- 13 dedup unit tests (DedupLookup + should_process edge cases)
- 5 integration tests (dedup suppresses, higher severity passes,
  resolved→firing passes, different host passes, health tracking)
- All 69 tests passing (30 parent + 39 webhook-receiver)

Implementation plan updated: Tasks 5 and 6 marked complete.
Storm protection module (storm_protection.py):
- StormTracker: in-memory tracker for detecting alert storms
  - Single-host: >3 alerts per host within 30s → bundle
  - Cross-host: same alert type across >=2 hosts within 60s → bundle
- AlertBundle: dataclass with to_alert() for downstream conversion
  - storm_single_host: inherits critical if any bundled alert is critical
  - storm_cross_host: always critical severity
  - Preserves alert count and host info in raw_labels

Server integration:
- WebhookHandler accepts StormTracker in constructor
- Processing pipeline: dedup → storm check → write to JSONL
- Bundled alerts replace individual alerts in output
- Health endpoint tracks alerts_bundled count
- Response includes bundled_ids list

Tests:
- 15 storm protection unit tests (thresholds, severity, expiry, reset)
- 4 integration tests (single-host bundling, cross-host bundling,
  health tracking, bundled_ids in response)
- 2 AlertBundle tests (single/cross-host, to_alert preserves count)
- All 90 tests passing (30 parent + 60 webhook-receiver)

Implementation plan updated: Task 7 marked complete.
1. SQLite connection leak in dedup.py (critical)
   - Wrapped sqlite3.connect() in contextlib.closing() to ensure
     connection is closed even if conn.execute() raises.
   - sqlite3 context manager handles transactions, not closing —
     contextlib.closing() is needed for the latter.

2. alerts_bundled in response was cumulative, not per-request
   - Added local bundled_count variable, incremented per bundle
   - Response now returns bundled_count (per-request) matching
     the pattern of alerts_deduped (len(deduped_ids))
   - Health endpoint still returns cumulative self.alerts_bundled

3. Added note in storm_protection.py about in-memory state
   - Documented that StormTracker resets on restart as acceptable
     trade-off (persisting state adds complexity for rare edge case)
@itenev
itenev merged commit 09b6f87 into main Jun 3, 2026
2 checks passed
@itenev
itenev deleted the feat/alert-storm-protection branch June 3, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant