feat: Phase 5 design doc and pre-testing hardening

Dockerfile

@@ -0,0 +1,29 @@
+# Dockerfile for Hermes Agent
+FROM python:3.12-slim
+
+WORKDIR /app
+
+# Install uv for fast, reliable dependency resolution
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
+
+# Copy project files
+COPY pyproject.toml uv.lock* ./
+COPY src/ ./src/
+
+# Install dependencies and the package
+RUN uv sync --frozen --no-dev
+
+# Set environment variables
+ENV PYTHONUNBUFFERED=1
+ENV ALERTS_JSONL_PATH=/var/hermes/alerts.jsonl
+ENV ALERT_OFFSET_PATH=/var/hermes/alerts.jsonl.offset
+ENV METRICS_PORT=8091
+
+# Create directory for persistent state
+RUN mkdir -p /var/hermes
+
+# Expose metrics port
+EXPOSE 8091
+
+# Run the processor loop as the main entry point
+CMD ["uv", "run", "python", "-m", "agentic_node_ops.processor"]

docs/prometheus-alerts.yml

@@ -0,0 +1,25 @@
+groups:
+  - name: hermes-self-monitoring
+    rules:
+      # Hermes Agent Silent
+      # Fires if the hermes_alive metric is absent (process crashed) or 0 (shut down) for 2 minutes.
+      - alert: HermesAgentSilent
+        expr: absent(hermes_alive) or hermes_alive == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Hermes agent not responding — alerts may be missed"
+          description: "The Hermes agent heartbeat metric is absent or zero. Check the hermes-agent container logs."
+
+      # Webhook Receiver Down
+      # Fires if the webhook_receiver_up metric is absent or 0 for 30 seconds.
+      # Note: webhook-receiver should expose this metric (e.g., via prometheus-client or textfile).
+      - alert: WebhookReceiverDown
+        expr: absent(webhook_receiver_up) or webhook_receiver_up == 0
+        for: 30s
+        labels:
+          severity: critical
+        annotations:
+          summary: "Alertmanager webhook receiver is down"
+          description: "The webhook receiver is not responding. Alerts will be queued in jsonl but not processed until it recovers."

docs/runbook-synthesis.md

@@ -0,0 +1,72 @@
+# Runbook Synthesis Design
+
+## Objective
+Automatically generate or refine YAML runbooks based on historical incident data, operator corrections, and successful runbook outcomes. This transforms reactive incident response into proactive, self-improving automation.
+
+## Core Principle
+**Synthesis is advisory, never autonomous.** Generated runbooks are drafted to a `runbooks/drafts/` directory and require explicit human review and promotion before they can be executed by the agent.
+
+## Minimum Data Requirements
+Synthesis is only triggered when sufficient statistical confidence exists. The minimum thresholds are:
+- **Incident Volume**: At least 10 resolved incidents of the same `alert_type` + `host` (or `alert_type` globally for host-agnostic issues).
+- **Resolution Consistency**: >= 70% of those incidents must have `outcome = "resolved"`.
+- **Action Consistency**: A specific `action_id` or sequence of actions must appear in >= 60% of the successful resolutions.
+
+If these thresholds are not met, the synthesis job logs a skip message and exits cleanly.
+
+## Synthesis Algorithm
+1. **Query Candidate Patterns**: 
+   ```sql
+   SELECT alert_type, host, COUNT(*) as incident_count
+   FROM incidents 
+   WHERE outcome = 'resolved' 
+   GROUP BY alert_type, host 
+   HAVING incident_count >= 10;
+   ```
+2. **Extract Successful Actions**: For each candidate pattern, query `runbook_outcomes` and `incidents.actions_taken` to find the most frequently successful actions.
+3. **Incorporate Operator Corrections**: Query `operator_corrections` for the same `alert_type` + `host`. If operators consistently note a specific diagnostic step (e.g., "always check peer count first"), elevate it to a Tier 1 diagnostic in the draft.
+4. **Generate Draft YAML**: Construct a `Runbook` object matching the schema in `runbook-spec.md`, populating `trigger`, `diagnostics`, and `actions` based on the extracted patterns.
+5. **Write to Drafts**: Save the generated YAML to `runbooks/drafts/{alert_type}_{host}_draft.yaml` with a header comment indicating the data source and confidence score.
+
+## Output Format
+Draft runbooks include metadata for human reviewers:
+```yaml
+# AUTO-GENERATED DRAFT
+# Generated: 2025-01-15T10:00:00Z
+# Confidence Score: 85% (based on 12 resolved incidents)
+# Top successful action: restart_consensus (8/12 times)
+# Operator corrections applied: 2
+
+id: consensus_desync_auto
+description: Auto-generated runbook for consensus desync based on historical success patterns.
+trigger:
+  alert_type: consensus_desync
+diagnostics:
+  - id: check_peers
+    cmd: "curl -s http://consensus:5052/eth/v1/node/peer_count"
+    description: "Check peer count (frequently cited in operator corrections)"
+actions:
+  - id: restart_consensus
+    description: "Restart consensus client (successful in 67% of resolved incidents)"
+    cmd: "docker restart consensus"
+    risk: medium
+    reversible: true
+    requires_approval: true
+```
+
+## Human Review Gate
+1. A daily cron job (or systemd timer) runs the synthesis script.
+2. If new drafts are generated, Hermes sends a Tier 1 Discord notification to the operator: *"New runbook draft generated: `consensus_desync_auto.yaml`. Review required."*
+3. The operator reviews the draft, edits if necessary, and moves it to the root `runbooks/` directory to activate it.
+4. Once promoted, the synthesis job tracks its performance via the standard `runbook_outcomes` table, creating a closed feedback loop.
+
+## Implementation Files
+- Create: `src/agentic_node_ops/synthesis.py` (orchestrator and YAML generator)
+- Create: `tests/test_synthesis.py`
+- Create: `runbooks/drafts/.gitkeep`
+
+---
+
+## Related Documents
+- [Runbook Spec](runbook-spec.md) — target schema for generated drafts
+- [Memory and Feedback](memory-and-feedback.md) — source tables (`incidents`, `operator_corrections`, `runbook_outcomes`)

pyproject.toml

@@ -10,6 +10,7 @@ requires-python = ">=3.12"
 dependencies = [
     "httpx>=0.27",
     "pyyaml>=6.0",
+    "prometheus-client>=0.20",
 ]
 
 [project.optional-dependencies]

runbooks/client_crash.yaml

@@ -0,0 +1,33 @@
+id: client_crash
+description: Ethereum client container has crashed or exited unexpectedly.
+trigger:
+  alert_type: client_crash
+diagnostics:
+  - id: check_container_status
+    cmd: "docker inspect --format='{{.State.Status}}' {{ container }}"
+    description: "Verify current container state (should be 'exited' or 'dead')"
+    timeout: 5s
+  - id: check_container_logs
+    cmd: "docker logs {{ container }} --tail 100"
+    description: "Inspect last 100 lines of container logs for OOM or panic"
+    timeout: 10s
+  - id: check_host_resources
+    cmd: "docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'"
+    description: "Check if host is under resource pressure (OOM killer)"
+    timeout: 5s
+actions:
+  - id: restart_client
+    description: "Restart the crashed client container"
+    cmd: "docker start {{ container }}"
+    risk: low
+    reversible: true
+    requires_approval: true
+    approval_timeout: 10m
+  - id: investigate_oom
+    description: "If OOM suspected, check dmesg and consider increasing memory limits"
+    cmd: "dmesg -T | grep -i oom"
+    risk: low
+    reversible: true
+    requires_approval: true
+    requires_explicit_unlock: true
+    approval_timeout: 30m

runbooks/validator_duty_misses.yaml

@@ -0,0 +1,32 @@
+id: validator_duty_misses
+description: Validator is missing attestation or proposal duties.
+trigger:
+  alert_type: validator_duty_misses
+diagnostics:
+  - id: check_validator_status
+    cmd: "curl -s http://validator:5042/eth/v1/validator/status"
+    description: "Check validator client status and connected consensus client"
+    timeout: 5s
+  - id: check_consensus_sync
+    cmd: "curl -s http://consensus:5052/eth/v1/node/syncing"
+    description: "Verify consensus client is fully synced"
+    timeout: 5s
+  - id: check_docker_logs
+    cmd: "docker logs validator --tail 50"
+    description: "Check recent validator client logs for errors"
+    timeout: 10s
+actions:
+  - id: restart_validator
+    description: "Restart the validator client to reconnect to consensus"
+    cmd: "docker restart validator"
+    risk: low
+    reversible: true
+    requires_approval: true
+    approval_timeout: 15m
+  - id: restart_consensus
+    description: "Restart consensus client if validator is healthy but consensus is stuck"
+    cmd: "docker restart consensus"
+    risk: medium
+    reversible: true
+    requires_approval: true
+    approval_timeout: 15m

src/agentic_node_ops/processor.py

@@ -13,6 +13,8 @@
 from pathlib import Path
 from typing import Optional
 
+from prometheus_client import Gauge, start_http_server
+
 from .context import build_hermes_context
 from .database import Database
 from .dispatcher import NotificationDispatcher
@@ -24,6 +26,19 @@
 ALERT_OFFSET_PATH = os.environ.get(
     "ALERT_OFFSET_PATH", "/var/hermes/alerts.jsonl.offset"
 )
+METRICS_PORT = int(os.environ.get("METRICS_PORT", "8091"))
+
+# Prometheus metrics
+HERMES_ALIVE = Gauge("hermes_alive", "Hermes agent heartbeat (1 = alive, 0 = silent)")
+
+
+def _start_metrics_server() -> None:
+    """Start Prometheus metrics HTTP server in a background thread."""
+    try:
+        start_http_server(METRICS_PORT)
+        log.info("Prometheus metrics server started on port %d", METRICS_PORT)
+    except Exception as e:
+        log.error("Failed to start metrics server: %s", e)
 
 
 def read_offset(path: str) -> int:
@@ -186,4 +201,22 @@ def run_processor_loop(
         poll_interval: Seconds to wait between polling cycles when queue is empty
     """
     log.info("Starting alert processor loop (poll interval: %ss)", poll_interval)
+
+    # Start Prometheus metrics server
+    _start_metrics_server()
+
+    # Set initial heartbeat
+    HERMES_ALIVE.set(1)
+
     asyncio.run(_run_loop_async(db, dispatcher, poll_interval))
+
+    # Clear heartbeat on shutdown
+    HERMES_ALIVE.set(0)
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+    run_processor_loop()