RCAgent

Kubernetes Root Cause Analysis agent. Scans cluster pods, diagnoses health issues, and generates triage reports — with persistent memory that learns from past incidents.

What it does

RCAgent automates Kubernetes cluster triage through a three-phase pipeline:

1. Orchestrate — Scans all pods across namespaces, creates work units in CockroachDB
2. Worker — Checks each pod for health issues (CrashLoopBackOff, OOMKilled, Pending, excessive restarts)
3. Synthesize — An LLM agent investigates findings using tools (pod details, logs, events), recalls similar past incidents from memory, and produces a triage report

The agent uses CockroachDB as its persistent memory layer — every report, reasoning trace, and embedding is stored and retrieved across runs. When the agent sees a familiar pattern, it references past incidents to accelerate diagnosis.

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────────┐
│ Orchestrator │────▶│   Worker    │────▶│   Synthesizer   │
│ (K8s scan)   │     │ (pod check) │     │ (LLM + tools)   │
└─────────────┘     └─────────────┘     └────────┬────────┘
                                                  │
                                         ┌────────▼────────┐
                                         │   CockroachDB   │
                                         │  (memory layer) │
                                         └─────────────────┘

Tech Stack

• Language: Python 3.13
• LLM: OpenRouter (Gemma 4 26B) — chat/reasoning
• Embeddings: Jina AI (jina-embeddings-v3) — 768-dim vectors via pgvector
• Database: CockroachDB — work queue, findings, reports, agent memory, vector search
• Kubernetes: Official Python client — pod health checks via K8s API
• Slack: slack-bolt (Socket Mode) — slash commands for triggering sweeps

Setup

1. Clone and install

git clone https://github.com/vichekaoeun/RCAgent.git
cd RCAgent
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Configure environment

Create a .env file:

COCKROACH_DB_URL=postgresql://user:pass@host:port/dbname?sslmode=require
SLACK_BOT_TOKEN=xoxb-your-bot-token
SLACK_APP_TOKEN=xapp-your-app-token
OPENROUTER_API_KEY=sk-or-your-key
JINA_API_KEY=jina-your-key

3. Initialize database

python main.py init

4. Start a Kubernetes cluster

minikube start

Usage

CLI

python main.py sweep         # Full pipeline: init → orchestrate → worker → synthesize
python main.py orchestrate   # Scan pods, create work units
python main.py worker        # Process work units, check pod health
python main.py synthesize    # Run LLM agent on findings (with memory)

Slack Bot

python main.py bot

Command	Description
/triage	Run full sweep and post triage report
/findings	List current findings
/queue	Show remaining pending work units
/history	Show last 5 triage reports from memory
/memory	Show agent's last 20 reasoning actions

How Memory Works

CockroachDB is the persistent memory that makes the agent useful across runs:

• incident_embeddings — Vector embeddings of past reports. Before each synthesis, the agent queries for similar past incidents and injects them into the LLM context.
• triage_reports — Full report text with timestamps. Stores every report for audit and retrieval.
• agent_memory — Step-by-step reasoning traces. Records which tools the agent called and what it found, building institutional knowledge.
• findings — Raw pod health findings from the worker phase.
• work_units — Distributed work queue with FOR UPDATE SKIP LOCKED for concurrent workers.

Project Structure

RCAgent/
├── main.py              # CLI entry point
├── orchestrator.py      # K8s pod discovery, work unit creation
├── worker.py            # Pod health checks, finding writes
├── synthesizer.py       # LLM agent loop with tool calling and memory
├── tools.py             # LLM-callable tools (pod details, logs, events)
├── k8s_client.py        # Kubernetes API health checks
├── db.py                # CockroachDB connection and schema
├── work_queue.py        # Work queue with FOR UPDATE SKIP LOCKED
├── slack_bot.py         # Slack bot with slash commands
└── requirements.txt     # Python dependencies