🛡️ RT-GIDS: Real-Time GPU-Accelerated Intrusion Detection System

A production-ready, real-time network intrusion detection system powered by XGBoost machine learning, featuring a modern Next.js dashboard and comprehensive threat analytics.

Features • Quick Start • Architecture • API • Contributing

📋 Table of Contents

Overview
Features
Architecture
Tech Stack
Prerequisites
Installation
Quick Start
Project Structure
Machine Learning Models
API Documentation
Frontend Dashboard
Configuration
Usage Examples
Development
Contributing
License

🎯 Overview

RT-GIDS (Real-Time GPU-Accelerated Intrusion Detection System) is an advanced network security monitoring platform that combines machine learning-based threat detection with real-time visualization and analytics. The system processes network traffic in real-time, identifies malicious patterns using XGBoost classifiers, and provides comprehensive threat intelligence through an intuitive web dashboard.

Key Highlights

Real-Time Detection: Sub-second threat identification with WebSocket streaming
Dual ML Models: Full-feature and lightweight realtime models for different use cases
Modern Dashboard: Next.js 14 with Three.js 3D visualizations and real-time charts
Production Ready: CPU-only mode with in-memory state management (no Docker/GPU required)
Comprehensive Analytics: Threat trends, attack frequency, top attackers, and incident overview
AI-Powered Narratives: GPT-4o-mini integration for intelligent threat summaries
SHAP Explainability: Model interpretability for security analysts

✨ Features

🔍 Threat Detection

Real-time packet analysis with XGBoost-based classification
Multi-attack type detection: Port scans, brute force, SQL injection, DDoS, malware C2 traffic
Confidence scoring with severity classification (Low/Medium/High/Critical)
Automated threat blocking and IP tracking
Anomaly detection with statistical analysis

📊 Analytics & Visualization

Live threat radar with 3D globe visualization
Attack frequency charts with time-series trends
Top attackers geolocation mapping
Incident overview with critical event tracking
System diagnostics with health monitoring
Model insights with SHAP feature importance

🎨 Modern UI/UX

Dark/Light theme support
Responsive design with mobile compatibility
Real-time WebSocket updates
Audio notifications for critical threats
Smooth animations with Framer Motion
Accessibility features (reduced motion, low performance mode)

🔧 Developer Experience

Local simulation mode (no external dependencies)
Hot reload for both frontend and backend
TypeScript for type safety
Comprehensive API with OpenAPI documentation
Attack seeder for testing and demos

🏗️ Architecture

flowchart LR
    subgraph Client
        FE["Frontend<br/>Next.js dashboard"]
    end
    subgraph Server
        BE["Backend<br/>FastAPI"]
        TS["Telemetry service"]
    end
    ML["ML models<br/>XGBoost"]
    R[("FakeRedis (in-memory)<br/>or real Redis")]

    FE <-->|"HTTP + WebSocket"| BE
    BE -->|"load and infer"| ML
    BE --> TS
    TS --> R
    ML -->|"threat metrics"| TS

Component Flow

Data Ingestion: Network events posted to /ingest endpoint
ML Inference: XGBoost models classify threats in real-time
Telemetry Processing: Events aggregated and normalized
WebSocket Streaming: Real-time updates pushed to frontend
Dashboard Rendering: React components visualize threats
Analytics: Historical trends and patterns analyzed

🛠️ Tech Stack

Backend

FastAPI 0.109+ - Modern async Python web framework
XGBoost 2.1.0 - Gradient boosting for threat classification
Pandas 2.1+ - Data manipulation and preprocessing
NumPy 1.26+ - Numerical computations
SHAP 0.41+ - Model explainability
OpenAI 1.12+ - AI-powered threat narratives
Uvicorn - ASGI server with WebSocket support
Prometheus Client - Metrics collection

Frontend

Next.js 14.2 - React framework with App Router
TypeScript 5.0 - Type-safe development
React 19 - UI library
Three.js 0.169 - 3D visualizations (Threat Globe)
Recharts 2.15 - Data visualization
Framer Motion 11 - Animations
Tailwind CSS 3.4 - Utility-first styling
Radix UI - Accessible component primitives
Axios 1.7 - HTTP client
Zustand/SWR - State management and data fetching

ML & Data Science

XGBoost - Primary ML framework
scikit-learn - Model evaluation and preprocessing
SHAP - Feature importance and explainability
Matplotlib/Seaborn - Visualization

Development Tools

ESLint - Code linting
Playwright - E2E testing
Lighthouse - Performance auditing
Stylelint - CSS linting

📦 Prerequisites

Python 3.10 or higher
Node.js 18+ and npm or pnpm
Git for cloning the repository
Optional: OPENAI_API_KEY for AI narrative summaries

System Requirements

OS: Windows 10+, Linux, or macOS
RAM: 4GB minimum (8GB recommended)
Storage: 2GB free space
Network: Local network access for testing

🚀 Installation

1. Clone the Repository

git clone https://github.com/Veladicodes/Real-Time-GPU-Based-Intrusive-Detection-System.git
cd Real-Time-GPU-Based-Intrusive-Detection-System

2. Backend Setup

# Navigate to backend directory
cd backend

# Create virtual environment
python -m venv .venv

# Activate virtual environment
# Windows:
.venv\Scripts\activate
# Linux/macOS:
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

3. Frontend Setup

# Navigate to frontend directory
cd frontend

# Install dependencies
npm install
# or
pnpm install

4. ML Models (Optional)

If you want to train your own models:

# Navigate to ML training directory
cd RealTime_IDS/scripts

# Preprocess CICIDS2017 dataset
python preprocess_gpu.py

# Train models
python train_gpu.py

Pre-trained models are included in RealTime_IDS/models/.

⚡ Quick Start

Option 1: Unified Launcher (Recommended)

# From project root
python run_rtgids.py

This runs docker compose up --build (backend + Redis), waits for /api/health, then starts the Next.js dev server and a background health monitor. Requires Docker; GPU device reservation in docker-compose.yml requires an NVIDIA GPU + nvidia-container-toolkit — on a machine without one, remove the deploy.resources block or build the cpu-runtime Dockerfile target instead.

Option 2: Manual Start

Start Backend

cd backend
python -m venv .venv
.venv\Scripts\activate  # Windows
# or
source .venv/bin/activate  # Linux/macOS

pip install -r requirements.txt
# On a machine without an NVIDIA GPU, install CPU-only torch instead of the
# large CUDA wheel pinned in requirements.txt:
#   pip install torch --index-url https://download.pytorch.org/whl/cpu

set RTGIDS_MODELS_DIR=..\RealTime_IDS\models  # Windows
# or
export RTGIDS_MODELS_DIR=../RealTime_IDS/models  # Linux/macOS

uvicorn app.main:app --reload --port 8000

A local Redis instance (default redis://localhost:6379/0) must be reachable; override with REDIS_URL. Auth defaults to RTGIDS_API_TOKEN=dev-token-abc and RTGIDS_JWT_SECRET=change-me-in-prod if unset — override both for anything beyond local development. See Security for JWT issuance and Redis ACLs.

Start Frontend

cd frontend
npm run dev
# or
pnpm dev

Seed Attack Data (Optional)

# From project root
python attack.py

Access the Dashboard

Frontend: http://localhost:3000
Backend API: http://localhost:8000
API Docs: http://localhost:8000/docs
Health Check: http://localhost:8000/api/health

📁 Project Structure

rt-gids/
├── backend/                    # FastAPI backend
│   ├── app/
│   │   ├── main.py            # Application entry point
│   │   ├── routers/           # API route handlers
│   │   │   ├── analytics.py   # Analytics endpoints
│   │   │   ├── logs.py         # Log streaming
│   │   │   ├── metrics.py      # System metrics
│   │   │   ├── threats.py      # Threat detection
│   │   │   └── ...
│   │   ├── services/          # Business logic
│   │   │   ├── ml_service.py   # ML inference
│   │   │   ├── telemetry_service.py  # Telemetry
│   │   │   ├── model_service.py      # Model management
│   │   │   └── ...
│   │   ├── utils/             # Utilities
│   │   │   ├── redis_url.py      # ACL-aware Redis connection URL builder
│   │   │   └── model_loader.py   # Model loading
│   │   └── schemas/           # Pydantic models
│   ├── redis/users.acl        # Redis ACL definitions (scoped rtgids-app user)
│   ├── requirements.txt       # Python dependencies
│   └── models/                # ML model storage
│
├── frontend/                   # Next.js frontend
│   ├── app/                   # Next.js App Router
│   │   ├── page.tsx           # Home page
│   │   ├── command-center/    # Main dashboard
│   │   ├── threat-analytics/  # Threat analytics page
│   │   └── ...
│   ├── components/             # React components
│   │   ├── command-center/    # Dashboard components
│   │   ├── threat/            # Threat visualization
│   │   ├── system/            # System diagnostics
│   │   └── ui/                # UI primitives
│   ├── hooks/                 # React hooks
│   │   ├── useThreatTelemetry.ts  # Telemetry hook
│   │   ├── useThreatAnalytics.ts  # Analytics hook
│   │   └── ...
│   ├── lib/                   # Utilities
│   │   └── api.ts             # API client
│   └── package.json           # Node dependencies
│
├── RealTime_IDS/              # ML training pipeline (standalone; not called by backend/)
│   ├── scripts/
│   │   ├── preprocess_gpu.py  # Data preprocessing
│   │   ├── train_gpu.py       # Model training
│   │   └── realtime_gpu_ids.py  # Live packet sniffing + auto-firewall-blocking demo
│   ├── models/                # Trained models
│   │   ├── xgb_gpu_ids.joblib      # Full model
│   │   └── xgb_realtime_ids.joblib # Realtime model
│   └── data/                  # Processed datasets
│
├── k8s/                        # Kubernetes manifests (Deployment/Service/HPA 2-10 pods)
├── .github/workflows/ci.yml    # GitHub Actions: backend pytest, frontend build, Docker build
├── attack.py                   # Synthetic attack generator (pushes to Redis directly)
├── run_rtgids.py                # Unified launcher (docker compose + frontend dev server)
└── README.md                  # This file

🤖 Machine Learning Models

Model Architecture

RT-GIDS uses XGBoost Gradient Boosting classifiers for threat detection:

1. Full IDS Model (`xgb_gpu_ids.joblib`)

Purpose: Comprehensive threat detection with all features
Features: 78 CICIDS2017 dataset features
Accuracy: 95.91% (held-out split) / 95.96% ± 0.20% (5-fold stratified CV)
AUC-ROC: 0.989 (held-out) / 0.991 ± 0.001 (5-fold CV)
Training Time: ~2.3 seconds
Use Case: Deep analysis, high accuracy requirements

2. Realtime Model (`xgb_realtime_ids.joblib`)

Purpose: Fast real-time packet classification
Features: 5 lightweight features
- packet_length
- protocol
- src_port
- dst_port
- ip_version
Accuracy: 90.50% (per RealTime_IDS/models/metrics.json)
Training Time: <0.1 second
Use Case: Live network monitoring, low latency

3. Heuristic Baseline (not a shipped model — evaluation reference only)

A depth-2 decision tree (2-3 threshold rules) on the same 5 lightweight features, evaluated purely as a sanity-check baseline the XGBoost models should beat.
Accuracy: 90.75%

All three numbers above come from an actual training run on the real CICIDS2017 dataset (see RealTime_IDS/models/metrics.json for hyperparameters and full feature list). On constrained hardware, each of the 8 CICIDS2017 CSVs is uniformly subsampled (RTGIDS_SAMPLE_FRAC, default 0.2 = 20%) before training to bound memory use — 527,732 rows were used for the numbers above. Set RTGIDS_SAMPLE_FRAC=1.0 to train on the full ~2.8M-row dataset given enough RAM. Hyperparameters were deliberately reduced in capacity (fewer trees, shallower depth, stronger L2) from the original defaults, which overfit to ~99.9%/~96.9%.

Training Pipeline

Step 1: Data Preprocessing

cd RealTime_IDS/scripts
python preprocess_gpu.py

Process:

Load CICIDS2017 CSV files
Clean and encode categorical features
Handle missing values and outliers
Create binary labels (Benign=0, Attack=1)
Output: features.csv and labels.csv

Step 2: Model Training

python train_gpu.py

Process:

Load preprocessed data
Train/test split (80/20, stratified)
Train full model with all features
Extract realtime features (5 features)
Train lightweight realtime model
Evaluate and save models
Generate metrics and visualizations

Output Files:

xgb_gpu_ids.json - Full model
xgb_realtime_ids.json - Realtime model
metrics.json - Training metrics
confusion_matrix_full.png - Confusion matrix
feature_importance_full.csv - Feature rankings

Model Hyperparameters

Full Model:

n_estimators=200
max_depth=8
learning_rate=0.1
subsample=0.8
colsample_bytree=0.8
tree_method="hist"  # CPU mode

Realtime Model:

n_estimators=150
max_depth=6
learning_rate=0.1
tree_method="hist"

Model Deployment

Models are automatically loaded by:

ModelService: Loads xgb_realtime_ids.json for real-time inference
MLService: Loads realtime model (preferred) or full model (fallback)

Models support:

Batch inference for performance
SHAP explainability for feature importance
Confidence scoring for threat severity

📡 API Documentation

Base URL

http://localhost:8000

Key Endpoints

Threat Detection

POST /ingest - Ingest network events
GET /api/threats/stats - Get threat statistics
GET /api/threats/recent?n=100 - Get recent threats
WebSocket /ws/logs?token=api::dev-token-abc - Real-time log stream

Analytics

GET /api/analytics/incidents-overview - Incident summary
GET /api/analytics/top-attackers?limit=5 - Top attacker IPs
GET /api/analytics/trends - Attack trends (15 minutes)
GET /api/analytics/attack-frequency?window=5m - Attack frequency

Metrics

GET /api/metrics/system - System metrics
GET /api/metrics/threat - Threat metrics
GET /api/metrics/anomalies - Anomaly detection

Model Insights

GET /api/model/info - Model information
POST /api/model/reload - Reload ML model
GET /api/shap/explain - SHAP feature importance

Health & Diagnostics

GET /api/health - Health check
GET /api/health/scan - System diagnostics
WebSocket /ws/telemetry - Real-time telemetry stream

Interactive API Docs

Visit http://localhost:8000/docs for Swagger UI with:

Endpoint documentation
Request/response schemas
Try-it-out functionality

🎨 Frontend Dashboard

Pages

1. Command Center (`/command-center`)

Incident Overview: Total attacks, notifications, critical events
Top Attackers: IP addresses with attack counts and geolocation
Attack Frequency Chart: Time-series visualization
AI Core Status: Model information and reload capability
System Summary: Real-time metrics bar

2. Threat Analytics (`/threat-analytics`)

Threat Radar: 3D radar visualization with attack points
Threat Globe: Interactive 3D globe with attacker locations
Threat Table: Detailed event log with filtering
Timeline: Attack timeline visualization

3. System Diagnostics (`/system-diagnostics`)

System Health: CPU, memory, disk usage
Network Stats: Packet counts, throughput
Model Status: ML model information
Service Status: Backend service health

4. Model Insights (`/model-insights`)

Feature Importance: SHAP-based feature rankings
Model Metrics: Accuracy, precision, recall
Prediction Examples: Sample predictions with explanations

Key Features

Real-time Updates: WebSocket streaming for live data
Responsive Design: Mobile and desktop support
Dark/Light Theme: System preference detection
Audio Notifications: Sound alerts for critical threats
Performance Controls: Reduced motion, low performance mode
Accessibility: WCAG-compliant components

⚙️ Configuration

Environment Variables

Backend

# Redis connection (plain, or ACL-scoped — see Security section below)
REDIS_URL=redis://localhost:6379/0
RTGIDS_REDIS_USER=rtgids-app        # optional, enables ACL-authenticated connection
RTGIDS_REDIS_PASSWORD=...           # required if RTGIDS_REDIS_USER is set

# Auth
RTGIDS_API_TOKEN=dev-token-abc      # shared secret exchanged for a JWT via /api/auth/token
RTGIDS_JWT_SECRET=change-me-in-prod
RTGIDS_JWT_ALG=HS256

# Model directory
RTGIDS_MODELS_DIR=/data/models      # e.g. ../RealTime_IDS/models for local dev

# OpenAI API (optional, for narrative summaries)
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o-mini

# Batch processing
RTGIDS_BATCH_MAX=32
RTGIDS_BATCH_TIMEOUT=0.05

Frontend

# Backend URL (optional, defaults to http://127.0.0.1:8000)
NEXT_PUBLIC_BACKEND_URL=http://127.0.0.1:8000

🔐 Security

JWT issuance (24h expiry)

Exchange the shared RTGIDS_API_TOKEN for a signed, short-lived JWT:

curl -X POST http://localhost:8000/api/auth/token \
  -H "Content-Type: application/json" \
  -d '{"api_token": "dev-token-abc"}'
# => {"access_token": "...", "token_type": "bearer", "expires_in": 86400}

Tokens are HS256-signed and expire exactly 24 hours (86400s) after issuance; RTGIDS_JWT_SECRET must be overridden from its change-me-in-prod default for anything beyond local development. Any JWT missing an exp claim is rejected outright, so a token can never grant indefinite access.

Redis ACLs

docker-compose.yml and k8s/redis-acl-configmap.yaml both start Redis with --aclfile users.acl, disabling the unrestricted default user and defining a scoped rtgids-app user limited to the exact commands/key patterns the backend uses (stream ops, get/set, pub/sub — no FLUSHALL, CONFIG, or ACL SETUSER). Set RTGIDS_REDIS_USER/RTGIDS_REDIS_PASSWORD to match the ACL file's credentials; override the placeholder password in both places before deploying.

Rate limiting

All routes are protected by a global 100/minute limit (slowapi), with two model-insights routes additionally capped at 1/10 seconds.

💡 Usage Examples

Ingest Network Event

TOKEN=$(curl -s -X POST http://localhost:8000/api/auth/token \
  -H "Content-Type: application/json" \
  -d '{"api_token": "dev-token-abc"}' | jq -r .access_token)

curl -X POST http://localhost:8000/api/logs/ingest \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "timestamp": "2024-01-15T10:30:00Z",
    "src_ip": "192.168.1.100",
    "dst_ip": "192.168.1.5",
    "dst_port": 80,
    "proto": "TCP",
    "bytes_in": 1024,
    "bytes_out": 2048,
    "is_attack": true,
    "label": "Port Scan"
  }'

Get Threat Statistics

curl http://localhost:8000/api/threats/stats

Get Top Attackers

curl http://localhost:8000/api/analytics/top-attackers?limit=10

WebSocket Connection (JavaScript)

const ws = new WebSocket('ws://127.0.0.1:8000/ws/logs?token=api::dev-token-abc');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Threat detected:', data);
};

Python Client Example

import requests

# Ingest event
response = requests.post(
    'http://localhost:8000/ingest',
    json={
        'src_ip': '10.0.0.1',
        'dst_ip': '192.168.1.1',
        'dst_port': 443,
        'proto': 'TCP',
        'is_attack': True
    }
)

# Get analytics
stats = requests.get('http://localhost:8000/api/analytics/incidents-overview').json()
print(f"Total attacks: {stats['total_attacks']}")

🔧 Development

Running in Development Mode

# Backend with hot reload
cd backend
uvicorn backend.app.main:app --reload --port 8000

# Frontend with hot reload
cd frontend
npm run dev

Running Tests

# Backend tests
cd backend
pytest

# Frontend tests
cd frontend
npm run test

# E2E tests
npm run verify:tier0

Code Quality

# Lint frontend
cd frontend
npm run lint

# Format Python (using black)
cd backend
black app/

# Type check
cd frontend
npm run type-check

Building for Production

# Build frontend
cd frontend
npm run build

# Start production server
npm run start

🤝 Contributing

Contributions are welcome! Please follow these steps:

Fork the repository
Create a feature branch: git checkout -b feature/amazing-feature
Make your changes and add tests
Commit with clear messages: git commit -m 'Add amazing feature'
Push to your branch: git push origin feature/amazing-feature
Open a Pull Request

Contribution Guidelines

Follow existing code style and conventions
Add tests for new features
Update documentation as needed
Ensure all tests pass
Write clear commit messages

Areas for Contribution

🐛 Bug fixes
✨ New features
📚 Documentation improvements
🎨 UI/UX enhancements
🚀 Performance optimizations
🧪 Test coverage
🌐 Internationalization

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

CICIDS2017 Dataset - For training data
XGBoost - Machine learning framework
FastAPI - Modern Python web framework
Next.js - React framework
Three.js - 3D graphics library
SHAP - Model explainability

📞 Support

Issues: GitHub Issues
Author: Adithya A on LinkedIn
Email: 10403adithya@gmail.com

Built by Adithya A

⭐ Star this repo if you find it helpful!

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
RealTime_IDS		RealTime_IDS
backend		backend
frontend		frontend
k8s		k8s
redis-data		redis-data
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
attack.py		attack.py
dc codes.txt		dc codes.txt
docker-compose.yml		docker-compose.yml
run_rtgids.py		run_rtgids.py
setup_rtgids.py		setup_rtgids.py

Latest commit

History

Folders and files

Repository files navigation

🛡️ RT-GIDS: Real-Time GPU-Accelerated Intrusion Detection System

📋 Table of Contents

🎯 Overview

Key Highlights

✨ Features

🔍 Threat Detection

📊 Analytics & Visualization

🎨 Modern UI/UX

🔧 Developer Experience

🏗️ Architecture

Component Flow

🛠️ Tech Stack

Backend

Frontend

ML & Data Science

Development Tools

📦 Prerequisites

System Requirements

🚀 Installation

1. Clone the Repository

2. Backend Setup

3. Frontend Setup

4. ML Models (Optional)

⚡ Quick Start

Option 1: Unified Launcher (Recommended)

Option 2: Manual Start

Start Backend

Start Frontend

Seed Attack Data (Optional)

Access the Dashboard

📁 Project Structure

🤖 Machine Learning Models

Model Architecture

1. Full IDS Model (xgb_gpu_ids.joblib)

2. Realtime Model (xgb_realtime_ids.joblib)

3. Heuristic Baseline (not a shipped model — evaluation reference only)

Training Pipeline

Step 1: Data Preprocessing

Step 2: Model Training

Model Hyperparameters

Model Deployment

📡 API Documentation

Base URL

Key Endpoints

Threat Detection

Analytics

Metrics

Model Insights

Health & Diagnostics

Interactive API Docs

🎨 Frontend Dashboard

Pages

1. Command Center (/command-center)

2. Threat Analytics (/threat-analytics)

3. System Diagnostics (/system-diagnostics)

4. Model Insights (/model-insights)

Key Features

⚙️ Configuration

Environment Variables

Backend

Frontend

🔐 Security

JWT issuance (24h expiry)

Redis ACLs

Rate limiting

💡 Usage Examples

Ingest Network Event

Get Threat Statistics

Get Top Attackers

WebSocket Connection (JavaScript)

Python Client Example

🔧 Development

Running in Development Mode

Running Tests

Code Quality

Building for Production

1. Full IDS Model (`xgb_gpu_ids.joblib`)

2. Realtime Model (`xgb_realtime_ids.joblib`)

1. Command Center (`/command-center`)

2. Threat Analytics (`/threat-analytics`)

3. System Diagnostics (`/system-diagnostics`)

4. Model Insights (`/model-insights`)