Feature/backend ingestion analytics optimizations

[mijinummi]

🛰️ Backend Hardening: Ingestion Recovery, TWAP Engine, WebSockets & Multi-Stage Docker

📝 Overview

This PR improves data reliability, realtime event broadcasting, price analytics, and production deployment efficiency across stellarflow-backend.

closes #714
closes #722
closes #727
closes #728

🛠️ Key Changes by Issue

1. Multi-Stage Docker Optimization (#728)

• Build Strategy: Implemented multi-stage Docker build utilizing python:3.11-slim.
• Footprint Reduction: Stripped non-essential compilation toolchains (gcc, g++, build-essential) from runtime layer, keeping total production image footprint under 200MB.
• Security: Enforced non-root appuser runtime execution context.

2. Multi-Channel WebSocket Broadcaster (#723)

• Architecture: Built ConnectionManager leveraging Redis Pub/Sub backend in app/websockets/manager.py.
• Topic Subscriptions: Added granular channel filtering (e.g., subscribe:pool:XLM-USDC).
• Connection Lifecycle: Implemented 30-second ping/pong heartbeat interval to automatically prune stale client connections.

3. Time-Weighted Average Price (TWAP) Engine (#722)

• Engine Implementation: Created TWAPEngine in app/analytics/twap.py computing 1-hour, 24-hour, and 7-day price signals.
• Outlier Protection: Added moving median outlier filtering that discards price spikes exceeding 50% variance.
• REST API: Exposed endpoint /api/v1/pools/{id}/twap.

4. Ingestion Recovery & Gap Detection (#714)

• Sequence Gap Detection: Added sequence tracking in SorobanListener to detect missing ledger blocks during WebSocket disconnections or high traffic.
• Automated Backfill: Dispatches background Celery backfill job dispatch_backfill_job to query missing ledger ranges via Soroban HTTP RPC.
• Idempotency: Enforced unique SHA-256 event_hash constraint on LedgerEvent records to prevent duplicate event insertion during backfills.

---

🧪 Testing & Verification

• Container Footprint: Verified final image size is under 200MB (docker images) and confirmed build compilers are absent in runtime layer.
• WebSocket Suite: Tested channel subscription/unsubscription logic and confirmed stale connection drop after missed 30s heartbeat.
• TWAP Calculations: Unit test verified removal of >50% price variance trade spikes and accurate time-weighted calculations (test_twap.py).
• Ingestion & Backfill: Unit test verified sequence jump detection triggers Celery backfill job dispatch with accurate ledger boundaries (test_soroban_listener.py).

pytest

[drips-wave]

@mijinummi Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits