Fluxora - Data Pipeline Manager

Fluxora is a production-ready Rails application for defining, validating, triggering, and monitoring ETL pipelines.

It combines:

• A polished Vue dashboard for pipeline operations.
• A Rails JSON API and ActiveJob execution engine.
• PostgreSQL persistence for pipelines, runs, and run events.
• Optional S3 and Redshift connector support.

Core Features

• Pipeline creation with guided connector-first forms.
• Built-in connection validation before save.
• Secure credential storage (encrypted at rest).
• Manual trigger and retry workflows.
• Run timeline, metrics, and status monitoring.

Technology Stack

• Backend: Rails 8.1, ActiveJob, PostgreSQL
• Frontend: Vue 3, Tailwind CSS, esbuild
• Queueing:
  • Development: :async
  • Production: Solid Queue-ready

Architecture Overview

• app/models/etl_pipeline.rb: pipeline definition and secure credential handling
• app/models/etl_run.rb: execution records and run metadata
• app/models/etl_run_event.rb: run timeline events
• app/jobs/pipeline_execution_job.rb: async ETL execution entrypoint
• app/services/etl/runner.rb: extract-transform-load orchestration
• app/services/etl/connectors/*: connector implementations
• app/services/etl/connection_validator.rb: pre-save connector validation
• app/controllers/api/v1/*: JSON API endpoints for dashboard and pipeline operations
• app/javascript/dashboard/*: Vue dashboard UI

Quick Start (Docker)

Prerequisite: Docker Desktop running.

1. Start services

docker compose up -d --build

2. Prepare database and seed demo data

docker compose run --rm web bin/rails db:migrate db:seed

3. Open the app

• App: http://localhost:3000
• Health: http://localhost:3000/up

Initial boot can take 15-30 seconds while app and asset watchers initialize.

Environment Configuration

Copy values from .env.example (for local dev, defaults are enough in most cases).

Important variables:

• DATABASE_URL
• AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
• REDSHIFT_HOST, REDSHIFT_DBNAME, REDSHIFT_USER, REDSHIFT_PASSWORD, REDSHIFT_PORT
• ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY
• ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY
• ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT

Connector Configuration Guide

Use the in-app Create Pipeline panel:

1. Pick source and destination from dropdowns.
2. Apply a suggested template.
3. Fill connector fields.
4. Click Validate Connection.
5. Create pipeline when source and destination validations pass.

Supported source types:

• sample
• s3
• postgres
• api
• custom

Supported destination types:

• postgres
• redshift
• s3
• custom

API Endpoints

• GET /api/v1/overview
• GET /api/v1/pipelines
• GET /api/v1/pipelines/:id
• POST /api/v1/pipelines
• PATCH /api/v1/pipelines/:id
• POST /api/v1/pipelines/:id/trigger
• POST /api/v1/pipelines/validate_connection
• GET /api/v1/runs
• GET /api/v1/runs/:id
• POST /api/v1/runs/:id/retry

Operational Commands

• Rebuild and restart:

docker compose up -d --build

• View logs:

docker compose logs web --tail=200
docker compose logs db --tail=200

• Stop stack:

docker compose down

Vercel Deployment

Fluxora can be deployed to Vercel as a serverless Rails app using @vercel/ruby.

Included project files:

• vercel.json
• api/index.rb

Steps

1. Import this repository in Vercel.
2. Set environment variables in Vercel project settings:

• DATABASE_URL (managed PostgreSQL connection string)
• RAILS_MASTER_KEY
• SECRET_KEY_BASE
• ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY
• ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY
• ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT
• optional connector variables (AWS_*, REDSHIFT_*)

3. Deploy.

Important Limits

• Vercel is serverless: no long-running worker process.
• In Vercel mode, jobs run inline (config.active_job.queue_adapter = :inline).
• Suitable for dashboard/API usage and light ETL tasks.
• For heavy/long ETL pipelines, deploy to a persistent platform (VM/container/Kubernetes).

Security Notes

• Pipeline credentials are encrypted at rest through Active Record Encryption.
• Do not commit .env or credential material.
• For production, set stable encryption keys via environment variables.

Project Hygiene

• .gitignore excludes generated artifacts, secrets, logs, and node modules.
• Docker Compose includes Postgres health checks and dependency gating.
• API routes expose only implemented operations.
• Unused scaffold artifacts from Rails defaults (Kamal hooks/config, placeholder folders) were removed.