Add local Mosquitto/TimescaleDB/Bento ingestion sandbox with Bento unit tests

[leonardogouvea]

Status

READY

Summary

Following up on Pablo's suggestion to start hands-on with the pieces likely to end up in the data pipeline, this sets up a local Mosquitto + TimescaleDB + Bento sandbox that simulates the ESP32 sensor readings end-to-end: MQTT broker receives the reading, Bento validates/transforms/deduplicates it, TimescaleDB stores it. Includes Bento's own unit tests and a Makefile to run them.

Type of change

• ✨ New feature (non-breaking change which adds functionality)
• 🛠️ Bug fix (non-breaking change which fixes an issue)
• ❌ Breaking change (fix or feature that would cause existing functionality to change)
• 🧹 Code refactor
• ✅ Build configuration change
• 📝 Documentation
• 🗑️ Chore

Changes

• Added docker-compose.yml: mosquitto, timescaledb, and bento services, with pinned image versions, healthchecks, and restart: unless-stopped.
• Added mosquitto/config/mosquitto.conf: broker config with unbounded message queueing for offline durable subscribers (Mosquitto's 1000-message default silently drops new messages once hit).
• Added timescaledb/init/01_schema.sql: hypertable schema for sensor readings, one row per ESP32 frame.
• Added bento/fertloops.yaml: the MQTT → transform → TimescaleDB pipeline. Validates incoming frames before mapping them, deduplicates redelivered readings (file-backed cache, fails open on cache errors), and falls back to writing to disk if the database insert fails instead of retrying forever. Includes embedded Bento unit tests (tests: block) covering the validation/rejection logic.
• Added scripts/publish_fake_reading.sh: publishes a fake ESP32 reading frame to Mosquitto, for manual testing.
• Added Makefile: make test-bento and make lint targets to run Bento's own test/lint tooling via Docker.
• Added .gitignore: excludes Bento's runtime dead-letter file and dedupe cache (bento/failed/), keeping only a .gitkeep.

Testing

Steps

1. docker compose up -d
2. make test-bento
3. bash scripts/publish_fake_reading.sh (or with a custom device ID as the first argument)
4. Check the data landed:

docker exec fertloops-timescaledb psql -U fertloops -d fertloops -c "SELECT time, dev_id, ph, ce FROM readings ORDER BY time DESC LIMIT 5;"

Results

Unit tests (make test-bento):

Test '/bento.yaml' succeeded

Data landed in the database (after step 3):

          time          |    dev_id     | ph  |   ce
------------------------+---------------+-----+--------
 2026-07-30 12:54:24+00 | PR:EXAMPLE:01 | 6.8 | 1850.3
(1 row)

Regressions

• N/A — this is the first pipeline code added to the repo, no existing services or configs are touched.

Notes for Reviewers

• Whatever publishes real readings later must use MQTT QoS ≥ 1. Tested QoS 0 (the default for most MQTT publish tools) and confirmed the broker doesn't queue anything for an offline subscriber at that QoS, store-and-forward silently does nothing.
• The time column is ingestion time (UTC), not the ESP32's own Timestamp field, since that field's timezone isn't documented. The device's original Timestamp is preserved unchanged inside the raw JSONB column for later reprocessing.
• Deduplication of redelivered MQTT messages happens in Bento (file-backed cache keyed on device ID + device timestamp), not as a database constraint, TimescaleDB rejects a unique index on a hypertable unless the partitioning column is part of it, which isn't usable here since ingestion time differs per redelivery.
• Tested killing the bento container to simulate a crash: restart: unless-stopped is configured, but didn't reliably bring the container back on this Docker Desktop/WSL2 setup. Worth re-verifying on whatever host this actually deploys to before relying on it.
• Credentials in docker-compose.yml/bento/fertloops.yaml are local-dev-only placeholders, not configured, but didn't reliably bring the container back on this Docker Desktop/WSL2 setup. Worth re-verifying on whatever host this actually deploys to before relying on it.
• Credentials in docker-compose.yml/bento/fertloops.yaml are local-dev-only placeholders, not meant to be reused anywhere real.

Checklist

• Scope: The referenced issue is addressed, and there are no unrelated changes.
• Compatibility: The changes are backwards compatible with existing API, storage, configuration...
• Documentation: Relevant documentation is added or updated.
• Testing: The steps/process to test this feature are clearly explained, including testing for regressions.
• Commits: Commit messages follow our guidelines, there are no fixup commits left.