Centralized bean catalog (Step 1): schema, migration, export, audit

[mrjunos]

What

Step 1 of the data-curation plan: a centralized catalog as the source of truth for the data, replacing the flat JSONL manifest's single-label / no-provenance model. The training pipeline is unchanged — it still reads a manifest, now exported from the catalog.

Backend: SQLite now, Postgres-portable via SQLModel. Images stay on disk; the catalog holds paths + labels + metadata, so it stays small as the data grows.

Schema (src/almendra/db/models.py)

• source / lot / bean / bean_view / defect_class + morphology_class / bean_defect.
• Multi-label by construction: bean_defect is a bean↔defect junction, so a bean can carry several defects — with is_primary (SCA most-severe), label_source (dataset|human_verified|model_weak) and a trust score.
• Rich provenance on lot: farm, producer, region, country, altitude, variety, species, process, harvest/wash/hulling dates, humidity (mostly null for public lots, full for private).
• Public/private separation: lot.provenance_type (public_dataset|proprietary).

Tooling

• almendra db {init,migrate,export-manifest,audit}.
• migrate.py imports the existing manifest idempotently (1507 beans → beans/views/defects).
• export.py writes a training manifest, gated on is_good + provenance (public-only by default; private opt-in with --all-provenance) + --min-trust. Each record carries the multi-label defects list plus the back-compat primary defect_class/defect_index.
• audit.py reports composition (by provenance/source/split/class/label-source) + integrity (license-blocked sources holding beans, beans missing views).
• BeanRecord gains a defects list (legacy manifests load fine — defaults to []).

Notes / deferred

• data/catalog.db is git-ignored and regenerable via almendra db migrate.
• DVC deferred: image-blob versioning waits on a chosen remote (dvc add restructures storage); the catalog doesn't depend on it. See docs/adr/0006-centralized-bean-catalog.md.
• Not in this PR (later steps): curation/dedup, new dataset adapters (coffee-adik ⭐, arabica-washed), and the multi-label model migration (BCE + multi-label metrics).

Verification

• uv run pytest -m "not e2e" → 73 passed, 1 skipped (e2e); ruff check . + format clean.
• almendra db migrate → 1507 beans, re-run skips all (idempotent); almendra db audit prints sane composition; db export-manifest round-trips the manifest.
• New tests/test_db.py: seed counts, idempotent migration, export round-trip, good/provenance gating, BeanRecord back-compat.

🤖 Generated with Claude Code