Skip to content

Let hflow catalog ui explore bucket-backed catalogs #305

Description

@kstonekuan

Current behavior

hflow catalog ui rejects every bucket URL before startup at src/hflow/cli.py:1287-1300. CatalogUiSettings and the polling lifecycle also require a local Path at src/hflow/catalog_ui.py:25-39 and src/hflow/catalog_ui.py:93-152.

open_catalog_connection() already accepts bucket catalog roots and syncs their append-only Parquet tables into the local mirror at src/hflow/curation.py:221-233 and src/hflow/curation.py:260-281. The browser is the remaining local-only catalog surface.

Definition of done

  • hflow catalog ui --catalog s3://..., gs://..., and az://... start against existing HFlow catalogs when the bucket extra and provider credentials are available.
  • Startup validates the existing remote catalog marker. Browsing never creates or changes objects in the bucket.
  • Catalog Parquet files sync into the existing local mirror before the DuckDB connection opens.
  • The polling lifecycle syncs newly appended catalog files and makes the first and later completed appends visible in the open UI without a restart.
  • A valid empty bucket catalog starts the UI and waits for its first completed append.
  • The startup output names the bucket catalog clearly and doesn't present the mirror directory as the source catalog.
  • DuckDB continues listening on loopback. SQL reads the local mirror and gets no direct object-store write surface.
  • Missing bucket dependencies, credentials, and remote catalog markers produce concise CLI errors with no traceback.
  • Add behavioral tests covering a remote catalog at startup, an empty remote catalog receiving its first append, a later append becoming visible, and no bucket writes during browsing.
  • Update docs/CATALOG.md with bucket usage, mirror behavior, and the existing loopback/SSH guidance.

Use the catalog's current append and mirror semantics. This issue doesn't introduce a stronger snapshot-isolation contract.

Non-goals

  • A custom HFlow frontend
  • Listening on a public network interface
  • Editing bucket objects through DuckDB SQL
  • Catalog compaction or a new catalog format

Validation

uv run ruff check
uv run ruff format --check
uv run ty check
uv run pytest tests/test_catalog_ui.py tests/test_cli.py tests/test_storage.py -q
uv run pytest -q

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    advancedNeeds codebase familiarity; not a starter issueenhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions