feat(server): read ClickHouse from a Malloy model via DuckDB

[jswir]

What

Adds clickhouseServers to the environment-level DuckDB connection, so a Malloy model can query a remote ClickHouse server:

{
  "name": "clickhouse",
  "type": "duckdb",
  "duckdbConnection": {
    "clickhouseServers": [
      {
        "name": "demo",
        "host": "localhost",
        "port": 8123,
        "database": "demo",
        "user": "malloy",
        "password": "${CLICKHOUSE_PASSWORD}"
      }
    ]
  }
}

Each entry registers a DuckDB table macro named after the server:

source: orders is clickhouse.sql("""
  SELECT * FROM demo('
    SELECT region, order_date, count() AS order_count, sum(amount) AS revenue
    FROM demo.orders GROUP BY region, order_date
  ')
""") extend {
  measure: total_revenue is revenue.sum()
  view: by_region is { group_by: region; aggregate: total_revenue }
}

Why this shape

ClickHouse's HTTP interface returns Parquet and DuckDB reads Parquet over HTTP, so a table macro is the entire integration — no driver, no Malloy dialect, no ClickHouse extension.

The chsql community extension packages the same idea as ch_scan, and it is worth saying why it is not used: its builds stop at DuckDB 1.3.2 while Malloy embeds 1.5.3 (INSTALL chsql FROM community 404s), and dumping its macro shows it concatenates the query without url_encode and takes a user but no password. So SELECT 1 + 1 returns HTTP 400 and any password-protected server returns 403. Both are fixed here.

Config declares only data-source intent, matching the existing attachedDatabases contract — Publisher owns the URL, the encoding, and the credentials.

Credential handling

Credentials become an HTTP Basic header stored in a DuckDB secret scoped to the server's base URL. They are deliberately kept out of two places:

• the query URL — ClickHouse records URLs in system.query_log
• the macro body — any model author can read that via duckdb_functions()

password goes through the existing ${ENV_VAR} substitution, so it need not sit in the config file. Tests assert the password appears in neither place, and that a wrong password fails rather than silently returning nothing.

The caveat, stated up front

Nothing is pushed down. The macro argument is exactly what ClickHouse runs; Malloy's filters and group_by are applied by DuckDB to the rows that came back. Aggregate inside the argument.

Measured on a 5M-row table, same results both ways:

	wall clock	over the wire
SELECT *, aggregate in DuckDB	1961 ms	1.63 MiB
aggregate in the ClickHouse query	156 ms	3.02 KiB

The docs, the example model, and the schema description all lead with this rather than leaving it to be discovered.

Behaviour change

An env-level DuckDB connection previously had to declare at least one attached database. It now requires at least one of attachedDatabases or clickhouseServers, and the error message names both. Two existing tests asserted the old wording and were updated (one in test:unit, one in tests/unit/duckdb, which is run by test:integration).

Validation

Bad config fails at startup with an actionable message rather than registering a macro nobody can call:

• name is not a SQL identifier → rejected (it is interpolated into DDL)
• name is a reserved word (primary) → rejected, since a model calling primary('…') would not parse. Checked against DuckDB's own duckdb_keywords() so it cannot drift from the parser across upgrades.
• duplicate names → rejected instead of the last one silently winning
• missing host → rejected

Testing

• clickhouse_servers.spec.ts (20 tests, always run) — exercises a real in-memory DuckDB rather than asserting on generated SQL strings, so DuckDB's parser and secret manager are the oracle. This caught a real bug during development: primary matched the identifier regex but failed to parse as DDL, which is what prompted the reserved-word check.
• clickhouse_servers.integration.spec.ts (5 tests, gated on CLICKHOUSE_TEST_HOST) — runs against a live ClickHouse: scalar round-trip, +/& survival, type mapping, server-side aggregation, and bad-password rejection.
• Full server unit suite: 3018 pass, 0 fail.
• Verified end to end through a running Publisher against ClickHouse 26.7.4 — REST query results match ClickHouse ground truth exactly.

Each unit test gets its own DuckDB database file. Malloy shares one DuckDB instance across :memory: connections whose config hashes match, so with :memory: the absence assertions passed alone and failed in the full suite as other spec files leaked macros and secrets in.

Docs & example

• docs/clickhouse.md — config, the aggregate-server-side rule with the measurements, the ClickHouse→DuckDB type mapping, and the DateTime timezone gotcha.
• examples/clickhouse/ — runnable package with a docker-compose.yml that seeds 50k rows, and two sources contrasting the pre-aggregated shape against the row-grain shape. Not in the default environment, since it needs a running ClickHouse.
• docs/connections.md updated — it stated that attached databases were the only configuration available and that one was required. Both are now false.

Notes for review

• The macro argument is trusted input. It is the model author's SQL, forwarded verbatim; Publisher does not parse or rewrite it. The intended posture is a read-only ClickHouse user granted only the tables you mean to expose, which the docs say.
• httpfs is already baked into the Publisher image (8/8 extensions), so this works under EXTENSION_FETCH_POLICY=local-only.
• Read path only; no writes.
• The generated API clients are gitignored, so api-doc.yaml is the only tracked source for the new schema.