feat(server): a package's index.malloy is its published surface

[jswir]

Revives the idea behind #988, which was closed as stale rather than rejected. This is not that branch rebased. It is a fresh implementation against current main, and it makes the one design decision #988 left open in the opposite direction.

Base note: this stacks on #1163, so the diff here is only its own 10 commits. It cannot merge before #1163 does. #1163 is what makes the three-answer table below true.

What this does

A package's published surface is its index.malloy. Put that file at the package root, import your models, export { … } what you publish. What it exports is what Publisher lists and what callers may query. No publisher.json key at all.

sales/
  publisher.json     { "name": "sales" }
  index.malloy       import "orders.malloy"
                     export { orders }
  orders.malloy      declares `orders` and `orders_staging`

orders_staging still compiles, and other models can import, join and extend it, but it is not listed and a direct query against it answers 404. create-malloy-package scaffolds the file, so a new package is curated from its first boot.

Where this departs from #988, and why

#988 made the convention curate listings only — it hid sources but never refused them, to protect a package that already had a root index.malloy from silently losing query access. That is the decision its closing note called "genuinely open".

That population is empty. Checked across everything available locally: 14 malloy-samples packages (none declare either key, none have an index.malloy), the three bundled examples (only governed-analytics declares explores, explicitly, so the convention never fires), and the one real index.malloy in a Malloyyo example repo — which is the only model file in that package, so the convention lists it and exposes everything it declares.

So the surface is the boundary, always, which is what the design note under Open questions in docs/malloyyo-dashboards-design.md proposed in the first place.

That reversal is mostly deletion. #988 added ~1,900 lines; exploresFromConvention, boundaryDeclared(), postLoadWarnings, resolvePatchedExploresOrigin, three warning builders and a 424-line integration spec all existed only because the convention produced a different kind of surface. Here it does not, and none of them appear.

The whole feature, stated once: the convention is a default for one manifest field. Nothing downstream records where a surface came from. That single sentence is why the diff to package.ts is 42 lines rather than 228 — exploresDeclared, exploreSet, both policy pushes, getInvalidExplores, listModels, setPackageMetadata and Environment.updatePackage all read packageMetadata.explores and cannot tell a filled-in value from a written one.

Deprecation, not removal

explores and queryableSources behave exactly as before. Both are marked deprecated in the OpenAPI spec and earn a load-time warning naming the replacement. Nothing is removed.

index.malloy does not replace queryableSources: "all", and its warning says so rather than advising a switch. "all" is the only way to curate listings without refusing queries, and a derived surface always enforces the boundary. That author is told to keep both keys.

Deleting the keys is deliberately out of scope, and not for diff size. publisher.json ignores unknown keys, so deleting the parsing would silently turn a curated, bounded package into an uncurated, unbounded one — governed-analytics's orders_base would go from refused to freely queryable with no error and no config change. That is a fail-open access change and deserves its own release, where the right behavior is to refuse to load a package still declaring the keys rather than ignore them.

examples/governed-analytics is converted

It was the only package in the repo declaring a curated surface, and it declared a two-file one — the shape the convention was least obviously able to express. It can, so publisher.json is back to a name and a description. That is the evidence explores can eventually be retired rather than merely deprecated.

Two things moved that no unit fixture covered, so a new spec drives the real package and pins both:

• Every query now enters through an import. Malloy marks imported entries exported: false and inlines their struct, which is the shape that once denied authorized callers by grafting a gate onto an entry nothing consults (authorize_import_hop.integration.spec.ts, whose header notes no other fixture put a gated source behind an import). This example now does, permanently, so the spec asserts the gate still tracks the caller across three different TENANTS values.
• The givens that drive the notebook's Parameters panel are declared in the imported files, not the entry model. Measured rather than assumed: all three reach index.malloy's surface. The spec pins the full list, because a narrowing there is not a patchable bug — it would mean reverting the conversion.

Three warnings had to change, and that is one bug not three

A package curated by its index.malloy has no explores key, so any message whose fix is "add it to explores" points at something that does not exist. Three sites said that:

• the dashboard withholding warning (adding an index.malloy to a package with dashboards/ withholds every dashboard, because a dashboard file is not something an index.malloy can export);
• the empty-surface warning (an import-only index.malloy exports nothing, so the package looks empty);
• and "explores": [], which is the documented opt-out and was being told to delete itself.

All three now name a remedy that exists, from one predicate derived from the tree rather than a stored origin. It only ever picks the wording of a remedy, never behavior.

Verified against a running server, not only in tests

Seven variants of one package served together, plus a gated package, plus a four-hop re-export chain:

	listed	queryable
no keys, no index	all 3 files	all 3
index.malloy exporting 2	index.malloy	2, via the surface
explores omitting the index	orders.malloy	1
explores: [] + index	all 4 files	all 3
index + queryableSources: "all"	index.malloy	all 3
only reports/index.malloy	all 4 files	all 3

• Three answers, all three rows: off the menu 404, locked 403, row-gated 200 with the caller's rows (empty is a 200, not an error).
• The oracle holds. A source both hidden and locked answers 404, byte-identical to a source that does not exist, whether or not the caller would have passed the lock. Only a listed locked source answers 403. Same at /compile.
• get_context tracks the surface, in enumeration and in ranked search, and every resource_id it returns round-trips through execute_query verbatim — on a convention package that model_path is the surface file, which is what keeps agents off a 404.
• Chains and joins: a source re-exported through four files stays queryable through the surface; a published source may join an unpublished one and the joined fields read normally. Curation hides a landing point, not a column.
• Versions: the same package published four times against a running server (uncurated → convention → widened export → opt-out), reloading each time. Listing, REST and get_context tracked every publish, no restart.

Gate: typecheck clean, 4044 server unit pass, 354 integration pass, skills/manifest/bundle/python green. The 35 config.spec failures reproduce at the base commit and are unrelated.

One thing reviewers should push back on if they disagree

export { … } governs what an importing file may see, which is Malloy's rule rather than Publisher's. So "hidden, not out of reach" means reachable through a file that exports it or that exports nothing — not through a file whose export omits it, which fails to compile with Reference to undefined object. That condition is now in the docs and the new skill. It is the one thing about this convention I would expect to surprise someone mid-migration.

🤖 Generated with Claude Code

[jswir]

Thanks both — all eleven items are in, and four of Kyle's were defects in fixes I had pushed the day before, which is the useful kind of catch.

Branch was CONFLICTING against the base, which matters more than it reads: GitHub stops dispatching CI on a conflicted PR, so the checks went quiet and looked green when nothing had run (DCO was the only check on the previous head). Merged the base in at 37ca95c and CI is running again.

@Sha-Bang — replies inline on each thread

Both blocking ones are fixed: a malformed explores now fails the load (refusing, the second of your two options, reasoning on the thread), and the rename advice is gone. The should-fix and the nit are fixed too. One half I deliberately did not build: a warning for the removal case. resolveExplores cannot tell "never had an index.malloy" from "had one until the last save" without state that persists the previous surface across loads, and that does not exist today. Details and a proposed home on that thread — happy to build it here if you want it in this PR.

@kylenesbit — all seven

The scaffolder exposure, both workflows:

1. xlsx recovery probe (model.custom.xlsx.malloy) — you were right that it was unrunnable both ways. The probe now goes in index.malloy and gets named in export { ... }, with the reason stated: raw SQL has to be part of a model, and the surface file is the only model a query can be addressed to.
2. Dashboards silently withheld — every scaffolded package's generated AGENTS.md now says the surface cannot publish a dashboard, before the author builds one, since the fix is a different curation shape rather than an extra line. malloy-dashboards gets it as step 0 of the build sequence, and its troubleshooting bullet covered only the "add index.malloy to a package that has dashboards" direction — the reverse is now the common case, because every scaffolded package has that file.

The four warnings:

3. "explores": [] with no index file now gets its own message instead of the deprecation, whose advice ("it curates and enforces exactly as the key does") is exactly the thing an empty array does not do.

4. Empty-surface remedy — fixed, same thread as Nathan's.

5. The withholding guard counted .malloynb. Filtered to MODEL_FILE_SUFFIX; it was contradicting the comment directly above it.

6. Broken-surface message — one message per surface now instead of one per file each claiming to be the whole surface, and the collateral count only counts models that actually compiled rather than every hidden .malloy.

7. docs/givens.md — both the sample and the introspection link now name index.malloy, with a line that givens declared in an imported file reach the surface unchanged.

One correction to my own earlier claim

I built and ran this against a live server, and it showed that the broken-index.malloy finding I fixed the day before was overstated. I said that case goes silent. It does not, on any path an author uses: first load fails the package into loadErrors, and the watcher, reload_package and ?reload=true all go through Environment.loadPackage, which keeps the last good model serving and reports stale: true with the compile error. My unit test only reached the "silent" state by calling reloadAllModels directly, which bypasses that handling.

The warning still has a home — the materialization and manifest rebind paths call reloadAllModels directly, install a placeholder, and never reach loadPackage, so they are the one way to get a package that is serving, is not stale, and exports nothing. Kept the code, scoped the claims to that path, in bd4cd06. Flagging it because I had described it as a broader bug than it is.

Verified against a running server, not just tests

Three scratch packages plus a second server for the boot case: convention package lists only index.malloy; query through the surface 200; off-surface model file 404 No queryable model; non-exported source through the surface 404 No queryable source; notebook beside index.malloy stays quiet; explores: [] lists everything with the new message; malformed explores at first boot leaves the package unserved with the message in loadErrors, so the hidden source in the fixture never becomes reachable; and the GET→PATCH round trip writes the description without persisting explores.

Typecheck and lint clean, 4166 server unit tests pass against the same 35 known config.spec failures, integration green. One perf test from the base (stays linear on the input that made the old one quadratic) fails under a loaded full run and passes in isolation.

Also resolved the AGENTS.md governance bullet in the merge, Kyle — you were right that it was the base's term and not this branch's. Kept the base's #(authorize) and this branch's index.malloy.

@Sha-Bang on the contract note: understood, and the manifest-reading blind spot is noted — that side is being fixed on your end, so nothing changes here for it.