feat(server): let a gateway list, show and run a package's files off its surface for the package's authors

[jswir]

A package with a root index.malloy hides every file that index.malloy does not export: listModels leaves those files out, GET …/models/{path} answers 404 for them, and a query to them answers 404. So the package's own authors can't read or try their own code through the API. In Credible, a modeler opening a curated package sees only index.malloy.

This adds includeOffSurface=true to those three routes. For one request, it serves the package as if it had no surface, the way queryableSources: "all" does for every request. #(authorize) and #(access_filter) still apply: it lifts curation, never the lock.

What changes

Route	With includeOffSurface=true
GET …/models	Lists every model file. Each entry carries a new onSurface boolean. Dashboard files stay on the dashboards route.
GET …/models/{path}	Returns a file off the surface instead of 404. It includes sourceText even when the text names a source the file does not publish. The compiled body is curated per file, as today.
POST …/models/{path}/query	Runs a file off the surface, and a source index.malloy doesn't export.

• Without the option, nothing changes. Every listing entry now carries onSurface: true for everything in a package with no surface.

• A bad value is refused with 400 on all three routes (anything other than true or false), through the existing booleanParamOr400.

• One definition. The option is a single shared components/parameters entry.

• The query change. Model.getQueryResults gets an includeOffSurface argument that treats the early surface check as cleared. The compiled backstop and the off-surface explanation hang off that check, so both are skipped for that request. Two other passes read the surface on their own, and the argument reaches them too:

  • the caller-join boundary pass is skipped, so a caller's own join to a hidden source runs;
  • the lock passes over aliases and caller joins keep a refused lock's 403. Without this, they turned it into a 404 for a hidden name.

  #(authorize) evaluation runs as before.

What does not change

• The notebook-cell route, the dashboards route, filter suggestions, the MCP tools and the legacy /projects/... aliases ignore the option. get_context and execute_query call without it.
• The query route does read it for any model path it is sent, including a dashboard or notebook file. That is the same reach queryableSources: "all" has.
• Indexing and materialization are unchanged.

Who may send it

Publisher is unauthenticated, so it does not decide. A gateway in front of it does. Credible's router forwards the option only for a caller with can_update_package (admins and modelers), and answers 403 to anyone else (ms2data/service#6609).

On the model GET the option shows more than queryableSources: "all" does. It returns the file's sourceText even when the text names a source the file does not publish, so a gateway passing it hands over #(authorize) expressions, given: defaults, connection names and the raw SQL of hidden sources. That is the point for an author, and it is why the gateway limits the option to people who can edit the package.

This is a plain query option rather than a shared-secret header like x-publisher-bypass-authorize. The bypass skips the lock (#(authorize)), and this doesn't: the surface is curation ("curation hides, authorize denies"). A source locked with #(authorize) false is still refused with the option, and a test pins it.

How it was checked

• Controller unit tests (model.controller.spec.ts):
  • With the option, assertFileOnSurface and showsFileText are skipped and the text is returned.
  • With false, both checks still apply.
• Real-package tests (explore_visibility.spec.ts), with a root index.malloy importing base.malloy:
  • The default listing is {index.malloy: true}. With the option it is {base.malloy: false, index.malloy: true}.
  • With the option, run: base_source on the hidden file and run: helper (a source index.malloy doesn't export) both return rows. Without the option, helper throws NotQueryableError.
  • With the option, a caller's own join to helper runs. Without it, it throws NotQueryableError.
  • A source locked with #(authorize) false still throws AccessDeniedError with the option, whether the query names it directly, through an alias (source: x is locked extend {}) or through a caller join.
  • A package with no surface marks every file onSurface: true either way.
• HTTP integration (tests/integration/index_convention/include_off_surface.integration.spec.ts, on the index-convention-test fixture), 4 of 4 pass:
  • the listing with and without the option;
  • 400 for yes, 1 and maybe;
  • a hidden file answers 404 by default, and with the option 200, with sourceText byte-for-byte the file on disk;
  • POST …/internal.malloy/query answers 404 by default and 200 with one row with the option.
• End to end, on a built server (bun run build:server-only, dist/server.mjs on a scratch server root), against the 23 index.malloy test packages from fix(server): make index.malloy the one list of what dashboards, notebooks and the model GET can read #1236. Each package is one surface shape: root index.malloy, layered re-exports, hidden joins, explores in each form, dashboards, a notebook, queryableSources: "all", and a hidden #@ persist source.
  • fix(server): make index.malloy the one list of what dashboards, notebooks and the model GET can read #1236's own checks: 54 of 54 pass. The default behavior is unchanged.
  • An oracle for this option: 337 of 337 pass. It checks against the files on disk, not other API responses, across 19 hidden files in 17 packages:
    • The listing with the option is every .malloy file minus served dashboards. onSurface is true exactly for the default listing.
    • Each hidden file's sourceText is byte-for-byte the file on disk.
    • A query to each hidden file's first source runs with the option (16 of them) and is refused without it. On idx-13, queryableSources: "all" already answers it without the option.
    • On-surface files return the same compiled body either way.
    • Bad values get 400. Notebooks and missing files still get 404.
    • 200 interleaved default and opted-in requests each answer for their own mode, so the option does not leak through shared state.
  • The oracle isn't passing for free. A build without this change fails its listing, onSurface and 400 checks and still passes fix(server): make index.malloy the one list of what dashboards, notebooks and the model GET can read #1236's 54.
  • MCP get_context is byte-identical for 8 packages between a build without this change and one with the listing and model-GET part. The query-route part doesn't touch that path.
• Mutation checks:
  • Removing the listing filter change and the controller skip makes the listing and model-GET tests fail.
  • Removing the query-route line makes the run test fail.
  • Undoing the caller-join skip, or either lock-pass change on its own, makes the run test fail.
• Server unit suite: 4614 pass, 35 fail. The same 35 config.spec / config.theme.spec failures happen on unmodified main under bun 1.4.
• Typecheck and lint are clean.
• The Python SDK build script passes. It rewrote formatting in four tracked files that this PR doesn't touch, and I left those out.
• Also run inside a local Credible stack. The workers ran the listing and model-GET part on v0.8.2, and requests went through the router. They behaved as above, and the workers stayed healthy.

🤖 Generated with Claude Code

[Sha-Bang]

No way to use this to skip restricted mode, #(authorize), #(access_filter) or givens: the flag only clears the surface check, and every gate after it runs as before. Three things don't match what the description and api-doc promise, plus a few doc nits. None of them block. Details inline. One more: the legacy /projects/... routes in server-old.ts neither honor nor 400 includeOffSurface, so it is silently ignored there. Probably fine for a compat alias, but it deserves one line in the docs.