Skip to content

Add github_path: load a model from a subdirectory of a repo - #110

Open
stefanoperenzoni wants to merge 2 commits into
malloydata:mainfrom
stefanoperenzoni:feat/github-path
Open

stefanoperenzoni wants to merge 2 commits into
malloydata:mainfrom
stefanoperenzoni:feat/github-path

Conversation

@stefanoperenzoni

Copy link
Copy Markdown

What

Datasets gain an optional github_path — a subdirectory within the GitHub repo where the model lives. GitHub-loaded models previously required index.malloy at the repo root, which excludes a common layout: a Malloy semantic layer living in a subdirectory of a larger data project (a monorepo with dbt/SQLMesh models, pipelines, etc. alongside the .malloy files).

With this change, a dataset configured with e.g. path=models/malloy resolves index.malloy, all of its imports, malloy-config.json, and the dashboards/ directory under that path.

How

  • New datasets.github_path column, NOT NULL DEFAULT '' — empty string means repo root, i.e. existing behavior, byte-for-byte.
  • GitHubURLReader gains an optional base path used only when building the Contents-API URL. Cache keys and stored file paths stay model-relative (index.malloy, dashboards/x.malloy), so Malloy import resolution and everything downstream are independent of where the model sits in the repo.
  • refreshGitHubModel and the dataset-create introspection prefix the config fetch, the dashboards listing, and dashboard component fetches with the path; provenance records it (github:owner/repo@branch/path).
  • The path is settable on dataset create and edit (API + an optional field in both forms) and returned by the dataset GET.
  • Manual migration drizzle/manual/0010_github_path.sql (idempotent, per the repo's migration convention); fresh installs get the column from the schema as usual.

Backward compatibility

Dark by default: every existing dataset gets '' and takes exactly the old code path (joinRepoPath('', p) === p). Old API clients that don't send the field are unaffected (zod default). No existing route's behavior changes; the only touched error message is the 404 hint text.

Tests

src/lib/github.test.ts (node:test, fetch stubbed, no network):

  • path normalization (slashes/whitespace/null)
  • join only when a base is set
  • reader with base path: prefixed fetch URL + model-relative cache key + cache hit
  • reader without base path: unchanged root behavior (regression guard)

Full preflight is green (engine + CLI typecheck/tests, server lint, next build, hosted integration test).

🤖 Generated with Claude Code

GitHub-loaded models previously required index.malloy at the repo root,
which excludes the common monorepo layout where the Malloy semantic layer
lives in a subdirectory alongside the rest of a data project.

Datasets gain an optional github_path (default "", i.e. the root — existing
behavior is unchanged byte-for-byte). The sync layer resolves index.malloy,
its imports, malloy-config.json, and dashboards/ under that path, while
cache keys and stored file paths stay model-relative, so import resolution
and everything downstream are independent of where the model sits in the
repo. The path is settable on dataset create/edit (API + UI) and recorded
in model provenance.

Includes an idempotent manual migration (drizzle/manual/0010) and unit
tests covering path normalization, prefixed fetching, and the unchanged
root-path behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: stefanoperenzoni <stefano.perenzoni@gmail.com>
The GitHub push webhook fires for every branch, and the route ignored the
payload — so registering it on an active repo (the monorepo case github_path
enables) minted a redundant model version for every feature-branch push.

Read the push payload's ref and skip refreshes for other branches. Calls
without a JSON ref (manual curl triggers) keep refreshing unconditionally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: stefanoperenzoni <stefano.perenzoni@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant