Add github_path: load a model from a subdirectory of a repo - #110
Open
stefanoperenzoni wants to merge 2 commits into
Open
stefanoperenzoni wants to merge 2 commits into
stefanoperenzoni wants to merge 2 commits into
Conversation
GitHub-loaded models previously required index.malloy at the repo root, which excludes the common monorepo layout where the Malloy semantic layer lives in a subdirectory alongside the rest of a data project. Datasets gain an optional github_path (default "", i.e. the root — existing behavior is unchanged byte-for-byte). The sync layer resolves index.malloy, its imports, malloy-config.json, and dashboards/ under that path, while cache keys and stored file paths stay model-relative, so import resolution and everything downstream are independent of where the model sits in the repo. The path is settable on dataset create/edit (API + UI) and recorded in model provenance. Includes an idempotent manual migration (drizzle/manual/0010) and unit tests covering path normalization, prefixed fetching, and the unchanged root-path behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: stefanoperenzoni <stefano.perenzoni@gmail.com>
stefanoperenzoni
force-pushed
the
feat/github-path
branch
from
July 22, 2026 12:23
cd3893a to
3dc3de5
Compare
The GitHub push webhook fires for every branch, and the route ignored the payload — so registering it on an active repo (the monorepo case github_path enables) minted a redundant model version for every feature-branch push. Read the push payload's ref and skip refreshes for other branches. Calls without a JSON ref (manual curl triggers) keep refreshing unconditionally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: stefanoperenzoni <stefano.perenzoni@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Datasets gain an optional
github_path— a subdirectory within the GitHub repo where the model lives. GitHub-loaded models previously requiredindex.malloyat the repo root, which excludes a common layout: a Malloy semantic layer living in a subdirectory of a larger data project (a monorepo with dbt/SQLMesh models, pipelines, etc. alongside the.malloyfiles).With this change, a dataset configured with e.g.
path=models/malloyresolvesindex.malloy, all of its imports,malloy-config.json, and thedashboards/directory under that path.How
datasets.github_pathcolumn,NOT NULL DEFAULT ''— empty string means repo root, i.e. existing behavior, byte-for-byte.GitHubURLReadergains an optional base path used only when building the Contents-API URL. Cache keys and stored file paths stay model-relative (index.malloy,dashboards/x.malloy), so Malloy import resolution and everything downstream are independent of where the model sits in the repo.refreshGitHubModeland the dataset-create introspection prefix the config fetch, the dashboards listing, and dashboard component fetches with the path; provenance records it (github:owner/repo@branch/path).drizzle/manual/0010_github_path.sql(idempotent, per the repo's migration convention); fresh installs get the column from the schema as usual.Backward compatibility
Dark by default: every existing dataset gets
''and takes exactly the old code path (joinRepoPath('', p) === p). Old API clients that don't send the field are unaffected (zod default). No existing route's behavior changes; the only touched error message is the 404 hint text.Tests
src/lib/github.test.ts(node:test, fetch stubbed, no network):Full preflight is green (engine + CLI typecheck/tests, server lint,
next build, hosted integration test).🤖 Generated with Claude Code