Multiple code locations in Dagster OSS with shared code (resources, utils)

[tspencerdata]

Hi all,
I’m looking for some guidance on repo structure best practices in Dagster OSS.

Context:

• I have multiple code locations, each of which requires its own Python environment
• I have a set of shared utilities, resources that are used across all code locations.

Current repo structure (simplified):

dagster_OSS/
├─ analytics/ # code location 1 (unique env)
│ ├─ defs/
│ ├─ lib/
| └─ pyproject.toml
├─ data_engineering/ # code location 2 (unique env)
│ ├─ defs/
│ ├─ lib/
│ ├─ resources/ # shared resources
│ ├─ utils/ # shared utilities
| └─ pyproject.toml
├─ deploy_files/
├─ workspace.yaml # defines the code locations
└─ pyproject.toml

Each code location may need libraries that would clash with others, so isolation is important. But I also want to avoid duplicating the shared utils/ and resources/ code across locations.

What’s the recommended way to structure repos when you need per-location env isolation but also shared code?

[valentinedwv]

Not fully sure... two options:

1. put shared code in one code location, say data_engineering, then import data_engineering as a python module into analytics
2. create a seperate module for the shared code, and import module

[zyd14]

I would highly recommend using uv workspaces for this, where each code location is a workspace and shared packages can be represented one or more workspaces within the same project. This makes it trivial to share common code across code locations. Just make sure all you don't have any shared code workspaces importing from code locations so as to avoid circular dependencies.

My company has a monorepo with about a half dozen code locations and a half dozen internal packages all structured as workspaces within a single UV project and it works amazingly well... within the codebase it feels like it's all just one big package but we can easily build and deploy code locations individually and can also slice out reusable shared code packages that are useful outside of Dagster as well.

Edit: just wanted to note that I personally try to avoid importing code from one code location into another code location, as those kinds of dependencies will cause scenarios where updating one code location requires also redeploying another code location. This can easily get you into weird situations where code location B is running with out-of-date code imported from an old version of code location A, because someone updated location A but didn't know they had to restart B as well (among other suboptimal scenarios).

[cmpadden]

Both recommendations here are great. Going to mark @zyd14's as the answer, but if you have any other questions please share.