Monorepos and Workspaces

[humphd]

I'm really loving pixi (it's amazing!), especially for my projects that include multiple languages. A number of these are monorepos, and I wanted to seek some guidance on how best to use pixi in a monorepo setup.

I'm coming from the JS/node.js ecosystem, where I'm used to using tools like pnpm and workspaces, such that I can have a top-level config (package.json and pnpm-workspace.yml) and lock file, with nested projects only needing their own config. Doing an install at the root of the monorepo recursively installs things, and using -r I can recursively run tasks throughout the monorepo.

How would you suggest I use pixi in such cases? Currently what I'm doing is using pixi at the root to install global tooling (node, python, etc) and pnpm for my JS sub-projects. For the Python sub-projects, I've opted for another pixi.toml and pixi.lock file within the sub-project, which handles things like installing requirements.txt as a task.

Installing everything at the root seems wrong, but I feel like there must be a better solution than what I'm doing. Has anyone done this and can provide guidance?

[ruben-arts]

Glad to hear you're enjoying it! 😃

The workspace story has already come to our minds multiple times, we're heavy users of the Cargo workspaces, as our main library https://github.com/mamba-org/rattler is using it.

We want to take what we can learn from that and apply it to pixi, pnpm indeed has a similar workspace setup but not build into the main configuration file, which is something we would definitely like to do.

This however is on our backlog, and some other features are having our priority right now. Not to discourage you of using pixi but I don't want to share unreachable deadlines kuchElonMuskkuch.

A solution for your situation right now depends on the use of your nested projects.

• If you're building a single running app that needs all the dependencies available at the moment of running the application I would say the pixi.toml at the top-level should be your best option.
• If you're using all projects separately, for instance a python back-end server and a JS front-end. Then I would advice separate them in different pixi projects which you just start by moving into those projects and activating them separately. This will help with bundling parts of your monorepo, e.g. into docker containers or separate packages(conda, pip, node, etc).

I would like this discussion to stay open and welcoming to ideas and wishes for a "Workspace" implementation" 💡

[humphd]

I get that this is a feature that needs thought and will take time to figure out. In the meantime, I'm glad to know that I'm not missing some obvious solution for getting a workable system. I agree with you re: optimizing for dockerizing parts of the monorepo, and using separate pixi.toml files per project.

Thanks for the quick reply and all the amazing work on pixi!

[majidaldo]

i've pushed this idea to the limit with conda https://github.com/pnnl/hydraconda. i had a venv for every "workspace" and deps b/w these workspaces.

pixi would form a much better basis.

[jacksonwb]

Would really love to see a way to have a project A defined in a pixi.toml. And then another project B defined next to it in source with its own pixi.toml that can point to project A via a path dependency.

Then be able to build and run project A by itself, or build and run project B that then automatically builds A. And ideally be able to create packages with correct named version dependencies for release.

[majidaldo]

Yup that's what I did above

[xiaoxiangmoe]

uv support workspace dependencies now.
https://docs.astral.sh/uv/concepts/dependencies/#workspace-member

Here is a example

.
├── packages
│   └── hello
│       ├── pyproject.toml
│       └── src
│           └── hello
│               ├── __init__.py
│               └── py.typed
├── pyproject.toml
├── src
│   └── main.py
└── uv.lock

[project]
name = "playground-monorepo"
version = "0.1.0"
description = "Add your description here"
requires-python = ">=3.13"
dependencies = [
    "hello",
]

[tool.uv.sources]
hello = { workspace = true }
[tool.uv.workspace]
members = ["packages/*"]

[cademirch]

Is there an equivalent to this using pixi?

[KevinEdry]

Any updates on this? We would really like to see a Pixi implementation of a monorepo.

[ruben-arts]

@hoetmaaiers, this works in both pixi.toml and pyproject.toml

[Andre-Medina]

Does the monorepo install dependencies from the packages respetive pyproject.toml defined deps?

Just quickly tested with my current configuration of:

- monorepo_root/
  - packages/
    - package_1/
      ...
      - pyproject.toml
    - package_2/
      ...
      - pyproject.toml
  ...
  - pixi.toml
  - pixi.lock 

And adding dependencies to the sub pyprojct.tomls doesn't add the packages to the pixi.lock. All the dependencies have to be within the root pixi.toml. This maybe an easy fix with some configuration change, but for my use case, I'm happy with just adding dependencies within different features within the pixi.toml file.

[hoetmaaiers]

Okay, this is also my experience. Ideally, packages as git submodules would install their dependencies and the root level should not need to have them installed.
I am not fully aware of the complexities behind this idea.

[ruben-arts]

Ah sorry, for the confusing response, this works with the "normal" pypi dependencies. If you depend on a pyproject.toml that has these fields it will only install the default pyproject.toml dependencies.

# This will be installed
[project]
dependencies = ["numpy"]

# These will not be installed
[tool.pixi.dependencies]
pytest = "*"
[tool.pixi.pypi-dependencies]
pytest = "*"

We're currently building a real conda based workspace workflow. We'll keep you posted. The current workaround would indeed be to manage features in the root project.

For now the features like

[hoetmaaiers]

Looking forward to the conda based workspace workflow!

[mantkiew]

I feel like the solution would be to expand the "solve groups" concept to be able to refer to features from other pixi.toml files for the individual repos. See "solve-group" in https://pixi.sh/v0.56.0/workspace/multi_environment/#feature-environment-set-definitions

Before pixi, I used micromamba and organized multiple repos around a common and shared environment. So, each repo would add their stuff to the common env. Of course, that didn't work well because of version conflicts and the lack of lockfiles in micromamba. So, joining a bunch of features from multiple pixi.toml into a single solve group would allow keeping everything globally consistent.