diff --git a/oeps/best-practices/oep-0018-bp-python-dependencies.rst b/oeps/archived/oep-0018-bp-python-dependencies.rst similarity index 98% rename from oeps/best-practices/oep-0018-bp-python-dependencies.rst rename to oeps/archived/oep-0018-bp-python-dependencies.rst index 253e3ec22..6de8f9545 100644 --- a/oeps/best-practices/oep-0018-bp-python-dependencies.rst +++ b/oeps/archived/oep-0018-bp-python-dependencies.rst @@ -8,13 +8,13 @@ OEP-18: Python Dependency Management +-----------------+--------------------------------------------------------+ | Title | Python Dependencies Management | +-----------------+--------------------------------------------------------+ -| Last Modified | 2023-03-06 | +| Last Modified | 2026-04-24 | +-----------------+--------------------------------------------------------+ | Authors | Jeremy Bowman | +-----------------+--------------------------------------------------------+ | Arbiter | Calen Pennington | +-----------------+--------------------------------------------------------+ -| Status | Approved | +| Status | Replaced | +-----------------+--------------------------------------------------------+ | Type | Best Practice | +-----------------+--------------------------------------------------------+ @@ -27,6 +27,11 @@ OEP-18: Python Dependency Management .. _open-edx-proposals#56 resolution: https://github.com/openedx/openedx-proposals/pull/56#pullrequestreview-116976355 +.. warning:: + + This OEP has been replaced and may be out of date. For the most up-to-date + information see :ref:`OEP-67 Standard Tools and Technologies`. + .. contents:: :local: :depth: 2 diff --git a/oeps/best-practices/oep-0067-bp-tools-and-technology.rst b/oeps/best-practices/oep-0067-bp-tools-and-technology.rst index cd62c39a9..5e7970828 100644 --- a/oeps/best-practices/oep-0067-bp-tools-and-technology.rst +++ b/oeps/best-practices/oep-0067-bp-tools-and-technology.rst @@ -11,7 +11,7 @@ OEP-67: Standard Tools and Technologies * - Title - Tools and Technology Standards * - Last Modified - - 2025-02-10 + - 2026-04-24 * - Authors - Feanil Patel * - Arbiter @@ -27,7 +27,8 @@ OEP-67: Standard Tools and Technologies * - Resolution - * https://github.com/openedx/openedx-proposals/pull/518 * - References - - :ref:`OEP-11 Front End Technology Standards` + - * Replaces :ref:`OEP-11 Front End Technology Standards` + * Replaces :ref:`OEP-18 Python Dependency Management` Abstract ******** @@ -327,6 +328,32 @@ they do not require disruptive migrations that would break existing MySQL installations. Support for these other database backends is not guaranteed and may require plugins or lag behind the official releases. +#. **pyproject.toml should be used to declare project metadata and dependencies** + + **Rationale**: `PEP 621`_ standardized project metadata in ``pyproject.toml``, + and `PEP 735`_ added dependency groups — a standard way to declare + context-specific dependencies (test, docs, CI, dev, etc.). Using + ``pyproject.toml`` as the single source of truth replaces the older pattern + of ``requirements/*.in`` files and ``setup.py`` / ``setup.cfg``, simplifies + tooling, and aligns with the broader Python ecosystem. + +#. **uv should be used for Python dependency locking and virtual environment management** + + **Rationale**: `uv`_ consolidates dependency resolution, locking, virtual + environment creation, and tool execution into a single fast tool. It replaces + `pip-tools`_ (``pip-compile`` / ``pip-sync``) and produces a single + ``uv.lock`` file from ``pyproject.toml`` dependency groups. The existing + ``make upgrade`` pattern is preserved — the target now runs + ``uv lock --upgrade`` instead of ``pip-compile --upgrade``. + + **Decision Record**: For details, see + :ref:`Use uv for Python dependency management`. + +.. _PEP 621: https://peps.python.org/pep-0621/ +.. _PEP 735: https://peps.python.org/pep-0735/ +.. _uv: https://docs.astral.sh/uv/ +.. _pip-tools: https://github.com/jazzband/pip-tools + Decisions ********* @@ -357,7 +384,7 @@ considered. :maxdepth: 1 :glob: -.. oep-0067/decisions/backend/* + oep-0067/decisions/backend/* @@ -368,11 +395,22 @@ Consequences * All ADRs under OEP-11 will be moved to be under this OEP instead. +* :ref:`OEP-18 Python Dependency Management` will be superseded by the backend + technology recommendations in this OEP. + * Future decisions for technology changes will require an ADR and an update to this OEP Change History ************** +2026-04-24 +========== + +* Add ``pyproject.toml`` and ``uv`` to Backend Technology Selection +* Add backend ADR for adopting uv (replacing pip-tools) +* Mark :ref:`OEP-18 Python Dependency Management` as superseded +* `Pull request #784 `_ + 2025-02-10 ========== diff --git a/oeps/best-practices/oep-0067/decisions/backend/0001-uv.rst b/oeps/best-practices/oep-0067/decisions/backend/0001-uv.rst new file mode 100644 index 000000000..25c83051d --- /dev/null +++ b/oeps/best-practices/oep-0067/decisions/backend/0001-uv.rst @@ -0,0 +1,304 @@ +.. _Use uv for Python dependency management: + +Use uv for Python dependency management +######################################## + +Status +****** + +Accepted + +Context +******* + +Since 2018, :ref:`OEP-18 Python Dependency Management` has defined the standard +workflow for Python dependencies in Open edX repositories: + +* Direct dependencies declared in ``requirements/*.in`` files + +* ``pip-compile`` (from `pip-tools`_) generates pinned ``requirements/*.txt`` + lockfiles + +* ``pip-sync`` installs exactly those pinned versions into a virtualenv + +* ``make upgrade`` orchestrates the whole process + +This workflow has served the project well, but the Python packaging ecosystem +has evolved significantly: + +* `PEP 621`_ (2021) standardized project metadata in ``pyproject.toml``, + making ``setup.py`` and ``setup.cfg`` legacy. + +* `PEP 735`_ (2024) added dependency groups to ``pyproject.toml``, providing a + standard way to declare context-specific dependencies (test, docs, CI, etc.) + — the same problem OEP-18 solved with multiple ``.in`` files. + +* `uv`_ emerged as a fast, unified tool that handles dependency resolution, + locking, virtual environment management, and tool execution in a single + binary. It natively understands ``pyproject.toml`` and PEP 735 dependency + groups. + +Meanwhile, ``pip-tools`` has not added support for PEP 735 dependency groups, +and continuing to use it means maintaining a parallel dependency declaration +system (``requirements/*.in``) alongside ``pyproject.toml``. Additionally, +`Jazzband`_ — the organization that maintains ``pip-tools`` — announced in +March 2026 that it is `sunsetting`_, leaving the long-term governance and +maintenance of ``pip-tools`` uncertain. + +Decision/Consequence +******************** + +Open edX Python repositories should use **uv** for dependency management: + +* **Declare dependencies in** ``pyproject.toml`` using PEP 621 metadata and + PEP 735 dependency groups. This replaces ``requirements/*.in`` files. + +* **Lock dependencies with** ``uv lock``, producing a single ``uv.lock`` file. + This replaces multiple ``requirements/*.txt`` files generated by + ``pip-compile``. + +* **Install dependencies with** ``uv sync``, which creates and manages the + virtual environment. This replaces ``pip-sync`` and manual ``python -m venv`` + steps. + +* **Run tools with** ``uv run`` when a tool is installed in the project's + virtualenv (e.g., ``uv run tox``, ``uv run pytest``). + +* **Preserve the** ``make upgrade`` **pattern** — the target now runs + ``uv lock --upgrade`` instead of ``pip-compile --upgrade``. + +For test matrices that need multiple versions of a dependency (e.g., Django), +use ``[tool.uv].conflicts`` with separate dependency groups and +``uv-venv-lock-runner`` in tox. This ensures fully reproducible, locked +resolutions for each matrix combination. + +Shared constraints (e.g., the global constraints file from ``edx-lint``) are +written into ``[tool.uv].constraint-dependencies`` in ``pyproject.toml`` via +the ``edx_lint write_uv_constraints`` command, since ``uv lock`` does not +accept a ``--constraint`` CLI flag. + +Rejected Alternatives +********************* + +1. **pip-tools (status quo)** + + pip-tools is mature and well-understood in the Open edX community. However, + it does not support PEP 735 dependency groups, requires a separate tool for + virtual environment management, and is significantly slower at dependency + resolution than uv. Continuing to use it means maintaining + ``requirements/*.in`` files alongside ``pyproject.toml``, duplicating + dependency declarations. The `sunsetting of Jazzband `_ also + raises questions about long-term maintenance. + +2. **Poetry** + + `Poetry `_ is a popular all-in-one tool, but it + is opinionated about project structure, uses a non-standard lock format + (``poetry.lock``), and has limited support for the kind of dependency groups + needed for Open edX code test matrices (e.g., multiple Django versions). Its + resolver has historically been slower than uv's. + +3. **PDM** + + `PDM `_ supports PEP 735 and produces a standard + lock format, but has a smaller community and less ecosystem adoption. uv has + gained broader momentum in the Python packaging space and is backed by the + same team (Astral) that maintains ``ruff``, which is already widely adopted + in the broader Python community. + +Examples +******** + +The following examples are drawn from `openedx/sample-plugin`_ and illustrate +the key patterns. See that repository for a complete working reference. + +Example pyproject.toml +====================== + +.. code-block:: toml + + [build-system] + requires = ["setuptools", "setuptools-scm>8.1"] + build-backend = "setuptools.build_meta" + + [project] + name = "openedx-sample-plugin" + description = "A sample backend plugin for the Open edX Platform" + requires-python = ">=3.12" + license = "Apache-2.0" + + dependencies = [ + "Django", + "djangorestframework", + "openedx-events", + ] + + # -- Dependency groups (PEP 735) ------------------------------------------ + # + # These replace the old requirements/*.in files. Each group maps to a usage + # context (testing, quality checks, docs, CI tooling, local dev). + + [dependency-groups] + # Framework-agnostic test deps. Not used directly — included by the + # version-specific groups below. + test-base = [ + "pytest-cov", + "pytest-django", + ] + + # Current default Django version. Used by quality, docs, and as the default + # test matrix entry. + test = [ + {include-group = "test-base"}, + "Django>=5.0,<6.0", + ] + + # Additional Django versions under test. Each gets its own group so uv can + # produce a separate locked resolution via [tool.uv].conflicts. + django60 = [ + {include-group = "test-base"}, + "Django>=6.0,<7.0", + ] + + quality = [ + {include-group = "test"}, + "edx-lint", + "pycodestyle", + ] + + doc = [ + {include-group = "test"}, + "Sphinx", + "doc8", + ] + + ci = [ + "tox", + "tox-uv", + ] + + dev = [ + {include-group = "quality"}, + {include-group = "ci"}, + ] + + # -- uv configuration ----------------------------------------------------- + + [tool.uv] + # Mutually exclusive groups: uv produces one locked resolution per entry. + conflicts = [ + [{group = "test"}, {group = "django60"}], + ] + + # DO NOT EDIT constraint-dependencies DIRECTLY. + # This list is managed by `edx_lint write_uv_constraints` + # and will be overwritten the next time `make upgrade` is run. + # - GLOBAL constraints: edit edx_lint/files/common_constraints.txt + # - REPO-SPECIFIC constraints: edit [tool.edx_lint].uv_constraints in this file + constraint-dependencies = [ + "Django<7.0", + "elasticsearch<7.14.0", + ] + + [tool.edx_lint] + # Repo-specific uv constraints merged with edx-lint's global constraints. + # Local entries override global ones for the same package. + # Run `make upgrade` to regenerate [tool.uv].constraint-dependencies. + uv_constraints = [ + "Django<7.0", # this repo tests Django 6.x; overrides global Django<6.0 + ] + +Example tox.ini +=============== + +.. code-block:: ini + + [tox] + envlist = py312-django{52,60},quality,docs + requires = + tox-uv>=1 + + [testenv] + runner = uv-venv-lock-runner + dependency_groups = + django52: test + django60: django60 + commands = + pytest {posargs} + + [testenv:quality] + runner = uv-venv-lock-runner + dependency_groups = + quality + commands = + pylint my_package tests + + [testenv:docs] + runner = uv-venv-lock-runner + dependency_groups = + doc + commands = + doc8 docs + make -e -C docs html + +Common commands +=============== + +.. code-block:: bash + + # Create a virtual environment + uv venv .venv --seed --python 3.12 + + # Install all dev dependencies (creates .venv if needed) + uv sync --group dev + + # Run tests via tox + uv run tox + + # Run a single tox environment + uv run tox -e py312-django60 + + # Upgrade all locked dependencies (the new "make upgrade") + uv lock --upgrade + + # Sync shared constraints from edx-lint into pyproject.toml + uv run --with edx-lint edx_lint write_uv_constraints + + # Full upgrade sequence (typically wrapped in a Makefile target) + uv run --with edx-lint edx_lint write_uv_constraints + uv lock --upgrade + +References +********** + +* `PEP 621`_ — Storing project metadata in pyproject.toml + +* `PEP 735`_ — Dependency Groups in pyproject.toml + +* `uv`_ documentation + +* `pip-tools`_ + +* `Jazzband`_ and their `sunsetting`_ announcement + +* `openedx/sample-plugin`_ — reference implementation + +* :ref:`OEP-18 Python Dependency Management` — the predecessor workflow this decision replaces + +Change History +************** + +2026-04-24 +========== + +* Document created +* `Pull request #784 `_ + +.. Cross-references +.. _pip-tools: https://github.com/jazzband/pip-tools +.. _PEP 621: https://peps.python.org/pep-0621/ +.. _PEP 735: https://peps.python.org/pep-0735/ +.. _uv: https://docs.astral.sh/uv/ +.. _Jazzband: https://jazzband.co/ +.. _sunsetting: https://jazzband.co/news/2026/03/14/sunsetting-jazzband +.. _openedx/sample-plugin: https://github.com/openedx/sample-plugin diff --git a/oeps/redirects.txt b/oeps/redirects.txt index d6b78c5af..3937d8188 100644 --- a/oeps/redirects.txt +++ b/oeps/redirects.txt @@ -25,3 +25,4 @@ "best-practices/oep-0013.rst" "archived/oep-0013.rst" "processes/oep-0056-proc-architectural-advisory-process.rst" "archived/oep-0056-proc-architectural-advisory-process.rst" "architectural-decisions/oep-0006-arch-context-xblock-fields.rst" "archived/oep-0006-arch-context-xblock-fields.rst" +"best-practices/oep-0018-bp-python-dependencies.rst" "archived/oep-0018-bp-python-dependencies.rst"