Switch to PEP 420 (Implicit namespace packages)

[icemac]

Currently we use pkg_resources-style namespace packages. Using pkg_resources is deprecated (see https://setuptools.pypa.io/en/latest/pkg_resources.html), it seems not yet clear when it is removed from setuptools.

The current future are PEP 420: Implicit namespace packages.

This requires:

• all the packages sharing a namespace to switch over to PEP 420.
• removing the __init__.py files existing to create a namespace
• removing namespace_packages declaration from setup.py
• maybe removal of the dependency on setuptools? (Maybe we could even switch to pyproject.toml.)

zopefoundation hosts the following namespaces:

• zope, zope.app (zope, zope.app – 117 repos #331) - 117 repos
• Products – 22 repos
• zc (zc, zc.recipe – 10 repos #346)
• keas (probably only some packages of that namespace) – all repos archived
• refline (probably only some packages of that namespace) - all repos archived
• grokui – all repos archived
• hurry – 1 repos (Switch to PEP 420 native namespace. hurry.query#18)
• cipher – all repos archived
• five – 4 repos – migrated
• megrok: 1 repos (Switch to implicit (PEP 420) namespace. megrok.strictrequire#8)

See the discussion in pypa/setuptools#3434 for some more details.

Suggestion:

• ✅ Try out on a small namespace where probably all packages are hosted by zopefoundation (grokcore?, z3c?, zope.app?)
• If this works fine (especially in zc.buildout), adapt the other namespaces.
  • This will require coordination with the Plone folks.
  • This would also require a big warning sign that other packages hosted outside zopefoundation but using the same namespaces also have to switch to PEP 420.

Cc-ing: @dataflake @mauritsvanrees @gforcada @mgedmin @jensens

See also https://packaging.python.org/en/latest/guides/packaging-namespace-packages/

[dataflake]

This looks like a ton of work... but if we could really get rid of setuptools it would be worth it. Setuptools is nothing but trouble at this point.

[mauritsvanrees]

I left two comments in the setuptools discussion, with some numbers. Let's see if we can get a timeline for when setuptools will drop support.

Zope and Plone can only move to native namespaces in a major release. Since Plone 6 was only released a few months ago, I expect it will take two to three years before we have Plone 7, which would need a Zope 6. We may need to speed that up. But I hope we can let this rest for another year.

If we know that for example setuptools 70 is going to drop support, I suppose we could add setuptools<70 in install_requires to prevent installation of incompatible packages side by side.

[jensens]

+1 to start the process of modernization here. Some remarks:

• I agree, this change is necessary at some point.
• I agree with @mauritsvanrees that this is a breaking change at some point and would need a new major Zope release.
• I think a good idea is to introduce this change incrementally:
  • first all controlled zope.* packages can be changed relatively easy, probably the same is valid for five.*, z3c.* and zc.* (and maybe some other in the list above I do not know/use). This might go into the Zope 5 release series.
  • With the Products.* namespace it gets difficult, this is a bunch of packages, including CMF, Plone and all add-ons. It needs tight coordination.
  • If we switch everything to PEP420 in Plone this would be possible only in a Plone 7.
• if this topic is touched I would switch to full to pyproject.yml w/o any setup.* along with it. For my company projects I use already PEP420 style packages along with pyproject.toml w/o any problems.
• ad getting rid of Setuptools: What is the idea behind this?
  • Setuptools is settled, widely used, has lots of tooling available and is relatively fast.
  • What is the alternative we want? We need some PEP 518 [build-system]. I am not that a fan of Poetry, did not try Flit or PDM.

[dataflake]

The idea of getting rid of setuptools was not based on any knowledge of replacements, but on some comments in that setuptools discussion on GitHub. It may not be possible. Where I come from is endless issues with the Zope packages that use C-code in conjunction with macOS and the different builds that exist for it (x86_64, aarch64 and universal2). I described that at #181.

[icemac]

I am not aware of an alternative to setuptools which can handle C code. But I am open to learn if there is one.

[mgedmin]

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

Are we talking about a flag day here, with major version bumps and coordinated releases for every zope.* package at the same time? (And similar efforts for zc.*, z3c.*, grok.*, collective.* namespaces)?

[d-maurer]

Marius Gedminas wrote at 2023-3-8 23:31 -0800:

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

As I read PEP-420 it does not provide a transistion path with `pkg_resources` but it does for `pkgutil.extend_path` -- with the limitation that in this case, `__path__` recomputation is not guaranteed (maybe: will not happen). It might be possible that `pkg_resources.declare_namespace` has been extended in modern `setuptools` versions to handle implicit namespace packages correctly. However, I am not sure: implicit namespace packages make `__path__` read-only which may make the task for `declare_namespace` much more difficult.

[d-maurer]

Marius Gedminas wrote at 2023-3-8 23:31 -0800:

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

I have had a second thought. When I understand PEP-420 correctly, then it modifies the import of package *p* as follows: * it looks for *p* on the current `__path__` * if it finds a directory *p* with `__init__`, then *p* is instantiated as a "normal" package * only if all directories for *p* lack a `__init__`, it is instantiated as a namespace package. This would imply: either all directories for a namespace package *p* lack an `__init__` or the existing `__init__`s must emulate the namespace package treatment. Accoding to PEP-420, `pkgutil.extend_path` can be expected to partially emulate the namespace package treatment (with the exception of support for subsequent `__path__` changes). Modern `setuptools` versions might have a `pkg_resources.declare_namespace` which does the same.

[jensens]

Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?

I did not read the PEPs in detail nor the code about it, but about a year ago I just tried it and it just did not work. Ago then, I found some source-on-the-internet stating exact: it does not mix (but currently I can not find it again).

[mauritsvanrees]

That would be the Python Packaging User Guide, which says:

Warning: While native namespace packages and pkgutil-style namespace packages are largely compatible, pkg_resources-style namespace packages are not compatible with the other methods. It’s inadvisable to use different methods in different distributions that provide packages to the same namespace.

It also gives one specific reason for using pkg_resources-style namespace packages:
"This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe."

Do we need that for some reason?
I guess not: in a lot of Zope and Plone package I see zip_safe=False, so the other way around.

[jensens]

"This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe."

Do we need that for some reason?

"It is very unlikely that the values of zip_safe will affect modern deployments that use pip for installing packages. Moreover, new users of setuptools should not attempt to create egg files using the deprecated build_egg command. Therefore, this flag is considered obsolete." https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html