Reliance on flit makes packaging incompatible with Void, Arch and Alpine Linux package systems

[ahesford]

The switch to flit in #352 has left the packaging source tarball without a setup.py. While one can use flit install to install packaging, the flit install command provides a means to choose between installing to the system site-packages (possibly in a virtualenv, by providing a custom path to the python executable) or to a per-user site-packages.

Void Linux xbps-src, Arch Build System and Alpine apk-tools package builders all expect to install Python packages into a specific destination directory that is empty except for the contents of the Python package; the distribution package is derived from the contents of this directory. For setuptools-based projects like packaging<20.5, this is accomplished by passing, e.g., --root $DESTDIR for Void Linux. flit install provides no similar mechanism. The closest one could come would be to construct a virtualenv at the $DESTDIR, then use flit install to populate, and then carefully remove all pieces of the virtualenv from $DESTDIR except for the installed package. It should be self evident that this is a horrible workaround.

If you are use flit build and flit publish to build wheels and source tarballs on PyPI, this problem should take care of itself, because both flit subcommands appear to create a setup.py in the source tarball by default. (Note, however, that the generated setup.py appears incorrect because it imports distutils.core.setup, then uses the install_requires and python_requires keywords that seem to be only recognized by setuptools.setup.) However, the PyPI source tarballs for packaging==20.5, packaging==20.6 and packaging==20.7 do not include setup.py. Thus, you are either creating a source tarball by some other means, or invoking flit build --no-setup-py or flit publish --no-setup-py.

Without a setup.py in the source tarball (either generated by flit build or flit publish, or manually created to parallel the package definition in pyproject.toml), new versions of packaging cannot be properly packaged for Void, Arch, Alpine and, I imagine, several other Linux distributions.

[pradyunsg]

IMO those packaging systems need to update themselves for the fact that PEP 517 is a thing in Python now.

We’re invoking python -m build (source code: https://github.com/pypa/build) for building our distributions and I don’t believe we’re packaging things incorrectly.

[uranusjr]

(Periodic reminder that a wheel installer would provide a straightforward workflow for this.)

[ahesford]

Supporting PEP 517 is fine, and moving away from setuptools or distutils is not ipso facto the issue here. As pointed out in response to takluyver/flit#380, flit really isn't really the problem here. I misunderstood the scope of that project.

Installing from the source tarball is possible using pip, which offers --root and --prefix arguments necessary to prepare a destination directory for production of system packages. Using pip, from a system packaging standpoint, presents a couple of obstacles:

1. By default, pip wants to fetch its own build dependencies. This is bad from a repeatability standpoint (I won't say "reproducability" because that has picked up the connotation of identical outputs that Void, at least, won't promise) because building the same distribution package at different times may pull different versions of build dependencies. This should be avoidable with the --no-build-isolation flag for pip.

2. Using pip with a locally built wheel causes the installation of a direct_url.json that encodes the location of the wheel or the source tree, and pip freeze will list that location rather than a version requirement. This helps repeatability for packages that are truly built and installed locally (provided the original wheel or source tree is preserved), but for system packages, prefering an ephemeral and inaccessible location on some build server to a version string actually discards information and prevents pip freeze from meaningfully describing the system python environment. The workaround here is to remove direct_url.json after installation and before the system package is produced, which is functional but perhaps a bit distasteful.

[FRidh]

In Nixpkgs this change is also causing trouble, but from a different point of view. We install everything from source, and with the change of packaging to use flit we end up with a dependency diagram that is no longer acyclic. Now, in principle this issue already existed before this change because pip and setuptools vendor their dependencies, but then at least downstreams would not have to deal with that.

Note however this issue is resolved as soon as flit uses toml instead of pytoml, but that has not been released yet. I've asked for a release. Until then, we can simply cherry-pick that commit and then things are fine. Thus, no need to change in from my point of view, it is just for your information.

[uranusjr]

Adapting PEP 517 means, essentially, to adapt wheels as the common installation medium. All build tools produce wheels. Pip (in PEP 517 mode) installs wheels. Other tools intending to install Python packages, if they choose to not use pip, would therefore need to understand how to build and install wheels. The wheel format is well-specified, and any tool can implement an interface to perform the process of putting files inside a wheel into any target location.

I would suggest this is a good chance for all the tools mentioned (xbps-src, apk-tools, etc.) to come together and discuss how they should migrate off the implicit dependency against setuptools, and “properly” support Python packages in its specified form. With Python packaging now being increasingly more and more based on standards (instead of specific implementations like setuptools), more and more packages are expected to move off setuptools, and delaying the progress will not make the transition easier down the road. People working on Python packaging generally do not involve too much in alternative tools (since it is enough work to keep eyes on one set of them), and having alternative players in the field would help us understand the challenges and take advice to improve the process early.

[FRidh]

I think that discussion is best kept at https://discuss.python.org/c/packaging.

[ahesford]

@FRidh I'm curious to see how Nixpkgs will handle this update (I see packaging there is still stuck at 20.4, as in Void), since you are even more concerned about reproducibility and will definitely need to freeze the build-time dependencies. Is --no-build-isolation sufficient?

More broadly, there isn't an inherent problem with relying on pip to install wheels as long as those wheels can be reliably produced from source code in a way that supports multiple architectures and cross-compilation. (This is not an issue for the pure-Python packaging, but will definitely become an issue for projects that want to abandon setuptools while still building compiled extensions. Note that flit does not even attempt to solve this problem.)

That pip produces PEP 610 output in direct_url.json without a mechanism to disable this output is a symptom of the real issue here: the Python package ecosystem is indifferent to distribution packaging. Sure, I can remove direct_url.json when producing a distribution package of some Python package; but this shouldn't have to be done. Moving packaging off of setuptools before pip and other build and installation tools fully accommodate the needs of system packaging is a step backward.

The Python community seems to acknowledge that packaging and distribution is a difficult problem, because Python-specific solutions to the packaging problem evolve dramatically. Developing an approach that fits together with distribution package systems, rather than attempting to stand in isolation, seems to offload some of this burden from the Python ecosystem to the distributions that want to bundle this software. Fundamentally, all Python packages should be representable as distribution-specific packages, because they need to fit in a larger dependency graph than the Python ecosystem alone can model.

[FRidh]

@FRidh I'm curious to see how Nixpkgs will handle this update (I see packaging there is still stuck at 20.4, as in Void),

I am updating right now NixOS/nixpkgs#105368. Those updates should land in a week, maybe two, in our unstable channel.

since you are even more concerned about reproducibility and will definitely need to freeze the build-time dependencies. Is --no-build-isolation sufficient?

We declare our dependencies and perform builds in a sandbox. We then indeed pass --no-build-isolation.

To build:

@pythonInterpreter@ -m pip wheel --no-index --no-deps --no-clean --no-build-isolation --wheel-dir dist .

To install:

@pythonInterpreter@ -m pip install -e . --prefix "$tmp_path" \
         --no-build-isolation >&2

Setuptools could still try to install, e.g. when using the deprecated python setup.py test. Before we used sandboxing we tried to capture all those cases. Nowadays sandboxing is the default with Nix, and we don't even bother anymore. Which sometimes is a pity though, e.g. when building with Nix in Docker on GitHub Actions. Anyway, we're getting off-topic.

[eli-schwartz]

IMO those packaging systems need to update themselves for the fact that PEP 517 is a thing in Python now.

@pradyunsg you, better than nearly anyone else alive, know precisely how much of a lie this is.

I get it that it takes time and effort to get https://github.com/pradyunsg/installer/ off the ground, and I don't mean to demand you volunteer "ALL YOUR TIME RIGHT NOW" to make free stuff for people.

But it would be really helpful for the purpose of discussion if you don't carefully miss the point by a country mile and suggest the existence of a tool to build wheels identical to https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl means distros don't need tools to install the darned thing.

You are intimately involved in the existing discussions as to why "those packaging systems" do not want to use pip, and generally cannot, and why PEP 517 only describes one half of a solution previously solved by "use the well-thought-out setuptools interface". I'm sure you're well aware by now that distros aren't helped one bit by PEP 517, which contrarily causes a lot of pain and heartbreak.

It's downright depressing to see this hostile attitude toward distros that are literally asking for one thing and one thing only: do not drop support for the old way of doing things until after a new way is implemented.

Distros are not necessarily experts at writing deep integrations into the guts of python PEPs. But we are very much experts at tying together build systems and system dependencies, then deploying your software to tens of thousands of people.

Making fundamental components of the python packaging ecosystem incapable of being installed using distro-compatible tooling, then performing victim blaming ("it's your fault for not writing your own build system to hook into our buildsystem, we don't care if you can package this, distros suck because they don't PEP 517"), is not exactly what I call welcoming.

Once again, it seems we're back to cavalierly saying "screw you" to distros.

[eli-schwartz]

BTW: python -m build going through the pep517 module, while no doubt capable of producing sdists suitable for consumption by pip install, is not a suitable way to create sdists for uploading to PyPI.
("yay for pep517 standard build_sdist()")

The flit_core module pep517 buildapi (not the flit module + command line tool) uses this: https://github.com/takluyver/flit/blob/6a2a8c6462e49f584941c667b70a6f48a7b3f9ab/flit_core/flit_core/sdist.py#L78

class SdistBuilder:
    """Builds a minimal sdist
    These minimal sdists should work for PEP 517.
    The class is extended in flit.sdist to make a more 'full fat' sdist,
    which is what should normally be published to PyPI.
    """

[FFY00]

(Periodic reminder that a wheel installer would provide a straightforward workflow for this.)

It would in fact solve this, though I do wish all core packaging packages used all the same backend to make bootstrapping easier. Otherwise, we have to manually bootstrap all the PEP517 backends required and then build the packages. It helps that flit_core has no dependencies, but it is still an extra thing to bootstrap. One extra step here plus one extra step there and one extra step in another place in the end get all summed up and make a very real difference.

Unless there are is a logical reason to choose flit, I urge the maintainers to reconsider and move back to setuptools.

I could open a discussion on Discourse to propose choosing one build backend for all core packaging packages.

[dstufft]

I don't think it should be too bad to ensure that the sdist is built with a compatibility shim setup.py. It's not zero cost though, as it means this project ends up on the hook for dealing with support requests that end up stemming from a secondary install mechanism.

That being said, downstream projects are going to have to figure out how to deal with a post 517 world sooner or later, and while I think the ideal situation is that there's some standard tool for helping with that, at the end of the day all of these projects are largely handled by volunteers, so there's a reasonable chance the different moving pieces move at different paces, and that might not be available.

[FFY00]

That being said, downstream projects are going to have to figure out how to deal with a post 517 world sooner or later, and while I think the ideal situation is that there's some standard tool for helping with that, at the end of the day all of these projects are largely handled by volunteers, so there's a reasonable chance the different moving pieces move at different paces, and that might not be available.

The more likely workflow would be using pypa/build, to build the wheels, and pradyunsg/installer, to install them. We still have the issue I presented above when building via PEP517, I think we should make sure all these tools and their dependencies use the same PEP517 backend.
I see no logical reason to use flit here over setuptools, neither the issue nor the PR presented any reasoning, it was just a "let's do it". pypa/packaging does not need anything from flit that setuptools doesn't provide, it was just a matter of personal preference... Which I do not believe is good enough reason to break the current workflows or add more complexity when doing it the PEP517 way. I sincerely do not understand why we are unnecessarily breaking things.