Running tests in Fedora packaging

[LecrisUT]

Lately I have been reviewing a bunch of jaraco.* packages in Fedora, and every now-and-then I get issues with running the tests where the non-src-layout interferes with /usr/bin/pytest calls. Sometimes I work around it by changing the --import-mode prepend, sometimes I have to run python3 -m pytest.

One recent example was in jaraco.packaging where it errors as:

Details

__________________ [doctest] packaging.metadata.hunt_down_url __________________
019 
020 Given project metadata, figure out what the package URL is.
021 
022 >>> hunt_down_url(load('.'))
UNEXPECTED EXCEPTION: ModuleNotFoundError("No module named 'packaging.version'")
Traceback (most recent call last):
  File "/usr/lib64/python3.13/doctest.py", line 1395, in __run
    exec(compile(example.source, filename, "single",
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                 compileflags, True), test.globs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<doctest packaging.metadata.hunt_down_url[0]>", line 1, in <module>
  File "/builddir/build/BUILD/python-jaraco-packaging-10.2.2-build/jaraco_packaging-10.2.2/jaraco/packaging/metadata.py", line 15, in load
    return util.project_wheel_metadata(source_dir, isolated, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/build/util.py", line 42, in project_wheel_metadata
    with DefaultIsolatedEnv() as env:
         ~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/build/env.py", line 90, in __enter__
    self._env_backend = _PipBackend()
                        ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/build/env.py", line 153, in __init__
    self._create_with_virtualenv = not self._has_valid_outer_pip and self._has_virtualenv
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/functools.py", line 1037, in __get__
    val = self.func(instance)
  File "/usr/lib/python3.13/site-packages/build/env.py", line 162, in _has_valid_outer_pip
    return _has_dependency('pip', '22.3')
  File "/usr/lib/python3.13/site-packages/build/env.py", line 46, in _has_dependency
    from packaging.version import Version
ModuleNotFoundError: No module named 'packaging.version'
/builddir/build/BUILD/python-jaraco-packaging-10.2.2-build/jaraco_packaging-10.2.2/jaraco/packaging/metadata.py:22: UnexpectedException

Basically it interpreted jaraco.packaging as packaging

I hope that a src-layout can mitigate the need for --import-mode importlib, but also it might need relative imports all around as well?

[jaraco]

I'm disinterested in implementing the src layout. I've long bemoaned the undesirable effect it has to move the primary functionality of the project into a subdirectory. For similar reasons, I'm also not a fan of the flat layout, which is why I've published the essential layout, which goes into a bit more detail about the problems with src and flat layouts. That said, I'm not ready yet to migrate projects to Coherent and the essential layout.

I've put a lot of work into trying to structure projects so they work consistently across all of them. I've learned from pytest that the import-mode of importlib is the preferred way forward (other modes are legacy and should only be used if necessary), so I'm also disinterested in switching to those less-preferred modes. Let's see if we can find a better solution.

As a side note, when switching to the coherent system and the essential layout, these issues all go away because the import finder is dead simple.

The good news is that I've put a lot of work into making sure all of the packages' tests are running under pytest using the importlib mode, so we know it's possible to run the tests using that mode.

It's not yet clear to me what it is about the Fedora packaging scheme that's violating the pytest discovery mechanism. Are you able to describe the steps one could take to simulate what Fedora packaging is doing so we could replicate the issues with a package like jaraco.packaging?

[LecrisUT]

👍 to whatever layout works best for you and the skeleton, but could we investigate the source of failure together and try to find some downstream-only patches that could help workaround these issues.

Well one issue would be with the outdated version of pytest in epel 8 or 9, which I don't think they can be updated in Fedora repos. At least the Fedora branches seem to work fine, but I can't confirm if it's because the importlib is working correctly or if the cwd python modules are being picked up, the latter of which we want to avoid.

Basically the workflow is pip wheel -> install wheel to staging directory -> pytest with PYTHONPATH pointing to staging directory. The test is meant to make sure all relevant python files are installed correctly and there should not be any files being accidentally picked up due to cwd (default is the extracted sdist). Any suggestions on how to ensure this part works well? We want to avoid cding to a dummy folder because the pytest.ini would not be picked up.

My current thoughts is if it's possible to modify the structure downstream-only. E.g. because setuptools is too old on epel9 to support PEP621 (because it's a fundamental dependency for packages like pip), I've resorted to switching the build backend to hatchling (which is more updated and supports PEP621) by patching the pyproject.toml. Wouldn't it be possible to patch these similarly for these packages?

There are a few ways to reproduce the build environment in Fedora and epel:

• container building with --no-build-isolation and installing the python packages manually
• building the downstream recipes from src.fedoraproject.org within copr and following the buuld.log
• adding a CI automation through packit which mimics all Fedora packaging environments in the upstream repo. I can show this setup in a PR of a repo and see what your thoughts are of its workflow. This would be the top preference because you would be able to debug issues directly in the git repo

Let me know how I can best help you investigate this issue or if there are any Fedora packaging questions I can help with.

Edit: added more details on Fedora build workflow

[jaraco]

I'm just getting caught up on old emails. Sorry for the delay.

E.g. because setuptools is too old on epel9 to support PEP621 (because it's a fundamental dependency for packages like pip), I've resorted to switching the build backend to hatchling (which is more updated and supports PEP621) by patching the pyproject.toml. Wouldn't it be possible to patch these similarly for these packages?

That sounds like a possibility, though I'm unsure what setuptools-specific behaviors might be problemmatic. For most projects I maintain, they are pure-Python packages, so are likely suitable for migration to hatchling. I'd be interested to know how well that works - I might be interested in migrating skeleton projects to hatchling (maybe even Setuptools itself O_O).

Let me know how I can best help you investigate this issue or if there are any Fedora packaging questions I can help with.

Either option 3 or 1 seem fairly straightforward. If you can prepare the PR or put together a Dockerfile that I can use to replicate the issue, I'll be happy to investigate the issue.