Fix Python Problems when using pip --no-build-isolation

[biergaizi]

Introduction

The current design of the Python extension allows the use of --no-build-isolation to bypass Python's own package management for several purposes, for example, it's suggested as the solution if pip cannot find the build-time dependency successfully, or if no network access to PyPI is available.

However, after further testing during the development of a new build script for openEMS-Project, it was discovered that --no-build-isolation does not actually work on many systems due to several problems.

This Pull Request introduces fixes to these problems.

On Linux-based systems, we also create a fresh network namespace without any network adapters, to create a severe test to ensure --no-build-isolation fulfills its intended purpose of installing Python pip packages, even without network access to PyPI.

Commits

python: define license_expression in setup.py instead.

setuptools v77 changed the format of the "license" field, without giving it a new name, making it a breaking change. A SPDX string is illegal according to setuptools v76 and lower, while the deprecated table format is illegal according to setuptools v77 and later.

This makes it difficult for users to build packages using --no-build-isolation, since most systems are still shipping earlier setuptools. Define license or license_expression in setup.py instead, and mark the license in pyproject.toml as dynamic [1].

See:

• [1] [Docs] Need migration guide for license expression (PEP 639) about TOML-table-based project.license deprecation if v77 unavailable  pypa/setuptools#4903

python/bootstrap: raise a custom RuntimeError

If bootstrap/setuptools_build_meta_custom.py fails because user has disabled build isolation in preference to a system-package build and the setuptools package is missing, the error message is difficult to understand. Raise our own RuntimeError instead.

python: improve setuptools_scm handling

The original setuptools_scm handling had compatibility problems under pip --no-build-isolation. Because the version_file attribute was unsupported before setuptools_scm v8, pip crashes. In --no-build-isolation mode, pip uses existing packages without any check, so setuptools_scm cannot be disabled by us by any means (even if we know it's broken).

setuptools_scm documentation suggests that new code should obtain package version via importlib.metadata.version(), the version_file field is largely obsolete anyway. Thus, this commit changes the implementation to obtain the __version__ string via importlib instead, the __version__.py file is removed. Instead, __fallback_version__.py is used if setuptools_scm fails, or if importlib is unsupported by the Python version installed.

python: add Cython to sys search path on macOS

The Cython package provided by macOS Homebrew is meant for building other Homebrew packages, so it's not available in the system's search path, making it useless. This commit adds Homebrew's Cython to the
search path to simplify pip --no-build-isolation installation.

CI: Test pip --no-build-isolation

This commit tests CI/CD with pip --no-build-isolation to ensure this use case actually works. On Linux-based systems, we also create a fresh network namespace without any network adapters, to ensure that --no-build-isolation fulfills its intended purpose of installing Python pip packages even without network access.

[biergaizi]

Merge ready, together with thliebig/openEMS#200.

Hopefully these patchsets will be the last refinement of the Python build process, which finally allows me to unblock openEMS-Projects's new Python workflow.

Ignore the "AlmaLinux Latest" test failure for now. It was caused by the broken VTK package in the Fedora EPEL repo. I've already reported the bug to the upstream.

[thliebig]

Thanks for all your effort @biergaizi but at the same time the amount of work required to get this pip workflow running seems a bit scary? I really have no idea what all the issues and problems are you are/were fighting...

[biergaizi]

If one wants to support only the latest systems running cutting-edge pip and Python, 50% of the code in setup.py can be deleted [1]. If it doesn't work, one would simply claim that their environment is unsupported, come back when they have fixed their environment (usually by compiling a new Python, and upgrading pip in an isolated user-level virtual environment). Officially, the Python policy is to only support the latest pip version, all other pip versions immediately become unsupported as soon as a new pip is released, and this is the convention used by most developers: packaging for the best conditions.

The work's focus was to add additional logic into setup.py, so that everything works by itself on even less common systems and environment combinations (even if end-user has an old pip version, an old h5py package with a non-standard name, a Cython installed to a non-standard location, the user decides to use a venv, the user decides not to use a venv, the machine has no network access to PyPI). These problems still exist back in the setup.py era, but they were typically worked around by the user manually (who know what they're doing), they were just never considered by us. But from the past records of GitHub Discussions and Issues, some users have only limited system management or Python knowledge, so I decide to move most logic into code with minimum intervention needed by end-users to increase their chance of success, or packaging for the worst conditions.

The GitHub CI script also wants developers to run the same commands on all systems, rather than running a separate workaround for each system. If one has to code the workaround anyway, writing it directly to setup.py fixes the same problem for end-users, not just the CI pipeline.

[1] I wanted to say 80%, but I realized that it would be false. The CXXFLAGS and LDFLAGS detection are still the largest functions in setup.py, unless one wants end-users to set them manually. In comparison a pure Python project doesn't need to worry about it.