Consider using the stable API when building wheels for CPython

[mattip]

I noticed your tweet about having lots of wheels (thanks for the PyPy one BTW). Perhaps you could consider creating stable ABI wheels? I think if you add

[bdist_wheel]
py-limited-api = cp34

to the setup.cfg then python setup.py bdist_wheel will build something like MarkupSafe-1.1.1-cp34-abi3-macosx_10_6_intel.whl which any cpython>-3.4 will support on macosx. This would save having to create a cpython-3.4, cpython3.5, ... wheels and would also mean you would future proof yourself for any new versions of cpython.

You might need to adjust the CI wheel build to only build the wheel once, since the various versions of CPython will build a wheel with the same name, I think bdist_wheel is unhappy if a wheel already exists.

[davidism]

We only support 3.6+, I'm assuming I could use cp36 there? Unfortunately, I still know very little about platform wheels, it's why I rely on cibuildwheel to do everything for me. @joerick, does cibuildwheel support this?

[joerick]

We don't support it directly, there's an open issue pypa/cibuildwheel#78, but you could set py-limited-api in setup.cfg, and then CIBW_BUILD=cp36-*, and you'd get one wheel that supports everything CPython 3.6+.

I'm actually not sure how PyPy would factor into this - @mattip, does PyPy support limited-api, or would you still need platform-specific wheels there?

[mattip]

We only support 3.6+, I'm assuming I could use cp36 there?

Yes. You might want to adjust your setup.cfg to mention cp36+, on PyPI there are wheels for 3.4, 3.5 and the identifier on the left also has those versions.

does PyPy support limited-api, or would you still need platform-specific wheels there?

Sorry, not yet. We have not really discussed how to do this. So it still needs a pypy-wheel per python version.

[joerick]

does PyPy support limited-api, or would you still need platform-specific wheels there?

Sorry, not yet. We have not really discussed how to do this. So it still needs a pypy-wheel per python version.

Makes sense. So then I think py-limited-api = cp36 in setup.cfg and CIBW_BUILD=cp36-* pp* would work... though I can't make any promises! Hopefully we'll have some official support some day :)

[davidism]

Thanks, I just subscribed to that issue. I'll try out those settings, I guess I would test it by trying to install the wheel on other Pythons? I'm still not 100% confident because I only have access to Linux x64 and Windows for testing.

[henryiii]

I guess I would test it by trying to install the wheel on other Pythons?

Yes, I'd do it one of two ways for now: for GHA, upload-artifact the built wheel, then make a matrix job that depends on the previous job, downloads the wheel with download-artifact, then runs setup-python and then runs the tests. Very similar to how you might release, see https://scikit-hep.org/developer/gha_wheels#publishing, but test instead of publish. Or if you like repeating yourself, make a chain of setup-pythons followed by installing and running the tests.

The other way, and generic for other platforms, would be to use nox. You'd setup-python (GHA, or however your other platform handles it) for all the versions you are interested in, one after the other, then you'd have a noxfile.py something like this:

import nox

@nox.session(python=["3.6", "3.7", "3.8", "3.9"])
def test_wheel(session):
    session.install("pytest")
    session.install("./wheelhouse/*.whl") # Might need to manually expand
    session.run("pytest")

Eventually cibuildwheel might use the "other" selected python platforms to just test the wheel produced on the earliest one, but this should work today.

[davidism]

I just tried building a cp36-abi3 wheel, then installed it on Python 3.7, and I was not able to import the speedups. Examining the wheel, the speedups were still named _speedups.cpython-36m-x86_64-linux-gnu.so, so I'm not sure how Python 3.7 would know it can use that.

Maybe I'm doing something wrong, it seems like the extension should be named _speedups.abi3.so. I put py-limited-api = cp36 in setup.cfg, ran python setup.py bdist_wheel, then pip installed the resulting wheel (which was tagged cp36-abi3) in a 3.7 venv. from markupsafe import _speedups results in an import error.

[mattip]

did you clean out any possible previous build?

[davidism]

I tried cloning a new copy of the repo, same result.

Also just tried removing some of the custom stuff in setup.py (which doesn't seem particularly necessary now anyway), since I thought that might be affecting it, but same result.

31dcd43 on the new-cibuildwheel-features branch has the py-limited-api line.

[henryiii]

The new features are only providing better support for the limited api in a matrix; cibuildwheel shouldn't be controlling the name before or after the changes (I think). It's definitely not going to fix something you can't do by hand.

Are you use you might not have used py-limited-api instead of py_limited_api? I'm also not completely sure that setting is available in setup.cfg, I've always seen it in setup.py, like for libsass.

[henryiii]

Ahh, nevermind, py_limited_abi needs to be set on the Extension - it isn't (only?) a setup() arg. So that has to be in the setup.py.

[mattip]

Ahh, right. I was patterning myself after this setup.cfg but forgot about this line in the setup.py

[davidism]

Thanks, that explains why the name isn't fixed. :-) Wish there was more documentation about this, I couldn't find anything in setuptools or wheel.

Just tried the extra line, I can build the cp36-abi tag then install and import it in 3.7.