Replace `subpar` with a cross-platform solution

[UebelAndre]

🚀 feature request

Description

A clear and concise description of the problem or missing capability...

I opened this as a feature request because I otherwise didn't know how to raise open the issue

Right now the rules use subpar to generate some executable which are then committed back and reused for other rules:
https://github.com/bazelbuild/rules_python/blob/c37ba2215eccab53ae1da5f827a335281f81b9e1/packaging/BUILD#L45-L62

Is there a reason these couldn't be converted to using Bazel's python_zip_file functionality to accomplish the same thing? (introduced in bazelbuild/bazel#9453)

[UebelAndre]

If this is possible, would it solve for the outstanding issue on #344

[UebelAndre]

is the generated zip (zipapp? PEP-0441) cross platform? Does anyone know it's limitations there? I'm pretty surprised this is not a more advertised feature.

[hrfuller]

The .zip won't be cross-platform. Any rules_python pip dependencies are fetched only for the local platform. In some cases with universal wheels they could be cross-platform, but it isn't a guarantee. Having said that and without in depth knowledge I don't think subpar's are truly cross-platform either.

[UebelAndre]

Is there currently a cross platform implementation for self contained Python apps?

[bshashank]

The .zip won't be cross-platform. Any rules_python pip dependencies are fetched only for the local platform. In some cases with universal wheels they could be cross-platform, but it isn't a guarantee. Having said that and without in depth knowledge I don't think subpar's are truly cross-platform either.

I agree, subpar and --build_python_zip behave in a similar manner wrt cross-platform. The one advantage of subpar is that you can ask for the par to be unzipped before execution - which helps Python discover compiled code (.so or .dlls) more easily.

Since pip and wheel are both pure-python, I think both subpar and --build_python_zip would work in the same manner.

[UebelAndre]

Except in that the zips are producible on windows and the pars are not, right?

[hrfuller]

FWIW if cross-platform support is a necessity, which it may not be here, https://github.com/pantsbuild/pex has cross-platform support.

[UebelAndre]

Nice! Are there Bazel rules for that?

Also, do all Python rules currently work on all platforms? I thought the par files were a bottleneck for windows

[shridevinb]

I need to build .par files on windows. is there a workaround for it apart from using --build_python_zip?

[UebelAndre]

For what it's worth, I found @apache_incubator_heron//tools/rules/pex which seems to be a fork of https://github.com/benley/bazel_rules_pex at first glance but perhaps those rules can be used to finally address #2 ?

[UebelAndre]

I also came across https://github.com/arrdem/rules_zapp which seems like it might solve for some of the needs here.

[groodt]

The two .par files in the repo are only used to support the legacy pip_import mechanism as far as I recall.

Maybe @brandjon or other maintainers can confirm if this can be dropped in the next release and then the need for the subpar dependency in these rules is removed.

If not, the .par files used as part of rules_python only take a dependency on pure Python wheels (pip, wheel, setuptools).

See here.

The .par files could be replaced by --build_python_zip. No additional dependencies would be required. It would work cross-platform because these dependencies are pure Python. Note: it would NOT make all the rules automatically cross-platform. It would only make the rules "execute" on Windows, which is a step forward I guess.

If I recall correctly, the shebang in a zip from --build_python_zip is #!/usr/bin/env python, which is supported on POSIX and also the Windows Python launcher. Of course, be careful of Python2 vs Python 3 on the PATH (but you don't use python2 anymore right? ... right?)

This does of course require Python to be installed on the PATH of the host and is not hermetic / toolchain aware, but this is true of the current situation with subpar as well as far as I understand.

This is why I think the easiest path forward is to remove the legacy pip_import mechanism.