pip.parse parsing all requirements files fails on platform specific wheel downloads

[keith]

🐞 bug report

Affected Rule

pip_parse

Is this a regression?

Yes, works in 0.40.0 with parse_all_requirements_files = False.

Description

This is a new issue for #2450 since I can't reopen that one.

🔬 Minimal Reproduction

Checkout this branch #2449 on a non linux x86_64 machine, run bazel query 'deps(...)'

🔥 Exception or Error

===== stdout start =====
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
===== stdout end =====
===== stderr start =====
ERROR: Could not find a version that satisfies the requirement torch==2.4.1+cpu (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.6.0, 2.6.0+cpu)
ERROR: No matching distribution found for torch==2.4.1+cpu
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python++python+python_3_11_aarch64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python++python+python_3_11_host/python', '-m', 'pip', '--isolated', 'download', '--only-binary=:all:', '--no-deps', '--index-url', 'https://pypi.org/simple', '--extra-index-url', 'https://download.pytorch.org/whl/cpu', '-r', '/tmp/tmpaeppdvmq']' returned non-zero exit status 1.
===== stderr end =====

🌍 Your Environment

Operating System: linux aarch64

Output of bazel version: 8.1.0

Rules_python version: f2941df

[keith]

primarily a new issue because all previous workarounds have been removed

[keith]

the example on that branch currently shows a universal requirements file where both of these entries exist:

torch==2.4.1 ; platform_machine != 'x86_64' \
torch==2.4.1+cpu ; platform_machine == 'x86_64' \

but the same goes for non-universal requirements files where you use requirements_by_platform and entirely separate files

[aignas]

Currently I am getting a lot of warnings with this example:

$ bazel query :test
Starting local Bazel server (8.1.0) and connecting to it...
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=1495132f30f722af1a091950088baea383fe39903db06b20e6936fd99402803e
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=30be2844d0c939161a11073bfbaf645f1c7cb43f62f46cc6e4df1c119fb2a798
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=36109432b10bd7163c9b30ce896f3c2cca1b86b9765f956a1594f0ff43091e2a
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=56ad2a760b7a7882725a1eebf5657abbb3b5144eb26bcb47b52059357463c548
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=fa27b048d32198cda6e9cff0bf768e8683d98743903b7e5d2b1f5098ded1d343
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=0c0a7cc4f7c74ff024d5a5e21230a01289b65346b27a626f6c815d94b4b8c955
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=1dd062d296fb78aa7cfab8690bf03704995a821b5ef69cfc807af5c0831b4202
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=2b03e20f37557d211d14e3fb3f71709325336402db132a1e0dd8b47392185baf
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=330e780f478707478f797fdc82c2a96e9b8c5f60b6f1f57bb6ad1dd5b1e7e97e
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=3a570e5c553415cdbddfe679207327b3a3806b21c6adea14fba77684d1619e97
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=3c99506980a2fb4b634008ccb758f42dd82f93ae2830c1e41f64536e310bf562
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=76a6fe7b10491b650c630bc9ae328df40f79a948296b41d3b087b29a8a63cbad
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=833490a28ac156762ed6adaa7c695879564fa2fd0dc51bcf3fdb2c7b47dc55e6
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=8800deef0026011d502c0c256cc4b67d002347f63c3a38cd8e45f1f445c61364
DEBUG: /home/aignas/.cache/bazel/_bazel_aignas/2f9c16c2e9c562c8dc36b29a0f32d3bb/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=c4f2c3c026e876d4dad7629170ec14fff48c076d6c2ae0e354ab3fdc09024f00
//:test

And it seems that the CPU packages are not fetched at all.

And I am suspecting that #2531 is the suspect. The reason is that:

1. We get the regular torch from the main PyPI and not search for it anymore.

However, I am not getting the same failure (because I am using linux right now) and it seems to be behaving...

What needs doing is changing the logic a little to continue searching for pkgs if we have not found the hashes of all of the packages. It probably would be more similar to how uv would behave? Does uv pip install -r requirements.txt works on both of the target platforms?

[aignas]

And after more thinking it does seem like in the #2531 I changed the behaviour from unsafe-best-match to first-index in the uv parlance: https://docs.astral.sh/uv/configuration/indexes/#searching-across-multiple-indexes

I see a few ways of dealing with this:

1. I revert the mentioned PR and this should fix your usecase. We are already matching by hashes, so in this case it is not a big deal.
2. I partially revert the mentioned PR and put it under a feature flag, where one can specify which strategy to use (first-index vs unsafe-best-match)
3. We implement the index overrides with env marker evaluation, like what uv has.

I think I like the combination of the 2 and 3 in this case. However, doing 1. just to fix the problem in the short term would be also good.

If someone is willing to submit a PR to do the revert, please add me as a reviewer.

[keith]

Does uv pip install -r requirements.txt works on both of the target platforms?

Yes.

Tested a quick revert here: #2628 but this didn't seem to fix the bazel query 'deps(...)' case on aarch64 linux. no change AFAICT.

[aignas]

Thanks for testing, really appreciate your help!

[keith]

I thought a bit more about the first-match vs unsafe-best-match comment and I don't think this is an ordering issue.

The command that's failing boils down to this:

echo "torch==2.4.1+cpu" > requirements.txt
python3 -m pip --isolated download --only-binary=:all: --no-deps --index-url https://pypi.org/simple --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt

And this fails on aarch64 linux since there is no wheel that satisfies this dependency.

If you manually add --platform linux_x86_64 it works since that is the compatible wheel platform.

So it seems to me that the bug is either this getting called at all, or this getting called without the platform arg that it should have. Wdyt?

[keith]

In my actual case atm I'm actually using requirements_by_platform, so it feels like in that example determining the correct --platform arg should be pretty reasonable. I'm not sure how that would work in the universal requirements file case, or when using experimental_target_platforms

[keith]

Looks like using --platform gets a bit more complicated because in torch's case you need linux_x86_64 not a manylinux variant, and for some other deps you need the opposite

[keith]

I guess that's a decision on the pip side to be strict about not letting those be ~the same, since uv happily normalizes the platform strings and still downloads it even if you pass a manylinux variant

[aignas]

Yeah, the platform tag is also something that you can specify multiple times to signal pip that your interpreter supports all of the platforms that you list. It is a very basic piece of machinery. So you could specify both, manylinux and linux platform strings here.

Regarding

So it seems to me that the bug is either this getting called at all, or this getting called without the platform arg that it should have. Wdyt?

I think that the line is getting called because you are doing bazel query. However, it is getting called because our simpleapi code could not find the wheel distributions in the index (due to what I mentioned previously) and is falling back to pip to download the right wheel, ideally we should use bazel downloader for this and this line at least would not error.

[keith]

ah ok thanks for explaining. i can poke at that.

one potential solution for this would be to support uv's --emit-index-url so you wouldn't have to reproduce their ordering logic here. it adds a comment like this:

jinja2==3.1.4 \
    --hash=sha256:4a3aee7acbbe7303aede8e9648d13b8bf88a429282aa6122a993f0ac800cb369 \
    --hash=sha256:bc5dd2abb727a5319567b7a813e6a2e7318c39f4f487cfe6c89c6f9c7d25197d
    # from https://pypi.org/simple