feat(bzlmod): cross-platform builds without experimental_index_url

[aignas]

With this change we finally generate the same lock file within the
legacy code pip.parse code path and it allows to slowly transition to
using the new code path as much as possible without user doing anything.

This moves the selection of the host-compatible lock file from the
extension evaluation to the build phase - note, we will generate extra
repositories here which might not work on the host platform, however, if
the users are consuming the whl_library repos through the hub repo
only, then everything should work. A known issue is that it may break
bazel query and in these usecases it is advisable to use cquery
until we have sdist cross-building from source fully working.

Summary:

• feat: reuse the render_pkg_aliases for when filename is not known
  but platform is known
• feat: support generating the extra config settings required
• feat: get_whl_flag_versions now generates extra args for the rules
• feat: make lock file generation the same irrespective of the host
  platform
• test: add an extra test with multiple requirements files
• feat: support cross-platform builds using download_only = True in
  legacy setups

Note, that users depending on the naming of the whl libraries will need
to start using extra_hub_aliases attribute instead to keep their
setups not relying on this implementation detail.

Fixes #2268
Work towards #260

[aignas]

@keith, could you please check to see if this is fixing the issues you were seeing previously? The CI is not fully green, but the examples and unit tests are passing, so I think this might be ready for a spin.

[aignas]

I am happy that my selection of the mock ctx.os values was already validating the bug in question.

[keith]

it's possible it's just fighting getting the lockfile updated, but with this i see:

% bz mod graph --lockfile_mode=update
ERROR: module extension "pip" from "@@rules_python~//python/extensions:pip.bzl" does not generate repository "pip_deps_311_torch", yet it is imported as "pip_deps_311_torch" in the usage at .../MODULE.bazel:237:20 (did you mean 'pip_deps_311_munch'?). Type 'bazel help mod' for syntax and help.

looks like i hit that with or without experimental_index_url

[rickeylev]

download_only=True is also an option, right? Assuming one is OK with not having sdists

[aignas]

It could be, but I haven't tested with regular requirements files, let me add a unit test to see how that would get wired through.

EDIT: With a few extra modifications, I think we can actually support that. This means that we can resolve #260 and have the stabilization for experimental_index_url done as a separate ticket. I've added extra docs on this, please check. :)

[aignas]

it's possible it's just fighting getting the lockfile updated, but with this i see:

% bz mod graph --lockfile_mode=update
ERROR: module extension "pip" from "@@rules_python~//python/extensions:pip.bzl" does not generate repository "pip_deps_311_torch", yet it is imported as "pip_deps_311_torch" in the usage at .../MODULE.bazel:237:20 (did you mean 'pip_deps_311_munch'?). Type 'bazel help mod' for syntax and help.

looks like i hit that with or without experimental_index_url

@keith, I should have explained better in the PR message/changelog, that with this change you will have multiple torch variants registered and they will be pip_deps_311_torch__{'_'.join(target_platforms)}. The spoke repository naming is an implementation and should not be relied on too much. What are you using from the spoke repository? Some headers? In this case the best option would be to use @pip_deps//torch and whl_filegroup or similar to do that, what do you think?

[rickeylev]

That is only needed if cc_library is unhappy with an empty file.

cc_library is OK with empty srcs

[aignas]

OK, quick status update - once #2369 is merged, I will update this PR to be
backwards compatible (i.e. users will have to opt into the new behaviour on
0.38.0) and then merge this PR. Then we can cut a release. Once that is done,
we can have our 1.0 I think.

[aignas]

@keith, could you please check this PR again? I think it should work for your usecase now.

[keith]

Yes sorry I'm OOO but I will as soon as I'm back next week

[aignas]

Since the behaviour is opt-in, I'll merge the PR now and we can make further fixes later, but I am still looking forward to feedback next week.

[keith]

I think I could probably make whl_filegroup work, but it definitely breaks some use cases we're using today. For example today in my additive_build_content_file I do this:

_LIBS = [
    lib
    for lib in glob(
        [
            "site-packages/torch/lib/*.so*",
            "site-packages/torch.libs/*.so*",
            "site-packages/torch/lib/*.dylib",
        ],
        allow_empty = True,
        exclude = [
            "site-packages/torch/lib/*libgomp*",  # Handled below
            # Intel only, seem to be unnecessary for our use case
            "**/libtorchbind_test*",
            "**/libbackend_with_compiler*",
            "**/libjitbackend_test*",
        ],
    )
]

And then I expand multiple targets based on the _LIBS list below using list comprehensions. I don't think this has a clear 1:1 since if I had these as a filegroup it would not be the same

[keith]

seems like one option is this:

    extra_hub_aliases = {
        "torch": [
            "stock-torch",
            "stock-torch-headers",
            "stock-torch-api",
            "torchlib",
        ],
        "torchvision": ["torchvisionpy"],
    },

which pretty much gets me back to what i had before, is that one not recommended?