feat(uv): Add a clean-sheet dependency implementation

[arrdem]

Design doc

An implementation of pip based on consuming PEP-751 [1] like lockfiles. Specifically uv lockfiles, which contain internal dependency graph information that the PEP-751 specification labels optional.

Follows in the footsteps of rules_js's pnpm support by consuming a lockfile which contains enough information to produce materialize dependencies without performing any repository time operations which could be platform dependent.

Features

• Supports cross-platform builds of wheels
• Supports hermetic source builds of wheels
• Automatically handles dependency cycles
• Creates unified pip hubs which span virtualenv/dependency solution boundaries
• Pip library targets from deactivated venvs are incompatible
• Platform constraints on pip libraries do not prevent the creation of a library target
• Allows for --editable-like workflows
• Supports console_scripts entrypoints*

Example

# MODULE.bazel content
uv = use_extensioin("@aspect_rules_py//uv:extesion.bzl", "uv")
uv.declare_hub(hub_name = "pip")
    
uv.declare_venv(hub_name = "pip", venv_name = "a")
uv.lockfile(hub_name = "pip", venv_name = "a", lockfile = "third_party/py/venvs/uv-a.lock")

uv.declare_venv(hub_name = "pip", venv_name = "b")
uv.lockfile(hub_name = "pip", venv_name = "b", lockfile = "third_party/py/venvs/uv-b.toml")

use_repo(pip, "pip")

# BUILD.bazel content

py_venv_binary(
  name = "foo",
  srcs = [
    "foo.py",
  ],
  main = "foo.py",
  deps = [
    "@pip//cowsay", # Pull cowsay from the configured venv
  ],
  venv = "a", # Configure the default venv to be "a"; may be overriden at the CLI
)

The active venv state can be overriden at the cli by specifying --'@pip//venv=b' here for instance, or by using transitions to(re) set that same flag.

Appendix

[1] https://peps.python.org/pep-0751/
[2] https://peps.python.org/pep-0751/#locking-build-requirements-for-sdists

Changes are visible to end-users: yes

• Searched for relevant documentation and updated as needed: yes/no
• Breaking change (forces users to change their own code or config): yes/no
• Suggested release notes appear below: yes/no

To do list

• Audit the comments/FIXMEs for accuracy
• Go over zbarsky's nits
• Create a quick and dirty uv_lock rule
• Document the extension
• Document that we consider the extension is unstable
• Replace pip.parse internally entirely
• Add support for annotating replacement of pip deps with internal builds (--editable / vendoring)
• Implement an "all deps for all venvs" data source comparable to the rules_python equivalent
• Go back over the interpreter compatibility machinery and align it with rules_python's config settings for now
• Go back over the interpreter feature flags and align it with rules_python's config settings for now fix(toolchains): correctly register musl/freethreaded toolchains for workspace bazel-contrib/rules_python#3314
• Investigate implementing a dist_info target comparable to that which rules_python generates There isn't a great way to do this because dist-info is in our world part of the installed package, so any such target is just ignoring the PyInfo details on the fileset.
• Provide a pattern for implementing/activating venv entrypoints
• Implement conditional activation of deps
• Implement an "all active deps" target comparable to the rules_python all list
• Look into platform conditional deps & how they get represented
• Flatten the git log
• Ensure python_version transition consistency with rules_python
• Consider a feature flag to turn on rules_python's package name normalization so that migration is easier.
• Implement extra activation
• Match the rules_python @hub//package[:package] syntax?
• Get a toml.bzl working
• Toolchainize the uv dependency

Test plan

• Manually test flipping the venv command line flag
• Manually test flipping the venv transition attr
• Create py_venv_tests covering that different versions of the same package can be concurrently configured via different venvs
• Create a py_venv_test covering that Airflow or another package with dependency cycles can be provisioned
• Create a py_venv_binary embedded in and transitioned for a Linux OCI container across arch boundaries
• Create a py_venv_test covering overriding a pip dep with a 1stparty target

[aspect-workflows]

test-os:linux-bzl:8 (Test)

All tests were cache hits

38 tests (100.0%) were fully cached saving 51s.

---

test-os:linux-bzl:latest (Test)

All tests were cache hits

38 tests (100.0%) were fully cached saving 51s.

---

test-os:linux-bzl:8 (Test)

e2e

All tests were cache hits

14 tests (100.0%) were fully cached saving 10s.

---

test-os:linux-bzl:latest (Test)

e2e

All tests were cache hits

14 tests (100.0%) were fully cached saving 10s.

---

test-os:linux-bzl:8 (Test)

examples/uv_pip_compile

All tests were cache hits

1 test (100.0%) was fully cached saving 335ms.

---

test-os:linux-bzl:latest (Test)

examples/uv_pip_compile

All tests were cache hits

1 test (100.0%) was fully cached saving 335ms.

[dzbarsky]

generally %s is faster than .format because it avoids a function call and has simpler (faster) arg conversion. Also if you have only a single substitution it helps even more because you don't even need to construct a tuple. Up to you where to use it, but at least for the functions in the extension that are called repeatedly I would strongly consider it. (Honestly I pretty much always use it unless I have a complex/multiline template with 4+ args and/or using an arg multiple times)

[dzbarsky]

since you said you love nits...i left a bunch of nits. feel free to apply or disregard as you see fit!

[dzbarsky]

nit: name = current_stage.name :)

[dzbarsky]

":" + current_Stage.condition or ":%s" % current_stage.condition are both faster and less chars :)

[dzbarsky]

it might be a bit clearer to put the brackets in the expression here, especially since I think the way you did it I think the trailing ] isn't indented correctly? Here's a similar thing I did with build_deps here

[dzbarsky]

I saw you use setdefault like this a bunch, I think you can tweak the pattern slightly to avoid repeated lookups:

hub_specs.setdefault(hub.hub_name, {})[mod.name] = 1

[dzbarsky]

I think you could combine this with the previous loop and make hub_specs store hub_name -> mod_name(just check if you already have an entry and add toproblems) instead of of hub_name -> mod_name -> 1`. I guess it would make it harder to handle 3 modules all using the same name, but meh? It would probably lead to simpler usage in the rest of the extension

[dzbarsky]

maybe clearer to use single quotes on the outside and/or repr the k/v to avoid inner ones?

[dzbarsky]

i guess you could create this depset once and reuse it below

[arrdem]

TODO: Can we rip the uv toolchain out?

[arrdem]

, For later, more of Ignas' fine code we can leverage https://github.com/bazel-contrib/rules_python/pull/2692/files, bazel-contrib/rules_python#2856