crate_universe: Render manifests with starlark

[uhthomas]

Fixes #1287

[UebelAndre]

Hey, I haven't done a real pass through this code yet but did see a ton of deletion I didn't expect. Why wouldn't this change initally be a new crates_vendor mode that writes the "Bazel lock files" and uses the new functionality here to render it into repository rules? It seems far less invasive and give folks a chance to easily transition their setups.

[uhthomas]

Hey, I haven't done a real pass through this code yet but did see a ton of deletion I didn't expect. Why wouldn't this change initally be a new crates_vendor mode that writes the "Bazel lock files" and uses the new functionality here to render it into repository rules? It seems far less invasive and give folks a chance to easily transition their setups.

Hmm, I suppose it makes sense to keep the existing functionally in addition to this?

The approach taken in this change differs dramatically as the build files aren't rendered, but instead interpreted which makes it fundamentally incompatible with the existing approach. Maintaining both would be tedious and strange.

I'm not at a computer at the minute so I can't give an example; the build files for the repository rules are almost all identical. They contain a snippet of the Cargo Bazel manifest and call a macro to interpret said snippet. Vendoring these would be pointless.

I mentioned this in the original issue: it is possible to see the result of the macros with bazel query --output=build @crate_universe_crate_index//... or bazel query --output=build @crate_universe_crate_index__toml-0.5.9//.... Might be helpful?

Happy to discuss and make changes where necessary :) Some work still needs to be done for feature parity.

[UebelAndre]

The approach taken in this change differs dramatically as the build files aren't rendered, but instead interpreted which makes it fundamentally incompatible with the existing approach. Maintaining both would be tedious and strange.

Perhaps I don't fully understand the changes being made. My current understanding of the fundamental change is this is a new set of functionality that could be added along side the existing functionality and things be deleted in a second step (making the initial implementation is less noisy). Could you link some updated docs that demonstrate the new workflow?

[uhthomas]

The approach taken in this change differs dramatically as the build files aren't rendered, but instead interpreted which makes it fundamentally incompatible with the existing approach. Maintaining both would be tedious and strange.

Perhaps I don't fully understand the changes being made. My current understanding of the fundamental change is this is a new set of functionality that could be added along side the existing functionality and things be deleted in a second step (making the initial implementation is less noisy). Could you link some updated docs that demonstrate the new workflow?

I'm on holiday this week unfortunately, though I would be happy to write some documentation when I'm back. The workflow should be almost identical, aside from bazel run @crate_universe_crate_index//:repin to repin instead of bazel sync.

I believe I wrote it this way because the cargo-bazel binary has been completely removed from the analysis phase. I guess it may be possible to write a new repository rule for this functionality instead of rewriting the old one?

[UebelAndre]

I'm on holiday this week unfortunately

Nothing unfortunate about being on holiday, enjoy the time off 😄.

I think this change needs more design discussion but understand the desire to remove the need for the cargo-bazel binary in the analysis phase. When you're back we can continue the conversation.

[uhthomas]

I'm back from holiday. Would you like to have a design discussion? It would be good to understand the problems and limitations with current system, what an ideal system would look like and how to get there.

Would it make more sense to build something based around bzlmod? There are some examples of how rules_js, rules_go/gazelle and rules_python are doing this:

• rules_js: https://github.com/aspect-build/rules_js/tree/aaae798f513c0a3784fbafb7e459da0ce971e704/e2e/bzlmod
• rules_go/gazelle: https://github.com/bazelbuild/bazel-central-registry/pull/129/files#diff-6cd9499b2a8d00d82ff339e9057167b612c510e020009e0c90cdbf73d8145755R24
• rules_python: bazel-contrib/rules_python@35391d9#diff-6136fc12446089c3db7360e923203dd114b6a1466252e71667c6791c20fe6bdcR9

side note: Need to considers forwarding environment variables as some git repositories may depend on ssh which may in turn depend on $PATH or $SSH_AUTH_SOCK.

[UebelAndre]

My expectation for this feature is to:

1. Add a new mode to crates_vendor, lock
2. Implement functionality for rendering BUILD files from the json file that represents the lock file that would now be output by crates_vendor.
3. Update the generated the crates module template to instead use the new rendering functionality to generate remote repository rules.

I think this is the most idiomatic form of what you're asking for. Updating/repinning is now an explicit step that does not happen in the analysis phase and the footprint of these dependencies are limited to just one file.

How does that sound?

[mering]

@uhthomas @UebelAndre Has there been any progress outside of this PR in the past two years?