From f7f04e3aa6e32a918d43476953f996335143fcd2 Mon Sep 17 00:00:00 2001 From: chrislovecnm Date: Mon, 3 Apr 2023 13:17:49 -0600 Subject: [PATCH 1/2] Updating documentation for bzlmod - Updated primary README.md to include documentation for using bzlmod or using a WORKSPACE file. - Updated gazelle/README.md to include documentation for only using bzmod and provided a link to the older docs. - Included other general updates for the gazelle documentation. --- README.md | 62 ++++++++++++++++++++++++++++++-- gazelle/README.md | 91 +++++++++++++++++++++++++++++++++++------------ 2 files changed, 128 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 07acaf8e19..98fe2c72fb 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,45 @@ contribute](CONTRIBUTING.md) page for information on our development workflow. ## Getting started +The next two sections cover using `rules_python` with bzlmod and +the older way of configuring bazel with a `WORKSPACE` file. + +### Using bzlmod + +To import rules_python in your project, you first need to add it to your +`MODULES.bazel` file, using the snippet provided in the +[release you choose](https://github.com/bazelbuild/rules_python/releases). + +#### Toolchain registration with bzlmod + +To register a hermetic Python toolchain rather than rely on a system-installed interpreter for runtime execution, you can add to the `MODULES.bazel` file: + +```python +# Find the latest version number here: https://github.com/bazelbuild/rules_python/releases +# and change the version number if needed in the line below. +bazel_dep(name = "rules_python", version = "0.20.0") + +pip = use_extension("@rules_python//python:extensions.bzl", "pip") + +pip.parse( + name = "pip", + requirements_lock = "//:requirements_lock.txt", +) + +use_repo(pip, "pip") + +# (Optional) Register a specific python toolchain instead of using the host version +python = use_extension("@rules_python//python:extensions.bzl", "python") + +use_repo(python, "python3_10_toolchains") + +register_toolchains( + "@python3_10_toolchains//:all", +) +``` + +### Using a WORKSPACE file + To import rules_python in your project, you first need to add it to your `WORKSPACE` file, using the snippet provided in the [release you choose](https://github.com/bazelbuild/rules_python/releases) @@ -53,7 +92,7 @@ http_archive( ) ``` -### Toolchain registration +#### Toolchain registration To register a hermetic Python toolchain rather than rely on a system-installed interpreter for runtime execution, you can add to the `WORKSPACE` file: @@ -118,6 +157,22 @@ target in the appropriate wheel repo. ### Installing third_party packages +#### Using bzlmod + +To add pip dependencies to your `MODULES.bazel` file, use the `pip.parse` extension, and call it to create the +central external repo and individual wheel external repos. + +```python +pip.parse( + name = "my_deps", + requirements_lock = "//:requirements_lock.txt", +) + +use_repo(pip, "pip") +``` + +#### Using a WORKSPACE file + To add pip dependencies to your `WORKSPACE`, load the `pip_parse` function, and call it to create the central external repo and individual wheel external repos. @@ -137,14 +192,15 @@ load("@my_deps//:requirements.bzl", "install_deps") install_deps() ``` +#### pip rules + Note that since `pip_parse` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no information about the Python toolchain and cannot enforce that the interpreter used to invoke pip matches the interpreter used to run `py_binary` targets. By default, `pip_parse` uses the system command `"python3"`. This can be overridden by passing the `python_interpreter` attribute or `python_interpreter_target` attribute to `pip_parse`. -You can have multiple `pip_parse`s in the same workspace. This will create multiple external repos that have no relation to -one another, and may result in downloading the same wheels multiple times. +You can have multiple `pip_parse`s in the same workspace. This will create multiple external repos that have no relation to one another, and may result in downloading the same wheels multiple times. As with any repository rule, if you would like to ensure that `pip_parse` is re-executed in order to pick up a non-hermetic change to your environment (e.g., diff --git a/gazelle/README.md b/gazelle/README.md index 0081701241..f19bba5f8a 100644 --- a/gazelle/README.md +++ b/gazelle/README.md @@ -1,21 +1,48 @@ # Python Gazelle plugin +[Gazelle](https://github.com/bazelbuild/bazel-gazelle) +is a build file generator for Bazel projects. It can create new BUILD.bazel files for a project that follows language conventions, and it can update existing build files to include new sources, dependencies, and options. + +Gazelle may be run by Bazel using the gazelle rule, or it may be installed and run as a command line tool. + This directory contains a plugin for [Gazelle](https://github.com/bazelbuild/bazel-gazelle) -that generates BUILD file content for Python code. +that generates BUILD files content for Python code. + +The following instructions are for when you use [bzlmod](https://docs.bazel.build/versions/5.0.0/bzlmod.html). +Please refer to older documentation that includes instructions on how to use Gazelle +without using bzlmod as your dependency manager. + +## Example -It requires Go 1.16+ to compile. +We have an example of using Gazelle with Python located [here](https://github.com/bazelbuild/rules_python/tree/main/examples/build_file_generation). -## Installation +## Adding Gazelle to your project -First, you'll need to add Gazelle to your `WORKSPACE` file. -Follow the instructions at https://github.com/bazelbuild/bazel-gazelle#running-gazelle-with-bazel +First, you'll need to add Gazelle to your `MODULES.bazel` file. +Get the current version of Gazelle from there releases here: https://github.com/bazelbuild/bazel-gazelle/releases/ -Next, we need to fetch the third-party Go libraries that the python extension -depends on. -See the installation `WORKSPACE` snippet on the Releases page: -https://github.com/bazelbuild/rules_python/releases +See the installation `MODULE.bazel` snippet on the Releases page: +https://github.com/bazelbuild/rules_python/releases in order to configure rules_python. + +You will also need to add the `bazel_dep` for configuration for `rules_python_gazelle_plugin`. + +Here is an example snippet of a `MODULE.bazel` file. + +```starlark +# The following stanza defines the dependency rules_python. +bazel_dep(name = "rules_python", version = "0.20.0") + +# The following stanza defines the dependency rules_python. +# For typical setups you set the version. +bazel_dep(name = "rules_python_gazelle_plugin", version = "0.20.0") + +# The following stanza defines the dependency rules_python. +bazel_dep(name = "gazelle", version = "0.30.0", repo_name = "bazel_gazelle") +``` +You will also need to do the other usual configuration for `rules_python`. In your +`MODULE.bazel` file. Next, we'll fetch metadata about your Python dependencies, so that gazelle can determine which package a given import statement comes from. This is provided @@ -157,11 +184,29 @@ Next, all source files are collected into the `srcs` of the `py_library`. Finally, the `import` statements in the source files are parsed, and dependencies are added to the `deps` attribute. -### Tests +### Unit Tests + +A `py_test` target is added to the BUILD file when gazelle encounters +a file named `__test__.py`. +Often Python unit test files are named with the suffix `_test`. +For example if we had a folder that is a package named "foo" we could have a Python file named `foo_test.py` +and gazelle would create a `py_test` block for the file. -Python test files are those ending in `_test.py`. +The following is an example of a `py_test` target that gazelle would add when +it encounters a file named `__test__.py`. + +```starlark +py_test( + name = "build_file_generation_test", + srcs = ["__test__.py"], + main = "__test__.py", + deps = [":build_file_generation"], +) +``` -A `py_test` target is added containing all test files as `srcs`. +You can control the naming convention for test targets by adding a gazelle directive named +`# gazelle:python_test_naming_convention`. See the instructions in the section above that +covers directives. ### Binaries @@ -170,16 +215,18 @@ of a Python program. A `py_binary` target will be created, named `[package]_bin`. -## Developing on the extension +## Developer Notes -Gazelle extensions are written in Go. Ours is a hybrid, which also spawns -a Python interpreter as a subprocess to parse python files. +Gazelle extensions are written in Go. This gazelle plugin is a hybrid, as it uses Go to execute a +Python interpreter as a subprocess to parse Python source files. +See the gazelle documentation https://github.com/bazelbuild/bazel-gazelle/blob/master/extend.md +for more information on extending Gazelle. -The Go dependencies are managed by the go.mod file. -After changing that file, run `go mod tidy` to get a `go.sum` file, -then run `bazel run //:update_go_deps` to convert that to the `gazelle/deps.bzl` file. -The latter is loaded in our `/WORKSPACE` to define the external repos -that we can load Go dependencies from. +If you add a new Go dependencies the plugin source code you need to "tidy" the go.mod file. +After changing that file, run `go mod tidy` or `bazel run @go_sdk//:bin/go -- mod tidy` +to update the go.mod and go.sum files. Then run `bazel run //:update_go_deps` to have gazelle +add the new dependenies to the deps.bzl file. The deps.bzl file is used as defined in our /WORKSPACE +to include the external repos Bazel loads Go dependencies from. -Then after editing Go code, run `bazel run //:gazelle` to generate/update -go_* rules in the BUILD.bazel files in our repo. +Then after editing Go code, run `bazel run //:gazelle` to generate/update the rules in the +BUILD.bazel files in our repo. From 1b42af1dba3ef0e9c2a0bfc1968c05602607a6a1 Mon Sep 17 00:00:00 2001 From: chrislovecnm Date: Mon, 3 Apr 2023 13:59:56 -0600 Subject: [PATCH 2/2] PR review updates --- README.md | 6 ++++-- gazelle/README.md | 12 ++++++------ 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 98fe2c72fb..089837de7d 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,8 @@ To register a hermetic Python toolchain rather than rely on a system-installed i # and change the version number if needed in the line below. bazel_dep(name = "rules_python", version = "0.20.0") +# You do not have to use pip for the toolchain, but most people +# will use it for the dependency management. pip = use_extension("@rules_python//python:extensions.bzl", "pip") pip.parse( @@ -60,7 +62,7 @@ pip.parse( use_repo(pip, "pip") -# (Optional) Register a specific python toolchain instead of using the host version +# Register a specific python toolchain instead of using the host version python = use_extension("@rules_python//python:extensions.bzl", "python") use_repo(python, "python3_10_toolchains") @@ -168,7 +170,7 @@ pip.parse( requirements_lock = "//:requirements_lock.txt", ) -use_repo(pip, "pip") +use_repo(pip, "my_deps") ``` #### Using a WORKSPACE file diff --git a/gazelle/README.md b/gazelle/README.md index f19bba5f8a..e9a8052353 100644 --- a/gazelle/README.md +++ b/gazelle/README.md @@ -20,7 +20,7 @@ We have an example of using Gazelle with Python located [here](https://github.co ## Adding Gazelle to your project First, you'll need to add Gazelle to your `MODULES.bazel` file. -Get the current version of Gazelle from there releases here: https://github.com/bazelbuild/bazel-gazelle/releases/ +Get the current version of Gazelle from there releases here: https://github.com/bazelbuild/bazel-gazelle/releases/. See the installation `MODULE.bazel` snippet on the Releases page: @@ -28,7 +28,7 @@ https://github.com/bazelbuild/rules_python/releases in order to configure rules_ You will also need to add the `bazel_dep` for configuration for `rules_python_gazelle_plugin`. -Here is an example snippet of a `MODULE.bazel` file. +Here is a snippet of a `MODULE.bazel` file. ```starlark # The following stanza defines the dependency rules_python. @@ -41,7 +41,7 @@ bazel_dep(name = "rules_python_gazelle_plugin", version = "0.20.0") # The following stanza defines the dependency rules_python. bazel_dep(name = "gazelle", version = "0.30.0", repo_name = "bazel_gazelle") ``` -You will also need to do the other usual configuration for `rules_python`. In your +You will also need to do the other usual configuration for `rules_python` in your `MODULE.bazel` file. Next, we'll fetch metadata about your Python dependencies, so that gazelle can @@ -188,8 +188,8 @@ dependencies are added to the `deps` attribute. A `py_test` target is added to the BUILD file when gazelle encounters a file named `__test__.py`. -Often Python unit test files are named with the suffix `_test`. -For example if we had a folder that is a package named "foo" we could have a Python file named `foo_test.py` +Often, Python unit test files are named with the suffix `_test`. +For example, if we had a folder that is a package named "foo" we could have a Python file named `foo_test.py` and gazelle would create a `py_test` block for the file. The following is an example of a `py_test` target that gazelle would add when @@ -222,7 +222,7 @@ Python interpreter as a subprocess to parse Python source files. See the gazelle documentation https://github.com/bazelbuild/bazel-gazelle/blob/master/extend.md for more information on extending Gazelle. -If you add a new Go dependencies the plugin source code you need to "tidy" the go.mod file. +If you add new Go dependencies to the plugin source code, you need to "tidy" the go.mod file. After changing that file, run `go mod tidy` or `bazel run @go_sdk//:bin/go -- mod tidy` to update the go.mod and go.sum files. Then run `bazel run //:update_go_deps` to have gazelle add the new dependenies to the deps.bzl file. The deps.bzl file is used as defined in our /WORKSPACE