Skip to content

fix: compile_pip_requirements test from external repositories #1124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
f0rmiga merged 1 commit into from
Apr 25, 2023
Merged

fix: compile_pip_requirements test from external repositories #1124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .bazelci/presubmit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -508,3 +508,41 @@ tasks:
name: ignore_root_user_error integration tests on Windows
working_directory: tests/ignore_root_user_error
platform: windows

integration_compile_pip_requirements_test_from_external_repo_ubuntu_min:
<<: *minimum_supported_version
name: compile_pip_requirements test from external repo on Ubuntu using minimum supported Bazel version
working_directory: tests/compile_pip_requirements_test_from_external_workspace
platform: ubuntu2004
shell_commands:
# Assert that @external_repository//:requirements_test does the right thing.
- "bazel test @external_repository//..."
integration_compile_pip_requirements_test_from_external_repo_ubuntu:
name: compile_pip_requirements test from external repo on Ubuntu
working_directory: tests/compile_pip_requirements_test_from_external_workspace
platform: ubuntu2004
shell_commands:
# Assert that @external_repository//:requirements_test does the right thing.
- "bazel test @external_repository//..."
integration_compile_pip_requirements_test_from_external_repo_debian:
name: compile_pip_requirements test from external repo on Debian
working_directory: tests/compile_pip_requirements_test_from_external_workspace
platform: debian11
shell_commands:
# Assert that @external_repository//:requirements_test does the right thing.
- "bazel test @external_repository//..."
integration_compile_pip_requirements_test_from_external_repo_macos:
name: compile_pip_requirements test from external repo on macOS
working_directory: tests/compile_pip_requirements_test_from_external_workspace
platform: macos
shell_commands:
# Assert that @external_repository//:requirements_test does the right thing.
- "bazel test @external_repository//..."
integration_compile_pip_requirements_test_from_external_repo_windows:
name: compile_pip_requirements test from external repo on Windows
working_directory: tests/compile_pip_requirements_test_from_external_workspace
platform: windows
shell_commands:
# Assert that @external_repository//:requirements_test does the right thing.
- "bazel test @external_repository//..."

2 changes: 1 addition & 1 deletion examples/build_file_generation/gazelle_python.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,4 +115,4 @@ manifest:
pip_repository:
name: pip
use_pip_repository_aliases: true
integrity: 85f073e37e31339508aaaf5e0d5472adae5148fd5f054e9cc586343c026660e1
integrity: 030d6d99b56c32d6577e616b617260d0a93588af791269162e43391a5a4fa576
Copy link
Collaborator

@aignas aignas Mar 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means that the integrity will change when users upgrade to a new rules_python version and recompile their requirements. @f0rmiga, what do you think about this? I am wondering if this is the right behaviour to need to regenerate the gazelle_python.yaml manifest when the requirements.txt changes but the contents themselves do not.

Copy link
Contributor

@chrislovecnm chrislovecnm Apr 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Rasrack, I agree with @aignas. Can we do this without forcing an upgrade? Are you able to follow his recommendation?

Copy link
Contributor Author

@Rasrack Rasrack Apr 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrislovecnm I don't think it's possible as the integrity will change as the annotations in the lock file change. However, my opinion is that the lock file should be generated without annotations but I think that is a separate issue.

Copy link
Member

@f0rmiga f0rmiga Apr 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integrity doesn't change on every rules_python upgrade. I made it so that only when the source code of the modules mapping, or the locked requirements.txt change.

2 changes: 1 addition & 1 deletion examples/build_file_generation/requirements_lock.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ click==8.1.3 \
flask==2.2.2 \
--hash=sha256:642c450d19c4ad482f96729bd2a8f6d32554aa1e231f4f6b4e7e5264b16cca2b \
--hash=sha256:b9c46cc36662a7949f34b52d8ec7bb59c0d74ba08ba6cb9ce9adc1d8676d9526
# via -r ./requirements.in
# via -r requirements.in
importlib-metadata==5.2.0 \
--hash=sha256:0eafa39ba42bf225fc00e67f701d71f85aead9f878569caf13c3724f704b970f \
--hash=sha256:404d48d62bba0b7a77ff9d405efd91501bef2e67ff4ace0bed40a0cf28c3c7cd
Expand Down
2 changes: 1 addition & 1 deletion examples/build_file_generation/requirements_windows.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ colorama==0.4.6 \
flask==2.2.2 \
--hash=sha256:642c450d19c4ad482f96729bd2a8f6d32554aa1e231f4f6b4e7e5264b16cca2b \
--hash=sha256:b9c46cc36662a7949f34b52d8ec7bb59c0d74ba08ba6cb9ce9adc1d8676d9526
# via -r ./requirements.in
# via -r requirements.in
importlib-metadata==5.2.0 \
--hash=sha256:0eafa39ba42bf225fc00e67f701d71f85aead9f878569caf13c3724f704b970f \
--hash=sha256:404d48d62bba0b7a77ff9d405efd91501bef2e67ff4ace0bed40a0cf28c3c7cd
Expand Down
12 changes: 6 additions & 6 deletions examples/bzlmod/requirements_lock.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,12 @@ platformdirs==2.6.0 \
pylint==2.15.9 \
--hash=sha256:18783cca3cfee5b83c6c5d10b3cdb66c6594520ffae61890858fe8d932e1c6b4 \
--hash=sha256:349c8cd36aede4d50a0754a8c0218b43323d13d5d88f4b2952ddfe3e169681eb
# via -r ./requirements.in
# via -r requirements.in
python-dateutil==2.8.2 \
--hash=sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86 \
--hash=sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9
# via
# -r ./requirements.in
# -r requirements.in
# s3cmd
python-magic==0.4.27 \
--hash=sha256:c1ba14b08e4a5f5c31a302b7721239695b2f0f058d125bd5ce1ee36b9d9d3c3b \
Expand Down Expand Up @@ -120,11 +120,11 @@ pyyaml==6.0 \
requests==2.25.1 \
--hash=sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804 \
--hash=sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e
# via -r ./requirements.in
# via -r requirements.in
s3cmd==2.1.0 \
--hash=sha256:49cd23d516b17974b22b611a95ce4d93fe326feaa07320bd1d234fed68cbccfa \
--hash=sha256:966b0a494a916fc3b4324de38f089c86c70ee90e8e1cae6d59102103a4c0cc03
# via -r ./requirements.in
# via -r requirements.in
setuptools==65.6.3 \
--hash=sha256:57f6f22bde4e042978bcd50176fdb381d7c21a9efa4041202288d3737a0c6a54 \
--hash=sha256:a7620757bf984b58deaf32fc8a4577a9bbc0850cf92c20e1ce41c38c19e5fb75
Expand All @@ -136,7 +136,7 @@ six==1.16.0 \
tabulate==0.9.0 \
--hash=sha256:0095b12bf5966de529c0feb1fa08671671b3368eec77d7ef7ab114be2c068b3c \
--hash=sha256:024ca478df22e9340661486f85298cff5f6dcdba14f3813e8830015b9ed1948f
# via -r ./requirements.in
# via -r requirements.in
tomli==2.0.1 \
--hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
--hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
Expand Down Expand Up @@ -224,4 +224,4 @@ wrapt==1.14.1 \
yamllint==1.28.0 \
--hash=sha256:89bb5b5ac33b1ade059743cf227de73daa34d5e5a474b06a5e17fc16583b0cf2 \
--hash=sha256:9e3d8ddd16d0583214c5fdffe806c9344086721f107435f68bad990e5a88826b
# via -r ./requirements.in
# via -r requirements.in
12 changes: 6 additions & 6 deletions examples/bzlmod/requirements_windows.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,12 @@ platformdirs==2.6.0 \
pylint==2.15.9 \
--hash=sha256:18783cca3cfee5b83c6c5d10b3cdb66c6594520ffae61890858fe8d932e1c6b4 \
--hash=sha256:349c8cd36aede4d50a0754a8c0218b43323d13d5d88f4b2952ddfe3e169681eb
# via -r ./requirements.in
# via -r requirements.in
python-dateutil==2.8.2 \
--hash=sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86 \
--hash=sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9
# via
# -r ./requirements.in
# -r requirements.in
# s3cmd
python-magic==0.4.27 \
--hash=sha256:c1ba14b08e4a5f5c31a302b7721239695b2f0f058d125bd5ce1ee36b9d9d3c3b \
Expand Down Expand Up @@ -124,11 +124,11 @@ pyyaml==6.0 \
requests==2.25.1 \
--hash=sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804 \
--hash=sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e
# via -r ./requirements.in
# via -r requirements.in
s3cmd==2.1.0 \
--hash=sha256:49cd23d516b17974b22b611a95ce4d93fe326feaa07320bd1d234fed68cbccfa \
--hash=sha256:966b0a494a916fc3b4324de38f089c86c70ee90e8e1cae6d59102103a4c0cc03
# via -r ./requirements.in
# via -r requirements.in
setuptools==65.6.3 \
--hash=sha256:57f6f22bde4e042978bcd50176fdb381d7c21a9efa4041202288d3737a0c6a54 \
--hash=sha256:a7620757bf984b58deaf32fc8a4577a9bbc0850cf92c20e1ce41c38c19e5fb75
Expand All @@ -140,7 +140,7 @@ six==1.16.0 \
tabulate==0.9.0 \
--hash=sha256:0095b12bf5966de529c0feb1fa08671671b3368eec77d7ef7ab114be2c068b3c \
--hash=sha256:024ca478df22e9340661486f85298cff5f6dcdba14f3813e8830015b9ed1948f
# via -r ./requirements.in
# via -r requirements.in
tomli==2.0.1 \
--hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
--hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
Expand Down Expand Up @@ -228,4 +228,4 @@ wrapt==1.14.1 \
yamllint==1.28.0 \
--hash=sha256:89bb5b5ac33b1ade059743cf227de73daa34d5e5a474b06a5e17fc16583b0cf2 \
--hash=sha256:9e3d8ddd16d0583214c5fdffe806c9344086721f107435f68bad990e5a88826b
# via -r ./requirements.in
# via -r requirements.in
8 changes: 4 additions & 4 deletions examples/pip_install/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
boto3==1.14.63 \
--hash=sha256:25c716b7c01d4664027afc6a6418a06459e311a610c7fd39a030a1ced1b72ce4 \
--hash=sha256:37158c37a151eab5b9080968305621a40168171fda9584d50a309ceb4e5e6964
# via -r ./requirements.in
# via -r requirements.in
botocore==1.17.63 \
--hash=sha256:40f13f6c9c29c307a9dc5982739e537ddce55b29787b90c3447b507e3283bcd6 \
--hash=sha256:aa88eafc6295132f4bc606f1df32b3248e0fa611724c0a216aceda767948ac75
Expand Down Expand Up @@ -84,7 +84,7 @@ pyyaml==6.0 \
s3cmd==2.1.0 \
--hash=sha256:49cd23d516b17974b22b611a95ce4d93fe326feaa07320bd1d234fed68cbccfa \
--hash=sha256:966b0a494a916fc3b4324de38f089c86c70ee90e8e1cae6d59102103a4c0cc03
# via -r ./requirements.in
# via -r requirements.in
s3transfer==0.3.7 \
--hash=sha256:35627b86af8ff97e7ac27975fe0a98a312814b46c6333d8a6b889627bcd80994 \
--hash=sha256:efa5bd92a897b6a8d5c1383828dca3d52d0790e0756d49740563a3fb6ed03246
Expand All @@ -100,11 +100,11 @@ six==1.16.0 \
tree-sitter==0.20.0 ; sys_platform != "win32" \
--hash=sha256:1940f64be1e8c9c3c0e34a2258f1e4c324207534d5b1eefc5ab2960a9d98f668 \
--hash=sha256:51a609a7c1bd9d9e75d92ee128c12c7852ae70a482900fbbccf3d13a79e0378c
# via -r ./requirements.in
# via -r requirements.in
urllib3==1.25.11 \
--hash=sha256:8d7eaa5a82a1cac232164990f04874c594c9453ec55eef02eab885aa02fc17a2 \
--hash=sha256:f5321fbe4bf3fefa0efd0bfe7fb14e90909eb62a48ccda331726b4319897dd5e
# via botocore
yamllint==1.26.3 \
--hash=sha256:3934dcde484374596d6b52d8db412929a169f6d9e52e20f9ade5bf3523d9b96e
# via -r ./requirements.in
# via -r requirements.in
6 changes: 3 additions & 3 deletions examples/pip_install/requirements_windows.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
boto3==1.14.63 \
--hash=sha256:25c716b7c01d4664027afc6a6418a06459e311a610c7fd39a030a1ced1b72ce4 \
--hash=sha256:37158c37a151eab5b9080968305621a40168171fda9584d50a309ceb4e5e6964
# via -r ./requirements.in
# via -r requirements.in
botocore==1.17.63 \
--hash=sha256:40f13f6c9c29c307a9dc5982739e537ddce55b29787b90c3447b507e3283bcd6 \
--hash=sha256:aa88eafc6295132f4bc606f1df32b3248e0fa611724c0a216aceda767948ac75
Expand Down Expand Up @@ -84,7 +84,7 @@ pyyaml==6.0 \
s3cmd==2.1.0 \
--hash=sha256:49cd23d516b17974b22b611a95ce4d93fe326feaa07320bd1d234fed68cbccfa \
--hash=sha256:966b0a494a916fc3b4324de38f089c86c70ee90e8e1cae6d59102103a4c0cc03
# via -r ./requirements.in
# via -r requirements.in
s3transfer==0.3.7 \
--hash=sha256:35627b86af8ff97e7ac27975fe0a98a312814b46c6333d8a6b889627bcd80994 \
--hash=sha256:efa5bd92a897b6a8d5c1383828dca3d52d0790e0756d49740563a3fb6ed03246
Expand All @@ -103,4 +103,4 @@ urllib3==1.25.11 \
# via botocore
yamllint==1.26.3 \
--hash=sha256:3934dcde484374596d6b52d8db412929a169f6d9e52e20f9ade5bf3523d9b96e
# via -r ./requirements.in
# via -r requirements.in
6 changes: 3 additions & 3 deletions examples/pip_parse/requirements_lock.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,11 @@ pyyaml==6.0 \
requests==2.25.1 \
--hash=sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804 \
--hash=sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e
# via -r ./requirements.in
# via -r requirements.in
s3cmd==2.1.0 \
--hash=sha256:49cd23d516b17974b22b611a95ce4d93fe326feaa07320bd1d234fed68cbccfa \
--hash=sha256:966b0a494a916fc3b4324de38f089c86c70ee90e8e1cae6d59102103a4c0cc03
# via -r ./requirements.in
# via -r requirements.in
setuptools==65.6.3 \
--hash=sha256:57f6f22bde4e042978bcd50176fdb381d7c21a9efa4041202288d3737a0c6a54 \
--hash=sha256:a7620757bf984b58deaf32fc8a4577a9bbc0850cf92c20e1ce41c38c19e5fb75
Expand All @@ -92,4 +92,4 @@ urllib3==1.26.13 \
# via requests
yamllint==1.26.3 \
--hash=sha256:3934dcde484374596d6b52d8db412929a169f6d9e52e20f9ade5bf3523d9b96e
# via -r ./requirements.in
# via -r requirements.in
2 changes: 1 addition & 1 deletion examples/pip_parse_vendored/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ idna==3.4 \
requests==2.28.1 \
--hash=sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983 \
--hash=sha256:8fefa2a1a1365bf5520aac41836fbee479da67864514bdb821f31ce07ce65349
# via -r ./requirements.in
# via -r requirements.in
urllib3==1.26.13 \
--hash=sha256:47cc05d99aaa09c9e72ed5809b60e7ba354e64b59c9c173ac3018642d8bb41fc \
--hash=sha256:c083dd0dce68dbfbe1129d5271cb90f9447dea7d52097c6e0126120c521ddea8
Expand Down
4 changes: 2 additions & 2 deletions examples/pip_repository_annotations/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ idna==3.4 \
requests[security]==2.28.1 \
--hash=sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983 \
--hash=sha256:8fefa2a1a1365bf5520aac41836fbee479da67864514bdb821f31ce07ce65349
# via -r ./requirements.in
# via -r requirements.in
urllib3==1.26.13 \
--hash=sha256:47cc05d99aaa09c9e72ed5809b60e7ba354e64b59c9c173ac3018642d8bb41fc \
--hash=sha256:c083dd0dce68dbfbe1129d5271cb90f9447dea7d52097c6e0126120c521ddea8
# via requests
wheel==0.38.4 \
--hash=sha256:965f5259b566725405b05e7cf774052044b1ed30119b5d586b2703aafe8719ac \
--hash=sha256:b60533f3f5d530e971d6737ca6d58681ee434818fab630c83a734bb10c083ce8
# via -r ./requirements.in
# via -r requirements.in
6 changes: 3 additions & 3 deletions python/pip_install/private/test/requirements_parser_tests.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,11 +177,11 @@ pyyaml==6.0 \
requests==2.25.1 \
--hash=sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804 \
--hash=sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e
# via -r ./requirements.in
# via -r requirements.in
s3cmd==2.1.0 \
--hash=sha256:49cd23d516b17974b22b611a95ce4d93fe326feaa07320bd1d234fed68cbccfa \
--hash=sha256:966b0a494a916fc3b4324de38f089c86c70ee90e8e1cae6d59102103a4c0cc03
# via -r ./requirements.in
# via -r requirements.in
six==1.16.0 \
--hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 \
--hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
Expand All @@ -192,7 +192,7 @@ urllib3==1.26.7 \
# via requests
yamllint==1.26.3 \
--hash=sha256:3934dcde484374596d6b52d8db412929a169f6d9e52e20f9ade5bf3523d9b96e
# via -r ./requirements.in
# via -r requirements.in

# The following packages are considered to be unsafe in a requirements file:
setuptools==59.6.0 \
Expand Down
3 changes: 2 additions & 1 deletion python/pip_install/requirements.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def compile_pip_requirements(
# where it appears, which is to say, in @rules_python
pip_compile = Label("//python/pip_install/tools/dependency_resolver:dependency_resolver.py")

loc = "$(rootpath {})"
loc = "$(rlocationpath {})"

args = [
loc.format(requirements_in),
Expand All @@ -99,6 +99,7 @@ def compile_pip_requirements(
requirement("importlib_metadata"),
requirement("zipp"),
requirement("more_itertools"),
Label("//python/runfiles:runfiles"),
] + extra_deps

tags = tags or []
Expand Down
Loading