having setuptools_scm>6 installed globally breaks pip_install

[implr]

Affected Rule

The issue is caused by the rule: pip_install

Description

If you have setuptools_scm installed on your system, it will register itself as a setuptools plugin (because /usr/.../site_packages/ is still on sys.path). Then the part in

      File "/tmp/tmpsov87r8j/setuptools-tmp/setuptools/dist.py", line 740, in finalize_options
        ep.load()(self)

will attempt to import it. setuptools_scm>6 requires setuptools>=45, rules_python bundles setuptools 44, so that import fails therefore pip wheel fails.

🔬 Minimal Reproduction

Basically any pip_install invocation which requires building wheels will fail, as long as setuptools_scm>6 is on your global python path.

🔥 Exception or Error

Error in fail: pip_import failed: Collecting setuptools==44 (from -r /home/bartek/dev/pejs/hscloud/third_party/py/requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/f9/d3/955738b20d3832dfa3cd3d9b07e29a8162edb480bf988332f5e6e48ca444/setuptools-44.0.0-py2.py3-none-any.whl
  Saved ./setuptools-44.0.0-py2.py3-none-any.whl
Collecting arrow==0.14.5 (from -r /home/bartek/dev/pejs/hscloud/third_party/py/requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/4f/c6/32df2c68e02e2d6b4457223fa499634edabb2d4ff74f00087ffff49b4be4/arrow-0.14.5-py2.py3-none-any.whl
  Saved ./arrow-0.14.5-py2.py3-none-any.whl
Collecting asn1crypto==0.24.0 (from -r /home/bartek/dev/pejs/hscloud/third_party/py/requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
  Saved ./asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting bcrypt==3.1.5 (from -r /home/bartek/dev/pejs/hscloud/third_party/py/requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/31/4b/4057d0716e7170c29ff12e19791eb6037422620835e4a58a01d4790e56d1/bcrypt-3.1.5-cp34-abi3-manylinux1_x86_64.whl
  Saved ./bcrypt-3.1.5-cp34-abi3-manylinux1_x86_64.whl
Collecting blinker==1.4 (from -r /home/bartek/dev/pejs/hscloud/third_party/py/requirements.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/1b/51/e2a9f3b757eb802f61dc1f2b09c8c99f6eb01cf06416c0671253536517b6/blinker-1.4.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "", line 1, in 
      File "/tmp/pip-build-t0bmbmpr/blinker/setup.py", line 10, in 
        setup(name="blinker",
      File "/tmp/tmpsov87r8j/setuptools-tmp/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/tmp/tmpsov87r8j/setuptools-tmp/setuptools/__init__.py", line 132, in _install_setup_requires
        dist = distutils.core.Distribution(dict(
      File "/tmp/tmpsov87r8j/setuptools-tmp/setuptools/dist.py", line 447, in __init__
        _Distribution.__init__(self, {
      File "/usr/lib/python3.9/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/tmp/tmpsov87r8j/setuptools-tmp/setuptools/dist.py", line 740, in finalize_options
        ep.load()(self)
      File "/tmp/tmpsov87r8j/setuptools-tmp/pkg_resources/__init__.py", line 2442, in load
        self.require(*args, **kwargs)
      File "/tmp/tmpsov87r8j/setuptools-tmp/pkg_resources/__init__.py", line 2465, in require
        items = working_set.resolve(reqs, env, installer, extras=self.extras)
      File "/tmp/tmpsov87r8j/setuptools-tmp/pkg_resources/__init__.py", line 791, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.VersionConflict: (setuptools 44.0.0 (/tmp/tmpsov87r8j/setuptools-tmp), Requirement.parse('setuptools>=45'))

🌍 Your Environment

Operating System:

  
gentoo
  

Output of bazel version:

  
Build label: 3.7.2- (@non-git)
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Sat Apr 24 11:55:50 2021 (1619265350)
Build timestamp: 1619265350
Build timestamp as int: 1619265350
  

Rules_python version:

  
0.0.3
  

I realize this is fairly old, but setuptools is still pinned to 44 on master. I might try to reproduce with rules_python master later.

[implr]

I made a minimal reproduction gist which fails on ubuntu 20.04 https://gist.github.com/implr/21794a58d87474ddd996027abd79ffee
Ubuntu still installs setuptools 44 by default, so use venv to get the newer problematic version.

wget https://gist.github.com/implr/21794a58d87474ddd996027abd79ffee/archive/a0de83fc248320ef0d1567c2f0450285800fb5d2.zip
unzip a0de83fc248320ef0d1567c2f0450285800fb5d2.zip
cd 21794a58d87474ddd996027abd79ffee-a0de83fc248320ef0d1567c2f0450285800fb5d2/

bazel build @pydeps//...  # succeeds

bazel clean --expunge
bazel shutdown
# make sure pip doesn't reuse the wheel it just built
rm -r ~/.cache/pip/wheels/*
python3 -m venv venv
. venv/bin/activate
pip install setuptools_scm
bazel build @pydeps//...  # fails

[thundergolfer]

So even if the plugin isn't used setuptools will load it? Searching around the internet I can't see a way to block the loading. sigh, non-hermetic toolchain strikes again.

[implr]

This seems to be the case, but I don't know enough about setuptools to say for certain. You'll probably start seeing this more often as distros update.

[hrfuller]

@thundergolfer I think we can upgrade setuptools now that we have officially sunset python2 #422 or will that break the legacy pip_import rule?

[hrfuller]

If anyone still needs older setuptools for python2 they can update their workspace to use an older set of bootstrapping deps, or an older release.

[hrfuller]

@implr can you see if latest master, which has a newer setuptools version now, fixes this?

[implr]

Yes, it works for us, tested on multiple distros. Thanks.

[thundergolfer]

@implr The default toolchain is non-hermetic so remains vulnerable to these kind of issues. Recommend switching to an in-build Python interpreter and hooking that up as your py_runtime. Unfortunately, at the moment there's not much in the way of documentation on how to do this simply, so individual exploration will probably be needed. The latest work I've done in this area is documented at https://thundergolfer.com/bazel/python/2021/06/25/a-basic-python-bazel-toolchain/.