Unable To Install Tensorflow on macOS Big Sur

[timkofu]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Big Sur 11.1
• Poetry version: 1.1.4

Issue

poetry add tensorflow fails in a fresh Python 3.8.7 .venv with the final exception distutils.errors.CompileError: command 'clang' failed with exit status 1.

pip install tensorflow fails with the error Could not find a version that satisfies the requirement tensorflow (from versions: none).

I upgraded pip and setuptools to the latest releases pip install -U pip setuptools and rerun poetry add; It failed with the same exception, and the following error:

ERROR: Command errored out with exit status 1: /private/tmp/test_poetry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"'; __file__='"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-record-be6ffsck/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/test_poetry/.venv/include/site/python3.8/grpcio Check the logs for full command output.

However, pip install tensorflow now works:

python -c "import tensorflow;print(tensorflow.__version__)"
2.4.0

@timkofu this is a platform specific error that doesn't affect poetry.

Most scientific packages broke with BigSur (Numpy, Pandas, Tensorflow) as there is breaking changes both for the intel and Apple silicon based MacBooks.

To get any of the aformentioned packages running again you'll have to revert to Catalina, run it containerized/vm or wait until fixes have made it downstream for Apple's breaking changes.

You can confirm this yourself in a minute or two by:

• Creating a new virtual environment python -m virtualenv testenv && source testenv/bin/activate
• Installing your affected packages pip install numpy tensorflow pandas etc
• Try to run a file that just imports the affected packages, if it still breaks its 100% for sure on your system 👍

(added this comment as I see you're running multiple versions of python. Python2 might never have been updated, for that it's important to remember Python2 is a dead and no longer maintained language .)

[timkofu]

@alexpdr In my last few sentences, I showed that it does work with an updated pip and setuptools, but not with poetry. Im current working with plain activate .venv/bin/activate and deactivate. Pandas, Tensorflow, Seaborn et al are all working well with the latest pip (which I demonstrated with the Tensorflow import in the Tensorflow case).

Also, I'm only running Python 3.8.7 on this venv, no Python 2. The last code is printing the Tensorflow version, showing it imported successfully with no errors when installed via pip.

----> grpc/grpc#24677
You might be able to find additional help here, but I'd resubmit the error message on the framework/tool that produces the error (which here is grpc)

ERROR: Command errored out with exit status 1: /private/tmp/test_poetry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"'; file='"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-record-be6ffsck/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/test_poetry/.venv/include/site/python3.8/grpcio Check the logs for full command output.

[timkofu]

Had seen that thread before reporting this; it's where I got the idea that using (the latest) plain pip would solve the DS stack (and other) install issues. Specifically this comment. "It's working fine with pip, have you tried that?" will probably be the first comment.

The activate .venv/bin/activate and deactivate (and manually managing the prod and dev deps) workflow is working well for us, but Poetry is such a great tool, and we like reporting issues as soon as we find them, so we can all keep having great DX.

[finswimmer]

Which versions of pip and setuptools are installed in the venv created by poetry?

[loganjhennessy]

I came across the same issue when attempting to install google-cloud-bigquery which also depends on grpcio.

There is a lot going on here, but the offending line in grpcio is https://github.com/grpc/grpc/blob/master/setup.py#L358.

The return value from sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') has always been a str (e.g. 10.14, 10.15, etc.) but in Big Sur, it is possible to get a MACOSX_DEPLOYMENT_TARGET value of just 11. Config values that look like integers are interpreted as integers by cpython, see python/cpython#23622 and python/cpython#23556. So on Big Sur, sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') returns an int. The subsequent call to pkg_resources.parse_version() fails and generates the aforementioned error.

I was able to get this to work by downgrading my Python distribution to 3.7 since, for whatever reason, sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') returns 11.0, which gets passed through as a str.

I don't know exactly where the fix for this should be made. It is frustrating that the value gets cast to an int. One solution would be to ask that the return value from that call in grpcio be cast to str before being passed to pkg_resources.parse_version().

By the way, I'm using Brew for this. Someone mentioned hardcoding the value in the Brew formula as a workaround (see python/cpython#23556 (comment)) although I don't think this is a good long-term solution.

[timkofu]

@finswimmer with the venv created with poetry env use ~/.pyenv/versions/3.8.7/bin/python:

>>> import pip, setuptools
>>> pip.__version__
'20.2.2'
>>> setuptools.__version__
'49.6.0'

[nacaclanga]

Wheel installation was fixed in packaging (pypa/packaging#319) and the newer version is used in pip (pypa/pip#9138), but not in poetry or poetry-core. I suspect that this kind of issue could be fixed by providing a new version of poetry-core, where the vendored packaging dependency is updated to the latest release and then fix the requirements of poetry itself.

[damienrj]

I think this would also solve the issue with needing pip 20.3.0> to be able to download wheels.
#3591

[bajohn]

I had a similar issue as described by @loganjhennessy , regarding google-cloud-bigquery being unable to install grpcio.

A quick fix that worked for me was running

export SYSTEM_VERSION_COMPAT=1

Afterwards, clang errors were no longer thrown and I was able to install via pipenv. Not sure if this fix applies to the original issue but hope it helps!

[timkofu]

This is now working perfectly with release 1.1.5.