Poetry uses wrong Python version

[joooeey]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux Mint 20.1
• Poetry version: 1.1.6
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/joooeey/78317c68c58d8aaa2d452c02212031f3

Issue

We're developing a project on a few machines with several different Python versions installed. We're using Poetry to make sure we have the same development environment. But unfortunately Poetry doesn't check if the right Python version is installed. I expect to get an error or a warning if Poetry isn't running on the Python version specified in the pyproject.toml file (Python 3.9). But Poetry lets this slip, so when working on my laptop the other day I only got a failure at runtime when suddenly tons of errors with type annotation popped up. It took me a while to figure out this was because I was using an old Python version (3.8).

To reproduce, copy the pyproject.toml linked above and run the following commands:

(base) lukas@ThinkPad-T560:~/Desktop/poetry_test$ poetry install
Updating dependencies
Resolving dependencies... (0.1s)
Writing lock file
(base) lukas@ThinkPad-T560:~/Desktop/poetry_test$ poetry run python
Python 3.8.5 (default, Sep  4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
(base) lukas@ThinkPad-T560:~/Desktop/poetry_test$ poetry env info
Virtualenv
Python:         3.8.5
Implementation: CPython
Path:           /home/lukas/.cache/pypoetry/virtualenvs/edeo-sensing-iAOxxtc0-py3.8
Valid:          True
System
Platform: linux
OS:       posix
Python:   /home/lukas/anaconda3
(base) lukas@ThinkPad-T560:~/Desktop/poetry_test$ poetry --version
4Poetry version 1.1.6
(base) lukas@ThinkPad-T560:~/Desktop/poetry_test$ 4

I would have expected Python to run on 3.9 or if it can't find that version I would expect it to display an error or warning during poetry install.

See also:

• Closed, similar issue: No method to control the python version used #522
• An older attempt to fix this, but not applicable if you want to develop on multiple machines: New env command and sub commands #731
• Another merged PR that was supposed to fix this if I understand it correctly: Improve environment management #1477

[NickBeeuwsaert]

I'm having this issue too, it looks like when poetry looks for a compatible python version, it calls whichever python3 is on the path, and if that matches, then it will call "virtualenv --python python3 ...", and when virtualenv runs detection it will first check if sys.executable is a valid match for python3, before searching the path for python3.

Minimal example with pyenv:

$ pyenv install 3.9.5
$ pyenv install 3.8.10
$ 
$ # force poetry to be called with python 3.8 (also assuming pyproject.toml requried python ^3.9)
$ ~/.pyenv/versions/3.8.10/bin/python3.8 ~/.poetry/bin/poetry run python --version
The currently activated Python version 3.8.10 is not supported by the project (^3.9).
Trying to find and use a compatible version.
Using python3 (3.9.2)
Creating virtualenv ttst in /tmp/test/.venv
Python 3.8.10

I'm only seeing this problem on FreeBSD 13, and not on Linux, probably because I'm using pyenv, so when I call poetry in my project (which has a compatible .python-version with my pyproject.toml) poetry is then being called with python3.9, but on freeBSD I have to alias poetry to /usr/local/bin/python3.8 ~/.poetry/bin/poetry since poetry doesn't vendor in dependencies for python3.9 on FreeBSD

[sotte]

Well, I can reproduce this issue like so (see below). I seems that poetry does not pick up the right python version when pyenv is used.

❯ cd projects/
~/projects
❯ pyenv --version
pyenv 2.0.0
~/projects
❯ which pyenv
/home/stefan/.pyenv/bin/pyenv
~/projects
❯ poetry --version
Poetry version 1.1.5
~/projects
❯ which poetry
/home/stefan/.local/bin/poetry
~/projects
❯ pyenv versions
  system
  3.10.0b1
* 3.9.4 (set by /home/stefan/.pyenv/version)
~/projects
❯ python --version
Python 3.9.4
~/projects
❯ which python
/home/stefan/.pyenv/shims/python
~/projects
❯ python3 --version
Python 3.9.4
~/projects
❯ which python3
/home/stefan/.pyenv/shims/python3
~/projects
❯ poetry new demo
Created package demo in demo
~/projects
❯ cd demo/
~/projects/demo is 📦 v0.1.0 via 🐍 v3.9.4
❯ python --version
Python 3.9.4
~/projects/demo is 📦 v0.1.0 via 🐍 v3.9.4
❯ vim pyproject.toml
~/projects/demo is 📦 v0.1.0 via 🐍 v3.9.4 took 16s
❯ cat pyproject.toml
[tool.poetry]
name = "demo"
version = "0.1.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

~/projects/demo is 📦 v0.1.0 via 🐍 v3.9.4
❯ poetry shell
The currently activated Python version 3.8.6 is not supported by the project (^3.9).
Trying to find and use a compatible version.
Using python3 (3.9.4)
Creating virtualenv demo-om1cyA5z-py3.9 in /home/stefan/.cache/pypoetry/virtualenvs
Spawning shell within /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9
source /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/activate.fish
Welcome to fish, the friendly interactive shell
Type `help` for instructions on how to use fish
~/projects/demo is 📦 v0.1.0 via 🐍 v3.9.4
❯ source /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/activate.fish
~/projects/demo is 📦 v0.1.0 via 🐍 v3.8.6 (demo-om1cyA5z-py3.9)
❯ python --version
Python 3.8.6
❯ python3 --version
Python 3.8.6
~/projects/demo is 📦 v0.1.0 via 🐍 v3.8.6 (demo-om1cyA5z-py3.9) 
❯ which python3
/home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/python3
~/projects/demo is 📦 v0.1.0 via 🐍 v3.8.6 (demo-om1cyA5z-py3.9) 
❯ which python
/home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/python
❯ l /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/python
Permissions Size User   Date Modified Name
lrwxrwxrwx    18 stefan  7 Jun 10:25   /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/python -> /usr/bin/python3.8
~/projects/demo is 📦 v0.1.0 via 🐍 v3.8.6 (demo-om1cyA5z-py3.9) 
❯ l /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/python3
Permissions Size User   Date Modified Name
lrwxrwxrwx     6 stefan  7 Jun 10:25   /home/stefan/.cache/pypoetry/virtualenvs/demo-om1cyA5z-py3.9/bin/python3 -> python

[ggicci]

@sdispater Shall we prefer Pythons installed by pyenv than the system's? Which can solve this issue, IMO. And we don't need to inspect pyenv's shims scripts.

Update: The root cause is mixing the interpreter discovery logic between poetry and virtualenv. So, better to solve it by by-passing the conflict logic. See my followed comment below :)

[NickBeeuwsaert]

It looks like you could use virtualenvs --try-first-wtith option to fix this, if this is an appropriate solution let me know, I'd be happy to submit a PR for it

[ggicci]

I reproduced this and give the explanation as follows:

❯ /usr/bin/python3 ~/.poetry/bin/poetry --version           
Poetry version 1.1.6

❯ /usr/bin/python3 -V                       
Python 3.6.9

❯ /usr/bin/python3 ~/.poetry/bin/poetry shell -vvv
The currently activated Python version 3.6.9 is not supported by the project (^3.9).  ----------> TRY-1
Trying to find and use a compatible version. 
Trying python3      ----------> TRY-2
Using python3 (3.9.4)
Creating virtualenv demo-zBGpEbJs-py3.9 in /home/mingjietang/.cache/pypoetry/virtualenvs
Using virtualenv: /home/mingjietang/.cache/pypoetry/virtualenvs/demo-zBGpEbJs-py3.9
Spawning shell within /home/mingjietang/.cache/pypoetry/virtualenvs/demo-zBGpEbJs-py3.9

Poetry tries the "currently activated Python executable" first to see if the version is compatible. See TRY-1 above. This is the Python running the poetry script. Here, the version is 3.6.9, thus incompatible.

Then poetry will try "system Python executables" to see if there's a compatible executable. The version info is gathered from running a shell call to the "executables". Here's the trying commands:

python3 -V
python3.9 -V
python3.8 -V
...
python2 -V
python2.7 -V
...

See TRY-2 above. Since python3 is of version 3.9.4, which is thought of compatible to this project. Then poetry will create venv by using this identifier (python3) by calling virtualenv. This is the ROOT CAUSE of the bug here.

In virutualenv's builtin discovery method, the interpreter where virtualenv is installed takes precedence.

discovery:
discover and provide a target interpreter
--discovery {builtin} interpreter discovery method (default: builtin)
-p py, --python py interpreter based on what to create environment (path/identifier) - by default use the interpreter where > the tool is installed - first found wins (default: [])
--try-first-with py_exe try first these interpreters before starting the discovery (default: [])

So, we can solve this problem by passing the path of the Python executable to virtualenv rather than by using a simple identifider, i.e python3, python3.9, etc. Which is involved in this commit b4b816e.

[ggicci]

As @NickBeeuwsaert mentioned above, another option from virtualenv can be utlized. But it's enough to use --python option if we pass the path to the Python executable directly.

[redeemefy]

even though I had the variable virtualenvs.in-project=true, and pyenv global 3.9.0 the project root .venv gets created with 3.7.8. When I run poetry env info virtualenv and python path are pointing to 3.7.8. As a work around for now I did the following:

• rm -rf .venv
• rm poetry.lock
• pyenv global 3.9.0
• python -m venv .venv
• source .venv/bin/activate
• python -m pip install --upgrade pip
• poetry env info Now has the correct information
• poetry install

[dr-igor]

I think I have this same issue, and would like to point out that it causes the example given in the documentation of how to use pyenv to not actually work.

(If my issue is different than this issue, let me know, and I will create a separate issue).

Despite installing and specifying 3.9.8 using pyenv, poetry creates a virtualenv using miniconda3 (3.8.11):

 pyenv which python
/Users/igor/.pyenv/versions/miniconda3-latest/bin/python
 pyenv versions
  system
  3.10.0
  3.8.6
* miniconda3-latest (set by /Users/igor/.pyenv/version)
  miniconda3-latest/envs/house_offer_analyzer
 poetry init

This command will guide you through creating your pyproject.toml config.

Package name [poetry-env-test]:
Version [0.1.0]:
Description []:
Author [Igor ***, n to skip]:
License []:
Compatible Python versions [^3.8]:

Would you like to define your main dependencies interactively? (yes/no) [yes] n
Would you like to define your development dependencies interactively? (yes/no) [yes] n
Generated file

[tool.poetry]
name = "poetry-env-test"
version = "0.1.0"
description = ""
authors = ["Igor ***"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes] y
 pyenv install 3.9.8
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.8.tar.xz...
-> https://www.python.org/ftp/python/3.9.8/Python-3.9.8.tar.xz
Installing Python-3.9.8...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.9.8 to /Users/igor/.pyenv/versions/3.9.8

 pyenv local 3.9.8
 poetry add click
Creating virtualenv poetry-env-test-uX9f7DhA-py3.8 in /Users/igor/Library/Caches/pypoetry/virtualenvs
Using version ^8.0.3 for click

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing click (8.0.3)
 poetry env info

Virtualenv
Python:         3.8.11
Implementation: CPython
Path:           /Users/igor/Library/Caches/pypoetry/virtualenvs/poetry-env-test-uX9f7DhA-py3.8
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /Users/igor/.pyenv/versions/miniconda3-latest
 pyenv versions
  system
  3.10.0
  3.8.6
* 3.9.8 (set by /Users/igor/Documents/dev/poetry-env-test/.python-version)
  miniconda3-latest
  miniconda3-latest/envs/house_offer_analyzer
 echo $PATH
/Users/igor/.pyenv/shims /Users/igor/.pyenv/versions/miniconda3-latest/bin /Users/igor/.pyenv/versions/miniconda3-latest/condabin /opt/homebrew/bin /opt/homebrew/sbin /Users/igor/.platformio/penv/bin /usr/local/bin /usr/bin /bin /usr/sbin /sbin /Library/Apple/usr/bin /usr/local/MacGPG2/bin /Users/igor/.local/bin

[SlapDrone]

Ubuntu 20.04 on WSL, Poetry version 1.1.13

Just bumping to say this one stung me too (3.9.5 used globally by pyenv and specified in pyproject.toml pointed to the system 3.8).

To reproduce, did a poetry run python to get into my interpreter, grabbed sys.executable and put it through readlink -f. Sure enough it was pointing to /usr/bin/python3.8.

The workaround by @redeemefy above has fixed for now.

[jonananas]

I have a similar problem on OSX, if I use the system version in pyproject.toml, the python used will be the newest one installed, not the system one. E.g. if python = "~3.9" and system is 3.9.12, but I have 3.10.2 installed:

poetry install
Creating virtualenv myproject-Sco6aV2r-py3.10...
...
poetry shell
(myproject-Sco6aV2r-py3.10) ➜  myproject python3 --version
Python 3.10.2

Workaround: poetry env use:

➜  myproject poetry env use /usr/local/opt/python@3.9/bin/python3
Creating virtualenv myproject-Sco6aV2r-py3.9 ...
(myproject-Sco6aV2r-py3.9) ➜  myproject python3 --version
Python 3.9.12

NOTE: If I try a version other than system, poetry install will fail with an appropriate message, which it what led me to the workaround.