Dependency resolution picks wrong version of url dependency when lock file exists

[bnorick]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04 (WSL)
• Poetry version: 1.1.10
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/bnorick/942580f0f5dcf13afbd87968a5d9b2af

TLDR

In my last comment, I identified what the cause of the bug was and made a hacky fix to workaround the issue. Someone more familiar with poetry should probably make the appropriate fix.

Issue

When a library depends on another library via a path which has a url dependency with multiple python version constraints, the dependency resolver appears to always picks the lowest python version of the dependency regardless of the python version in use by the poetry environment if a lock file exists. Deleting the lock file "resolves" the issue, and the correct version of the dependency is installed, however the generated lock file causes the issue again.

Let me demonstrate with a minimal example. The gist linked above includes two pyproject.toml files and the directory structure I am using. From a high level:

• Two libraries in sibling directories
• The first library has a "complex" url dependency with multiple urls and matching python version constrains
• The second library depends on the first via a path dependency

To set up this minimal repro, I created the libraries with poetry new lib1 --src and poetry new lib2 --src. I then manually edited the appropriate pyproject.toml files, adding the "complex" url dependency for lib1 and the lib1 dependency for lib2.

The directory structure looks like this:

.
├── lib1
│   ├── README.rst
│   ├── pyproject.toml
│   ├── src
│   │   └── lib1
│   │       └── __init__.py
│   └── tests
│       ├── __init__.py
│       └── test_lib1.py
└── lib2
    ├── README.rst
    ├── pyproject.toml
    ├── src
    │   └── lib2
    │       └── __init__.py
    └── tests
        ├── __init__.py
        └── test_lib2.py

Initially no lock file exists and the installation is successful.

./lib2$ pyenv shell 3.7.12  # enable python 3.7.12
./lib2$ poetry env use `which python`  # tell poetry to use the correct python version
Creating virtualenv lib2-sdBY6ERp-py3.7 in $HOME/.cache/pypoetry/virtualenvs
Using virtualenv: $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7

./lib2$ poetry env info  # verify that poetry is using python 3.7.12

Virtualenv
Python:         3.7.12
Implementation: CPython
Path:           $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7
Valid:          True

System
Platform: linux
OS:       posix
Python:   $HOME/.pyenv/versions/3.7.12

./lib2$ poetry install
Updating dependencies
Resolving dependencies... (15.8s)

Writing lock file

Package operations: 54 installs, 0 updates, 0 removals
[... snip ...]
  • Installing ray (2.0.0.dev0 https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl)
[... snip ...]

Installing the current project: lib2 (0.1.0)

Now, let's remove the environment (as if I've just cloned this repository, for example) and try installing while the lock file exists.

./lib2$ rm -rf `poetry env info --path`

./lib2$ pyenv shell 3.7.12  # enable python 3.7.12
./lib2$ poetry env use `which python`  # tell poetry to use the correct python version
Creating virtualenv lib2-sdBY6ERp-py3.7 in $HOME/.cache/pypoetry/virtualenvs
Using virtualenv: $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7

./lib2$ ls -l poetry.lock  # verify that the lock file exists
-rwxrwxrwx 1 user group 90805 Sep 22 21:45 poetry.lock

./lib2$ poetry install -vvv
Using virtualenv: $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 54 installs, 0 updates, 0 removals, 6 skipped
[... snip ...]
  • Installing ray (2.0.0.dev0 https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl): Installing...
  • Installing ray (2.0.0.dev0 https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl): Failed

  EnvCommandError

  Command ['$HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7/bin/pip', 'install', '--no-deps', '$HOME/.cache/pypoetry/artifacts/5b/e5/2d/a7ac74ee969b881a903e0c31a789da58ab7a771be56c4256a34ce1266a/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'] errored with the following return code 1, and output:
  ERROR: ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl is not a supported wheel on this platform.
  WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
  You should consider upgrading via the '$HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7/bin/python -m pip install --upgrade pip' command.


  at ~/.local/pipx/venvs/poetry/lib/python3.6/site-packages/poetry/utils/env.py:1180 in _run
      1176│                 output = subprocess.check_output(
      1177│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1178│                 )
      1179│         except CalledProcessError as e:
    → 1180│             raise EnvCommandError(e, input=input_)
      1181│
      1182│         return decode(output)
      1183│
      1184│     def execute(self, bin, *args, **kwargs):

As you can see, poetry tried to install the python 3.6 version of the ray url dependency from lib1. I should note also that I did update pip as the warning recommends, but as expected this issue is unrelated to that.

Removing the lock file (which was generated from the successful installation) and running poetry install again is successful however.

./lib2$ rm poetry.lock

./lib2$ poetry install -vvv
Using virtualenv: $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7
Updating dependencies
Resolving dependencies...
[... snip ...]
   1: Version solving took 0.646 seconds.
   1: Tried 1 solutions.
   0: Complete version solving took 9.931 seconds with 6 overrides
   0: Resolved with overrides: ({Package('ray', '2.0.0.dev0', features=frozenset({'default'}), source_type='url', source_url='https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f
102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.16)>}}), ({Package('ray', '2.0.0.dev0', features=frozenset({'default'}), source_type='url', source_url='https://s3-us-w
est-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.16)>}, Package('ray', '2.0.0.dev0', source_type
='url', source_url='https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.16)>}}), ({P
ackage('ray', '2.0.0.dev0', features=frozenset({'default'}), source_type='url', source_url='https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-m
anylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.16)>}, Package('ray', '2.0.0.dev0', source_type='url', source_url='https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24
f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('ray', '2.0.0.dev0', features=frozenset({'default'}), source_type='url', source_url='https://s3-u
s-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('ray', '2.0.0.dev0', fe
atures=frozenset({'default'}), source_type='url', source_url='https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'n
umpy': <Dependency numpy (>=1.19.3)>}, Package('ray', '2.0.0.dev0', source_type='url', source_url='https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-
cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('ray', '2.0.0.dev0', features=frozenset({'default'}), source_type='url', source_url='https://s3-us-west-2.amazonaws.com/ray-w
heels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('ray', '2.0.0.dev0', source_type='url', source_url='htt
ps://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl'): {'numpy': <Dependency numpy (>=1.16)>}})

Writing lock file

Finding the necessary packages for the current system
Package operations: 3 installs, 0 updates, 0 removals, 51 skipped
[... snip ...]
  • Installing ray (2.0.0.dev0 https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl)
[... snip ...]

Installing the current project: lib2 (0.1.0)
  - Building package lib2 in editable mode
  - Adding lib2.pth to $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7/lib/python3.7/site-packages for ./lib2
  - Adding the lib2-0.1.0.dist-info directory to $HOME/.cache/pypoetry/virtualenvs/lib2-sdBY6ERp-py3.7/lib/python3.7/site-packages

Interestingly, there are a lot of references to the python 3.6 version of ray even in the output of the dependency resolution (prior to the "Writing lock file" line) even in this case, however it does install the correct version.

[bnorick]

This is the first time I've poked around in poetry source, so someone else can probably get to the bottom of this much faster than me. That said, I think I've narrowed it down to a particular solver.solve call. Looking at the operation associated with the ray package in ops before and after installer.py:305 when a lock file does and doesn't exist shows the following:

# poetry.lock exists before poetry install
## before
<Install ray (2.0.0.dev0 https://s3-us-
west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl)>
## after
<Install ray (2.0.0.dev0 https://s3-us-
west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl)>

# poetry.lock doesn't exist before poetry install
## before
<Install ray (2.0.0.dev0 https://s3-us-
west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl)>
## after
<Install ray
(2.0.0.dev0 https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl)>

When the lock file exists, the dependency doesn't change versions and poetry tries to install the wrong version. When the lock file doesn't exist, the dependency changes to the correct version.

[bnorick]

Oh, looks like the lock file is just... incorrectly written?

The issue can be manually resolved by editing the lib1 package in lib2/poetry.lock from

[[package]]
name = "lib1"
version = "0.1.0"
description = ""
category = "dev"
optional = false
python-versions = "^3.6.1"
develop = true

[package.dependencies]
ray = {url = "https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl", extras = ["default"], markers = "python_version >= \"3.6\" and python_version < \"3.7\" or python_version >= \"3.7\" and python_version < \"3.8\""}

to

[[package]]
name = "lib1"
version = "0.1.0"
description = ""
category = "dev"
optional = false
python-versions = "^3.6.1"
develop = true

[package.dependencies]
ray = [
    {url = "https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl", extras = ["default"], markers = "python_version >= \"3.6\" and python_version < \"3.7\""},
    {url = "https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl", extras = ["default"], markers = "python_version >= \"3.7\" and python_version < \"3.8\""}
]

The only change here is to the package.dependencies.

[bnorick]

It looks like this block erroneously merges the two URLDependency instances for ray dependencies into a single dependency,

Both share the same dep.constraint of <Version 2.0.0.dev0>, so they get merged and only the URLDependency which appears first (the python 3.6 one) is retained but with an incorrect marker of

transitive_marker:<MarkerUnion python_version >= "3.6" and python_version < "3.7" or python_version >= "3.7" and python_version < "3.8">

I still haven't totally grokked all of the poetry internals, but it seems to me that URLDependencies should not be merged in this way.

With that in mind, I tried what is probably a hacky fix to provider.py: if deps is a list of only URLDependency instances, then pass it directly through to the overrides generation step. You can see the modified file in this gist (which, as noted in the original issue, corresponds to poetry 1.1.10). I tabbed in a bunch of lines and added three lines:

            if all(isinstance(dep, URLDependency) for dep in deps):
                _deps = deps
            else:

This makes the generated lock file work correctly for fresh installation, as it now contains the python 3.7 version of ray in the locked dependencies of lib1:

[[package]]
name = "lib1"
version = "0.1.0"
description = ""
category = "dev"
optional = false
python-versions = "^3.6.1"
develop = true

[package.dependencies]
ray = {url = "https://s3-us-west-2.amazonaws.com/ray-wheels/master/cd22a7d1bbf38f66aa8b735459319ff24f102a20/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl", extras = ["default"], markers = "python_version >= \"3.7\" and python_version < \"3.8\""}

I would imagine that this issue also affects vcs/file/directory dependencies. My hack won't work for those cases, obviously.

[bnorick]

Hey @abn, finswimmer told me to ping you. I did the work of finding the cause of this bug, and I am happy to work on a fix for this, just need a bit of guidance on what you think the best course of action is. Thanks!

[GeorgeWillaman]

Just bumping this. Experienced the same issue, removing the generated lock file and regenerating works. Any updates on maybe a PR?

[radoering]

Should be resolved in master via #5715

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.