Poetry v1.1.6 does not respect markers or version specifiers when generating lockfile

[tigerhawkvok]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 v20H2 build 19042.867
• Poetry version: 1.1.6
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/tigerhawkvok/b7e1e54872983148d6599b1180cac67f

Issue

This worked on my prior Poetry build (possibly as old as v0.12, but I updated today to v1.1.6). Neither the python nor the markers key seems to be respected in pyproject.toml.

An illustrative screenshot:

Full sample:
poetry_dep_bug.zip

If I manually edit my lockfile, I can recover the expected behaviour

[[package]]
name = "pycurl"
version = "7.43.0.4"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.6, < 3.7"

[package.source]
type = "file"
url = "pycurl-7.43.0.4-cp36-cp36m-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.4"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.7, < 3.8"

[package.source]
type = "file"
url = "pycurl-7.43.0.4-cp37-cp37m-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.4"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.8, < 3.9"

[package.source]
type = "file"
url = "pycurl-7.43.0.4-cp38-cp38-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.5"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.9, < 3.10"

[package.source]
type = "file"
url = "pycurl-7.43.0.5-cp39-cp39-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.6"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">=3.5"
marker = "platform_system != 'Windows'"

[tigerhawkvok]

Local testing shows that the marker was preserved at least as recently as 1.0.5, though the version resolution per-file was still broken:

[[package]]
category = "main"
description = "PycURL -- A Python Interface To The cURL library"
marker = "python_version ~= \"3.6.0\" and platform_system == \"Windows\" or python_version ~= \"3.7.0\" and platform_system == \"Windows\" or python_version ~= \"3.8.0\" and platform_system == \"Windows\" or python_version ~= \"3.9.0\" and platform_system == \"Windows\""
name = "pycurl"
optional = false
python-versions = "*"
version = "7.43.0.4"

[package.source]
reference = ""
type = "file"
url = "pycurl-7.43.0.4-cp36-cp36m-win_amd64.whl"

It's possible this never really worked but it at least picked the version on your system that you created a venv for in 0.12.16:

[[package]]
category = "main"
description = "PycURL -- A Python Interface To The cURL library"
marker = "python_version >= \"3.7.0\" and python_version < \"3.8.0\" or python_version >= \"3.8.0\" and python_version < \"3.9.0\" or python_version >= \"3.9.0\" and python_version < \"3.10.0\" or python_version >= \"3.6.0\" and python_version < \"3.7.0\""
name = "pycurl"
optional = false
python-versions = "*"
version = "7.43.0.4"

[package.source]
reference = ""
type = "file"
url = "pycurl-7.43.0.4-cp37-cp37m-win_amd64.whl"

[rschmied]

Not 100% sure if this is the same issue or if I should open a separate one... but I'm seeing something very similar since I moved my env from 3.6.8 to 3.8.5.

In my case this manifests itself with the "typing-extensions" package which is a dependency of various packages I use. For most of them the dependency is declared as

typing-extensions = ">=3.7.4"

However, yarl declares it as

[[package]]
name = "yarl"
version = "1.6.3"
description = "Yet another URL library"
category = "main"
optional = false
python-versions = ">=3.6"

[package.dependencies]
idna = ">=2.0"
multidict = ">=4.0"
typing-extensions = {version = ">=3.7.4", markers = "python_version < \"3.8\""}

which results in `typing-extensions' not being installed on Python 3.8.5... resulting in

12:49:29  Ignoring typing-extensions: markers 'python_full_version >= "3.6.8" and python_full_version < "4.0.0" and python_version >= "3.6" and python_version < "3.8"' don't match your environment

when installing (via poetry export --format=requirements.txt >requirements.txt and pip3 install -r requirements.txt. It seems as if Poetry takes the version restriction of one package (yarl) and extrapolates the version restriction to all packages. However, in my case, aiohttp still has a hard requirement on the package (even though they should not, I'd say) and fails without it being present.

Update:

looking at the dependencies from yarl, there's no "marker"... Who/What/When creates those?

$ poetry show yarl
name         : yarl
version      : 1.6.3
description  : Yet another URL library

dependencies
 - idna >=2.0
 - multidict >=4.0
 - typing-extensions >=3.7.4
$ 

[tgolsson]

I've got a fork that's solving some of the issue re: missing markers; master...tgolsson:ts/markers-in-lockfile. However, while doing the final cleanup and fixing the last tests I noticed that this was part of an intentional change by @sdispater two years ago, #2361.

While I understand the motivations of that PR and (some of) what it solves, it leads to forgetting about "top-level" dependencies with markers in the actual pyproject.toml. My PR adds those back and then makes the version solver deal with them again (likely in a far too simplistic manner in light of the original PR). However, the lockfiles that are currently generated without my changes are broken if I'm using markers to toggle versions in my pyproject.toml:

torch = [
    { version = "1.10.2+cu111", markers = "sys_platform != 'darwin'" },
    { version = "1.10.2+cpu", markers = "sys_platform == 'darwin'" }
]

Without the proper information in the lockfile both torch deps are identical.

One alternative solution route would be to generate the project itself into the lockfile, so if our project is foo we'd depend on both, with the proper markers at that level. However, that is a bigger change, and I'm quite sure it'd be both a breaking change in compatibility between poetry.lock and poetry, in both directions.

I realize a lot of time has passed @sdispater, but do you remember enough to have any input on solutions here?

(Edit: I think that the issue @rschmied posted above is different from markers for primary dependencies, and my fix would have no effect. That seems like a bug in the constraint solving itself.)

[tgolsson]

OK; so more investigation and... On latest poetry it seems like if I ensure that there's no overlap anywhere in my versions; it works out fine. I was getting tricked by my fixes because I'd started with one case and then started changing Poetry to fix what was likely an error in my pyproject.toml, and then I eventually fixed that along the way but kept digging.

So - I started with the following:

torch = [
    { version = "1.10.2+cu111", source="foo", markers = "sys_platform != 'darwin'" },
    { version = "1.10.2", markers = "sys_platform == 'darwin'" }
]

And this indeed exhibits some bad behavior, at least when toggling which version belongs on which platform. In particular, 1.10.2+cu111 is a valid subversion for 1.10.2, so there's no fundamental difference between a naked 1.10.2 and the cu111. So, that's what I tried to solve. Later on after doing more experiments I'd unwittingly switched to the +cpu version for some other testing to see if using another source would work better. This changed the version resolution functionality, because the local tags are different.

I thus think this issue is aimed at solving the wrong problem -- the markers not being in poetry.lock isn't the source of this problem because they are seemingly read from pyproject.toml. I think there's good reasons to have them in poetry.lock for other reasons, but my testing over the last two days conclude that it's not required for poetry install to correctly work with markers.

Now, one important difference between this and OP's post is that I'm using 1.2.0b1 to test, and the issue was reported for 1.1.6. However, there's clearly bugs with version resolution still -- for example, if I test

torch = [
    { version = "1.10.2+cu111", source="foo", markers = "sys_platform != 'linux'" },
    { version = "1.10.2", source="bar", markers = "sys_platform == 'linux'" }
]

It'll helpfully offer only +rocm local tags... Not quite what I had in mind. This might be related to @rschmied's issues above.

Anyways, long story short: I went into a rabbit hole hunting a one year old bug, did a lot of work that was unnecessary, and finally undid everything. As redemption I tested your zipfile @tigerhawkvok and with fresh installed poetry==1.2.0b1 and Python 3.8, I get

$ poetry install
Creating virtualenv poetry-dep-bug-XteSQS2G-py3.8 in <snip>\AppData\Local\pypoetry\Cache\virtualenvs
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pycurl (7.43.0.4 <snip>/poetry_dep_bug/pycurl-7.43.0.4-cp38-cp38-win_amd64.whl)

Which seems to be what I should expect.

[dimbleby]

#5715, probably

[john-sandall]

I'm having similar issues with environment markers for torch dependencies being ignored when generating poetry.lock, if I add them in manually it all works fine. It may be because I'm using a local wheel and therefore seems more related to this issue: #2765 (comment)

Leading me to this unpleasant (hopefully very temporary!) solution: https://gist.github.com/john-sandall/05f03d6e39978c1aef29e00dd61e253a

[neersighted]

Duplicate #2765 for the original issue -- the people chiming in in the comments have a different issue that should be fixed