Picking wrong platform installing a dependency with multiple constraints

[leon19]

• I am on the 1.0.5 Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04
• Poetry version: 1.0.5
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/leon19/42bf22ddc791e0500c614d461c32a486

Issue

Markers are not working properly with multiple constraints dependencies as stated in the docs

I am trying to install a dependency from an URL based on the platform, different URLs for every platform (markers = "sys_platform == 'linux'") and poetry is picking the URL from the platform darwin when I'm under a linux platform

I have not tested yet what happens on a Darwin platform. Will update the issue when I do

Update:

If I reverse the order of the dependencies, the first line the darwin url, it works on Linux. The same behaviour applies on Darwin

[abn]

@leon19 try chaging it to the following (this is only a partial workaround - see below).

guildai = [
  {url = "https://files.pythonhosted.org/packages/10/ed/ffc1a004da1a272131e776f4b1d24d980b1dbb25b3bbaea4aff5ff7ef538/guildai-0.7.0rc7-cp37-cp37m-manylinux1_x86_64.whl", platform = "linux"},
  {url = "https://files.pythonhosted.org/packages/17/bc/2bc49e33d9343e689723c0b67aa45653d79fc4b5342c12bd7836aca58e7b/guildai-0.7.0rc7-cp37-cp37m-macosx_10_14_x86_64.whl", platform = "darwin"}
]

This will work if you are on a linux environment. I did notice that the generated lockfile will only contain the first specified url (in this case command will succeed since I was working on a linux machine and linux platform was specified first). If I were to change the order, it will fail again. However, when building the wheel generated the correct metadata.

Requires-Dist: guildai @ https://files.pythonhosted.org/packages/10/ed/ffc1a004da1a272131e776f4b1d24d980b1dbb25b3bbaea4aff5ff7ef538/guildai-0.7.0rc7-cp37-cp37m-manylinux1_x86_64.whl; sys_platform == "linux"
Requires-Dist: guildai @ https://files.pythonhosted.org/packages/17/bc/2bc49e33d9343e689723c0b67aa45653d79fc4b5342c12bd7836aca58e7b/guildai-0.7.0rc7-cp37-cp37m-macosx_10_14_x86_64.whl; sys_platform == "darwin"

From what I can tell, when markers are explicitly used the constraints get populated differently.

Issue 1: This pertains to the use of "url". When duplicate dependencies are being merged, url and markers are ignored resulting in only the first entry to be used (in this case the linux url). Ideally, the dependency graph should be branched due to the existance of the platform marker and the url should be considered when merging duplicates.

Additionally, during investigation, I had a quick look at how this was being processed. It did raise a few questions (which might be separate issue(s) to be dealt with).

When markers are explicitly provided (eg: markers = "sys_platform == 'win32'", python = ">=3.6");

Issue 2: the python version specified is ignored if markers are explicitly specified (unsure if this is expected behaviour).

poetry/poetry/packages/package.py

Lines 349 to 359 in ae6d64d

	if not markers:
	marker = AnyMarker()
	if python_versions:
	dependency.python_versions = python_versions
	marker = marker.intersect(
	parse_marker(
	create_nested_marker(
	"python_version", dependency.python_constraint
	)
	)
	)

Issue 3: sole use of markers causes solver to be stuck in an infinite loop.
As an example, this works.

pypiwin32 = [
    { version = "220", platform = "win32", python = ">=3.6"},
    { version = "219", platform = "win32", python = "<3.6"}
]

However, this causes an infinite loop.

pypiwin32 = [
    { version = "220", markers = "sys_platform == 'win32' and python_version >= '3.6'"},
    { version = "219", markers = "sys_platform == 'win32' and python_version < '3.6'"}
]

@finswimmer would be good to get your thoughts on the above.

[leon19]

@abn I tried using platform instead of markers as suggested but it still does not work for me

Note: If I reverse the order of the dependencies, the first line the darwin url, it works on linux (I haven't tested on mac yet)

[abn]

@leon19 looks like it is non-deterministic.

[breki]

I have a similar problem. python-ldap package won't build on Windows, so I have to use an unofficial wheel file. I wanted to tell Poetry that it should use the official package for Linux and the locally stored wheel for Windows:

python-ldap = [
    { markers = "sys_platform == 'linux'", version = "*" },
    { markers = "sys_platform == 'win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }

but, judging from the poetry.lock file, it seems markers are then merged and just determine whether the library should be installed at all:

[[package]]
category = "main"
description = "Python modules for implementing LDAP clients"
marker = "sys_platform == \"linux\" or sys_platform == \"win32\""
name = "python-ldap"

I've also asked a question on StackOverflow.

[abn]

@breki try this instead.

python-ldap = [
    { platform = "linux", version = "*" },
    { platform = "win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }
]

[breki]

@breki try this instead.

python-ldap = [
    { platform = "linux", version = "*" },
    { platform = "win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }
]

@abn, tried that one as well, no difference.

[eterna2]

any solution so far? I am having problem because of pyTorch. They did not release a windows package. And my dev env is a crappy windows, but my production env is linux.

It is very hard to make my docker build to work with poetry.

[iramisvalentincapco]

@abn or @eterna2 any new findings or workarounds for this issue?

I am encountering the same behavior using multiple constraints for a package using git and path dependencies based on platform.

The poetry export -f requirements.txt command seems to generate the requirements in the following way: given multiple constraints, poetry will simply use the first entry in the list as the sole dependency and attach any constraints/markers onto it via a union (markers separated by "or") This means that regardless of if the given constraints should conflict (which is the intention of the different constraints), it will simply install the first listed dependency if it meets any of the given constraints.

From my understanding, this is not the expected behavior, as multiple constraints dependencies are meant to allow for defining different locations for retrieving a given dependency.