Poetry package install time seems longer than installing with pip

[MichaelAquilina]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
  If an exception occurs when executing a command, I executed it again in debug mode (-vvv option). (Does not apply)

• OS version and name: docker image (python:3-7-stretch) running on CircleCI
• Poetry version: 0.11.2
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/MichaelAquilina/S4/blob/5e3a103/pyproject.toml

Issue

I am reasonably sure that installing packages using poetry install instead of pip install takes significantly longer.

To compare. Here is a pip install taking 35 seconds: https://circleci.com/gh/MichaelAquilina/S4/464#action-103

Here is another build with poetry install and the same requirements taking 1:27 seconds: https://circleci.com/gh/MichaelAquilina/S4/521#action-104

In both cases, both dev and non dev requirements were installed.

[sdispater]

Thanks for your interest in Poetry!

This is expected because Poetry orders the packages to install so that the deepest packages in the dependency graph are installed first to avoid errors at installation time. This requires sequential installation of the packages which takes longer but is more "secure".

Also, Poetry checks hashes of installed packages for security reasons and due to the way pip works, Poetry has to generate a temporary requirements.txt file to make pip check hashes. This adds an overhead which explains the difference between the two tools.

[MichaelAquilina]

Thanks for taking the time to explain :)

Would it be possible for poetry to just download the target packages in parallel and install in sequence to speed up the process?

[sdispater]

I intend to improve the installation part of Poetry sometime in the future to speed things up, yes. I can't give you an ETA though, the only thing I can tell you is that it will likely be after the 1.0 milestone since I want to stabilize Poetry before changing critical parts, like this one.

I'll be sure to keep you posted if anything changes on that front.

[MichaelAquilina]

That sounds great @sdispater. Of course I fully understand your reasoning and agree that ensuring stability is a lot more important than performance.

Thanks for the great work on Poetry!

[dbarrosop]

Let me start by saying I recently discovered poetry and I think it's awesome :)

Now, is there any progress on this? I just migrated a couple of projects to poetry and my container builds went from 2s to 5 minutes each. The reason why it's so slow is twofold:

1. When building containers I'd do something like this:

ADD requirements.txt /tmp
RUN pip install -r requirements.txt

ADD . /project
...

so I could cache that layer. With poetry I have to do:

ADD ./project
RUN cd /project && poetry install

Which means I need to install the dependencies over and over even though I haven't changed a single dependency. I am trying to do something equivalent with:

ADD poetry.lock /tmp
ADD pyproject.toml /tmp
RUN cd /tmp && /root/.poetry/bin/poetry update --no-dev

But I am not sure if this is the right thing to do but does the job. Feels extremely hacky so I'd appreciate some advice :P

2. The second issue is the one described here. A clean install takes ~30s with pip and ~5minutes with poetry :( Monitoring the network I can see how pip downloads everything very fast at a few MB/s while poetry does it one by one never reaching more than a dozen KB/s.

[DanCardin]

Re this previous comment, it would be ideal if there was an interface to install when only the lock file is available, so that we can continue to make good use of docker layer caching. Because I haven't been able to get an install to succeed when the source isn't also available.

[hvdklauw]

Just did a comparison for out 190 package requirement project (according to poetry) and the time difference between pipenv en poetry was the following:

Poetry (prerelease):   201.54 real       138.98 user        43.05 sys
Pipenv:                        75.50 real       200.14 user        51.65 sys
Pip with hashes:         83.60 real        53.16 user        21.66 sys

Now yes, pipenv probably doesn't take installation order into account and just installs them concurrently and yes I have had the probably where a package needed it's dependencies to install correctly (which shouldn't be the case, but sometimes you don't have control) but still, I think if you look at the dependency tree you can run a lot of leaves and branches in parallel without this being an issue because they don't meet up until the end.

Still, pip with a requirement file with hashes is a lot faster....

[pmav99]

Still, pip with a requirement file with hashes is a lot faster

It feels like you are comparing apples to oranges here. Pip doesn't resolve dependencies. Poetry install does.

[aequitas]

However those dependencies are only resolved during the lock phase, the install phase (that is performed more often than locks) just installs the packages.

We could consider to do some time expensive ordering/parallelization calculations during the lock phase?

[hvdklauw]

Yeah, the dependency resolving taking long is fine, then doing a poetry export -f requirements.txt to generate the requirements file with all dependencies and installing that with pip is faster then just using pure poetry.

Like I said: that misses the ordering in installing which seems to be only needed if not doing it causes issues.

Maybe the solution would be to give the install command a flag (and environment variable) to install concurrently to speed it up if ordering is not important. Then the default will be the slower and safe, but if you know it's not needed it will speed up CI build times significantly.

[hvdklauw]

Also I just saw this in the pip documentation, stating that they also install packages in order.. now I'm really confused about what poetry is doing to make it go so much slower...

[matthiasgoergens]

@hvdklauw Pip documentation just says that they are installing in topological order. That would still allow parallelism for packages that are not forced into a specific order by a dependency relationship.

(But looking at the discussions around pip, they don't do parallel installs either.)