DEV speed up pip install -r requirements-dev.txt by nearly 2000%

[MarcoGorelli]

• closes DEV pip install -r requirements-dev.txt takes...54 minutes? #50239 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

it looks like it may not be necessary to list boto3 and botocore, and they get pulled in anyway by s3fs and moto

EDIT:

yup, this works! and we can stop listing even more unpinned dependencies!

• botocore and boto3 are pulled in by s3fs
• beautifulsoup4 is pulled in by nbsphinx
• coverage is pulled in by pytest-cov
• fsspec is pulled in by s3fs
• gitdb is pulled in by GitPython
• ipykernel is pulled in by ipywidgets
• ipython is pulled in by ipywidgets
• jinja2 is pulled in by moto
• nbformat is pulled in by nbsphinx
• psutil is pulled in by ipywidgets
• pyyaml is pulled in by dask
• requests is pulled in by gcsfs
• sphinx is pulled in by nbsphinx

By removing them, pip install -r requirements-dev.txt goes from taking over 54 minutes, to just under 3 minutes!

EDIT2: only removing boto3 and botocore. Timing is still under 3 minutes 🥳

[jorisvandenbossche]

I am slightly -1 on this. I think it is clearer that any package that we use directly is listed in the requirements (which eg was not the case for aiobotocore). This is more explicit and also more robust (in case some other package changes its dependencies).

It is pip's "fault" that it cannot handle this properly. Of course, given that this makes the pip install so unusable, we can be pragmatic and help that on our side. But then I would personally limit it to removing the biggest culprits (eg just removing boto3/botocore already resolves most of it?), and keep being explicit for the others.

[MarcoGorelli]

But then I would personally limit it to removing the biggest culprits (eg just removing boto3/botocore already resolves most of it?)

OK, sure, as long as we can do that because pip installation is currently unusable

[mroeschke]

Agree with @jorisvandenbossche's reasoning. Would be good to keep an issue open regarding adding back boto3 and botocore if pip can more quickly resolve this in the future. Maybe also adding lower pins on all our dependencies would help the solver?

[MarcoGorelli]

Agree with @jorisvandenbossche's reasoning. Would be good to keep an issue open regarding adding back boto3 and botocore if pip can more quickly resolve this in the future. Maybe also adding lower pins on all our dependencies would help the solver?

sure, done #50300

[phofl]

Can we backport? Otherwise the docker change might take forever