DeprecationWarnings in vendored requests

[mvanbaak]

The vendored requests library is emitting DeprecationWarnings:

lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/_collections.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableMapping

This has been fixed upstream here: psf/requests#4501

Since I do not know how you handle the vendored requests, I have no idea how to make a PR to fix it. sorry.

Can someone pick this up? It's very annoying while running tests. Or give me a hint how to create a PR for the vendored requests, and I'll create one.

[JordonPhillips]

What are you doing to trigger that? botocore moved off of using the vendored requests a while ago, it's only kept in for backwards compatibility of certain exceptions. I can't get the warning to trigger by using botocore itself.

[mvanbaak]

Runnning pytest for a project that uses botocore is enough to trigger this warning.

[joguSD]

I've confirmed this, using 3.7.1 and pytest. Simply importing a botocore session is enough to trigger this. I think we need to come up with a more long term plan for handling our vendored version of requests.

[schlomo]

Annoying for any pytest user, would be nice if you can fix it. Maybe just try importing from collections.abc and fall back to importing from collections? Python style is just try and catch error instead of checking for Python versions.

[zgoda-mobica]

Python 3.7.2, pytest-4.0.2

The only boto3 related imports in the whole code base:

import boto3
from boto3 import Session
from botocore.client import Config
from botocore.exceptions import ClientError

[TomAugspurger]

Here's a simple example

import warnings
warnings.simplefilter("error", DeprecationWarning)

import boto3

That raises with

$ python bug.py
Traceback (most recent call last):
  File "bug.py", line 4, in <module>
    import boto3
  File "/private/tmp/tmp/lib/python3.7/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/private/tmp/tmp/lib/python3.7/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/session.py", line 29, in <module>
    import botocore.configloader
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/compat.py", line 25, in <module>
    from botocore.exceptions import MD5UnavailableError
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/exceptions.py", line 15, in <module>
    from botocore.vendored import requests
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/vendored/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/vendored/requests/packages/__init__.py", line 3, in <module>
    from . import urllib3
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 38, in <module>
    from .response import HTTPResponse
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 9, in <module>
    from ._collections import HTTPHeaderDict
  File "/private/tmp/tmp/lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/_collections.py", line 1, in <module>
    from collections import Mapping, MutableMapping
  File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
  File "/usr/local/Cellar/python/3.7.2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/collections/__init__.py", line 52, in __getattr__
    DeprecationWarning, stacklevel=2)
DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working

Version info:

boto3            1.9.73
botocore         1.12.73

[TomAugspurger]

botocore moved off of using the vendored requests a while ago

It seems to be imported at the top level in

botocore/botocore/exceptions.py

Line 15 in 2bd9d91

from botocore.vendored import requests

Should that be in a

try:
    import requests
except ImportError:
    from botocore.vendored import requests

?

[therc]

I triggered a different, but related problem somewhere else:

File "exedir/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 12, in <module>
    from Queue import LifoQueue, Empty, Full
ImportError: cannot import name LifoQueue

This is complicated to trigger, as I suspect it requires a) running on a case-insensitive FS (don't ask...) and b) having a queue.py (note the lower case) file somewhere in your search path.

Anyway, my problem is probably obviated by recent changes in source layout in urllib3 (1.23 and greater).