HTTPie ignores system certificates

HTTPie ignores system certificates

http --debug -j https://example_using_my_ca.com

HTTPie 0.9.3
HTTPie data: /home/lukas/.httpie
Requests 2.10.0
Pygments 1.6
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] linux

>>> requests.request(**{'allow_redirects': False,
 'auth': None,
 'cert': None,
 'data': '',
 'files': DataDict(),
 'headers': {'Accept': b'application/json',
             'Content-Type': b'application/json',
             'User-Agent': b'HTTPie/0.9.3'},
 'method': 'get',
 'params': ParamsDict(),
 'proxies': {},
 'stream': True,
 'timeout': 30,
 'url': 'https://example_using_my_ca.com',
 'verify': True})

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.4/ssl.py", line 365, in wrap_socket
    _context=self)
  File "/usr/lib/python3.4/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib/python3.4/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 604, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/http", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/httpie/core.py", line 115, in main
    response = get_response(args, config_dir=env.config.directory)
  File "/usr/local/lib/python3.4/dist-packages/httpie/client.py", line 48, in get_response
    response = requests_session.request(**kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

For reference, curl works fine: curl https://example_using_my_ca.com

Would it make sense to use ssl.get_default_verify_paths() to get the default paths?

I would propose following behaviour:

If --verify is passed a parameter other than no or yes, pass the parameter through to requests.

If --verify is set to yes:

1. if REQUESTS_CA_BUNDLE is set, pass True to requests verify.
2. elif environment variable ssl.get_default_verify_paths().openssl_cafile_env is set, pass that to requests verify
3. elif environment variable ssl.get_default_verify_paths().openssl_capath_env is set pass that to requests verify
4. elif ssl.get_default_verify_paths().capath is not None, pass that to requests verify
5. else pass True to requests verify

but please note that ssl.get_default_verify_paths is available only since python3.4

[sigmavirus24]

@luv not only is that only available in 3.4+ and 2.7.10+ but it doesnt' work on every platform. This is why requests is working on its own issue to resolve this for people. Please stop constantly commenting on httpie. This is a concern that exists in requests and not httpie IMO.

@sigmavirus24 wtf man? No idea what you mean by constantly commenting, but whatever, I see you are not even a httpie contributor. I assume just a stupid troll :/

a quick glance at the curl source code to see what a working implementation looks like ....

Disregarding all those amiga and VMS ifdefs and support for many different ssl libraries, it's actually pretty stupid and it is not using openssl X509_* lookup methods (as, for example, used in get_default_verify_paths() in Python SSL module).

Instead, curl simply iterates over a bunch of known locations at the compile time (see this m4dness https://github.com/curl/curl/blob/master/acinclude.m4#L2560 ) and then explicitly supports overriding with (CURL_CA_BUNDLE and) SSL_CERT_DIR and SSL_CERT_FILE environment variables at runtime (again not using stuff like X509_get_default_cert_file_env()). That's it.

So what about implementing the same approach in pure python? (Yes it looks quick and dirty as hell but it works for curl!) but adding support for windows (ssl.enum_certificates ?) and OS X (not sure here but python seems to use the apple-supplied openssl library on OS X 10.6+ so that should be fine already!).

[jkbrzt]

@luv thanks for the report but please be more respectful to other members. @sigmavirus24 is a core developer of requests on which httpie relies for all HTTP and without which it wouldn't even exist, so his view is extremely relevant.

please note this wont get fixed in requests until 3.0.0 release ... not even SSL_CERT_FILE is going to be supported because of possible compatibility issues https://github.com/kennethreitz/requests/pull/2903

btw. I didn't mean to be disrespectful I really thought s/he was trolling :) I went out of my way to help find a fix for an issue and someone who has not even once contributed to the project (I checked HTTPie's contributors because I was really suprised by such a hostile reaction) was basically telling me to shut up :/

[sigmavirus24]

@luv I was not telling you to shut up. There are 763 people subscribed to this issue. 100% of them might receive emails for it (but it's probably more like 70% or ~534). That means you generated (in the 2 hours after the issue was opened) ~1602 emails (possibly more or less). In other words, you're spamming people with stream of conscious posts like this. The best way to document new information when no one has replied to you is by editing the original post.

Finally code contributions are not the only contribution to a project. If you want to see all the contributions on a project you might use a project like octohatrack. I've responded to several requests related bugs on httpie for @jkbrzt because they don't quite have the time to follow requests' development. When I see a requests related bug here, I respond because 99% of the time, it's already being dealt with. Luckily for @jkbrzt, I've been treated as poorly (and a few times worse) before so I have thicker skin.

As a side note, "they" is as many characters as "s/he" and far more generally applicable since it can be used to refer to a single person as well.

so are we getting this fixed?

[sigmavirus24]

@jkbrzt I would call this a feature, not a bug. It's intentional that Requests works identically across Windows, *nixes, and BSDs and in fact, many system distributions of Requests actively remove that behaviour and point towards the system certificate store/bundle. So if users need this behaviour, they can use the version of Requests packaged by their system distributors with HTTPie (and possibly the system packaged HTTPie).

As for "getting this fixed", it depends on your definition of fixed. This a feature that will be added to Requests. At which point HTTPie will get the behaviour for free. If HTTPie instead deems this a higher priority (which @jkbrzt may do as they see fit) they can duplicate development effort to create a release which does this sooner.

As it seems you're using linux, @luv, you can take advantage of the fact that your distribution of linux almost certainly has a version of Requests that uses the system certificate store today. Requests is packaged in every distribution of linux I have checked and pretty much every one of those distributions uses their system certificate store. If that is not the case, perhaps you should be filing a bug with the Requests package maintainer of that distribution.

same prob with HTTPie installed via apt (ubuntu 14.04lts)

[sigmavirus24]

@luv that's surprising because I know for a fact that requests on 14.04 and 16.04 use the system certificate store. How did you install Requests?

I purged all "requests" and "httpie" versions to verify the ubuntu distribution of httpie is used and you are right I was still using httpie from PyPI. Ubuntu httpie fails with "ImportError: cannot import name is_windows" which is a known-issue.

I think I forgot to run "pip3 uninstall" and ran only "pip uninstall" as I had to use python3 to get working ssl in the first place.

I apologize to 761 persons to "spam" them with description of my httpie setup.