Streamed request with preset Host header sends two Host headers

[mjpieters]

The following code sends two host headers:

import requests

def gen(): yield b'binary data'

requests.post(
    'http://localhost:3000/', 
    data=gen(),
    headers={'Host': 'example.com'}
)

(I tested with nc -vvl 3000 on OS X).

Headers received:

POST / HTTP/1.1
Host: localhost:3000
User-Agent: python-requests/2.18.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Host: example.com
Transfer-Encoding: chunked

This is because when chunking a request, we don't tell the httplib / http.client library to skip the host header if one is already set. From the adapters.HTTPAdapter.send() implementation:

                    low_conn.putrequest(request.method,
                                        url,
                                        skip_accept_encoding=True)

which leaves the skip_host parameter set to False. It probably should use skip_host='host' in request.headers here.

System Information

>>> requests.help.main()
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.0.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.3"
  },
  "platform": {
    "release": "16.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010006f",
    "version": "17.3.0"
  },
  "requests": {
    "version": "2.18.1"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}

[nateprewitt]

Yep, this looks like a miss from #3637. Since we're not using urllib3's chunked request interface, we never got the change integrated in Requests.

The quick path here is to carry the host dedupe logic from urrlib3 into that try block. The better but likely more involved solution would be to try to use urllib3's chunked logic in Requests. This is something we've discussed before but probably out of scope for this issue.

@requests/core I'm assuming the former option is the advisable one, but do you have different thoughts?

[Lukasa]

I think we can probably use urllib3’s implementation: it’s present in all our supported versions of urllib3 I think.

[sigmavirus24]

If we're going to swap in urllib3's chunked logic, we need better end-to-end (socket level) tests of our chunking behaviour to make sure we don't drastically change things.

[jlherren]

Does anyone know a work-around to avoid this double host header?

I need to access a frustrating and honestly quite stupid API that I have no control over, which requires the host header to be some kind of information totally unrelated to the actual hostname. And it requires chunked encoding.

[mjpieters]

Currently the best work-around you have is to not use requests and use urllib3 directly.

[judofyr]

Any update on this?

[nateprewitt]

Linking #5391 back here as a potential fix as converting to urllib3's chunked functionality is unlikely to happen before a major version bump.

[sethmlarson]

Requests is now using urllib3's chunking API