Breaking change in 2.28.0 when using string enums as Headers (working in v2.27.1)

[bonastreyair]

When using string enums as key values for the headers dict parameter with requests.get() function in v2.28.0 it raises an InvalidHeader error but using previous version v2.27.1 works well without any error. It seems there was an unexpected breaking change with that release.

Expected Result

Dict keys for the headers parameter when using requests.get() are still valid if a string enum is used.

Actual Result

Instead it raises an InvalidHeader error stating that:

Header must be of type str or bytes, not enum.

Traceback (most recent call last):
  File "/home/yair/PycharmProjects/tests/requests_error.py", line 9, in <module>
    requests.get("http://URL", headers={CustomEnum.TRACE_ID: "90e85293-afd1-4b48-adf0-fa6daf02359e"})
  File "/home/yair/.local/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/yair/.local/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/yair/.local/lib/python3.8/site-packages/requests/sessions.py", line 573, in request
    prep = self.prepare_request(req)
  File "/home/yair/.local/lib/python3.8/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
  File "/home/yair/.local/lib/python3.8/site-packages/requests/models.py", line 369, in prepare
    self.prepare_headers(headers)
  File "/home/yair/.local/lib/python3.8/site-packages/requests/models.py", line 491, in prepare_headers
    check_header_validity(header)
  File "/home/yair/.local/lib/python3.8/site-packages/requests/utils.py", line 1037, in check_header_validity
    raise InvalidHeader(
requests.exceptions.InvalidHeader: Header part (<CustomEnum.TRACE_ID: 'X-B3-TraceId'>) from {<CustomEnum.TRACE_ID: 'X-B3-TraceId'>: '90e85293-afd1-4b48-adf0-fa6daf02359e'} must be of type str or bytes, not <enum 'CustomEnum'>

Reproduction Steps

from enum import Enum
import requests


class CustomEnum(str, Enum):
    TRACE_ID = "X-B3-TraceId"


requests.get("http://URL", headers={CustomEnum.TRACE_ID: "90e85293-afd1-4b48-adf0-fa6daf02359e"})

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "charset_normalizer": {
    "version": "2.0.4"
  },
  "cryptography": {
    "version": "2.8"
  },
  "idna": {
    "version": "3.2"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.10"
  },
  "platform": {
    "release": "5.13.0-48-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010106f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.28.0"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.26.6"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}

[bonastreyair]

I know it is easily solvable by making use of the CustomEnum.TRACE_ID.value to use the stored string instead of using the enum directly with CustomEnum.TRACE_ID... but it worked before with v2.27.1, so it feels like an unexpected breaking change... 👀

[nateprewitt]

Thanks for the report, @bonastreyair! So I think this is on the edge of support. Headers have always been defined to be a str or bytes and we've been programmatically enforcing that for header values for 6 years. While this worked in 2.27.1, it wasn't an intentional function of the headers param. I'd like to see how widespread this issue is before we make a decision on cases like this.

In the meantime, Requests has no special logic for Enums, so casting it to a string prior to calling Requests will not affect behavior. That would be the recommendation while we gauge impact. Otherwise, 2.27.1 will continue to work as it did originally.

[paulclarkaranz]

Here's another instance fwiw: taverntesting/tavern#788
In this case it's just a subclass of str being passed in as header values: https://github.com/taverntesting/tavern/blob/ede36ca062b546cfaf3e6b0940fbac30733586a4/tavern/util/formatted_str.py

[FelixWohlfrom]

We also have similar issues in company internal tools.
Isn't the explicit check for str and byte contradicting the duck typing approach of python? At least previously the code was working fine, even though we passed e.g. xml elements that were then implicitly converted to strings.

[sigmavirus24]

We also have similar issues in company internal tools.
Isn't the explicit check for str and byte contradicting the duck typing approach of python? At least previously the code was working fine, even though we passed e.g. xml elements that were then implicitly converted to strings.

No. It's not contradicting that because any time we magically convert things for users, it tends to be wrong about 50% of the time. Being explicit in our documentation hasn't been effective and these validity checks are designed to help users not send junk that will almost certainly not do what they want. Much of what has been explicitly described above either for narrow cases but not for everyone. Enums for example are not required to have a string values. How does one consistently string-ify it then? Support for people subclassing built-in types notorious for breaking those subclasses is also not a logical choice because again it will surprise people who try it by breaking in a weird way it doing the opposite of what they expect

[jaypatrickhoward]

We encountered this same issue when passing header values whose type is a subclass of str. Might it make sense to allow subclasses to pass validation? In this case the requests library would not be doing any conversion for the user.

[nateprewitt]

I wrote up a patch for this on Friday. I've been waiting to see how widely this approach is used, because we've only had 4 reports with ~75% of downloads (50 million/week) now using 2.28.0. I think this falls into a very niche usage, but I agree bytes/str subclasses did work for both name and value prior to this release.

We're going to monitor for any other issues in 2.28.0 to determine if we need a patch release. I'd be willing to merge the fix above into that with a heavy caveat that this isn't strictly supported. Any breakages due to subclasses deviating in behavior from the base bytes/str type likely won't be fixed.

Long term, I think we're going to continue taking a much firmer stance on inputs because people do crazy things when the values aren't bytes/str and expect them to just work. This becomes much more problematic when the requests are being sent to systems corrupted due to incorrect "stringification". We can't reasonably handle every case, so we need to draw a line somewhere on what the APIs are intended to support to provide stability.

[jaypatrickhoward]

In our case we're interacting with Salesforce, whose API consists of XML messages. We parse a session ID out of the response from the authentication endpoint using lxml. This value is of type lxml.etree._ElementUnicodeResult which is a subclass of str. We pass it to the constructor of a class from salesforce-bulk. This class uses requests under the covers to communicate with Salesforce. It naively assumes that the session ID we supplied can be passed to requests as a header value without issue.

Our fix was to cast the session ID to an actual str before giving it to salesforce-bulk.

[kevinr-electric]

Just want to add support for this issue by saying we are also having this issue. Our use is basically:

class ContentType(str, Enum):
    JSON = "application/json"
    XML = "text/xml"

...
def execute_request(...):
...
    headers = {"Content-Type": ContentType.JSON}
    response = requests.request(
                url=url,
                method=method,
                params=params,
                headers=headers,
                data=data
            )

[nickswebsite]

Using enums for header names is a common (and generally regarded as a good) practice. +1.

In the ContentType enum given above, you have what looks like a string, acts like a string, and quacks like a string. It should be treated as a string. requests shouldn't be responsible for being fed misbehaving subclasses of builtin types IMO.

Anyway, just trying to add one to the 'user engagement' since that was mentioned as a consideration in the 2.28.1 release.

[nateprewitt]

Using enums for header names is a common (and generally regarded as a good) practice. +1.

I don't believe anything in this change precludes the use of enums in your code. Requests is being explicit of what we accept. enum.name becomes enum.name.value and everything works fine.

[briantist]

Also seeing this in an Ansible collection, since the test containers updated their requirements to allow a newer version of requests.

• token must be of type str or bytes, not <class 'ansible.utils.unsafe_proxy.AnsibleUnsafeText'> ansible-collections/community.hashi_vault#289