TypeError: HTTPConnectionPool.urlopen() got multiple values for argument 'headers'

[kjprice]

Describe your environment

OS: MacOSX
Python version: 3.10
Package version: 0.60b1

What happened?

The Instrumentation Bug (init.py:334-369):

• Line 334: _prepare_headers(kwargs) modifies kwargs in-place, setting kwargs["headers"]

• Line 365: Injects tracing headers into that modified headers dict

• Line 369: Calls wrapped(*args, **kwargs) with the modified kwargs

When urllib3's urlopen() makes a recursive call to itself (for retries, redirects, or connection pooling),
Python receives headers as BOTH:

• A positional argument (from args)

• A keyword argument (from modified kwargs)

Steps to Reproduce

We are using pinecone (pinecone[grpc]==7.0.0), which uses urlib3. This is a difficult bug to reproduce. We have only been able to recreate it in our production environment, so it might have something to do with the increased volume but we're not sure. However, we are pretty certain that we have uncovered what is going on (see the other section for more information).

Expected Result

We expected telemetry for urlib3 (used by Pinecone).

Actual Result

An error occurred. See a subset of the stacktrace here:

  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/pinecone/openapi_support/api_client.py", line 170, in __call_api
    response_data = self.request(
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/pinecone/openapi_support/api_client.py", line 386, in request
    return self.rest_client.POST(
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/pinecone/openapi_support/rest_utils.py", line 146, in POST
    return self.request(
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/pinecone/openapi_support/rest_urllib3.py", line 190, in request
    r = self.pool_manager.request(
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 143, in request
    return self.request_encode_body(
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 457, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/urllib3/__init__.py", line 369, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 942, in urlopen
    return self.urlopen(
  File "/home/appuser/app/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/urllib3/__init__.py", line 369, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
TypeError: HTTPConnectionPool.urlopen() got multiple values for argument 'headers'

Additional context

No response

Would you like to implement a fix?

None

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[xrmx]

@kjprice Thanks for reporting, do you have a reproducer (i.e. a call to urllib3 like pinecone does) at hand?

[rite7sh]

I dug into this quite a bit and I think the root cause is mutation of kwargs inside the urllib3 wrapper.

In instrumented_urlopen, _prepare_headers(kwargs) mutates the original kwargs in-place. In recursive internal calls inside urllib3 (e.g. retries, proxy tunneling, pooling as mentioned by @kjprice ), headers can be passed positionally, and mutating kwargs results in headers being passed both positionally and via keyword, which leads to:

TypeError: HTTPConnectionPool.urlopen() got multiple values for argument 'headers'

I was able to trigger deep recursive urlopen() call paths locally (proxy + retries), and confirmed that avoiding in-place mutation of kwargs prevents this class of failures.

Proposed fix: make a shallow copy of kwargs before injecting headers, and call the wrapped function with the copied dict instead, like :

new_kwargs = dict(kwargs)
headers = _prepare_headers(new_kwargs)
...
response = wrapped(*args, **new_kwargs)

Happy to open a PR with this change if this approach sounds reasonable.

[kjprice]

Thanks y'all! I played around with it and came up with this minimal example that should show the error:

# reproduce_buy.py
import urllib3

from opentelemetry.instrumentation.urllib3 import URLLib3Instrumentor

URLLib3Instrumentor().instrument()


pool = urllib3.HTTPConnectionPool("httpbin.org", port=80)

response = pool.urlopen(
    "GET",  # method - position 0
    "/",  # url - position 1
    None,  # body - position 2
    {"User-Agent": "test"},  # headers - position 3 (POSITIONAL!)
)

# pyproject.toml
[project]
name = "otel-error"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.10.13"
dependencies = [
    "opentelemetry-instrumentation-pinecone>=0.51.0",
]

uv run reproduce_buy.py

Traceback (most recent call last):
  File "/Users/kprice/repos/bhr-ai-labs-document-loader/otel-error/reproduce_buy.py", line 22, in <module>
    response = pool.urlopen(
  File "/Users/kprice/repos/bhr-ai-labs-document-loader/.venv/lib/python3.10/site-packages/opentelemetry/instrumentation/urllib3/__init__.py", line 369, in instrumented_urlopen
    response = wrapped(*args, **kwargs)
TypeError: HTTPConnectionPool.urlopen() got multiple values for argument 'headers'

[kjprice]

UPDATE: See PR.

@rite7sh I like your fix as it is minimal! However, I tried your solution but can still see the error. Maybe I don't fully understand your implementation though.

Here's the change I made which seems to fix the error:

# Line 333
method = _get_url_open_arg("method", args, kwargs).upper() 

# FIX: Handle headers being passed as positional argument (position 3)
# urllib3 signature: urlopen(self, method, url, body=None, headers=None, ...)
args_list = list(args)  # Convert to mutable list

# If headers was passed as positional arg (position 3), extract it
if len(args_list) > 3:
    positional_headers = args_list[3]
    args_list = args_list[:3] + args_list[4:]  # Remove headers from position 3
    args = tuple(args_list)
    # Merge with any headers in kwargs (kwargs takes precedence)
    if positional_headers is not None:
        if "headers" not in kwargs or kwargs["headers"] is None:
            kwargs = dict(kwargs)  # Make mutable copy
            kwargs["headers"] = positional_headers
        else:
            # Merge: kwargs headers override positional headers
            merged = dict(positional_headers) if isinstance(positional_headers, dict) else {}
            merged.update(kwargs["headers"])
            kwargs = dict(kwargs)
            kwargs["headers"] = merged

# Line 334
headers = _prepare_headers(kwargs)

It's not beautiful (I'm sure a helper method would be helpful).

@rite7sh what do you think?

[kjprice]

I created this PR #4143

This is my first time and honestly I don't have a huge amount of time to contribute to the issue, but I think this will solve the problem.

[rite7sh]

@kjprice That makes sense — thanks for digging into this and sharing the patch + repro, this definitely clarifies the remaining edge case.

I agree this approach looks correct. Handling positional headers explicitly and normalizing everything into kwargs before calling _prepare_headers feels like the safest way to avoid the double-passing issue, especially given how urllib3 internally recurses through urlopen().

If readability becomes a concern, we could potentially move this normalization logic into a small helper to keep instrumented_urlopen easier to follow, something along the lines of:

def _normalize_headers_args(args, kwargs):
    args_list = list(args)
    positional_headers = None

    if len(args_list) > 3:
        positional_headers = args_list[3]
        args_list = args_list[:3] + args_list[4:]

    new_kwargs = dict(kwargs)
    if positional_headers is not None:
        if new_kwargs.get("headers") is None:
            new_kwargs["headers"] = positional_headers
        else:
            merged = dict(positional_headers) if isinstance(positional_headers, dict) else {}
            merged.update(new_kwargs["headers"])
            new_kwargs["headers"] = merged

    return tuple(args_list), new_kwargs

Then in instrumented_urlopen:

args, kwargs = _normalize_headers_args(args, kwargs)
headers = _prepare_headers(kwargs)

Not strictly necessary, but might help keep the main wrapper logic cleaner.

One other thought: it might be worth adding a small regression test that explicitly passes headers positionally with instrumentation enabled, just to make sure we don’t regress on this edge case in the future.

Overall though, this looks solid — happy to help test this further if useful.

[rite7sh]

I created this PR #4143

This is my first time and honestly I don't have a huge amount of time to contribute to the issue, but I think this will solve the problem.

This looks great — using inspect.signature + bind to canonicalize the arguments is a really clean approach and makes the wrapper much more robust to future signature changes.

One small thought: it might be worth adding a regression test that explicitly passes headers positionally with instrumentation enabled, just to lock this behavior in and avoid regressions. Otherwise this approach looks solid to me @kjprice

[kjprice]

Hey @shalevr, thanks for getting PR #4144 merged!

Could we get a release of opentelemetry-instrumentation-urllib3 that includes this fix?
We're blocked on deploying this to production.

cc @aaronabbott @lzchen @kiorky

Thank you all very much! I'm incredibly impressed by how quickly we got this done!