[PULP-1036] /dev/urandom file handle leak for content app downloads (on_demand or streamed) results in: OSError: [Errno 24] Too many open files

[quba42]

Version

We have verified this affects Katello 4.18 (pulpcore-packaging 3.73 using Python 3.12) as well as Satellite 6.18.0 (which is based on these versions).
We have also verified it does not affect Katello 4.16 (pulpcore-packaging 3.63).
For Katello 4.19 (pulpcore-packaging 3.85) we sometimes see a open /dev/urandom but far more rarely than on 3.73 (to the point where we were not able to reach any limits).

Describe the bug

The bug occurs whenever something asks for a file from the content app (pulpcore-content process) that is not yet held by Pulp, because the policy was either on_demand or streamed. For on_demand it does not happen if the same file is downloaded from Pulp again, for streamed it happens every time.

What happens is that either pulpcore or some dependency, opens a file handle on /dev/urandom, that is never closed again. (Until the pulpcore-content worker is restarted).

If you start enough downloads, you will eventually run into a OSError: [Errno 24] Too many open files as a result. This may present as a "500 Internal server error" to users, but in the logs we see the following trace:

pulpcore-content[10625]: [2025-12-08 09:53:02 +0000] [10625] [ERROR] Error handling request
pulpcore-content[10625]: Traceback (most recent call last):
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/app/models/repository.py", line 567, in download_factory
pulpcore-content[10625]:     return self._download_factory
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]: AttributeError: 'AptRemote' object has no attribute '_download_factory'. Did you mean: 'download_factory'?
pulpcore-content[10625]: During handling of the above exception, another exception occurred:
pulpcore-content[10625]: Traceback (most recent call last):
pulpcore-content[10625]:   File "/usr/lib64/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
pulpcore-content[10625]:     resp = await request_handler(request)
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib64/python3.12/site-packages/aiohttp/web_app.py", line 567, in _handle
pulpcore-content[10625]:     return await handler(request)
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib64/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
pulpcore-content[10625]:     return await handler(request)
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/content/authentication.py", line 49, in authenticate
pulpcore-content[10625]:     return await handler(request)
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/cache/cache.py", line 354, in cached_function
pulpcore-content[10625]:     response = await self.make_entry(
pulpcore-content[10625]:                ^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/cache/cache.py", line 400, in make_entry
pulpcore-content[10625]:     response = await handler(*args, **kwargs)
pulpcore-content[10625]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/content/handler.py", line 268, in stream_content
pulpcore-content[10625]:     return await self._match_and_stream(path, request)
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/content/handler.py", line 700, in _match_and_stream
pulpcore-content[10625]:     return await self._stream_content_artifact(
pulpcore-content[10625]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/content/handler.py", line 841, in _stream_content_artifact
pulpcore-content[10625]:     response = await self._stream_remote_artifact(request, response, remote_artifact)
pulpcore-content[10625]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/content/handler.py", line 1139, in _stream_remote_artifact
pulpcore-content[10625]:     downloader = remote.get_downloader(
pulpcore-content[10625]:                  ^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/app/models/repository.py", line 634, in get_downloader
pulpcore-content[10625]:     download_factory = self.download_factory
pulpcore-content[10625]:                        ^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/app/models/repository.py", line 569, in download_factory
pulpcore-content[10625]:     self._download_factory = DownloaderFactory(self)
pulpcore-content[10625]:                              ^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/download/factory.py", line 74, in __init__
pulpcore-content[10625]:     self._session = self._make_aiohttp_session_from_remote()
pulpcore-content[10625]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pulpcore-content[10625]:   File "/usr/lib/python3.12/site-packages/pulpcore/download/factory.py", line 115, in _make_aiohttp_session_from_remote
pulpcore-content[10625]:     sslcontext.load_cert_chain(cert_file.name, key_file.name)
pulpcore-content[10625]: OSError: [Errno 24] Too many open files

Even before you run into the error, you can start to see files accumulating using for example:

for pid in $(ps aux | grep pulpcore-content | awk '{ print $2 }'); do echo "PID: $pid"; lsof -p "$pid" | grep urandom  | wc -l; done

To Reproduce
For us the fastest way to reproduce this in the Katello context was to have a smart-proxy server using policy "streamed", with some Pulp content on it, and then simply try to consume that Pulp content. Because of streamed, each access starts a new download from the main Katello server, and you start accumulating open file handles, that you can track using:

for pid in $(ps aux | grep pulpcore-content | awk '{ print $2 }'); do echo "PID: $pid"; lsof -p "$pid" | grep urandom  | wc -l; done

However, any workflow that consumes Pulp content that is yet to be downloaded (because of on_demand or streamed) should reproduce the error (for the affected Python/Pulp versions).

Expected behavior
File handles do not accumulate/are closed after use.

Additional context
It is possible that the error is within some Python library dependency, rather than pulpcore itself. However, it is a pulpcore workflow that breaks as a result. Katello 4.18 and satellite 6.18 appear to be affected.

[quba42]

FYI: It is possible to increase the limit (how many files the pulpcore-content processes are allowed to open), by editing the systemd service file /etc/systemd/system/pulpcore-content.service on Katello installations. You will need to add LimitNOFILE=4098 The default is 1024.

However, this just delays how long it takes to run into the issue, at which point you will need to restart the services to reset things.

We also found this: https://access.redhat.com/solutions/6970506

It gave us some ideas for debugging, but does not fix the issue.

[quba42]

I found this satellite issue that sounds somewhat similar: https://issues.redhat.com/browse/SAT-29428

[quba42]

So, it looks like we have a code change that fixes the issue, but it is an early draft that does not exactly explain why this worked in the past and has not yet received much vetting on any undesirable side effects. I am saying it still needs work, but definitely gives us more information. Definitely progress!