Yewtube needs httpx less than 0.28 #396811

gautaz · 2025-04-07T11:54:02Z

Hello,

Currently yewtube does not support httpx>=0.28.0, see:
#394572

These changes ensure that http=0.27.2 is used in order to work around this issue.

I am new to Python packages overriding, so if I am doing it wrong, please advise.

Things done

Add a 👍 reaction to pull requests you find important.

drupol · 2025-04-07T12:17:17Z

Hello,

Thanks for your PR, but I can't help but think that this kind of issue should be fixed upstream, not downstream. Perhaps I'm wrong here, but it feels weird to do that to fix something that should be fixed upstream.

gautaz · 2025-04-07T13:29:20Z

Hello,

Thanks for your PR, but I can't help but think that this kind of issue should be fixed upstream, not downstream. Perhaps I'm wrong here, but it feels weird to do that to fix something that should be fixed upstream.

I agree, currently the issue was closed on yewtube side, I was planning to ask for reopening it.

But in the meantime, as there is a workaround, don't you think that it would be beneficial for all other nixpkgs users to at least have the opportunity to use yewtube?

This is currently necessary to work around the following yewtube issue: mps-youtube/yewtube#1303

FliegendeWurst · 2025-04-07T13:51:53Z

Would be better to fix the root cause here: https://github.com/alexmercerind/youtube-search-python/blob/main/youtubesearchpython/core/requests.py#L25

Using the patch below. Code adapted using information from encode/httpx#2879

diff --git a/youtubesearchpython/core/requests.py b/youtubesearchpython/core/requests.py
index ea3ed7f..3eaa2a3 100644
--- a/youtubesearchpython/core/requests.py
+++ b/youtubesearchpython/core/requests.py
@@ -11,10 +11,10 @@ class RequestCore:
         self.proxy = {}
         http_proxy = os.environ.get("HTTP_PROXY")
         if http_proxy:
-            self.proxy["http://"] = http_proxy
+            self.proxy["http://"] = httpx.HTTPTransport(proxy=http_proxy)
         https_proxy = os.environ.get("HTTPS_PROXY")
         if https_proxy:
-            self.proxy["https://"] = https_proxy
+            self.proxy["https://"] = httpx.HTTPTransport(proxy=https_proxy)

     def syncPostRequest(self) -> httpx.Response:
         return httpx.post(
@@ -22,18 +22,18 @@ class RequestCore:
             headers={"User-Agent": userAgent},
             json=self.data,
             timeout=self.timeout,
-            proxies=self.proxy
+            mounts=self.proxy
         )

     async def asyncPostRequest(self) -> httpx.Response:
-        async with httpx.AsyncClient(proxies=self.proxy) as client:
+        async with httpx.AsyncClient(mounts=self.proxy) as client:
             r = await client.post(self.url, headers={"User-Agent": userAgent}, json=self.data, timeout=self.timeout)
             return r

     def syncGetRequest(self) -> httpx.Response:
-        return httpx.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'}, proxies=self.proxy)
+        return httpx.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'}, mounts=self.proxy)

     async def asyncGetRequest(self) -> httpx.Response:
-        async with httpx.AsyncClient(proxies=self.proxy) as client:
+        async with httpx.AsyncClient(mounts=self.proxy) as client:
             r = await client.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'})
             return r

Save as fix-httpx-proxies.patch and add to youtube-search-python's patches

FliegendeWurst · 2025-04-07T13:53:52Z

Note: the patch above cannot be upstreamed, since the corresponding repository is archived.

drupol · 2025-04-07T14:09:54Z

I definitely prefer the solution proposed by @FliegendeWurst ! (CC @amozeo)

gautaz · 2025-04-07T14:10:11Z

Note: the patch above cannot be upstreamed, since the corresponding repository is archived.

Ok, that explains why the current fix proposed on yewtube side is downgrading httpx.
In order to push the patch upstream, it would probably mean for the yewtube maintainers to take over the project youtube-search-python, I'll see how this goes on their side.

In the meantime (and it will probably take long on yewtube side), what is the best course of actions on nixpkgs side?
@FliegendeWurst Are you actually meaning to apply this patch on nixpkgs side in the youtube-search-python package?

FliegendeWurst · 2025-04-07T14:15:04Z

Are you actually meaning to apply this patch on nixpkgs side in the youtube-search-python package?

Yes, this will benefit other users of the package too. Save the patch in pkgs/development/python-modules/youtube-search-python/fix-httpx-proxies.patch, and add this to youtube-search-python's default.nix

patches = [
  # Update usage of deprecated and removed proxies argument.
  # https://github.com/encode/httpx/pull/2879
  # https://github.com/NixOS/nixpkgs/pull/396811
  ./fix-httpx-proxies.patch
];

gautaz · 2025-04-07T14:17:00Z

Yes, this will benefit other users of the package too. Save the patch in pkgs/development/python-modules/youtube-search-python/fix-httpx-proxies.patch, and add this to youtube-search-python's default.nix

Ok, I'll create a new PR for this and will close this one when the new one is ready.
I'll keep you posted.

drupol · 2025-04-07T14:18:30Z

You can do everything within this PR.

amozeo · 2025-04-07T14:21:53Z

I don't use yewtube package myself. I like the proposed solution of patching youtube-search-python.

gautaz · 2025-04-07T14:42:51Z

You can do everything within this PR.

@drupol I just saw your message, I have already started a new branch, my apologies.

Seems like the patch for youtube-search-python needs additional changes, I got this while testing with yewtube:

post() got an unexpected keyword argument 'mounts'

I am looking into it.

gautaz · 2025-04-07T15:26:40Z

It seems the right patch would be:

--- a/youtubesearchpython/core/requests.py
+++ b/youtubesearchpython/core/requests.py
@@ -11,10 +11,10 @@ class RequestCore:
         self.proxy = {}
         http_proxy = os.environ.get("HTTP_PROXY")
         if http_proxy:
-            self.proxy["http://"] = http_proxy
+            self.proxy["http://"] = httpx.HTTPTransport(proxy=http_proxy)
         https_proxy = os.environ.get("HTTPS_PROXY")
         if https_proxy:
-            self.proxy["https://"] = https_proxy
+            self.proxy["https://"] = httpx.HTTPTransport(proxy=https_proxy)

     def syncPostRequest(self) -> httpx.Response:
         return httpx.post(
@@ -22,18 +22,17 @@ class RequestCore:
             headers={"User-Agent": userAgent},
             json=self.data,
             timeout=self.timeout,
-            proxies=self.proxy
         )

     async def asyncPostRequest(self) -> httpx.Response:
-        async with httpx.AsyncClient(proxies=self.proxy) as client:
+        async with httpx.AsyncClient(mounts=self.proxy) as client:
             r = await client.post(self.url, headers={"User-Agent": userAgent}, json=self.data, timeout=self.timeout)
             return r

     def syncGetRequest(self) -> httpx.Response:
-        return httpx.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'}, proxies=self.proxy)
+        return httpx.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'}, mounts=self.proxy)

     async def asyncGetRequest(self) -> httpx.Response:
-        async with httpx.AsyncClient(proxies=self.proxy) as client:
+        async with httpx.AsyncClient(mounts=self.proxy) as client:
             r = await client.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'})
             return r

yewtube seems to behave well with this one.
I'll push the PR as soon as it's ready.

FliegendeWurst · 2025-04-07T15:28:19Z

I think proxy=self.proxy.get("https://") would be more correct for httpx.post and httpx.get calls

FliegendeWurst · 2025-04-07T15:29:11Z

Though it may be safer to just remove the proxies argument, yes. Certainly easier to test.

gautaz · 2025-04-07T15:42:10Z

And I took the time to read the httpx docs, so the right patch is finally:

--- a/youtubesearchpython/core/requests.py
+++ b/youtubesearchpython/core/requests.py
@@ -11,29 +11,28 @@ class RequestCore:
         self.proxy = {}
         http_proxy = os.environ.get("HTTP_PROXY")
         if http_proxy:
-            self.proxy["http://"] = http_proxy
+            self.proxy["http://"] = httpx.HTTPTransport(proxy=http_proxy)
         https_proxy = os.environ.get("HTTPS_PROXY")
         if https_proxy:
-            self.proxy["https://"] = https_proxy
+            self.proxy["https://"] = httpx.HTTPTransport(proxy=https_proxy)

     def syncPostRequest(self) -> httpx.Response:
-        return httpx.post(
+        return httpx.Client(mounts=self.proxy).post(
             self.url,
             headers={"User-Agent": userAgent},
             json=self.data,
-            timeout=self.timeout,
-            proxies=self.proxy
+            timeout=self.timeout
         )

     async def asyncPostRequest(self) -> httpx.Response:
-        async with httpx.AsyncClient(proxies=self.proxy) as client:
+        async with httpx.AsyncClient(mounts=self.proxy) as client:
             r = await client.post(self.url, headers={"User-Agent": userAgent}, json=self.data, timeout=self.timeout)
             return r

     def syncGetRequest(self) -> httpx.Response:
-        return httpx.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'}, proxies=self.proxy)
+        return httpx.Client(mounts=self.proxy).get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'})

     async def asyncGetRequest(self) -> httpx.Response:
-        async with httpx.AsyncClient(proxies=self.proxy) as client:
+        async with httpx.AsyncClient(mounts=self.proxy) as client:
             r = await client.get(self.url, headers={"User-Agent": userAgent}, timeout=self.timeout, cookies={'CONSENT': 'YES+1'})
             return r

I should have started with this in the first place 🤣.

gautaz · 2025-04-07T15:51:26Z

The new pull request is available, see #396845.
I am closing this one.

nix-owners bot requested a review from fgaz April 7, 2025 12:02

yewtube: override httpx dependency to 0.27.2

6a43d37

This is currently necessary to work around the following yewtube issue: mps-youtube/yewtube#1303

gautaz force-pushed the yewtube-needs-httpx-less-than-0.28 branch from 7bd7749 to 6a43d37 Compare April 7, 2025 13:34

gautaz mentioned this pull request Apr 7, 2025

Error fetching data. Possible network issue. post() got an unexpected keyword argument 'proxies' mps-youtube/yewtube#1303

Open

gautaz mentioned this pull request Apr 7, 2025

python3Packages.youtube-search-python: patch for httpx >= 0.28 #396845

Merged

13 tasks

gautaz closed this Apr 7, 2025

Uh oh!

Yewtube needs httpx less than 0.28 #396811

Yewtube needs httpx less than 0.28 #396811

Uh oh!

Conversation

gautaz commented Apr 7, 2025

Things done

Uh oh!

drupol commented Apr 7, 2025

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

FliegendeWurst commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FliegendeWurst commented Apr 7, 2025

Uh oh!

drupol commented Apr 7, 2025

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

FliegendeWurst commented Apr 7, 2025

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

drupol commented Apr 7, 2025

Uh oh!

amozeo commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

FliegendeWurst commented Apr 7, 2025

Uh oh!

FliegendeWurst commented Apr 7, 2025

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

gautaz commented Apr 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FliegendeWurst commented Apr 7, 2025 •

edited

Loading

amozeo commented Apr 7, 2025 •

edited

Loading