SOCKS5 Proxy Support for OPTIMADE Client.

[hongyi-zhao]

Hi there,

Does the OPTIMADE Python client support SOCKS5 proxies (e.g., socks5h://127.0.0.1:16667)?

I need to use optimade-get from a network that requires routing through a SOCKS proxy, but couldn't find any proxy configuration options in the documentation.

Is there a supported way to configure the client to use a proxy, either through environment variables or API parameters?

Regards,
Zhao

[ml-evs]

I'm not overly familiar with SOCKS5 (nor can I easily test it) but you could override the default OptimadeClient._http_client to use your proxy (looking at the bottom of this page: https://www.python-httpx.org/advanced/proxies/), something like:

client = OptimadeClient(http_client=httpx.AsyncClient(proxy="socks5h://127.0.0.1:16667"))

potentially after installing the socks extra for httpx pip install httpx[socks] (according to their docs).

[hongyi-zhao]

Hi @ml-evs,

Based on your hints, I tried the following method:

"""
Test code demonstrating issue with socks5h proxy in OptimadeClient
"""

import httpx
from optimade.client import OptimadeClient

# Attempt to use socks5h proxy with OptimadeClient
try:
    print("Creating client with socks5h proxy...")
    client = OptimadeClient(
        http_client=httpx.AsyncClient(
            proxy="socks5h://127.0.0.1:16667"
        )
    )
    
    # This line won't be reached due to initialization error
    results = client.get('elements HAS "Si"')
    print(results)
    
except Exception as e:
    print(f"ERROR: {type(e).__name__}: {e}")

The error is as follows:

Creating client with socks5h proxy...
ERROR: ValueError: Unknown scheme for proxy URL URL('socks5h://127.0.0.1:16667')

I need to use socks5h:// specifically (for remote DNS resolution), but it appears httpx only supports the socks5:// scheme. Is there any workaround for this in OPTIMADE, or would I need to patch httpx to support this scheme?

[ml-evs]

It looks to me like this has already been added encode/httpx#3178 in httpx 0.28. Which version are you using? We only seem to be enforcing httpx >= 0.23 and in fact are testing with 0.28.1 in our CI.

I'm not against adding a proxy option to the main optimade-python-tools package if this is working, but as I said its very hard for me to test.

Another option would be to try using a requests.Session rather than httpx (they should expose the same API as far as we are concerned, and requests seems to support SOCKS5), it just won't be async, so you'll have to also set use_async=False.

[ml-evs]

Just another note when you are testing, currently the pre-query aggregation of providers does not use the http client that you configure (this could be considered a bug), so you'll have to test with explicitly set base URLs.

e.g.,

client = OptimadeClient(base_urls=["optimade.materialsproject.org", ...], ...)

[hongyi-zhao]

Now, I have updated httpx as follows and this fix the error shown above:

$ pip install -U httpx httpx-socks

But I meet the error as follows:

In [1]: import httpx
   ...: from optimade.client import OptimadeClient
   ...: 
   ...: # 创建客户端，明确指定base_urls而不是使用自动发现
   ...: client = OptimadeClient(
   ...:     # 明确指定需要查询的数据库URLs，避免预查询聚合器的问题
   ...:     base_urls=[
   ...:         "https://optimade.materialsproject.org",
   ...:         "https://optimade.odbx.science"
   ...:         # 添加其他你需要的数据库
   ...:     ],
   ...:     # 配置httpx客户端使用socks5h代理
   ...:     http_client=httpx.AsyncClient(
   ...:         proxy="socks5h://127.0.0.1:16667"
   ...:     )
   ...: )
   ...: 
   ...: # 执行查询
   ...: results = client.get('elements HAS "Si"')
   ...: print(results)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 5
      2 from optimade.client import OptimadeClient
      4 # 创建客户端，明确指定base_urls而不是使用自动发现
----> 5 client = OptimadeClient(
      6     # 明确指定需要查询的数据库URLs，避免预查询聚合器的问题
      7     base_urls=[
      8         "https://optimade.materialsproject.org",
      9         "https://optimade.odbx.science"
     10         # 添加其他你需要的数据库
     11     ],
     12     # 配置httpx客户端使用socks5h代理
     13     http_client=httpx.AsyncClient(
     14         proxy="socks5h://127.0.0.1:16667"
     15     )
     16 )
     18 # 执行查询
     19 results = client.get('elements HAS "Si"')

File ~/.pyenv/versions/3.11.1/envs/datasci/lib/python3.11/site-packages/optimade/client/client.py:247, in OptimadeClient.__init__(self, base_urls, max_results_per_provider, headers, http_timeout, max_attempts, use_async, silent, exclude_providers, include_providers, exclude_databases, http_client, verbosity, callbacks, skip_ssl)
    245 if http_client:
    246     self._http_client = http_client
--> 247     if issubclass(self._http_client, httpx.AsyncClient):
    248         if not self.use_async and self.__strict_async:
    249             raise RuntimeError(
    250                 "Cannot use synchronous mode with an asynchronous HTTP client, please set `use_async=True` or pass an asynchronous HTTP client."
    251             )

TypeError: issubclass() arg 1 must be a class

[hongyi-zhao]

Based on my tries, it seems that the following two methods work:

Method 1: Modify global socket setting:

import socks
import socket
from optimade.client import OptimadeClient

# 保存原始socket以便之后恢复
original_socket = socket.socket

# 设置全局SOCKS5代理（rdns=True表示使用远程DNS解析，相当于socks5h）
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 16667, rdns=True)
socket.socket = socks.socksocket

# 创建客户端，不需要特殊配置代理
client = OptimadeClient(
    base_urls=[
        "https://optimade.materialsproject.org",
        "https://optimade.odbx.science"
    ]
)

# 执行查询
results = client.get('elements HAS "Si"')
print(results)

# 可选：恢复原始socket
# socket.socket = original_socket

Method 2: Use Requests instead of HTTPX, as you have told:

import requests
from optimade.client import OptimadeClient

# 创建Session类
class ProxiedSession(requests.Session):
    def __init__(self):
        super().__init__()
        self.proxies = {
            "http": "socks5h://127.0.0.1:16667",
            "https": "socks5h://127.0.0.1:16667"
        }

# 使用非异步模式和Requests
client = OptimadeClient(
    base_urls=[
        "https://optimade.materialsproject.org",
        "https://optimade.odbx.science"
    ],
    http_client=ProxiedSession,  # 传递类
    use_async=False  # 必须设为False以配合requests
)

# 执行查询
results = client.get('elements HAS "Si"')
print(results)

[ml-evs]

Great, thanks for the info. I think the first approach would also work, but I gave you an incorrect snippet. The custom http client must be initialised by the OptimadeClient, so you'll need to do something like:

from functools import partial
from optimade.client import OptimadeClient
from httpx import AsyncClient

client = OptimadeClient(http_client=partial(AsyncClient, proxy="socks5h://xxxx"))

The global socket configuration seems like the cleanest approach though.

[hongyi-zhao]

Great, thanks for the info. I think the first approach would also work, but I gave you an incorrect snippet. The custom http client must be initialised by the OptimadeClient, so you'll need to do something like:

from functools import partial
from optimade.client import OptimadeClient
from httpx import AsyncClient

client = OptimadeClient(http_client=partial(AsyncClient, proxy="socks5h://xxxx"))

I tried the above method, but still failed:

In [1]: import asyncio
   ...: from functools import partial
   ...: from optimade.client import OptimadeClient
   ...: from httpx import AsyncClient
   ...: 
   ...: async def main():
   ...:     # 创建OptimadeClient，使用socks5h代理
   ...:     client = OptimadeClient(
   ...:         base_urls=["https://optimade.materialsproject.org"],
   ...:         http_client=partial(AsyncClient, proxy="socks5h://127.0.0.1:16667"),
   ...:         use_async=True
   ...:     )
   ...: 
   ...:     # 执行查询
   ...:     results = await client.get('elements HAS "Si"')
   ...:     print(results)
   ...: 
   ...: # 运行异步函数
   ...: asyncio.run(main())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 19
     16     print(results)
     18 # 运行异步函数
---> 19 asyncio.run(main())

File ~/.pyenv/versions/3.11.1/lib/python3.11/asyncio/runners.py:190, in run(main, debug)
    186     raise RuntimeError(
    187         "asyncio.run() cannot be called from a running event loop")
    189 with Runner(debug=debug) as runner:
--> 190     return runner.run(main)

File ~/.pyenv/versions/3.11.1/lib/python3.11/asyncio/runners.py:118, in Runner.run(self, coro, context)
    116 self._interrupt_count = 0
    117 try:
--> 118     return self._loop.run_until_complete(task)
    119 except exceptions.CancelledError:
    120     if self._interrupt_count > 0:

File ~/.pyenv/versions/3.11.1/lib/python3.11/asyncio/base_events.py:653, in BaseEventLoop.run_until_complete(self, future)
    650 if not future.done():
    651     raise RuntimeError('Event loop stopped before Future completed.')
--> 653 return future.result()

Cell In[1], line 8, in main()
      6 async def main():
      7     # 创建OptimadeClient，使用socks5h代理
----> 8     client = OptimadeClient(
      9         base_urls=["https://optimade.materialsproject.org"],
     10         http_client=partial(AsyncClient, proxy="socks5h://127.0.0.1:16667"),
     11         use_async=True
     12     )
     14     # 执行查询
     15     results = await client.get('elements HAS "Si"')

File ~/.pyenv/versions/3.11.1/envs/datasci/lib/python3.11/site-packages/optimade/client/client.py:247, in OptimadeClient.__init__(self, base_urls, max_results_per_provider, headers, http_timeout, max_attempts, use_async, silent, exclude_providers, include_providers, exclude_databases, http_client, verbosity, callbacks, skip_ssl)
    245 if http_client:
    246     self._http_client = http_client
--> 247     if issubclass(self._http_client, httpx.AsyncClient):
    248         if not self.use_async and self.__strict_async:
    249             raise RuntimeError(
    250                 "Cannot use synchronous mode with an asynchronous HTTP client, please set `use_async=True` or pass an asynchronous HTTP client."
    251             )

TypeError: issubclass() arg 1 must be a class

In [2]: import httpx
   ...: import asyncio
   ...: 
   ...: async def test_proxy():
   ...:     async with httpx.AsyncClient(proxy="socks5h://127.0.0.1:16667") as client:
   ...:         try:
   ...:             response = await client.get("https://httpbin.org/ip")
   ...:             print(f"代理正常工作: {response.json()}")
   ...:         except Exception as e:
   ...:             print(f"代理测试失败: {type(e).__name__}: {e}")
   ...: 
   ...: asyncio.run(test_proxy())
代理测试失败: KeyError: b'socks5h'

[ml-evs]

To summarise and close this issue, it seems like the proxy can be most effectively handled via global socket settings, rather than requiring changes to this package. This discussion did unsurface a related issue #2275, but I am not in a rush to fix it.

Thanks again for reporting and for the workaround!