dvuploader doesn't work with custom proxy settings

[de-ranit]

I had my data on a remote data storage which is meant to be used with HPC. Our HPC uses custom HTTP/HTTPS proxies. When I was trying to push my code directly from the remote data storage to a dataverse-based data hosting platform using dvuploader 0.2.3 , I was getting following error:

Traceback (most recent call last):
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/connector.py", line 1115, in _wrap_create_connection
    sock = await aiohappyeyeballs.start_connection(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 71, in start_connection
    sock = await _connect_sock(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 163, in _connect_sock
    await loop.sock_connect(sock, address)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/futures.py", line 285, in __await__
    yield self  # This tells Task to wait for completion.
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/futures.py", line 196, in result
    raise exc
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/client.py", line 703, in _request
    conn = await self._connector.connect(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/connector.py", line 548, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/connector.py", line 1056, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/connector.py", line 1400, in _create_direct_connection
    raise last_exc
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/connector.py", line 1369, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/connector.py", line 1112, in _wrap_create_connection
    async with ceil_timeout(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/async_timeout/__init__.py", line 179, in __aexit__
    self._do_exit(exc_type)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/async_timeout/__init__.py", line 265, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/Data/dir/data_upload.py", line 33, in <module>
    dvuploader.upload(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/dvuploader/dvuploader.py", line 133, in upload
    asyncio.run(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/dvuploader/directupload.py", line 72, in direct_upload
    upload_results = await asyncio.gather(*tasks)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/asyncio/tasks.py", line 234, in __step
    result = coro.throw(exc)
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/dvuploader/directupload.py", line 133, in _upload_to_store
    ticket = await _request_ticket(
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/dvuploader/directupload.py", line 198, in _request_ticket
    async with session.get(url) as response:
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/client.py", line 1425, in __aenter__
    self._resp: _RetType = await self._coro
  File "path/to/miniforge3/envs/gen_env_2024/lib/python3.10/site-packages/aiohttp/client.py", line 707, in _request
    raise ConnectionTimeoutError(
aiohttp.client_exceptions.ConnectionTimeoutError: Connection timeout to host https://something.com/api/datasets/:persistentId/uploadurls?key=xxxxxxx&persistentId=doi:xxxxx.xxxx/xxxxx&size=3

Following, Client session docs and proxy support docs, I set trust_env=True in L56 and 90 of directupload.py to make dvuploader 0.2.3 work for connection with custom proxies.

[JR-1991]

@de-ranit Thank you for submitting the issue! We've recently migrated from aiohttp to httpx for various reasons, primarily due to its improved usability over aiohttp.

We really appreciate you bringing up the proxy concern. This is something we hadn't considered before, but it's very useful. To address this, I've opened PR #23, which now includes an optional proxy address. Could you please take a look at the PR and test it out?

[de-ranit]

Hi, Thank you very much for your quick response, and for adding this. I downloaded your branch add-proxy-option to try it. However, I am unable to install it as python3 -m pip install . after cd into python-dvuploader as it is missing a setup.py. How shall I install it if I want to install it from the source?

[JR-1991]

@de-ranit, thank you for your prompt reply. We have transitioned to using pyproject.toml for our building process, eliminating the need for setup.py. You can install the package via pip with the following command:

pip install .

Please let me know if this works for you.

[de-ranit]

@JR-1991, Thanks for the instruction. It works perfectly.

Surprisingly, even if I don't pass a proxy argument to dvuploader.upload, now it still works, and I can upload a file. I don't know if it is expected behaviour. It also works if I specify an argument, e.g., proxy="http://localhost:8030" to dvuploader.upload. I also now see that there is the possibility to use different proxies for HTTP and HTTPS using a dictionary (https://www.python-httpx.org/advanced/proxies/).

[JR-1991]

@de-ranit this is odd, but at least it is working now 😁 I will add some more tests to check this.

I also now see that there is the possibility to use different proxies for HTTP and HTTPS using a dictionary

Indeed, that is pretty cool and far better than aiohttp.