Skip to content

feat(DEVC-1752): added requests.Session reuse; replaced third-party tenacity with builtin urllib3 Retry#100

Merged
kossman merged 26 commits intomasterfrom
feature/DEVC-1752-add-connection-keep-alive-and-socket-reuse
Jul 25, 2025
Merged

feat(DEVC-1752): added requests.Session reuse; replaced third-party tenacity with builtin urllib3 Retry#100
kossman merged 26 commits intomasterfrom
feature/DEVC-1752-add-connection-keep-alive-and-socket-reuse

Conversation

@kossman
Copy link
Copy Markdown
Contributor

@kossman kossman commented Jul 23, 2025

Rationale

DataApi is suffering of a large number of connections created by applications. The issue is that Python apps establish a new connection with each request. To fix this, we need to change how we handle HTTPS requests in the SDK.

Changes

So, in the requests library, we need to properly use Session objects https://requests.readthedocs.io/en/latest/user/advanced/
That will bring all that we badly need:

https://requests.readthedocs.io/en/latest/user/advanced/#keep-alive
https://requests.readthedocs.io/en/latest/user/advanced/#example-automatic-retries

What was done:

  1. Added requests.Session object for making any HTTP requests from Api(...)
  2. Added posibility to adjust some params related to connection pool:
    a. POOL_CONNECTIONS_COUNT - Total pools count
    b. POOL_MAX_SIZE - Max connections count per pool/host
    c. POOL_BLOCK - Wait until connection released or not (instantly raise an exception)
    d. MAX_RETRY_COUNT - If 0 then retires will be disabled, otherwise retrying logic will be used
  3. Move retrying logic from tenacity to internal urllib3.util.Retry(...)
  4. Removed redundant dependency tenacity from python-sdk
  5. Bump version for py3.13 to py3.13.3 at CI version matrix in order to fix broken tests for logging
  6. Bump version for fakeredis also according to failed some tests

@kossman kossman self-assigned this Jul 23, 2025
@kossman kossman marked this pull request as ready for review July 25, 2025 12:08
@kossman kossman requested a review from corva-alex-s July 25, 2025 12:08
Comment thread src/corva/api.py Outdated
Comment thread src/corva/api_utils.py Outdated
Comment thread src/corva/configuration.py Outdated
@kossman kossman merged commit beec970 into master Jul 25, 2025
16 checks passed
@kossman kossman deleted the feature/DEVC-1752-add-connection-keep-alive-and-socket-reuse branch July 25, 2025 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants