Skip to content

feat(deps,rpc): add retry logic to fix flaky connection issues in hive#2205

Merged
spencer-tb merged 4 commits intomainfrom
6110-exception-retries-with-tenacity
Sep 26, 2025
Merged

feat(deps,rpc): add retry logic to fix flaky connection issues in hive#2205
spencer-tb merged 4 commits intomainfrom
6110-exception-retries-with-tenacity

Conversation

@danceratopz
Copy link
Member

🗒️ Description

Add retries for RPC requests using tenacity. This is an alternative suggestion to #2195, in comparison this approach has:

  • Targeted retries: Only connection errors, not RPC/HTTP errors; Don't retry for malformed responses (i.e., app logic).
  • Exponential backoff: 0.5s → 4s to avoid overwhelming clients; allow them to get start-up in typical use case.
  • Improved logging: Automatic retry attempt logging with details.
  • Reusable: Decorator pattern can be applied elsewhere if needed (uses the de facto Python retry library).

🔗 Related Issues or PRs

Closes #1532.

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@danceratopz danceratopz added type:feat type: Feature scope:consume Scope: Consume command suite scope:execute Scope: Changes to the execute command labels Sep 25, 2025
@danceratopz
Copy link
Member Author

@felix314159 following up from the discussion here:

This is an implementation using tenacity, as an alternative to the existing approach, cf #2195.

@danceratopz danceratopz added the scope:deps Scope: Updates package dependencies label Sep 25, 2025
@danceratopz danceratopz changed the title feat(rpc): add retry logic to fix flaky connection issues in hive feat(deps,rpc): add retry logic to fix flaky connection issues in hive Sep 25, 2025
@spencer-tb spencer-tb force-pushed the 6110-exception-retries-with-tenacity branch from fc74e32 to f60e9d9 Compare September 26, 2025 11:57
Copy link
Collaborator

@spencer-tb spencer-tb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks :)

@spencer-tb spencer-tb merged commit e55fbce into main Sep 26, 2025
15 checks passed
@spencer-tb spencer-tb deleted the 6110-exception-retries-with-tenacity branch September 26, 2025 11:58
danceratopz added a commit that referenced this pull request Sep 29, 2025
#2205)

* chore(deps): add `tenacity` for retry logic

* feat(rpc): add rpc request retries with exp backoff

* docs: add retry logic via tenacity to coding standards

* docs: add changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope:consume Scope: Consume command suite scope:deps Scope: Updates package dependencies scope:execute Scope: Changes to the execute command type:feat type: Feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(consume): enable retries for RPC requests to reduce flakiness

2 participants