Skip to content

Conversation

@zghong
Copy link
Contributor

@zghong zghong commented Dec 25, 2024

Description

When executing large-payload insert queries by POST, the current retry and throughput-limit mechanisms read the POST body, which can be resource-consuming and may even cause CPU/Memory bottlenecks. This MR avoids unnecessary I/O operations in these two mechanisms:

  • For the retry mechanism: If the retry parameter is not set, the step of copying the body can be skipped.
  • For the throughput-limit mechanism: Instead of calculating the length of query, the Content-Length is read directly from POST header.

Closes #499, #428

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

Checklist

  • Linter passes correctly
  • Add tests which fail without the change (if possible)
  • All tests passing
  • Extended the README / documentation, if necessary

Does this introduce a breaking change?

  • Yes
  • No

Further comments

For the throughput-limit mechanism: The Content-Length header indicates the size of the request body, so the throughput of GETs will be ignored. This is logical, as GETs typically have minimal impact on throughput compared to POSTs.

@zghong
Copy link
Contributor Author

zghong commented Dec 25, 2024

Before:

image
image
image
image

After:

image
image
image
image

Copy link
Collaborator

@mga-chka mga-chka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice improvement

@mga-chka mga-chka merged commit 5007bbb into ContentSquare:master Feb 4, 2025
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] High CPU/Memory usage for large-payload insert queries

2 participants