[feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413

Hastyshell · 2024-10-24T10:02:09Z

Proposed changes

Usage

adjust limit

curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params}

Entry	Description
param	uint64 qps_limit
behavior	set qps_limit global default value
example	`curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000`

Entry	Description
param	uint64 qps_limit, string rpc_name
behavior	set RPC specific qps_limit
example	curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster

Entry	Description
param	uint64 qps_limit, string rpc_name, string instance_id
behavior	set instance qps_limit for specific RPC
example	`ccurl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"`

Entry	Description
param	uint64 qps_limit, string instance_id
behavior	set global qps_limit for specific instance
example	`curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"`

query limit

Entry	Description
param	none
behavior	query qps limit for all RPC interface
example	`curl http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit`

doris-robot · 2024-10-24T10:02:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-24T10:06:29Z

cloud/src/rate-limiter/rate_limiter.h


 #pragma once

 #include <brpc/server.h>


warning: 'brpc/server.h' file not found [clang-diagnostic-error]

#include <brpc/server.h> ^

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-25T09:43:52Z

cloud/test/rate_limiter_test.cpp

 // specific language governing permissions and limitations
 // under the License.

 #include "rate-limiter/rate_limiter.h"


warning: 'rate-limiter/rate_limiter.h' file not found [clang-diagnostic-error]

#include "rate-limiter/rate_limiter.h" ^

Hastyshell · 2024-10-25T12:12:58Z

run buildall

Hastyshell · 2024-10-25T16:26:44Z

run buildall

Hastyshell · 2024-10-28T04:28:50Z

run buildall

Hastyshell · 2024-10-31T12:39:40Z

run buildall

Hastyshell · 2024-10-31T12:43:15Z

run buildall

github-actions

clang-tidy made some suggestions

cloud/src/meta-service/meta_service_http.cpp

cloud/test/meta_service_http_test.cpp

Hastyshell · 2024-10-31T13:41:01Z

run buildall

gavinchou · 2024-11-01T14:42:49Z

Add some usage examples in description.

Hastyshell · 2024-11-05T13:16:32Z

run buildall

github-actions

clang-tidy made some suggestions

cloud/src/meta-service/meta_service_http.cpp

gavinchou · 2024-11-10T17:30:18Z

cloud/src/meta-service/meta_service_http.cpp

+                                           qps_limit_str, rpc_name, instance_id));
+    };
+
+    static auto parse_qps_limit =


it seems that abuse using lambdas

gavinchou · 2024-11-10T17:31:54Z

cloud/src/meta-service/meta_service_http.cpp

+        });
+    };
+
+    // for 8 element in true table of params, register processor cb


this part is confusing

Maybe I'll add some comments, just register different callback for different params pattern, and the "abuse" of lamda are just meant to adapt for this.

cloud/src/rate-limiter/rate_limiter.cpp

cloud/src/rate-limiter/rate_limiter.h

cloud/src/meta-service/meta_service_http.cpp

Hastyshell · 2024-11-12T12:36:33Z

run buildall

…on meta service

Hastyshell · 2024-11-13T06:20:02Z

run buildall

github-actions · 2024-11-13T08:23:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-13T08:23:52Z

PR approved by anyone and no changes requested.

Hastyshell · 2024-11-13T09:09:07Z

run compile

Hastyshell · 2024-11-13T09:09:17Z

run cloud_p0

Hastyshell · 2024-11-13T09:09:22Z

run p0

Hastyshell · 2024-11-13T09:09:30Z

run external

…on meta service (#42413) ## Proposed changes Usage 1. adjust limit ``` curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params} ``` | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit | |behavior | set qps_limit global default value | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000```| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string rpc_name | |behavior | set RPC specific qps_limit | |example|curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string rpc_name, string instance_id | |behavior | set instance qps_limit for specific RPC | |example|```ccurl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"```| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string instance_id | |behavior | set global qps_limit for specific instance | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"```| 2. query limit | Entry | Description | | ----------- | ----------- | | param | none | |behavior | query qps limit for all RPC interface | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit```|

…c qps limit on meta service #42413 (#43884) Cherry-picked from #42413 Co-authored-by: Siyang Tang <tangsiyang2001@foxmail.com>

…ng (#44362) UT failed with coredump in PR #42413 but was not detected by CI. Fix the problem.

github-actions bot reviewed Oct 24, 2024

View reviewed changes

Hastyshell force-pushed the dynamic-ms-rate-limit branch from 928d926 to 453770f Compare October 25, 2024 09:39

github-actions bot reviewed Oct 25, 2024

View reviewed changes

Hastyshell force-pushed the dynamic-ms-rate-limit branch 2 times, most recently from cb4d256 to 063ca50 Compare October 25, 2024 12:08

Hastyshell force-pushed the dynamic-ms-rate-limit branch from 063ca50 to ebe7ddf Compare October 25, 2024 16:26

Hastyshell force-pushed the dynamic-ms-rate-limit branch from ebe7ddf to e3098ab Compare October 28, 2024 04:28

Hastyshell marked this pull request as ready for review October 28, 2024 06:13

dataroaring added dev/3.0.x p0_c labels Oct 29, 2024

Hastyshell marked this pull request as draft October 29, 2024 12:13

Hastyshell force-pushed the dynamic-ms-rate-limit branch from c6532ab to e541fab Compare October 31, 2024 12:42

github-actions bot reviewed Oct 31, 2024

View reviewed changes

cloud/src/meta-service/meta_service_http.cpp Show resolved Hide resolved

cloud/test/meta_service_http_test.cpp Show resolved Hide resolved

Hastyshell force-pushed the dynamic-ms-rate-limit branch from e541fab to c997f0d Compare October 31, 2024 13:40

Hastyshell marked this pull request as ready for review November 1, 2024 01:53

Hastyshell force-pushed the dynamic-ms-rate-limit branch from c997f0d to 1b6cbea Compare November 5, 2024 13:16

github-actions bot reviewed Nov 5, 2024

View reviewed changes

cloud/src/meta-service/meta_service_http.cpp Show resolved Hide resolved

gavinchou reviewed Nov 10, 2024

View reviewed changes

cloud/src/rate-limiter/rate_limiter.cpp Show resolved Hide resolved

gavinchou reviewed Nov 10, 2024

View reviewed changes

cloud/src/rate-limiter/rate_limiter.cpp Show resolved Hide resolved

gavinchou requested changes Nov 10, 2024

View reviewed changes

luwei16 approved these changes Nov 12, 2024

View reviewed changes

Hastyshell requested a review from gavinchou November 12, 2024 12:36

Hastyshell added 3 commits November 13, 2024 14:19

[feature](meta-service) Support querying and adjusting rpc qps limit …

4f7e830

…on meta service

refactor and support adjusting by instance id

6be65b4

fix reviewd problems

70f9607

Hastyshell force-pushed the dynamic-ms-rate-limit branch from 84e350b to 70f9607 Compare November 13, 2024 06:19

gavinchou approved these changes Nov 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024

github-actions bot added the reviewed label Nov 13, 2024

gavinchou merged commit 4783fc0 into apache:master Nov 13, 2024

github-actions bot mentioned this pull request Nov 13, 2024

branch-3.0: [feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413 #43884

Merged

gavinchou pushed a commit that referenced this pull request Nov 14, 2024

branch-3.0: [feature](meta-service) Support querying and adjusting rp…

663845a

…c qps limit on meta service #42413 (#43884) Cherry-picked from #42413 Co-authored-by: Siyang Tang <tangsiyang2001@foxmail.com>

gavinchou added dev/3.0.3-merged and removed dev/3.0.x labels Nov 14, 2024

Hastyshell mentioned this pull request Nov 20, 2024

[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setting #44362

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request Nov 21, 2024

[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…

012620c

…ng (#44362) UT failed with coredump in PR #42413 but was not detected by CI. Fix the problem.

github-actions bot pushed a commit that referenced this pull request Nov 21, 2024

[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…

63cd07d

…ng (#44362) UT failed with coredump in PR #42413 but was not detected by CI. Fix the problem.

gavinchou mentioned this pull request Nov 26, 2024

Release Note 3.0.3 #44522

Closed

[feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413

[feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413

Uh oh!

Conversation

Hastyshell commented Oct 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 24, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 25, 2024

Choose a reason for hiding this comment

Uh oh!

Hastyshell commented Oct 25, 2024

Uh oh!

Hastyshell commented Oct 25, 2024

Uh oh!

Hastyshell commented Oct 28, 2024

Uh oh!

Hastyshell commented Oct 31, 2024

Uh oh!

Hastyshell commented Oct 31, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Hastyshell commented Oct 31, 2024

Uh oh!

gavinchou commented Nov 1, 2024

Uh oh!

Hastyshell commented Nov 5, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gavinchou Nov 10, 2024

Choose a reason for hiding this comment

Uh oh!

gavinchou Nov 10, 2024

Choose a reason for hiding this comment

Uh oh!

Hastyshell Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hastyshell commented Nov 12, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Oct 24, 2024 •

edited

Loading