Skip to content

[fix](rpc) fix transfer large data and enable transfer_large_data_by_brpc by default #35770 #36167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 12, 2024
Merged

[fix](rpc) fix transfer large data and enable transfer_large_data_by_brpc by default #35770 #36167

dataroaring merged 1 commit into from
Jun 12, 2024

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Jun 12, 2024

cherry pick from #35770

@liaoxin01
[fix](rpc) fix transfer large data and enable transfer_large_data_by_…
48cc184 
…brpc by default (apache#35770)

Issue Number: close #xxx

when send large data by rpc，load will report the following error:
```
add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000.

```

The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use
the brpc HTTP method. However, the block data was not removed during
encoding, resulting in protobuf exceeding the length and encoding
failure.

The config of transfer_large_data_by_brpc was enabled by default on
branch-2.0, so we also enable it on master.
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Jun 12, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 12, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

TPC-H: Total hot run time: 49605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48cc1840f374f86ea719da1436c9ffc2055443c6, data reload: false

------ Round 1 ----------------------------------
q1	17681	4481	4375	4375
q2	2113	162	153	153
q3	10440	1888	1899	1888
q4	10367	1223	1295	1223
q5	8445	3866	3872	3866
q6	233	148	138	138
q7	2002	1602	1571	1571
q8	9554	2716	2698	2698
q9	13872	10175	10259	10175
q10	8659	3535	3508	3508
q11	410	253	246	246
q12	482	310	307	307
q13	18363	3942	4063	3942
q14	354	322	325	322
q15	500	456	453	453
q16	681	571	581	571
q17	1156	947	940	940
q18	7380	6804	7015	6804
q19	1797	1681	1652	1652
q20	540	303	294	294
q21	4447	4130	4038	4038
q22	544	442	441	441
Total cold run time: 120020 ms
Total hot run time: 49605 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4335	4380	4334	4334
q2	321	226	226	226
q3	4204	4157	4151	4151
q4	2729	2730	2739	2730
q5	7163	7057	7077	7057
q6	238	119	117	117
q7	3230	2814	2825	2814
q8	4373	4487	4486	4486
q9	16802	16851	16671	16671
q10	4226	4234	4281	4234
q11	762	698	702	698
q12	1019	845	852	845
q13	7193	3733	3733	3733
q14	443	422	423	422
q15	507	469	454	454
q16	730	699	693	693
q17	3812	3876	3841	3841
q18	8870	8824	8767	8767
q19	1763	1676	1685	1676
q20	2344	2112	2102	2102
q21	8487	8467	8472	8467
q22	1076	995	961	961
Total cold run time: 84627 ms
Total hot run time: 79479 ms

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8099/21424)
Line Coverage: 29.47% (66154/224486)
Region Coverage: 28.96% (34104/117777)
Branch Coverage: 24.81% (17505/70556)
Coverage Report: http://coverage.selectdb-in.cc/coverage/48cc1840f374f86ea719da1436c9ffc2055443c6_48cc1840f374f86ea719da1436c9ffc2055443c6/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

TPC-DS: Total hot run time: 205673 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48cc1840f374f86ea719da1436c9ffc2055443c6, data reload: false

query1	946	448	406	406
query2	6547	2899	2657	2657
query3	6920	221	206	206
query4	21042	18071	17972	17972
query5	19721	6634	6546	6546
query6	301	217	249	217
query7	4162	310	294	294
query8	455	411	431	411
query9	3162	2742	2652	2652
query10	407	312	302	302
query11	11264	10679	10749	10679
query12	131	83	82	82
query13	5598	696	711	696
query14	18036	13320	13706	13320
query15	364	248	257	248
query16	6471	309	267	267
query17	1676	1470	882	882
query18	2317	418	429	418
query19	216	154	156	154
query20	81	84	85	84
query21	190	101	105	101
query22	5193	5084	4941	4941
query23	33049	32112	32144	32112
query24	6984	6608	6545	6545
query25	528	447	425	425
query26	531	167	163	163
query27	1875	303	305	303
query28	6247	2389	2354	2354
query29	3032	2787	2681	2681
query30	257	175	180	175
query31	948	739	758	739
query32	71	67	59	59
query33	417	270	270	270
query34	900	482	509	482
query35	1141	978	901	901
query36	1322	1297	1361	1297
query37	89	61	62	61
query38	3090	2961	2943	2943
query39	1374	1359	1353	1353
query40	208	104	98	98
query41	47	45	46	45
query42	83	83	83	83
query43	796	720	688	688
query44	1161	727	722	722
query45	251	240	232	232
query46	1235	966	991	966
query47	1853	2036	1822	1822
query48	1044	712	719	712
query49	625	378	378	378
query50	862	588	614	588
query51	4890	4707	4722	4707
query52	102	91	88	88
query53	447	330	320	320
query54	2748	2464	2510	2464
query55	98	85	83	83
query56	225	222	210	210
query57	1142	1171	1144	1144
query58	218	204	203	203
query59	4238	3955	3833	3833
query60	224	199	202	199
query61	98	94	95	94
query62	875	478	463	463
query63	497	359	348	348
query64	2338	1487	1487	1487
query65	3709	3643	3638	3638
query66	817	370	374	370
query67	15607	15779	17113	15779
query68	8715	659	672	659
query69	582	361	358	358
query70	1622	1637	1623	1623
query71	409	327	320	320
query72	6521	3516	3498	3498
query73	761	328	327	327
query74	6354	5884	5993	5884
query75	5208	3816	3781	3781
query76	5234	1157	1216	1157
query77	897	255	252	252
query78	12815	11910	11715	11715
query79	7737	652	649	649
query80	1254	421	413	413
query81	500	250	248	248
query82	1569	99	100	99
query83	180	140	146	140
query84	272	73	72	72
query85	884	318	327	318
query86	322	298	319	298
query87	3301	3017	3059	3017
query88	5188	2391	2395	2391
query89	392	306	283	283
query90	1914	203	222	203
query91	192	160	142	142
query92	60	57	61	57
query93	5112	588	560	560
query94	732	207	204	204
query95	1123	1069	1052	1052
query96	664	350	337	337
query97	6498	6347	6386	6347
query98	196	188	182	182
query99	2918	972	925	925
Total cold run time: 313619 ms
Total hot run time: 205673 ms

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

ClickBench: Total hot run time: 31.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48cc1840f374f86ea719da1436c9ffc2055443c6, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.04
query4	1.78	0.07	0.08
query5	0.53	0.52	0.52
query6	1.24	0.62	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.52	0.47	0.49
query10	0.55	0.56	0.53
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.60
query14	0.78	0.79	0.78
query15	0.79	0.77	0.76
query16	0.36	0.36	0.38
query17	1.01	1.00	1.03
query18	0.24	0.23	0.25
query19	1.94	1.86	1.81
query20	0.01	0.01	0.01
query21	15.46	0.56	0.59
query22	2.06	2.30	1.93
query23	17.06	0.98	1.09
query24	3.63	2.64	0.91
query25	0.35	0.11	0.06
query26	0.56	0.15	0.14
query27	0.04	0.05	0.04
query28	7.91	0.76	0.72
query29	12.69	2.34	2.38
query30	0.61	0.56	0.52
query31	2.81	0.40	0.37
query32	3.38	0.52	0.51
query33	3.10	3.04	3.08
query34	15.26	4.78	4.80
query35	4.84	4.83	4.86
query36	1.07	1.02	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.15	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 102.22 s
Total hot run time: 31.05 s

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 48cc1840f374f86ea719da1436c9ffc2055443c6 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@dataroaring dataroaring merged commit 4228320 into apache:branch-2.0 Jun 12, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@liaoxin01
[fix](rpc) fix transfer large data and enable transfer_large_data_by_…
9c388e0 
…brpc by default apache#35770 (apache#36167)

cherry pick from apache#35770
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liaoxin01 @doris-robot @dataroaring