Skip to content

[branch-2.0](routine-load) add retry when get Kafka meta info and make get Kafka meta timeout configurable #37458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 3 commits into from
Jul 8, 2024
Merged

[branch-2.0](routine-load) add retry when get Kafka meta info and make get Kafka meta timeout configurable #37458

liaoxin01 merged 3 commits into from
Jul 8, 2024

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jul 8, 2024

pick #36619 and #35376

sollhui added 2 commits July 8, 2024 15:10
@sollhui
[improve](routine-load) add retry when get Kafka meta info (#35376)
3c89e1c 
If be down when FE send RPC `getInfo` or meet network error when be send
RPC to Kafka, routine load job will pause.
To keep routine load stable, add retry when get Kafka meta info.
@sollhui
[chore](routine-load) make get Kafka meta timeout configurable (#36619)
929c689 
Sometimes, the delay from be to Kafka is relatively high, which can
cause get info RPC timeout.

Make get Kafka meta timeout configurable allow users to customize
timeout times which can solve this issue.
@doris-robot
Copy link

doris-robot commented Jul 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-H: Total hot run time: 49851 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5ea81615346a7628319382b3cabcdd3201371110, data reload: false

------ Round 1 ----------------------------------
q1	17710	4439	4400	4400
q2	2076	167	154	154
q3	10273	1880	1935	1880
q4	10106	1260	1373	1260
q5	8513	3897	3915	3897
q6	254	140	129	129
q7	2053	1601	1596	1596
q8	9281	2747	2724	2724
q9	10855	10263	10143	10143
q10	8673	3521	3497	3497
q11	426	252	253	252
q12	462	306	311	306
q13	18346	4007	4038	4007
q14	350	338	337	337
q15	515	465	456	456
q16	671	566	581	566
q17	1146	995	956	956
q18	7217	6853	6969	6853
q19	1782	1664	1602	1602
q20	548	299	293	293
q21	4481	4251	4098	4098
q22	527	450	445	445
Total cold run time: 116265 ms
Total hot run time: 49851 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4370	4355	4355
q2	322	226	221	221
q3	4192	4154	4148	4148
q4	2751	2772	2756	2756
q5	7187	7130	7127	7127
q6	237	120	120	120
q7	3208	2867	2815	2815
q8	4357	4514	4519	4514
q9	16957	16775	16788	16775
q10	4246	4273	4309	4273
q11	787	684	676	676
q12	1047	863	829	829
q13	7124	3735	3731	3731
q14	456	429	424	424
q15	519	460	459	459
q16	731	682	676	676
q17	3880	3840	3848	3840
q18	8977	8843	8794	8794
q19	1750	1729	1651	1651
q20	2357	2157	2125	2125
q21	8603	8442	8419	8419
q22	1055	998	1008	998
Total cold run time: 85137 ms
Total hot run time: 79726 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TeamCity be ut coverage result:
Function Coverage: 37.88% (8112/21417)
Line Coverage: 29.55% (66457/224921)
Region Coverage: 29.01% (34248/118051)
Branch Coverage: 24.88% (17588/70690)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5ea81615346a7628319382b3cabcdd3201371110_5ea81615346a7628319382b3cabcdd3201371110/report/index.html

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-DS: Total hot run time: 204100 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5ea81615346a7628319382b3cabcdd3201371110, data reload: false

query1	938	424	388	388
query2	6550	2838	2650	2650
query3	6920	205	198	198
query4	20321	18027	18069	18027
query5	19718	6520	6538	6520
query6	274	215	226	215
query7	4153	293	306	293
query8	460	392	410	392
query9	3104	2661	2600	2600
query10	402	299	303	299
query11	11305	10808	10876	10808
query12	129	76	73	73
query13	5606	706	702	702
query14	17735	13325	13660	13325
query15	362	241	246	241
query16	6479	286	264	264
query17	1704	1470	867	867
query18	2324	422	412	412
query19	204	149	156	149
query20	80	78	80	78
query21	187	94	92	92
query22	5278	5035	5009	5009
query23	32671	31970	31924	31924
query24	6993	6514	6500	6500
query25	534	447	427	427
query26	531	163	163	163
query27	1877	302	299	299
query28	6162	2346	2326	2326
query29	2791	2797	2685	2685
query30	239	165	175	165
query31	893	740	737	737
query32	68	65	61	61
query33	413	267	252	252
query34	868	472	472	472
query35	1118	883	894	883
query36	1356	1133	1228	1133
query37	92	59	61	59
query38	3092	2931	2921	2921
query39	1363	1340	1307	1307
query40	210	92	96	92
query41	46	46	44	44
query42	85	85	87	85
query43	735	652	680	652
query44	1117	713	720	713
query45	252	234	233	233
query46	1233	970	985	970
query47	1751	2002	1910	1910
query48	997	727	695	695
query49	633	374	375	374
query50	859	622	635	622
query51	4751	4689	4648	4648
query52	85	80	75	75
query53	447	326	322	322
query54	2663	2481	2461	2461
query55	85	99	77	77
query56	214	218	213	213
query57	1273	1161	1108	1108
query58	223	207	217	207
query59	4150	4212	3943	3943
query60	210	190	195	190
query61	96	95	95	95
query62	783	485	603	485
query63	485	339	344	339
query64	2370	1523	1509	1509
query65	3657	3569	3550	3550
query66	811	381	370	370
query67	16057	15275	16353	15275
query68	8596	662	653	653
query69	573	342	334	334
query70	1519	1456	1416	1416
query71	408	321	310	310
query72	6629	3479	3470	3470
query73	750	324	320	320
query74	6309	5847	5864	5847
query75	4836	3733	3685	3685
query76	4866	1125	1218	1125
query77	708	265	262	262
query78	12600	11738	12833	11738
query79	10495	664	645	645
query80	1221	401	405	401
query81	499	240	232	232
query82	1151	104	104	104
query83	159	135	127	127
query84	258	71	72	71
query85	887	330	329	329
query86	317	305	317	305
query87	3250	3062	3070	3062
query88	4818	2277	2291	2277
query89	438	305	264	264
query90	1812	212	206	206
query91	173	138	138	138
query92	63	54	54	54
query93	5821	586	604	586
query94	614	207	208	207
query95	1116	1095	1054	1054
query96	643	322	328	322
query97	6550	6383	6402	6383
query98	192	173	175	173
query99	3128	827	844	827
Total cold run time: 312971 ms
Total hot run time: 204100 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

ClickBench: Total hot run time: 31.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5ea81615346a7628319382b3cabcdd3201371110, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.79	0.09	0.08
query5	0.54	0.53	0.53
query6	1.26	0.63	0.62
query7	0.02	0.01	0.00
query8	0.03	0.02	0.02
query9	0.52	0.49	0.50
query10	0.55	0.54	0.55
query11	0.13	0.09	0.09
query12	0.11	0.09	0.10
query13	0.62	0.62	0.61
query14	0.80	0.78	0.78
query15	0.78	0.76	0.76
query16	0.40	0.38	0.40
query17	1.03	0.99	1.03
query18	0.21	0.27	0.25
query19	1.83	1.85	1.86
query20	0.01	0.02	0.01
query21	15.46	0.55	0.54
query22	2.13	2.54	1.52
query23	17.37	1.11	0.99
query24	5.18	1.65	1.32
query25	0.37	0.12	0.06
query26	0.61	0.16	0.15
query27	0.05	0.05	0.04
query28	7.15	0.82	0.78
query29	12.63	2.37	2.37
query30	0.59	0.54	0.54
query31	2.82	0.39	0.38
query32	3.38	0.49	0.50
query33	3.09	3.08	3.06
query34	15.25	4.77	4.79
query35	4.85	4.85	4.85
query36	1.06	1.02	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.36 s
Total hot run time: 31.31 s

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5ea81615346a7628319382b3cabcdd3201371110 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@liaoxin01 liaoxin01 merged commit e7056c5 into apache:branch-2.0 Jul 8, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui @Baymine
[branch-2.0](routine-load) add retry when get Kafka meta info and mak…
55d98ae 
…e get Kafka meta timeout configurable (apache#37458)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sollhui @doris-robot @liaoxin01