Skip to content

[feature](cloud) Support multi follower fe in cloud #38388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 4 commits into from
Sep 5, 2024
Merged

[feature](cloud) Support multi follower fe in cloud #38388

dataroaring merged 4 commits into from
Sep 5, 2024

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Jul 25, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@deardeng
Copy link
Contributor Author

deardeng commented Jul 25, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

TPC-H: Total hot run time: 39963 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03019769802bb7c09d058c2a9fc2f9a631ec3c72, data reload: false

------ Round 1 ----------------------------------
q1	17643	4640	4369	4369
q2	2360	200	195	195
q3	10513	1264	1099	1099
q4	10146	775	786	775
q5	7546	2788	2713	2713
q6	216	138	139	138
q7	1017	622	596	596
q8	9224	1963	1917	1917
q9	9268	6994	6936	6936
q10	8751	3773	3821	3773
q11	491	251	250	250
q12	394	225	222	222
q13	17889	2976	2952	2952
q14	277	244	241	241
q15	536	476	479	476
q16	505	394	382	382
q17	979	663	670	663
q18	8174	7608	7342	7342
q19	4936	1048	1039	1039
q20	670	347	341	341
q21	5047	3251	4025	3251
q22	351	303	293	293
Total cold run time: 116933 ms
Total hot run time: 39963 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4502	4225	4225	4225
q2	371	264	277	264
q3	3011	2789	2734	2734
q4	1880	1613	1609	1609
q5	5298	5340	5334	5334
q6	218	134	134	134
q7	2119	1762	1740	1740
q8	3218	3373	3320	3320
q9	8925	8923	8841	8841
q10	3900	3674	3716	3674
q11	617	490	495	490
q12	785	615	605	605
q13	17453	3006	2968	2968
q14	303	269	268	268
q15	521	470	485	470
q16	469	431	422	422
q17	1778	1495	1468	1468
q18	7645	7503	7497	7497
q19	1663	1493	1514	1493
q20	1996	1793	1781	1781
q21	4899	4579	4787	4579
q22	576	518	542	518
Total cold run time: 72147 ms
Total hot run time: 54434 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

TPC-DS: Total hot run time: 172269 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03019769802bb7c09d058c2a9fc2f9a631ec3c72, data reload: false

query1	917	387	367	367
query2	6446	1953	1940	1940
query3	6657	204	217	204
query4	26941	17554	17471	17471
query5	4255	489	479	479
query6	280	170	162	162
query7	4585	288	286	286
query8	241	196	194	194
query9	8455	2431	2406	2406
query10	443	278	264	264
query11	10579	10044	10100	10044
query12	139	84	83	83
query13	1604	366	369	366
query14	10499	7920	7184	7184
query15	224	165	162	162
query16	7171	468	485	468
query17	1328	536	544	536
query18	1828	279	273	273
query19	186	139	140	139
query20	93	84	89	84
query21	206	98	101	98
query22	4236	4087	4015	4015
query23	33738	33191	33257	33191
query24	11247	2841	2944	2841
query25	645	371	377	371
query26	1341	155	152	152
query27	3030	267	275	267
query28	7359	2026	2015	2015
query29	899	413	436	413
query30	284	153	150	150
query31	963	741	740	740
query32	97	52	56	52
query33	764	345	320	320
query34	900	473	475	473
query35	907	730	732	730
query36	1124	939	932	932
query37	151	79	80	79
query38	2825	2752	2734	2734
query39	899	803	787	787
query40	277	116	120	116
query41	46	44	44	44
query42	117	104	97	97
query43	494	472	472	472
query44	1224	724	733	724
query45	209	173	175	173
query46	1094	717	755	717
query47	1833	1738	1752	1738
query48	373	306	295	295
query49	1131	436	446	436
query50	809	412	414	412
query51	6801	6769	6779	6769
query52	140	90	88	88
query53	257	179	182	179
query54	853	441	441	441
query55	76	79	74	74
query56	299	279	279	279
query57	1168	1040	1055	1040
query58	265	257	271	257
query59	2848	2776	2720	2720
query60	324	274	287	274
query61	97	95	92	92
query62	831	646	635	635
query63	209	189	188	188
query64	10158	2243	1711	1711
query65	3168	3128	3116	3116
query66	1037	355	323	323
query67	15381	14801	14748	14748
query68	4576	554	565	554
query69	441	310	306	306
query70	1085	1156	1017	1017
query71	418	273	263	263
query72	7539	5668	5958	5668
query73	784	327	333	327
query74	6027	5666	5557	5557
query75	3407	2735	2711	2711
query76	2904	942	947	942
query77	460	329	312	312
query78	9697	9315	8829	8829
query79	2707	531	530	530
query80	2135	499	499	499
query81	604	221	218	218
query82	743	138	137	137
query83	313	185	184	184
query84	269	79	79	79
query85	1997	322	307	307
query86	486	315	312	312
query87	3336	3074	3091	3074
query88	3851	2410	2397	2397
query89	403	283	284	283
query90	1791	191	197	191
query91	130	97	106	97
query92	70	52	51	51
query93	2004	552	552	552
query94	815	304	285	285
query95	354	274	269	269
query96	596	270	272	270
query97	3249	3037	2987	2987
query98	229	219	188	188
query99	1602	1282	1243	1243
Total cold run time: 281027 ms
Total hot run time: 172269 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2024

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03019769802bb7c09d058c2a9fc2f9a631ec3c72, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.04
query4	1.68	0.07	0.08
query5	0.51	0.48	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.05
query9	0.55	0.49	0.48
query10	0.54	0.56	0.55
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.58	0.58
query14	0.74	0.78	0.77
query15	0.85	0.81	0.82
query16	0.37	0.36	0.37
query17	0.96	1.04	0.99
query18	0.23	0.22	0.22
query19	1.77	1.64	1.70
query20	0.01	0.01	0.01
query21	15.42	0.76	0.66
query22	4.90	6.41	2.06
query23	18.24	1.44	1.21
query24	2.11	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.24	0.23
query28	13.33	1.01	1.00
query29	12.65	3.32	3.29
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.25	0.47	0.47
query33	2.89	2.90	2.91
query34	17.16	4.34	4.40
query35	4.38	4.45	4.46
query36	0.66	0.46	0.48
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.41 s
Total hot run time: 30.65 s

@github-actions
Copy link
Contributor

github-actions bot commented Jul 26, 2024

clang-tidy review says "All clean, LGTM! 👍"

@deardeng
Copy link
Contributor Author

deardeng commented Jul 26, 2024

run buildall

w41ter
w41ter reviewed Jul 31, 2024
View reviewed changes
yujun777
yujun777 approved these changes Aug 20, 2024
View reviewed changes
Copy link
Contributor

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 20, 2024

PR approved by anyone and no changes requested.

gavinchou
gavinchou reviewed Aug 30, 2024
View reviewed changes
gavinchou
gavinchou requested changes Sep 1, 2024
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@deardeng
Copy link
Contributor Author

deardeng commented Sep 3, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Sep 3, 2024

TPC-H: Total hot run time: 38092 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63cff667c51eec15b0cc4de5011085b0bbda162e, data reload: false

------ Round 1 ----------------------------------
q1	18047	4526	4353	4353
q2	2038	181	185	181
q3	11489	993	1198	993
q4	10177	722	747	722
q5	7747	2878	2788	2788
q6	239	138	142	138
q7	969	626	597	597
q8	9365	2106	2053	2053
q9	7063	6614	6637	6614
q10	6983	2241	2209	2209
q11	462	245	242	242
q12	394	219	221	219
q13	17750	3054	3096	3054
q14	271	230	238	230
q15	542	478	493	478
q16	583	536	515	515
q17	999	626	731	626
q18	7532	6886	6949	6886
q19	1410	1104	999	999
q20	706	337	338	337
q21	3944	2916	2844	2844
q22	1119	1033	1014	1014
Total cold run time: 109829 ms
Total hot run time: 38092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4348	4263	4263
q2	382	278	267	267
q3	2896	2670	2621	2621
q4	1903	1670	1668	1668
q5	5634	5697	5745	5697
q6	244	138	135	135
q7	2256	1809	1799	1799
q8	3286	3445	3448	3445
q9	8910	8870	8828	8828
q10	3571	3360	3429	3360
q11	638	516	529	516
q12	827	673	655	655
q13	14556	3168	3284	3168
q14	318	297	299	297
q15	544	492	475	475
q16	611	587	568	568
q17	1848	1582	1521	1521
q18	8260	8005	7867	7867
q19	1724	1496	1442	1442
q20	2140	1895	1920	1895
q21	5707	5326	5360	5326
q22	1145	1062	1026	1026
Total cold run time: 71802 ms
Total hot run time: 56839 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2024

TPC-DS: Total hot run time: 192673 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63cff667c51eec15b0cc4de5011085b0bbda162e, data reload: false

query1	1247	907	860	860
query2	6343	2046	1993	1993
query3	10755	3981	3896	3896
query4	60186	26974	23422	23422
query5	5313	506	500	500
query6	401	157	158	157
query7	5788	286	298	286
query8	294	213	238	213
query9	8809	2441	2455	2441
query10	499	275	262	262
query11	18165	15114	15525	15114
query12	153	103	96	96
query13	1589	402	411	402
query14	11116	7304	7531	7304
query15	221	180	185	180
query16	7388	499	487	487
query17	1105	604	553	553
query18	1746	291	297	291
query19	299	148	143	143
query20	127	105	106	105
query21	209	110	107	107
query22	4554	4511	4816	4511
query23	34060	33646	33188	33188
query24	5954	2869	2867	2867
query25	546	392	380	380
query26	685	152	154	152
query27	1795	276	277	276
query28	3915	2030	2022	2022
query29	627	404	403	403
query30	242	157	159	157
query31	909	766	779	766
query32	83	49	54	49
query33	474	281	303	281
query34	856	483	484	483
query35	804	700	696	696
query36	1072	954	926	926
query37	144	89	88	88
query38	4043	3834	3834	3834
query39	1448	1401	1371	1371
query40	199	115	113	113
query41	45	45	45	45
query42	112	92	98	92
query43	510	462	469	462
query44	1106	722	719	719
query45	196	206	167	167
query46	1078	743	779	743
query47	1869	1796	1801	1796
query48	360	300	298	298
query49	747	415	428	415
query50	806	418	403	403
query51	7036	6837	6780	6780
query52	96	87	85	85
query53	246	174	179	174
query54	559	459	458	458
query55	74	75	73	73
query56	272	252	246	246
query57	1202	1074	1045	1045
query58	227	250	237	237
query59	3116	2914	2828	2828
query60	291	269	268	268
query61	102	98	102	98
query62	746	641	622	622
query63	210	183	189	183
query64	2581	671	661	661
query65	3237	3155	3124	3124
query66	628	340	334	334
query67	15460	15269	15142	15142
query68	2973	560	546	546
query69	399	290	276	276
query70	1168	1153	1029	1029
query71	332	286	275	275
query72	6184	4056	4022	4022
query73	756	329	326	326
query74	9157	8952	9045	8952
query75	3346	2696	2729	2696
query76	1359	999	1007	999
query77	564	323	331	323
query78	9612	9637	9065	9065
query79	1048	525	519	519
query80	711	526	519	519
query81	469	237	237	237
query82	250	151	147	147
query83	179	230	148	148
query84	251	72	74	72
query85	674	297	279	279
query86	322	305	295	295
query87	4332	4296	4351	4296
query88	3173	2316	2317	2316
query89	374	286	288	286
query90	1995	190	188	188
query91	124	101	100	100
query92	59	50	50	50
query93	1029	525	529	525
query94	676	286	291	286
query95	322	297	253	253
query96	584	261	273	261
query97	3232	3061	3065	3061
query98	216	195	199	195
query99	1518	1258	1300	1258
Total cold run time: 304463 ms
Total hot run time: 192673 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2024

ClickBench: Total hot run time: 32.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63cff667c51eec15b0cc4de5011085b0bbda162e, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.53	0.50	0.50
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.48	0.48
query10	0.56	0.54	0.54
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.62	0.60	0.59
query14	2.04	2.10	2.07
query15	0.86	0.82	0.82
query16	0.37	0.38	0.38
query17	1.05	1.01	1.06
query18	0.21	0.19	0.21
query19	1.93	1.83	1.81
query20	0.01	0.02	0.01
query21	15.41	0.67	0.65
query22	4.00	7.09	2.58
query23	18.26	1.38	1.27
query24	2.08	0.24	0.21
query25	0.15	0.08	0.09
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.26	1.02	1.00
query29	12.64	3.31	3.27
query30	0.24	0.06	0.05
query31	2.86	0.40	0.39
query32	3.26	0.48	0.47
query33	2.99	3.02	3.02
query34	17.02	4.45	4.51
query35	4.53	4.48	4.53
query36	0.66	0.48	0.50
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.05
query40	0.14	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.75 s
Total hot run time: 32.85 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 0013c31 into apache:master Sep 5, 2024
gavinchou pushed a commit that referenced this pull request Sep 10, 2024
@deardeng @gavinchou
[feature](cloud) Support multi follower fe in cloud (#38388)
0e11380
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@w41ter w41ter w41ter approved these changes

+1 more reviewer

@yujun777 yujun777 yujun777 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged meta-change reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@deardeng @doris-robot @gavinchou @w41ter @yujun777 @dataroaring