Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #47313

@github-actions github-actions bot requested a review from dataroaring as a code owner January 24, 2025 09:54
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 24, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41439 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef6a4ee3677d9d9f5fa380bcbb317f886f4eda3f, data reload: false

------ Round 1 ----------------------------------
q1	17582	7645	7393	7393
q2	2057	167	177	167
q3	10611	1081	1165	1081
q4	10565	749	755	749
q5	7757	2887	2854	2854
q6	244	152	149	149
q7	1017	623	611	611
q8	9366	2041	2114	2041
q9	6681	6413	6448	6413
q10	6988	2302	2308	2302
q11	458	267	269	267
q12	438	217	217	217
q13	17779	2997	2990	2990
q14	240	214	220	214
q15	574	517	524	517
q16	637	597	576	576
q17	994	549	578	549
q18	7227	6612	6744	6612
q19	1413	1179	1208	1179
q20	486	208	200	200
q21	4213	3496	3375	3375
q22	1117	983	1001	983
Total cold run time: 108444 ms
Total hot run time: 41439 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7358	7388	7332	7332
q2	332	234	232	232
q3	3043	2956	2964	2956
q4	2123	1971	1825	1825
q5	5738	5782	5733	5733
q6	242	150	148	148
q7	2287	1830	1815	1815
q8	3350	3547	3448	3448
q9	8994	8978	8993	8978
q10	3630	3602	3574	3574
q11	591	487	495	487
q12	832	632	655	632
q13	8924	3194	3119	3119
q14	305	273	284	273
q15	581	521	541	521
q16	706	658	632	632
q17	1866	1658	1647	1647
q18	8340	7909	7794	7794
q19	1777	1628	1716	1628
q20	2174	1902	1899	1899
q21	5629	5336	5512	5336
q22	1164	1057	1058	1057
Total cold run time: 69986 ms
Total hot run time: 61066 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef6a4ee3677d9d9f5fa380bcbb317f886f4eda3f, data reload: false

query1	1326	913	909	909
query2	6251	2102	2073	2073
query3	10981	4397	4465	4397
query4	66735	27073	23487	23487
query5	5063	445	434	434
query6	411	177	171	171
query7	5557	300	303	300
query8	311	217	230	217
query9	8728	2697	2703	2697
query10	463	265	265	265
query11	17394	15237	15919	15237
query12	154	102	102	102
query13	1501	424	439	424
query14	9997	7359	7013	7013
query15	197	189	184	184
query16	7108	518	519	518
query17	1097	595	632	595
query18	1860	349	326	326
query19	212	162	163	162
query20	118	111	117	111
query21	207	110	107	107
query22	4943	4718	4526	4526
query23	34897	34118	34391	34118
query24	6105	3008	2940	2940
query25	537	433	424	424
query26	660	178	179	178
query27	1796	363	357	357
query28	4269	2515	2436	2436
query29	713	473	447	447
query30	239	161	164	161
query31	985	814	856	814
query32	72	61	59	59
query33	435	302	292	292
query34	910	516	550	516
query35	852	759	723	723
query36	1071	967	980	967
query37	127	83	77	77
query38	4138	4084	3985	3985
query39	1544	1496	1481	1481
query40	205	101	96	96
query41	48	47	49	47
query42	117	101	100	100
query43	535	507	507	507
query44	1160	813	812	812
query45	181	180	172	172
query46	1164	744	742	742
query47	2009	1925	1919	1919
query48	496	388	383	383
query49	703	380	394	380
query50	885	441	449	441
query51	7412	7238	7008	7008
query52	101	89	87	87
query53	261	182	180	180
query54	568	452	434	434
query55	76	76	77	76
query56	247	233	250	233
query57	1227	1131	1105	1105
query58	213	202	220	202
query59	3213	2884	2955	2884
query60	273	250	258	250
query61	111	109	107	107
query62	753	645	669	645
query63	224	187	189	187
query64	1364	651	673	651
query65	3235	3188	3224	3188
query66	690	322	306	306
query67	15950	15709	15775	15709
query68	3917	603	558	558
query69	424	264	269	264
query70	1194	1137	1143	1137
query71	349	260	256	256
query72	6443	4021	4219	4021
query73	768	348	347	347
query74	10231	8898	9003	8898
query75	3378	2684	2681	2681
query76	1888	1016	1106	1016
query77	520	274	277	274
query78	10714	9710	9624	9624
query79	1442	597	588	588
query80	842	431	423	423
query81	500	243	240	240
query82	1239	120	122	120
query83	199	144	197	144
query84	282	80	80	80
query85	869	315	297	297
query86	367	299	313	299
query87	4433	4468	4259	4259
query88	3971	2364	2324	2324
query89	417	285	295	285
query90	1999	190	190	190
query91	197	151	152	151
query92	68	51	51	51
query93	1850	541	548	541
query94	772	309	307	307
query95	357	254	252	252
query96	607	275	281	275
query97	3319	3195	3189	3189
query98	211	210	196	196
query99	1676	1302	1301	1301
Total cold run time: 318828 ms
Total hot run time: 197669 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef6a4ee3677d9d9f5fa380bcbb317f886f4eda3f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.51	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.05
query9	0.57	0.50	0.49
query10	0.56	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.60
query14	2.76	2.75	2.74
query15	0.90	0.83	0.81
query16	0.38	0.41	0.37
query17	1.06	1.02	1.01
query18	0.24	0.21	0.22
query19	1.89	1.77	1.98
query20	0.01	0.02	0.01
query21	15.38	0.62	0.57
query22	2.49	1.63	1.64
query23	17.05	0.90	0.74
query24	3.20	1.32	1.00
query25	0.18	0.04	0.14
query26	0.53	0.14	0.14
query27	0.04	0.03	0.05
query28	10.29	1.11	1.07
query29	12.62	3.34	3.35
query30	0.25	0.06	0.05
query31	2.85	0.40	0.39
query32	3.23	0.45	0.47
query33	3.04	3.04	3.06
query34	17.27	4.58	4.53
query35	4.65	4.52	4.61
query36	0.70	0.48	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.17 s
Total hot run time: 32.6 s

Since the DirMoveTask is executed asynchronously, the FE might send the
task again to ensure its completion eventually. But the rowsets
committed during two DirMoveTasks (if any) will be dropped, which causes
the data loss.

This PR adds a LOADED tag file to indicate that the snapshot has been
loaded into a tablet and should not be reloaded again.
@dataroaring dataroaring force-pushed the auto-pick-47313-branch-3.0 branch from ef6a4ee to e0e78fa Compare January 31, 2025 02:24
@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40777 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0e78fa2b6030774b15bb6ac2dc7479caf76049f, data reload: false

------ Round 1 ----------------------------------
q1	17577	7550	7211	7211
q2	2056	172	167	167
q3	10683	1106	1154	1106
q4	10562	738	730	730
q5	7733	2844	2791	2791
q6	236	144	142	142
q7	974	614	594	594
q8	9347	1998	2042	1998
q9	6607	6442	6500	6442
q10	7047	2366	2353	2353
q11	483	262	256	256
q12	401	214	219	214
q13	17797	2973	2989	2973
q14	241	207	214	207
q15	572	530	522	522
q16	699	584	580	580
q17	983	609	650	609
q18	7289	6697	6601	6601
q19	1418	1078	1006	1006
q20	486	205	198	198
q21	3993	3095	3257	3095
q22	1129	982	994	982
Total cold run time: 108313 ms
Total hot run time: 40777 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7264	7209	7212	7209
q2	332	242	240	240
q3	2893	2936	2915	2915
q4	2011	1844	1833	1833
q5	5754	5707	5752	5707
q6	229	143	141	141
q7	2298	1801	1836	1801
q8	3374	3516	3517	3516
q9	8765	8921	8757	8757
q10	3603	3552	3571	3552
q11	619	520	516	516
q12	811	613	579	579
q13	9225	3248	3175	3175
q14	315	278	284	278
q15	574	521	526	521
q16	699	653	648	648
q17	1852	1642	1615	1615
q18	8181	7720	7624	7624
q19	1670	1478	1546	1478
q20	2105	1866	1850	1850
q21	5470	5459	5369	5369
q22	1144	1076	1040	1040
Total cold run time: 69188 ms
Total hot run time: 60364 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198217 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0e78fa2b6030774b15bb6ac2dc7479caf76049f, data reload: false

query1	1285	941	880	880
query2	6227	2072	2043	2043
query3	10953	4548	4519	4519
query4	66354	29267	23653	23653
query5	4903	459	437	437
query6	417	177	169	169
query7	5670	310	312	310
query8	303	219	219	219
query9	9237	2664	2664	2664
query10	477	275	258	258
query11	17485	15256	15775	15256
query12	153	99	106	99
query13	1562	463	423	423
query14	9936	7047	7069	7047
query15	202	190	175	175
query16	7301	518	466	466
query17	1096	587	603	587
query18	1816	332	336	332
query19	245	170	169	169
query20	117	115	117	115
query21	210	106	110	106
query22	4721	4512	4474	4474
query23	34888	34188	34128	34128
query24	6703	2925	2963	2925
query25	543	424	425	424
query26	673	170	180	170
query27	2113	358	357	357
query28	4382	2462	2453	2453
query29	719	467	421	421
query30	236	157	162	157
query31	971	826	828	826
query32	68	55	57	55
query33	439	293	278	278
query34	923	500	527	500
query35	841	742	731	731
query36	1094	966	964	964
query37	123	69	77	69
query38	4073	4005	3988	3988
query39	1518	1476	1497	1476
query40	204	106	106	106
query41	51	51	49	49
query42	113	99	100	99
query43	530	501	506	501
query44	1154	845	830	830
query45	189	176	170	170
query46	1121	699	753	699
query47	2072	1898	1954	1898
query48	471	400	389	389
query49	754	388	389	388
query50	841	426	421	421
query51	7293	7206	7223	7206
query52	99	89	86	86
query53	253	183	175	175
query54	543	444	441	441
query55	73	75	77	75
query56	260	235	249	235
query57	1216	1121	1080	1080
query58	229	210	214	210
query59	3234	3136	2958	2958
query60	275	246	241	241
query61	107	110	104	104
query62	841	750	731	731
query63	220	185	185	185
query64	1394	671	619	619
query65	3266	3199	3223	3199
query66	639	325	319	319
query67	15922	15665	15669	15665
query68	4165	569	565	565
query69	416	269	264	264
query70	1190	1158	1147	1147
query71	330	252	250	250
query72	6427	3983	4028	3983
query73	748	347	351	347
query74	10106	9150	9068	9068
query75	3367	2652	2653	2652
query76	1825	1030	1048	1030
query77	493	269	268	268
query78	10433	9541	9606	9541
query79	1386	593	596	593
query80	881	432	417	417
query81	507	242	238	238
query82	1258	121	118	118
query83	254	145	144	144
query84	280	82	76	76
query85	886	293	297	293
query86	337	291	287	287
query87	4454	4349	4333	4333
query88	3716	2369	2333	2333
query89	416	298	294	294
query90	1928	191	185	185
query91	178	149	148	148
query92	63	49	51	49
query93	1884	547	551	547
query94	754	289	289	289
query95	353	254	252	252
query96	612	284	277	277
query97	3320	3197	3192	3192
query98	211	205	211	205
query99	1721	1430	1441	1430
Total cold run time: 319240 ms
Total hot run time: 198217 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0e78fa2b6030774b15bb6ac2dc7479caf76049f, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.64	0.10	0.10
query5	0.51	0.50	0.52
query6	1.13	0.74	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.50
query10	0.55	0.56	0.58
query11	0.14	0.10	0.11
query12	0.13	0.14	0.11
query13	0.60	0.61	0.60
query14	2.84	2.83	2.73
query15	0.89	0.82	0.83
query16	0.37	0.38	0.39
query17	1.05	1.05	1.08
query18	0.23	0.23	0.21
query19	1.83	1.79	2.01
query20	0.01	0.01	0.01
query21	15.35	0.57	0.54
query22	2.44	1.96	2.72
query23	17.06	0.92	0.88
query24	2.70	0.86	1.21
query25	0.14	0.30	0.07
query26	0.35	0.14	0.14
query27	0.06	0.04	0.04
query28	10.95	1.09	1.07
query29	12.54	3.21	3.21
query30	0.24	0.06	0.05
query31	2.86	0.39	0.39
query32	3.26	0.46	0.46
query33	3.02	3.01	3.07
query34	16.95	4.53	4.58
query35	4.58	4.56	4.57
query36	0.68	0.47	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.55 s
Total hot run time: 32.88 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 226067a into branch-3.0 Feb 19, 2025
25 of 29 checks passed
@github-actions github-actions bot deleted the auto-pick-47313-branch-3.0 branch February 19, 2025 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants