[Fix](autoinc) try fix concurrent load problem with auto inc column #36421

bobhan1 · 2024-06-17T15:13:52Z

increase the value of AutoIncrementGenerator.BATCH_ID_INTERVAL to reduce the number of writes to BDBJE. (the default value of config::auto_inc_prefetch_size_ratio is 10 and the default value of AutoIncIDBuffer::_batch_size is 4064, so mostly the request length is 40960)
only allow master fe to offer getAutoIncrementRange service
write editlog before update batchEndId in memory in getAutoIncrementRange
refactor AutoIncIDBuffer

branch-2.1 pick: #37027

doris-robot · 2024-06-17T15:13:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

bobhan1 · 2024-06-17T15:14:05Z

run buildall

github-actions · 2024-06-17T15:20:05Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2024-06-17T15:37:14Z

run buildall

github-actions · 2024-06-17T15:42:56Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring

LGTM

github-actions · 2024-06-17T15:47:52Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-17T15:47:54Z

PR approved by anyone and no changes requested.

doris-robot · 2024-06-17T16:01:14Z

TPC-H: Total hot run time: 39815 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 38154b40c2b397028c705e032ffea76eb60271d5, data reload: false

------ Round 1 ----------------------------------
q1	17621	4313	4285	4285
q2	2035	190	189	189
q3	10473	1148	1160	1148
q4	10203	906	726	726
q5	7465	2674	2695	2674
q6	226	137	135	135
q7	966	601	592	592
q8	9225	2045	2056	2045
q9	8946	6491	6471	6471
q10	8949	3692	3724	3692
q11	461	233	234	233
q12	398	237	221	221
q13	17755	2999	2987	2987
q14	263	222	214	214
q15	518	479	487	479
q16	521	382	383	382
q17	950	656	679	656
q18	7882	7412	7445	7412
q19	4244	1484	1573	1484
q20	652	332	315	315
q21	4925	3265	3141	3141
q22	395	334	337	334
Total cold run time: 115073 ms
Total hot run time: 39815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4272	4264	4264
q2	356	268	271	268
q3	2987	2731	3052	2731
q4	2008	1786	1710	1710
q5	5612	5469	5524	5469
q6	224	127	130	127
q7	2169	1857	1823	1823
q8	3252	3419	3390	3390
q9	8697	8666	8817	8666
q10	4095	3859	3734	3734
q11	583	478	466	466
q12	806	643	651	643
q13	16560	3301	3180	3180
q14	287	275	281	275
q15	527	485	494	485
q16	491	423	428	423
q17	1800	1490	1461	1461
q18	8151	7871	7676	7676
q19	2078	1529	1495	1495
q20	3060	1927	1866	1866
q21	5033	4932	4899	4899
q22	616	532	535	532
Total cold run time: 73754 ms
Total hot run time: 55583 ms

doris-robot · 2024-06-17T16:12:06Z

TPC-DS: Total hot run time: 173836 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 38154b40c2b397028c705e032ffea76eb60271d5, data reload: false

query1	931	383	373	373
query2	6445	2479	2322	2322
query3	6638	210	213	210
query4	18793	17545	17273	17273
query5	3617	490	458	458
query6	255	163	163	163
query7	4579	295	289	289
query8	317	299	281	281
query9	8543	2401	2392	2392
query10	580	303	294	294
query11	10644	10019	10015	10015
query12	122	92	89	89
query13	1639	365	363	363
query14	10161	7638	7876	7638
query15	238	192	197	192
query16	7823	275	302	275
query17	1814	558	528	528
query18	2020	289	281	281
query19	203	161	158	158
query20	96	87	84	84
query21	213	135	130	130
query22	4713	4503	4347	4347
query23	34061	33616	33735	33616
query24	10590	2757	2730	2730
query25	573	357	359	357
query26	728	156	147	147
query27	2202	308	311	308
query28	5956	2013	2028	2013
query29	842	615	602	602
query30	224	149	150	149
query31	934	722	746	722
query32	93	54	53	53
query33	648	287	277	277
query34	912	461	457	457
query35	722	614	611	611
query36	1069	920	907	907
query37	143	71	75	71
query38	2839	2764	2729	2729
query39	870	794	809	794
query40	200	125	122	122
query41	56	51	49	49
query42	113	95	97	95
query43	570	565	539	539
query44	1102	724	736	724
query45	206	171	162	162
query46	1065	731	746	731
query47	1883	1760	1766	1760
query48	375	302	303	302
query49	857	415	457	415
query50	759	388	387	387
query51	6777	6670	6693	6670
query52	112	89	97	89
query53	359	283	293	283
query54	874	443	437	437
query55	76	71	74	71
query56	284	253	260	253
query57	1092	1027	1060	1027
query58	266	236	251	236
query59	3581	3081	3044	3044
query60	296	267	267	267
query61	93	92	92	92
query62	620	453	469	453
query63	322	294	289	289
query64	8561	2233	1721	1721
query65	3161	3096	3106	3096
query66	748	319	329	319
query67	15349	14805	14800	14800
query68	4509	530	546	530
query69	523	378	350	350
query70	1207	1103	1189	1103
query71	433	270	269	269
query72	7341	5548	5437	5437
query73	734	325	322	322
query74	5890	5525	5800	5525
query75	3405	2694	2702	2694
query76	2760	940	877	877
query77	478	302	292	292
query78	10298	9798	9677	9677
query79	2554	530	511	511
query80	1146	505	457	457
query81	590	214	220	214
query82	789	104	106	104
query83	227	169	168	168
query84	234	84	86	84
query85	1543	299	294	294
query86	477	312	319	312
query87	3287	3068	3102	3068
query88	4072	2362	2355	2355
query89	475	387	385	385
query90	1670	200	194	194
query91	130	104	103	103
query92	65	51	53	51
query93	2043	489	495	489
query94	1054	194	197	194
query95	433	390	325	325
query96	584	267	273	267
query97	3186	3024	3070	3024
query98	224	204	203	203
query99	1160	837	827	827
Total cold run time: 267068 ms
Total hot run time: 173836 ms

doris-robot · 2024-06-17T16:13:46Z

TeamCity be ut coverage result:
Function Coverage: 36.48% (9004/24679)
Line Coverage: 28.02% (73752/263203)
Region Coverage: 27.49% (38305/139335)
Branch Coverage: 24.19% (19527/80732)
Coverage Report: http://coverage.selectdb-in.cc/coverage/38154b40c2b397028c705e032ffea76eb60271d5_38154b40c2b397028c705e032ffea76eb60271d5/report/index.html

doris-robot · 2024-06-17T16:17:37Z

ClickBench: Total hot run time: 30.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 38154b40c2b397028c705e032ffea76eb60271d5, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.06	0.06
query4	1.66	0.10	0.08
query5	0.51	0.50	0.51
query6	1.12	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.79	0.76	0.79
query15	0.83	0.82	0.82
query16	0.37	0.37	0.37
query17	0.98	0.98	0.99
query18	0.20	0.27	0.24
query19	1.79	1.73	1.77
query20	0.01	0.01	0.02
query21	15.42	0.68	0.66
query22	4.41	6.69	2.13
query23	18.23	1.51	1.28
query24	2.17	0.24	0.21
query25	0.15	0.09	0.10
query26	0.26	0.18	0.18
query27	0.08	0.09	0.08
query28	13.21	1.02	0.99
query29	12.63	3.22	3.26
query30	0.26	0.07	0.06
query31	2.88	0.38	0.37
query32	3.29	0.48	0.46
query33	2.89	2.90	2.95
query34	16.83	4.40	4.42
query35	4.56	4.52	4.49
query36	0.65	0.45	0.48
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.15
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.32 s
Total hot run time: 30.79 s

zhannngchen

pls add description for your patch

github-actions · 2024-06-20T12:27:56Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-20T12:31:15Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-20T12:41:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-22T02:39:40Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-06-24T09:11:44Z

run buildall

doris-robot · 2024-06-24T10:08:05Z

TPC-H: Total hot run time: 39823 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cdebf692f971fbf261ce116689e2ca5276cdd50, data reload: false

------ Round 1 ----------------------------------
q1	17939	4731	4271	4271
q2	2006	195	190	190
q3	10564	1115	1164	1115
q4	10241	841	765	765
q5	7468	2653	2582	2582
q6	222	133	129	129
q7	946	591	601	591
q8	9222	2072	2051	2051
q9	9058	6528	6482	6482
q10	8917	3697	3705	3697
q11	478	248	258	248
q12	490	224	220	220
q13	17818	2961	3001	2961
q14	272	232	226	226
q15	516	474	491	474
q16	497	370	370	370
q17	975	661	740	661
q18	8166	7484	7579	7484
q19	4978	1457	1449	1449
q20	643	323	345	323
q21	4946	3792	3195	3195
q22	394	339	356	339
Total cold run time: 116756 ms
Total hot run time: 39823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4283	4202	4202
q2	390	257	258	257
q3	3022	2917	2876	2876
q4	2045	1700	1727	1700
q5	5566	5491	5477	5477
q6	220	129	128	128
q7	2195	1852	1858	1852
q8	3283	3401	3463	3401
q9	8751	8666	8813	8666
q10	4048	3896	3724	3724
q11	591	481	512	481
q12	810	639	619	619
q13	17038	3115	3148	3115
q14	318	273	286	273
q15	516	490	476	476
q16	473	423	444	423
q17	1856	1511	1498	1498
q18	8007	7844	7746	7746
q19	1839	1529	1560	1529
q20	3051	1933	1879	1879
q21	6237	4922	4934	4922
q22	642	558	560	558
Total cold run time: 75229 ms
Total hot run time: 55802 ms

doris-robot · 2024-06-24T10:19:13Z

TPC-DS: Total hot run time: 175165 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cdebf692f971fbf261ce116689e2ca5276cdd50, data reload: false

query1	940	388	369	369
query2	6446	2352	2226	2226
query3	6625	210	214	210
query4	23009	17492	17430	17430
query5	3626	476	465	465
query6	245	161	163	161
query7	4581	305	290	290
query8	312	298	306	298
query9	8451	2473	2454	2454
query10	582	283	297	283
query11	10624	10309	10235	10235
query12	114	83	82	82
query13	1641	364	357	357
query14	10121	6928	7809	6928
query15	244	191	194	191
query16	7853	283	284	283
query17	1867	572	540	540
query18	1990	287	284	284
query19	207	158	157	157
query20	93	92	87	87
query21	212	130	127	127
query22	4453	4290	4309	4290
query23	34097	33642	33702	33642
query24	10806	2884	3017	2884
query25	606	415	404	404
query26	727	160	158	158
query27	2309	334	337	334
query28	5519	2177	2162	2162
query29	899	653	649	649
query30	250	158	159	158
query31	977	777	850	777
query32	95	56	54	54
query33	650	302	303	302
query34	866	488	477	477
query35	760	671	657	657
query36	1131	986	995	986
query37	143	75	71	71
query38	3005	2900	2869	2869
query39	891	825	818	818
query40	215	135	125	125
query41	54	51	56	51
query42	109	102	99	99
query43	590	539	532	532
query44	1091	727	744	727
query45	193	162	158	158
query46	1070	722	706	706
query47	1910	1798	1792	1792
query48	361	321	297	297
query49	845	402	417	402
query50	781	389	394	389
query51	6920	6761	6703	6703
query52	101	95	89	89
query53	360	289	279	279
query54	876	448	447	447
query55	76	74	73	73
query56	279	256	259	256
query57	1154	1046	1090	1046
query58	258	266	249	249
query59	3391	3266	3320	3266
query60	298	263	282	263
query61	92	90	93	90
query62	601	447	447	447
query63	311	301	288	288
query64	8512	2249	1736	1736
query65	3131	3098	3096	3096
query66	738	333	316	316
query67	15648	15027	15079	15027
query68	4600	539	539	539
query69	490	305	323	305
query70	1139	1092	1175	1092
query71	376	279	283	279
query72	7975	5687	5452	5452
query73	741	332	327	327
query74	6009	5589	5570	5570
query75	3384	2668	2675	2668
query76	2379	944	885	885
query77	665	309	286	286
query78	10393	9864	9898	9864
query79	4162	519	518	518
query80	1058	468	473	468
query81	559	221	228	221
query82	772	104	102	102
query83	329	178	175	175
query84	275	101	90	90
query85	1323	289	279	279
query86	459	322	309	309
query87	3279	3158	3093	3093
query88	4849	2454	2457	2454
query89	559	377	380	377
query90	1824	186	191	186
query91	129	102	103	102
query92	59	50	48	48
query93	5165	548	529	529
query94	1162	194	190	190
query95	405	317	317	317
query96	606	270	266	266
query97	3209	3044	3091	3044
query98	224	203	190	190
query99	1145	842	859	842
Total cold run time: 277635 ms
Total hot run time: 175165 ms

doris-robot · 2024-06-24T10:24:45Z

ClickBench: Total hot run time: 30.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6cdebf692f971fbf261ce116689e2ca5276cdd50, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.51	0.49	0.47
query6	1.16	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.53	0.50	0.49
query10	0.53	0.54	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.75	0.79	0.82
query15	0.82	0.80	0.81
query16	0.36	0.36	0.35
query17	0.95	0.99	1.00
query18	0.20	0.30	0.20
query19	1.85	1.72	1.69
query20	0.01	0.01	0.01
query21	15.42	0.67	0.66
query22	3.70	8.13	1.91
query23	18.28	1.44	1.28
query24	2.13	0.22	0.23
query25	0.15	0.08	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.23	1.04	1.00
query29	12.62	3.34	3.31
query30	0.26	0.06	0.05
query31	2.85	0.38	0.39
query32	3.28	0.48	0.49
query33	2.87	2.88	2.92
query34	17.02	4.43	4.49
query35	4.59	4.51	4.53
query36	0.65	0.45	0.47
query37	0.18	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.83 s
Total hot run time: 30.6 s

doris-robot · 2024-06-27T16:28:40Z

TPC-H: Total hot run time: 39732 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 98918127496fe7919b4b6c1b4e890fac597fe39a, data reload: false

------ Round 1 ----------------------------------
q1	18363	4657	4288	4288
q2	2032	197	194	194
q3	10547	1205	1013	1013
q4	10248	790	841	790
q5	7546	2677	2569	2569
q6	224	138	137	137
q7	966	599	611	599
q8	9230	2118	2101	2101
q9	8987	6540	6514	6514
q10	8962	3758	3750	3750
q11	456	237	242	237
q12	414	240	242	240
q13	17853	2993	2959	2959
q14	272	226	217	217
q15	535	475	480	475
q16	490	389	375	375
q17	992	700	648	648
q18	8190	7463	7355	7355
q19	3874	1459	1431	1431
q20	649	340	329	329
q21	4966	3161	3981	3161
q22	415	353	350	350
Total cold run time: 116211 ms
Total hot run time: 39732 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4457	4280	4237	4237
q2	364	256	271	256
q3	3023	2885	2893	2885
q4	2007	1774	1699	1699
q5	5673	5517	5524	5517
q6	227	129	131	129
q7	2231	1896	1856	1856
q8	3309	3462	3457	3457
q9	8739	8750	8842	8750
q10	4129	3909	3735	3735
q11	595	498	501	498
q12	865	675	660	660
q13	15939	3183	3146	3146
q14	304	264	280	264
q15	541	505	492	492
q16	505	425	437	425
q17	1824	1528	1486	1486
q18	8338	8040	7797	7797
q19	1836	1607	1733	1607
q20	3056	1915	1857	1857
q21	5824	5012	4978	4978
q22	639	546	584	546
Total cold run time: 74425 ms
Total hot run time: 56277 ms

doris-robot · 2024-06-27T16:39:52Z

TPC-DS: Total hot run time: 174800 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 98918127496fe7919b4b6c1b4e890fac597fe39a, data reload: false

query1	930	379	393	379
query2	6479	2585	2374	2374
query3	6643	209	211	209
query4	19499	17721	17382	17382
query5	3675	486	489	486
query6	258	172	165	165
query7	4592	305	302	302
query8	308	308	298	298
query9	8449	2483	2437	2437
query10	568	307	301	301
query11	10444	10044	9985	9985
query12	118	85	85	85
query13	1641	358	350	350
query14	9477	7486	7032	7032
query15	240	188	189	188
query16	7742	286	261	261
query17	1531	533	536	533
query18	1956	278	272	272
query19	200	153	162	153
query20	92	85	83	83
query21	220	147	133	133
query22	4284	3958	4127	3958
query23	33927	34103	33741	33741
query24	10951	2886	2889	2886
query25	589	387	370	370
query26	725	161	160	160
query27	2305	323	331	323
query28	6051	2151	2145	2145
query29	876	622	637	622
query30	261	164	152	152
query31	971	775	751	751
query32	97	53	56	53
query33	646	302	304	302
query34	877	498	498	498
query35	760	677	626	626
query36	1127	953	987	953
query37	153	78	74	74
query38	2969	2848	2832	2832
query39	902	856	862	856
query40	230	137	139	137
query41	58	57	61	57
query42	112	106	104	104
query43	601	587	564	564
query44	1085	720	736	720
query45	197	173	178	173
query46	1072	762	715	715
query47	1853	1747	1768	1747
query48	379	309	300	300
query49	863	465	432	432
query50	760	397	402	397
query51	6946	6822	6788	6788
query52	103	94	98	94
query53	398	294	301	294
query54	899	467	454	454
query55	76	77	74	74
query56	305	282	294	282
query57	1134	1035	1059	1035
query58	266	258	281	258
query59	3451	3347	3225	3225
query60	329	300	299	299
query61	110	109	115	109
query62	593	440	460	440
query63	321	308	302	302
query64	8633	2388	1880	1880
query65	3181	3095	3104	3095
query66	746	337	339	337
query67	15415	14879	14863	14863
query68	4691	540	543	540
query69	601	325	348	325
query70	1168	1106	1142	1106
query71	410	297	284	284
query72	8606	5602	5364	5364
query73	753	330	329	329
query74	5909	5513	5595	5513
query75	3751	2653	2724	2653
query76	2756	896	919	896
query77	710	299	307	299
query78	10593	9841	9853	9841
query79	8275	521	513	513
query80	1314	471	462	462
query81	561	225	218	218
query82	1388	110	100	100
query83	381	174	171	171
query84	274	87	84	84
query85	1932	297	318	297
query86	457	310	314	310
query87	3297	3109	3119	3109
query88	5014	2448	2434	2434
query89	482	383	379	379
query90	1881	197	188	188
query91	131	101	100	100
query92	63	50	49	49
query93	6329	530	516	516
query94	1258	190	188	188
query95	411	311	327	311
query96	607	271	265	265
query97	3213	3044	3047	3044
query98	222	208	197	197
query99	1327	860	830	830
Total cold run time: 282414 ms
Total hot run time: 174800 ms

doris-robot · 2024-06-27T16:45:26Z

ClickBench: Total hot run time: 31.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 98918127496fe7919b4b6c1b4e890fac597fe39a, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.66	0.10	0.09
query5	0.50	0.54	0.48
query6	1.15	0.73	0.73
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.54	0.50	0.49
query10	0.54	0.53	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	0.77	0.80	0.78
query15	0.86	0.82	0.82
query16	0.37	0.36	0.35
query17	1.00	0.98	1.03
query18	0.22	0.25	0.23
query19	1.82	1.72	1.68
query20	0.02	0.01	0.01
query21	15.42	0.82	0.76
query22	4.13	7.05	2.44
query23	18.20	1.37	1.27
query24	2.11	0.24	0.22
query25	0.15	0.08	0.09
query26	0.27	0.17	0.17
query27	0.08	0.08	0.09
query28	13.30	1.02	1.00
query29	12.56	3.33	3.32
query30	0.28	0.07	0.05
query31	2.86	0.39	0.40
query32	3.27	0.47	0.48
query33	2.90	2.90	2.94
query34	17.02	4.50	4.48
query35	4.47	4.56	4.50
query36	0.66	0.47	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.18	0.15	0.15
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.24 s
Total hot run time: 31.37 s

dataroaring

LGTM

bobhan1 · 2024-06-28T03:33:04Z

run build all

github-actions · 2024-06-28T03:38:37Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2024-06-28T06:36:40Z

run build all

bobhan1 · 2024-06-28T06:36:49Z

run buildall

doris-robot · 2024-06-28T08:09:43Z

TPC-H: Total hot run time: 40513 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95f675031bdba2fb65ac04a3cc1f42a95ab0e87b, data reload: false

------ Round 1 ----------------------------------
q1	17956	5033	4285	4285
q2	2026	192	208	192
q3	10526	1213	1132	1132
q4	10212	849	750	750
q5	7503	2666	2613	2613
q6	215	132	135	132
q7	946	603	592	592
q8	9216	2092	2079	2079
q9	8973	6522	6504	6504
q10	8968	3685	3734	3685
q11	446	239	230	230
q12	450	227	232	227
q13	17761	3012	2975	2975
q14	278	213	216	213
q15	522	486	474	474
q16	504	371	368	368
q17	968	723	689	689
q18	8144	7455	7416	7416
q19	5575	1555	1447	1447
q20	664	314	311	311
q21	4902	3859	3913	3859
q22	407	341	340	340
Total cold run time: 117162 ms
Total hot run time: 40513 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4386	4219	4201	4201
q2	376	263	264	263
q3	2991	2929	2926	2926
q4	2029	1710	1703	1703
q5	5531	5549	5491	5491
q6	217	127	129	127
q7	2232	1837	1806	1806
q8	3311	3441	3424	3424
q9	8720	8779	8691	8691
q10	4164	3656	3801	3656
q11	597	502	511	502
q12	804	664	664	664
q13	17230	3174	3199	3174
q14	302	296	277	277
q15	509	502	493	493
q16	491	437	448	437
q17	1810	1492	1482	1482
q18	8276	7975	7875	7875
q19	1903	1790	1674	1674
q20	3205	1866	1889	1866
q21	5191	4653	4969	4653
q22	768	587	557	557
Total cold run time: 75043 ms
Total hot run time: 55942 ms

doris-robot · 2024-06-28T08:20:43Z

TPC-DS: Total hot run time: 174043 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95f675031bdba2fb65ac04a3cc1f42a95ab0e87b, data reload: false

query1	953	383	378	378
query2	6404	2476	2310	2310
query3	6632	207	211	207
query4	19726	17780	17297	17297
query5	3561	492	487	487
query6	257	180	163	163
query7	4583	302	288	288
query8	300	316	302	302
query9	8513	2462	2448	2448
query10	547	299	275	275
query11	10644	10108	10097	10097
query12	119	87	81	81
query13	1645	385	384	384
query14	10207	7644	6273	6273
query15	233	186	186	186
query16	7774	271	264	264
query17	1896	547	531	531
query18	1955	271	269	269
query19	191	154	154	154
query20	89	79	80	79
query21	217	130	123	123
query22	4404	4050	4049	4049
query23	33869	33627	33449	33449
query24	11066	3043	2832	2832
query25	626	401	374	374
query26	1138	169	158	158
query27	2309	341	325	325
query28	5903	2208	2184	2184
query29	878	631	642	631
query30	257	157	155	155
query31	976	779	756	756
query32	92	56	56	56
query33	761	299	289	289
query34	1060	484	497	484
query35	769	635	636	635
query36	1122	971	980	971
query37	153	75	78	75
query38	2958	2837	2956	2837
query39	901	856	839	839
query40	213	128	127	127
query41	55	51	54	51
query42	105	101	105	101
query43	608	590	564	564
query44	1261	758	748	748
query45	194	170	166	166
query46	1072	731	732	731
query47	1861	1779	1770	1770
query48	378	297	301	297
query49	846	405	429	405
query50	772	396	394	394
query51	6926	6788	6747	6747
query52	110	91	97	91
query53	357	289	286	286
query54	919	455	446	446
query55	76	74	75	74
query56	295	261	262	261
query57	1140	1055	1057	1055
query58	251	274	267	267
query59	3440	3291	3216	3216
query60	333	281	277	277
query61	93	93	96	93
query62	613	447	461	447
query63	329	302	297	297
query64	8771	2268	1764	1764
query65	3177	3218	3107	3107
query66	768	338	336	336
query67	15611	15083	15045	15045
query68	4546	547	605	547
query69	493	435	400	400
query70	1172	1166	1139	1139
query71	412	294	292	292
query72	7474	5664	5733	5664
query73	742	329	326	326
query74	5840	5480	5565	5480
query75	3500	2704	2710	2704
query76	2214	942	952	942
query77	460	312	314	312
query78	10133	9840	9674	9674
query79	2279	549	534	534
query80	1847	472	489	472
query81	590	220	223	220
query82	790	105	105	105
query83	281	165	167	165
query84	263	91	84	84
query85	1905	290	280	280
query86	488	332	317	317
query87	3303	3084	3071	3071
query88	4178	2363	2380	2363
query89	480	392	392	392
query90	1729	193	191	191
query91	130	100	107	100
query92	59	49	53	49
query93	2436	526	527	526
query94	1085	189	185	185
query95	464	314	319	314
query96	595	273	264	264
query97	3252	3094	3030	3030
query98	231	216	217	216
query99	1305	853	827	827
Total cold run time: 270938 ms
Total hot run time: 174043 ms

doris-robot · 2024-06-28T08:26:15Z

ClickBench: Total hot run time: 31.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95f675031bdba2fb65ac04a3cc1f42a95ab0e87b, data reload: false

query1	0.04	0.02	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.09	0.10
query5	0.50	0.49	0.49
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.54	0.54	0.53
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.81	0.78
query15	0.83	0.83	0.81
query16	0.37	0.36	0.38
query17	1.05	1.05	1.00
query18	0.25	0.25	0.24
query19	1.89	1.79	1.83
query20	0.01	0.01	0.01
query21	15.42	0.76	0.66
query22	4.02	6.31	2.30
query23	18.31	1.45	1.27
query24	2.05	0.23	0.21
query25	0.16	0.09	0.08
query26	0.26	0.17	0.18
query27	0.09	0.08	0.08
query28	13.29	1.01	1.00
query29	12.65	3.34	3.30
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.28	0.48	0.48
query33	2.88	2.86	2.93
query34	17.08	4.38	4.39
query35	4.49	4.47	4.56
query36	0.66	0.48	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.19	0.16	0.16
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.39 s
Total hot run time: 31.05 s

zhannngchen

LGTM

github-actions · 2024-06-28T09:32:06Z

PR approved by at least one committer and no changes requested.

…36421) 1. increase the value of `AutoIncrementGenerator.BATCH_ID_INTERVAL` to reduce the number of writes to BDBJE. (the default value of `config::auto_inc_prefetch_size_ratio` is 10 and the default value of `AutoIncIDBuffer::_batch_size` is 4064, so mostly the request length is 40960) 2. only allow master fe to offer `getAutoIncrementRange` service 3. write editlog before update `batchEndId` in memory in `getAutoIncrementRange ` 4. refactor `AutoIncIDBuffer`

…h auto inc column #36421" (#37027) ## Proposed changes pick #36421

bobhan1 force-pushed the try-fix-auto-inc branch from a771e13 to 38154b4 Compare June 17, 2024 15:36

dataroaring previously approved these changes Jun 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2024

github-actions bot added the reviewed label Jun 17, 2024

zhannngchen requested changes Jun 18, 2024

View reviewed changes

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 18, 2024

dataroaring added the dev/2.1.x label Jun 19, 2024

bobhan1 dismissed dataroaring’s stale review via 1f3c240 June 20, 2024 12:22

bobhan1 force-pushed the try-fix-auto-inc branch 2 times, most recently from 1f3c240 to 633dab0 Compare June 20, 2024 12:25

bobhan1 force-pushed the try-fix-auto-inc branch from 6cdebf6 to c73fd0c Compare June 24, 2024 11:06

dataroaring approved these changes Jun 27, 2024

View reviewed changes

bobhan1 added 8 commits June 28, 2024 11:13

tmp

2979436

tmp

ea6c872

check master in getAutoIncrementRange service

32d75ea

refactor autoinf_buffer async fetching mechanism

86b5da1

tmp save, refactor autoinc_buffer.cpp, remove current_volume

506c483

refactor autoinc_buffer

696a3c6

reduce test concurrency

1f06a05

fix AutoIncrementGenerator

95f6750

bobhan1 force-pushed the try-fix-auto-inc branch from 9891812 to 95f6750 Compare June 28, 2024 03:32

zhannngchen approved these changes Jun 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024

dataroaring merged commit 1059604 into apache:master Jun 28, 2024

bobhan1 mentioned this pull request Jun 29, 2024

[branch-2.1] Pick "[Fix](autoinc) try fix concurrent load problem with auto inc column #36421" #37027

Merged

dataroaring pushed a commit that referenced this pull request Jun 30, 2024

[branch-2.1] Pick "[Fix](autoinc) try fix concurrent load problem wit…

12dddfc

…h auto inc column #36421" (#37027) ## Proposed changes pick #36421

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jun 30, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

[Fix](autoinc) try fix concurrent load problem with auto inc column #36421

[Fix](autoinc) try fix concurrent load problem with auto inc column #36421

Uh oh!

Conversation

bobhan1 commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

bobhan1 commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

bobhan1 commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

github-actions bot commented Jun 22, 2024

Uh oh!

dataroaring commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 24, 2024

Uh oh!

doris-robot commented Jun 27, 2024

Uh oh!

doris-robot commented Jun 27, 2024

Uh oh!

doris-robot commented Jun 27, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

bobhan1 commented Jun 28, 2024

Uh oh!

github-actions bot commented Jun 28, 2024

Uh oh!

bobhan1 commented Jun 28, 2024

Uh oh!

bobhan1 commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

doris-robot commented Jun 28, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 28, 2024

Uh oh!

Reviewers

bobhan1 commented Jun 17, 2024 •

edited

Loading