Skip to content

Conversation

@Baymine
Copy link
Contributor

@Baymine Baymine commented Jul 15, 2024

Proposed changes

Updated the default setting of sys_log_mode in fe.conf from NORMAL to ASYNC for better performance.

…onf from NORMAL to ASYNC for better performance.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

caiconghui
caiconghui previously approved these changes Jul 15, 2024
Copy link
Contributor

@caiconghui caiconghui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@caiconghui
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40120 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 49cc89aa80a6f27490bff591619e238b262cfed5, data reload: false

------ Round 1 ----------------------------------
q1	17852	4503	4437	4437
q2	2267	193	198	193
q3	10455	1240	1149	1149
q4	10197	810	755	755
q5	7525	2681	2649	2649
q6	229	140	140	140
q7	959	605	589	589
q8	9224	2067	2072	2067
q9	8349	6589	6551	6551
q10	8611	3837	3758	3758
q11	445	241	242	241
q12	399	230	233	230
q13	17770	3012	3005	3005
q14	285	237	231	231
q15	527	478	501	478
q16	488	397	382	382
q17	957	629	702	629
q18	8177	7685	7440	7440
q19	8521	1517	1465	1465
q20	667	319	329	319
q21	4855	3125	3195	3125
q22	350	291	287	287
Total cold run time: 119109 ms
Total hot run time: 40120 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4421	4307	4270	4270
q2	381	259	267	259
q3	3058	2804	2760	2760
q4	1863	1631	1603	1603
q5	5283	5324	5340	5324
q6	221	131	131	131
q7	2098	1752	1776	1752
q8	3201	3405	3322	3322
q9	8427	8385	8357	8357
q10	3971	3707	3717	3707
q11	565	489	489	489
q12	781	617	593	593
q13	17539	2976	3028	2976
q14	305	281	279	279
q15	519	482	485	482
q16	488	416	432	416
q17	1815	1489	1479	1479
q18	7588	7537	7506	7506
q19	1646	1493	1604	1493
q20	2023	1829	1798	1798
q21	4827	4602	4643	4602
q22	570	518	489	489
Total cold run time: 71590 ms
Total hot run time: 54087 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173011 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 49cc89aa80a6f27490bff591619e238b262cfed5, data reload: false

query1	921	366	362	362
query2	6467	1851	1760	1760
query3	6638	209	220	209
query4	27921	17863	17288	17288
query5	4192	497	476	476
query6	266	172	163	163
query7	4593	297	285	285
query8	236	194	190	190
query9	8593	2376	2388	2376
query10	452	305	287	287
query11	10683	10210	10252	10210
query12	140	86	90	86
query13	1662	380	380	380
query14	9658	7791	7764	7764
query15	235	172	171	171
query16	7860	322	314	314
query17	1789	572	547	547
query18	1996	291	287	287
query19	208	155	160	155
query20	96	86	87	86
query21	207	126	130	126
query22	4371	4039	3899	3899
query23	34027	33261	33341	33261
query24	12159	2952	2924	2924
query25	665	386	392	386
query26	1805	158	154	154
query27	2969	279	273	273
query28	7507	1956	1976	1956
query29	1114	712	606	606
query30	289	150	149	149
query31	950	743	746	743
query32	98	56	54	54
query33	774	286	303	286
query34	990	493	497	493
query35	685	595	578	578
query36	1098	919	912	912
query37	283	89	81	81
query38	2873	2791	2746	2746
query39	851	825	819	819
query40	277	125	120	120
query41	62	44	45	44
query42	123	99	102	99
query43	498	472	461	461
query44	1192	732	722	722
query45	194	163	160	160
query46	1090	743	718	718
query47	1875	1777	1774	1774
query48	367	300	291	291
query49	1199	424	413	413
query50	780	399	405	399
query51	6964	6845	6854	6845
query52	108	91	94	91
query53	362	291	296	291
query54	986	445	441	441
query55	75	74	75	74
query56	319	269	266	266
query57	1189	1036	1039	1036
query58	264	251	251	251
query59	2847	2578	2533	2533
query60	303	288	280	280
query61	111	95	94	94
query62	836	648	662	648
query63	325	302	288	288
query64	10519	2207	1662	1662
query65	3254	3132	3102	3102
query66	1366	334	325	325
query67	15405	14862	14982	14862
query68	4685	550	520	520
query69	620	414	338	338
query70	1184	1156	1167	1156
query71	447	286	289	286
query72	7821	5231	6269	5231
query73	774	325	326	325
query74	6023	5705	5645	5645
query75	4002	2772	2731	2731
query76	3121	943	893	893
query77	721	322	316	316
query78	9719	9164	9143	9143
query79	4223	509	516	509
query80	1133	465	474	465
query81	577	223	225	223
query82	1492	155	139	139
query83	305	167	171	167
query84	279	90	93	90
query85	1564	364	300	300
query86	415	317	320	317
query87	3328	3166	3210	3166
query88	5227	2476	2455	2455
query89	502	369	380	369
query90	1926	193	193	193
query91	129	99	103	99
query92	57	49	52	49
query93	4958	501	491	491
query94	1194	217	211	211
query95	405	307	317	307
query96	604	272	275	272
query97	3198	3070	3035	3035
query98	230	195	191	191
query99	1572	1247	1277	1247
Total cold run time: 293646 ms
Total hot run time: 173011 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 49cc89aa80a6f27490bff591619e238b262cfed5, data reload: false

query1	0.05	0.04	0.03
query2	0.09	0.04	0.05
query3	0.21	0.05	0.05
query4	1.69	0.07	0.06
query5	0.50	0.48	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.53	0.50	0.48
query10	0.55	0.56	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.57
query14	0.77	0.78	0.76
query15	0.86	0.80	0.81
query16	0.37	0.37	0.36
query17	0.98	0.96	1.01
query18	0.22	0.21	0.22
query19	1.82	1.83	1.83
query20	0.01	0.01	0.01
query21	15.40	0.75	0.66
query22	4.06	8.47	1.48
query23	18.28	1.35	1.30
query24	2.11	0.24	0.21
query25	0.16	0.08	0.08
query26	0.29	0.21	0.22
query27	0.45	0.23	0.23
query28	13.29	1.00	0.99
query29	12.64	3.29	3.31
query30	0.25	0.06	0.06
query31	2.85	0.40	0.39
query32	3.29	0.49	0.47
query33	2.93	2.94	2.87
query34	17.10	4.37	4.34
query35	4.45	4.44	4.44
query36	0.65	0.46	0.49
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.68 s
Total hot run time: 30.24 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

morningman
morningman previously approved these changes Jul 15, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need modify Config.java too

…ig.java from NORMAL to ASYNC for better performance.
@Baymine Baymine dismissed stale reviews from morningman and caiconghui via caff078 July 15, 2024 08:12
@yiguolei
Copy link
Contributor

run buildall

@caiconghui
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39648 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 131b2937f6e7e3b85103e3757ed2bb80ac6758ea, data reload: false

------ Round 1 ----------------------------------
q1	17837	4457	4351	4351
q2	2319	196	188	188
q3	11704	1144	1074	1074
q4	10745	722	807	722
q5	7663	2703	2675	2675
q6	221	142	138	138
q7	972	606	612	606
q8	9257	2100	2119	2100
q9	8962	6541	6568	6541
q10	8734	3756	3759	3756
q11	454	239	239	239
q12	393	221	218	218
q13	17776	2957	2970	2957
q14	276	239	243	239
q15	531	492	480	480
q16	492	391	377	377
q17	953	643	752	643
q18	8046	7378	7284	7284
q19	5625	1504	1349	1349
q20	650	314	328	314
q21	5038	3111	3217	3111
q22	357	286	288	286
Total cold run time: 119005 ms
Total hot run time: 39648 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4384	4264	4300	4264
q2	376	272	272	272
q3	3019	2751	2825	2751
q4	1975	1732	1715	1715
q5	5572	5606	5564	5564
q6	234	132	130	130
q7	2194	1803	1830	1803
q8	3288	3442	3379	3379
q9	8788	8797	8843	8797
q10	4143	3889	3873	3873
q11	584	508	503	503
q12	789	646	626	626
q13	16696	3096	2999	2999
q14	319	289	280	280
q15	533	512	498	498
q16	480	449	455	449
q17	1784	1519	1499	1499
q18	8003	7899	7896	7896
q19	1738	1634	1582	1582
q20	2185	1887	1893	1887
q21	10330	4631	4800	4631
q22	610	537	495	495
Total cold run time: 78024 ms
Total hot run time: 55893 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173904 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 131b2937f6e7e3b85103e3757ed2bb80ac6758ea, data reload: false

query1	797	396	386	386
query2	6388	2010	1944	1944
query3	6492	202	214	202
query4	24865	17708	17539	17539
query5	3816	513	477	477
query6	259	169	162	162
query7	4497	316	295	295
query8	248	198	200	198
query9	8405	2417	2406	2406
query10	449	283	286	283
query11	10899	10156	10202	10156
query12	128	92	83	83
query13	1590	384	380	380
query14	10363	7700	7254	7254
query15	218	167	167	167
query16	7530	327	325	325
query17	1302	556	557	556
query18	1833	300	293	293
query19	199	154	170	154
query20	94	86	81	81
query21	211	131	129	129
query22	4307	4079	4065	4065
query23	34116	33710	33658	33658
query24	12130	2873	2853	2853
query25	674	402	397	397
query26	1779	153	156	153
query27	2956	282	279	279
query28	7436	2034	2011	2011
query29	1128	656	642	642
query30	268	158	160	158
query31	956	749	753	749
query32	105	55	59	55
query33	778	317	322	317
query34	934	490	493	490
query35	687	588	596	588
query36	1119	953	961	953
query37	281	82	87	82
query38	2896	2779	2717	2717
query39	898	810	814	810
query40	260	129	127	127
query41	49	49	47	47
query42	121	100	98	98
query43	527	484	443	443
query44	1204	728	726	726
query45	195	162	161	161
query46	1096	745	716	716
query47	1911	1805	1784	1784
query48	372	296	297	296
query49	1010	419	428	419
query50	779	392	395	392
query51	7024	6877	6846	6846
query52	103	97	103	97
query53	362	296	288	288
query54	974	463	466	463
query55	77	76	74	74
query56	315	386	271	271
query57	1177	1062	1071	1062
query58	259	264	251	251
query59	2961	2649	2919	2649
query60	336	275	292	275
query61	96	111	94	94
query62	810	658	631	631
query63	317	291	303	291
query64	10573	2239	1691	1691
query65	3285	3121	3158	3121
query66	1361	329	330	329
query67	15534	14981	15163	14981
query68	7930	553	563	553
query69	719	377	334	334
query70	1233	1115	1064	1064
query71	555	284	287	284
query72	8716	5354	5694	5354
query73	817	325	326	325
query74	6201	5681	5739	5681
query75	5294	2701	2696	2696
query76	4867	966	919	919
query77	826	321	306	306
query78	13335	10065	9052	9052
query79	9961	531	531	531
query80	882	496	485	485
query81	595	240	240	240
query82	335	144	137	137
query83	335	174	170	170
query84	278	85	86	85
query85	738	315	306	306
query86	435	329	324	324
query87	3359	3155	3122	3122
query88	5530	2365	2394	2365
query89	505	407	384	384
query90	2070	201	195	195
query91	134	102	102	102
query92	62	51	50	50
query93	2469	524	515	515
query94	1283	221	219	219
query95	407	319	315	315
query96	617	278	274	274
query97	3261	3043	3031	3031
query98	220	204	197	197
query99	1542	1260	1219	1219
Total cold run time: 302630 ms
Total hot run time: 173904 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 131b2937f6e7e3b85103e3757ed2bb80ac6758ea, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.48	0.50
query6	1.14	0.74	0.73
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.54	0.48	0.49
query10	0.56	0.53	0.54
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.58	0.57
query14	0.75	0.78	0.78
query15	0.84	0.80	0.81
query16	0.36	0.36	0.35
query17	1.01	0.98	0.98
query18	0.23	0.22	0.22
query19	1.94	1.73	1.83
query20	0.01	0.02	0.01
query21	15.40	0.74	0.66
query22	4.14	6.64	2.32
query23	18.31	1.34	1.32
query24	2.10	0.23	0.22
query25	0.15	0.08	0.09
query26	0.32	0.21	0.21
query27	0.46	0.23	0.22
query28	13.31	1.01	0.99
query29	12.61	3.27	3.24
query30	0.25	0.06	0.06
query31	2.85	0.38	0.38
query32	3.31	0.49	0.47
query33	2.83	2.86	2.92
query34	16.99	4.32	4.33
query35	4.38	4.41	4.45
query36	0.65	0.47	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.03
query40	0.16	0.12	0.11
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.91 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 2f0e574 into apache:master Jul 16, 2024
@yiguolei
Copy link
Contributor

Do not pick to 2.1 now, we should check it on master.

seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
…onf for better performance (apache#37793)

## Proposed changes

<!--Describe your changes.-->
Updated the default setting of `sys_log_mode` in `fe.conf` from `NORMAL`
to `ASYNC` for better performance.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…onf for better performance (#37793)

## Proposed changes

<!--Describe your changes.-->
Updated the default setting of `sys_log_mode` in `fe.conf` from `NORMAL`
to `ASYNC` for better performance.
morningman added a commit that referenced this pull request Aug 19, 2024
followup #37793
NORMAL:  sync flush + with location
ASYNC(default): async flush + with location
BRIEF: async flush + without location
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
followup #37793
NORMAL:  sync flush + with location
ASYNC(default): async flush + with location
BRIEF: async flush + without location
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants