[enhencement](config) change default write buffer size #30749

liaoxin01 · 2024-02-02T06:51:09Z

Proposed changes

Issue Number: close #xxx

revert #28977
We recover memtable size from 100MB to 200MB. Reducing the Memtable size will result in loading large files with a -238 error message.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-02T06:51:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

liaoxin01 · 2024-02-02T06:51:44Z

run buildall

github-actions · 2024-02-02T07:00:06Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-02T07:22:46Z

TeamCity be ut coverage result:
Function Coverage: 36.17% (8625/23846)
Line Coverage: 28.22% (70521/249896)
Region Coverage: 27.24% (36397/133621)
Branch Coverage: 24.03% (18641/77582)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6b85496e35f404588a88773678c70f5beef15925_6b85496e35f404588a88773678c70f5beef15925/report/index.html

doris-robot · 2024-02-02T07:25:52Z

TPC-H: Total hot run time: 36992 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b85496e35f404588a88773678c70f5beef15925, data reload: false

------ Round 1 ----------------------------------
q1	17685	4444	4411	4411
q2	2036	134	129	129
q3	10601	906	934	906
q4	4653	716	782	716
q5	7679	2914	2900	2900
q6	180	121	122	121
q7	1085	702	696	696
q8	9245	2033	2011	2011
q9	7136	6331	6303	6303
q10	8070	2417	2407	2407
q11	415	207	216	207
q12	746	258	275	258
q13	17987	3282	3349	3282
q14	275	247	243	243
q15	522	493	500	493
q16	474	401	405	401
q17	936	568	524	524
q18	6790	5902	5986	5902
q19	1587	1443	1353	1353
q20	559	345	317	317
q21	6904	3126	3153	3126
q22	814	297	286	286
Total cold run time: 106379 ms
Total hot run time: 36992 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4530	4355	4389	4355
q2	332	239	227	227
q3	2967	2824	2772	2772
q4	1894	1715	1612	1612
q5	5179	5306	5250	5250
q6	201	115	114	114
q7	2187	1790	1792	1790
q8	3103	3214	3242	3214
q9	8286	8267	8382	8267
q10	5797	3589	3694	3589
q11	554	453	471	453
q12	769	568	570	568
q13	13255	3120	3080	3080
q14	273	262	263	262
q15	533	484	483	483
q16	493	480	479	479
q17	1840	1659	1667	1659
q18	8027	7689	7496	7496
q19	8875	1556	1539	1539
q20	2122	1914	1902	1902
q21	4822	4539	4565	4539
q22	576	451	472	451
Total cold run time: 76615 ms
Total hot run time: 54101 ms

doris-robot · 2024-02-02T07:37:05Z

TPC-DS: Total hot run time: 174324 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b85496e35f404588a88773678c70f5beef15925, data reload: false

query1	920	351	329	329
query2	6533	1937	1870	1870
query3	6697	200	201	200
query4	31618	22039	22041	22039
query5	4008	403	380	380
query6	243	157	155	155
query7	4607	266	260	260
query8	246	175	178	175
query9	8971	2311	2296	2296
query10	418	210	193	193
query11	18612	15552	15507	15507
query12	122	68	62	62
query13	1636	373	381	373
query14	9645	7323	7316	7316
query15	249	189	181	181
query16	7933	276	253	253
query17	1913	516	473	473
query18	2036	262	259	259
query19	350	129	135	129
query20	77	69	71	69
query21	199	135	122	122
query22	4771	4568	4544	4544
query23	32133	30220	30031	30031
query24	10685	2720	2797	2720
query25	538	314	299	299
query26	987	139	140	139
query27	2625	282	286	282
query28	7056	1851	1838	1838
query29	955	641	621	621
query30	282	132	135	132
query31	918	716	714	714
query32	93	57	49	49
query33	732	209	208	208
query34	1103	463	462	462
query35	854	791	752	752
query36	1073	954	938	938
query37	90	62	60	60
query38	3217	3063	3175	3063
query39	1315	1258	1247	1247
query40	197	88	89	88
query41	38	35	37	35
query42	88	82	94	82
query43	564	482	485	482
query44	1195	689	689	689
query45	187	178	170	170
query46	1037	649	636	636
query47	1639	1531	1529	1529
query48	386	305	327	305
query49	1111	279	291	279
query50	681	316	319	316
query51	5245	5140	5109	5109
query52	96	82	82	82
query53	328	261	258	258
query54	257	176	196	176
query55	81	79	80	79
query56	176	160	175	160
query57	993	857	919	857
query58	183	154	153	153
query59	2485	2390	2465	2390
query60	198	181	178	178
query61	90	83	82	82
query62	645	374	352	352
query63	279	272	270	270
query64	5202	3430	3633	3430
query65	3247	3200	3243	3200
query66	926	314	306	306
query67	14484	14277	14128	14128
query68	5468	509	475	475
query69	472	313	317	313
query70	1260	1250	1168	1168
query71	306	211	215	211
query72	6173	3162	2858	2858
query73	680	316	315	315
query74	6681	6200	6119	6119
query75	3094	2335	2331	2331
query76	3383	935	915	915
query77	373	232	227	227
query78	9256	8817	8641	8641
query79	2398	494	490	490
query80	1377	315	303	303
query81	540	195	195	195
query82	887	79	81	79
query83	213	116	114	114
query84	234	71	77	71
query85	2174	342	331	331
query86	499	302	297	297
query87	3424	3245	3288	3245
query88	3965	2188	2162	2162
query89	432	354	357	354
query90	1920	185	184	184
query91	145	119	117	117
query92	59	49	45	45
query93	2374	415	423	415
query94	1379	156	162	156
query95	455	353	352	352
query96	614	315	314	314
query97	4236	4138	4129	4129
query98	216	195	182	182
query99	974	655	703	655
Total cold run time: 284159 ms
Total hot run time: 174324 ms

doris-robot · 2024-02-02T07:42:32Z

ClickBench: Total hot run time: 30.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b85496e35f404588a88773678c70f5beef15925, data reload: false

query1	0.03	0.03	0.02
query2	0.05	0.02	0.02
query3	0.23	0.06	0.07
query4	1.65	0.10	0.10
query5	0.53	0.51	0.51
query6	1.22	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.56	0.50	0.49
query10	0.57	0.54	0.57
query11	0.12	0.09	0.08
query12	0.11	0.09	0.10
query13	0.61	0.60	0.60
query14	0.78	0.79	0.82
query15	0.79	0.78	0.77
query16	0.39	0.38	0.39
query17	0.99	1.03	1.04
query18	0.21	0.27	0.20
query19	1.89	1.79	1.80
query20	0.01	0.01	0.02
query21	15.41	0.58	0.55
query22	3.27	2.51	1.35
query23	16.90	0.78	0.81
query24	2.46	1.00	0.78
query25	0.32	0.17	0.16
query26	0.47	0.14	0.15
query27	0.07	0.04	0.05
query28	12.09	0.85	0.82
query29	12.55	3.21	3.15
query30	0.69	0.59	0.54
query31	2.78	0.35	0.35
query32	3.34	0.48	0.48
query33	3.23	3.24	3.26
query34	15.91	4.28	4.31
query35	4.32	4.34	4.27
query36	1.10	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.12
query41	0.08	0.01	0.01
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.14 s
Total hot run time: 30.39 s

doris-robot · 2024-02-02T07:45:40Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6b85496e35f404588a88773678c70f5beef15925 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.5 seconds inserted 10000000 Rows, about 689K ops/s

liaoxin01 · 2024-02-02T07:48:25Z

run p0

liaoxin01 · 2024-02-02T08:29:45Z

run buildall

github-actions · 2024-02-02T08:39:07Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-02T09:02:10Z

TPC-H: Total hot run time: 36630 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80afc2bc76c31547e4544474ef30ec47bf3da596, data reload: false

------ Round 1 ----------------------------------
q1	17673	4462	4398	4398
q2	2038	139	133	133
q3	10610	910	927	910
q4	4647	686	688	686
q5	7684	2861	2756	2756
q6	182	119	118	118
q7	1149	717	707	707
q8	9277	2002	2007	2002
q9	7218	6345	6297	6297
q10	8158	2424	2374	2374
q11	412	208	212	208
q12	707	282	271	271
q13	18014	3279	3234	3234
q14	277	249	245	245
q15	533	491	501	491
q16	476	427	409	409
q17	941	523	531	523
q18	6658	5872	5880	5872
q19	1554	1378	1338	1338
q20	582	336	326	326
q21	6776	3064	3029	3029
q22	794	303	307	303
Total cold run time: 106360 ms
Total hot run time: 36630 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4487	4382	4433	4382
q2	353	226	226	226
q3	3017	2836	2851	2836
q4	1846	1599	1607	1599
q5	5185	5170	5221	5170
q6	188	112	114	112
q7	2166	1804	1812	1804
q8	3069	3234	3241	3234
q9	8338	8283	8323	8283
q10	5818	3543	3566	3543
q11	543	476	472	472
q12	767	567	587	567
q13	15870	3071	3094	3071
q14	296	247	274	247
q15	537	499	492	492
q16	509	467	481	467
q17	1849	1660	1712	1660
q18	8183	7843	7649	7649
q19	10679	1544	1545	1544
q20	2145	1912	1914	1912
q21	5055	4759	4837	4759
q22	605	510	515	510
Total cold run time: 81505 ms
Total hot run time: 54539 ms

doris-robot · 2024-02-02T09:08:24Z

TeamCity be ut coverage result:
Function Coverage: 36.16% (8627/23856)
Line Coverage: 28.22% (70537/249923)
Region Coverage: 27.24% (36403/133641)
Branch Coverage: 24.02% (18640/77596)
Coverage Report: http://coverage.selectdb-in.cc/coverage/80afc2bc76c31547e4544474ef30ec47bf3da596_80afc2bc76c31547e4544474ef30ec47bf3da596/report/index.html

doris-robot · 2024-02-02T09:13:47Z

TPC-DS: Total hot run time: 173867 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 80afc2bc76c31547e4544474ef30ec47bf3da596, data reload: false

query1	3095	347	338	338
query2	8539	1949	1947	1947
query3	8172	226	206	206
query4	32433	22115	21956	21956
query5	4316	360	422	360
query6	250	165	161	161
query7	4605	272	285	272
query8	231	178	173	173
query9	9097	2287	2263	2263
query10	412	202	212	202
query11	18178	15211	15329	15211
query12	136	74	72	72
query13	1620	411	410	410
query14	9344	6850	6919	6850
query15	299	177	177	177
query16	8164	250	241	241
query17	2195	540	504	504
query18	2197	268	262	262
query19	371	137	134	134
query20	79	73	79	73
query21	199	134	123	123
query22	4856	4585	4664	4585
query23	31636	30044	29931	29931
query24	10520	2748	2801	2748
query25	577	337	335	335
query26	1185	140	144	140
query27	2928	300	300	300
query28	7382	1850	1836	1836
query29	861	600	592	592
query30	276	134	140	134
query31	910	689	733	689
query32	88	55	49	49
query33	733	215	212	212
query34	1114	438	454	438
query35	869	761	757	757
query36	1040	933	914	914
query37	115	58	58	58
query38	3191	3119	3089	3089
query39	1300	1262	1235	1235
query40	182	93	89	89
query41	37	35	33	33
query42	101	87	89	87
query43	516	479	485	479
query44	1052	682	691	682
query45	184	178	170	170
query46	1050	646	632	632
query47	1631	1440	1518	1440
query48	412	346	354	346
query49	1076	271	271	271
query50	760	358	361	358
query51	5239	5150	5127	5127
query52	93	91	77	77
query53	324	267	265	265
query54	270	203	216	203
query55	74	77	74	74
query56	214	202	197	197
query57	984	905	936	905
query58	205	175	179	175
query59	2434	2321	2457	2321
query60	236	217	217	217
query61	83	80	78	78
query62	659	343	358	343
query63	286	270	275	270
query64	5149	3631	3682	3631
query65	3242	3217	3224	3217
query66	812	306	318	306
query67	14292	14078	14127	14078
query68	5460	557	534	534
query69	494	330	323	323
query70	1283	1267	1255	1255
query71	338	244	258	244
query72	6480	2837	2662	2662
query73	689	315	323	315
query74	6570	6216	6214	6214
query75	3049	2346	2315	2315
query76	3469	934	944	934
query77	359	221	220	220
query78	9179	8704	8502	8502
query79	2756	484	494	484
query80	2115	357	336	336
query81	546	198	198	198
query82	877	83	73	73
query83	299	127	122	122
query84	284	81	80	80
query85	2189	335	325	325
query86	481	306	285	285
query87	3420	3163	3205	3163
query88	4239	2305	2312	2305
query89	441	366	353	353
query90	1971	161	164	161
query91	148	119	120	119
query92	58	41	41	41
query93	3812	469	463	463
query94	1297	175	172	172
query95	7918	7571	348	348
query96	583	270	272	270
query97	4269	4136	4157	4136
query98	210	203	197	197
query99	1190	680	670	670
Total cold run time: 301583 ms
Total hot run time: 173867 ms

doris-robot · 2024-02-02T09:19:11Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 80afc2bc76c31547e4544474ef30ec47bf3da596, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.52	0.52
query6	1.19	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.02	0.02
query9	0.54	0.50	0.50
query10	0.55	0.54	0.56
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.59	0.61	0.61
query14	0.77	0.79	0.79
query15	0.79	0.77	0.77
query16	0.38	0.38	0.36
query17	0.99	1.00	0.98
query18	0.22	0.24	0.24
query19	1.89	1.73	1.77
query20	0.01	0.01	0.01
query21	15.40	0.56	0.55
query22	2.82	2.29	1.98
query23	17.21	0.98	0.91
query24	2.34	1.17	0.25
query25	0.28	0.24	0.18
query26	0.47	0.15	0.14
query27	0.06	0.04	0.04
query28	11.77	0.88	0.82
query29	12.48	3.15	3.24
query30	0.64	0.53	0.54
query31	2.78	0.35	0.35
query32	3.36	0.47	0.48
query33	3.22	3.26	3.19
query34	15.95	4.27	4.25
query35	4.32	4.39	4.33
query36	1.11	1.06	1.05
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.12
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.33 s
Total hot run time: 30.55 s

doris-robot · 2024-02-02T09:22:16Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 80afc2bc76c31547e4544474ef30ec47bf3da596 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

dataroaring

LGTM

github-actions · 2024-02-02T11:22:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-02-02T11:22:17Z

PR approved by anyone and no changes requested.

sollhui

LGTM

liaoxin01 added dev/2.1.0 and removed dev/2.1.0 labels Feb 2, 2024

[enhencement](config) change default write buffer size

80afc2b

liaoxin01 force-pushed the write_buffer_size branch from 6b85496 to 80afc2b Compare February 2, 2024 08:29

dataroaring approved these changes Feb 2, 2024

View reviewed changes

dataroaring added the dev/2.1.0 label Feb 2, 2024

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 2, 2024

github-actions bot added the reviewed label Feb 2, 2024

sollhui approved these changes Feb 2, 2024

View reviewed changes

dataroaring merged commit c7b439a into apache:master Feb 2, 2024

yiguolei pushed a commit that referenced this pull request Feb 3, 2024

[enhencement](config) change default write buffer size (#30749)

bc8c187

liaoxin01 deleted the write_buffer_size branch February 6, 2024 12:26

[enhencement](config) change default write buffer size #30749

[enhencement](config) change default write buffer size #30749

Uh oh!

Conversation

liaoxin01 commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

liaoxin01 commented Feb 2, 2024

Uh oh!

github-actions bot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

liaoxin01 commented Feb 2, 2024

Uh oh!

liaoxin01 commented Feb 2, 2024

Uh oh!

github-actions bot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

doris-robot commented Feb 2, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 2, 2024

Uh oh!

github-actions bot commented Feb 2, 2024

Uh oh!

sollhui left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liaoxin01 commented Feb 2, 2024 •

edited

Loading