Skip to content

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jul 9, 2024

On 64G machine, warning water mark is 12.8G. when system available memory reaches the warning water mark, load memtable will flush, this is not a good idea.

mem_limit default 10%, except for the memory used by the Linux kernel , buffer may be around 5%, so low water mark=min(6.4G, MemTotal * 5%) looks better.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Jul 9, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40489 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c69d162c7fca2a5a5b2b9a8e328bc46a9090c80c, data reload: false

------ Round 1 ----------------------------------
q1	17631	4332	4314	4314
q2	2018	197	194	194
q3	10462	1135	1170	1135
q4	10195	764	827	764
q5	7515	2699	2643	2643
q6	223	140	141	140
q7	954	599	612	599
q8	9216	2081	2076	2076
q9	8761	6490	6470	6470
q10	8976	3769	3714	3714
q11	458	244	242	242
q12	471	220	223	220
q13	17816	2955	2972	2955
q14	264	227	225	225
q15	531	469	471	469
q16	481	371	379	371
q17	967	667	737	667
q18	8053	7461	7347	7347
q19	5348	1520	1502	1502
q20	670	322	330	322
q21	5038	3787	3959	3787
q22	407	333	341	333
Total cold run time: 116455 ms
Total hot run time: 40489 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4379	4228	4211	4211
q2	385	277	269	269
q3	2994	2925	2836	2836
q4	1994	1744	1683	1683
q5	5683	5522	5473	5473
q6	226	136	134	134
q7	2170	1913	1843	1843
q8	3266	3419	3421	3419
q9	8694	8699	8792	8699
q10	4074	3807	3776	3776
q11	599	513	506	506
q12	847	671	636	636
q13	16207	3216	3221	3216
q14	299	276	274	274
q15	551	496	481	481
q16	482	428	428	428
q17	1812	1499	1512	1499
q18	8174	8121	7908	7908
q19	1890	1758	1640	1640
q20	2153	1884	1886	1884
q21	10136	4795	4702	4702
q22	628	566	571	566
Total cold run time: 77643 ms
Total hot run time: 56083 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173323 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c69d162c7fca2a5a5b2b9a8e328bc46a9090c80c, data reload: false

query1	911	370	371	370
query2	6450	2297	2318	2297
query3	6638	206	227	206
query4	28162	17542	17331	17331
query5	3669	466	484	466
query6	254	163	168	163
query7	4593	284	280	280
query8	321	306	307	306
query9	8533	2417	2401	2401
query10	593	304	291	291
query11	11959	10048	10075	10048
query12	116	90	81	81
query13	1644	378	364	364
query14	10193	7359	7665	7359
query15	259	192	189	189
query16	7842	321	308	308
query17	1796	558	514	514
query18	2006	274	270	270
query19	191	145	142	142
query20	86	80	78	78
query21	200	135	126	126
query22	4574	4026	3901	3901
query23	34016	33305	33574	33305
query24	10881	2850	3003	2850
query25	571	399	396	396
query26	728	148	146	146
query27	2272	284	283	283
query28	6047	2119	2139	2119
query29	904	611	633	611
query30	247	151	158	151
query31	987	763	769	763
query32	97	54	52	52
query33	750	301	293	293
query34	975	483	504	483
query35	678	589	573	573
query36	1131	962	992	962
query37	147	82	80	80
query38	2983	2868	2812	2812
query39	909	830	857	830
query40	209	131	121	121
query41	58	52	58	52
query42	111	103	101	101
query43	625	560	547	547
query44	1229	729	724	724
query45	195	156	159	156
query46	1087	694	737	694
query47	1854	1753	1786	1753
query48	378	292	294	292
query49	831	408	402	402
query50	771	392	391	391
query51	6911	6768	6759	6759
query52	113	91	93	91
query53	358	285	288	285
query54	885	442	435	435
query55	73	74	71	71
query56	291	277	273	273
query57	1135	1048	1080	1048
query58	245	237	247	237
query59	3478	3264	2929	2929
query60	301	271	273	271
query61	99	92	94	92
query62	793	671	644	644
query63	322	296	294	294
query64	9058	2183	1653	1653
query65	3129	3121	3118	3118
query66	757	331	327	327
query67	15659	14904	14900	14900
query68	4516	528	518	518
query69	550	400	394	394
query70	1186	1115	1180	1115
query71	391	281	280	280
query72	7576	5050	5335	5050
query73	749	319	315	315
query74	5891	5534	5475	5475
query75	3345	2659	2650	2650
query76	2150	959	999	959
query77	467	297	306	297
query78	9683	11866	9355	9355
query79	2543	514	512	512
query80	1825	467	487	467
query81	604	215	216	215
query82	285	131	131	131
query83	322	167	163	163
query84	284	91	83	83
query85	706	312	309	309
query86	475	315	323	315
query87	3294	3067	3053	3053
query88	3632	2349	2407	2349
query89	490	376	371	371
query90	1839	187	183	183
query91	134	100	100	100
query92	60	48	48	48
query93	1066	487	495	487
query94	1081	207	209	207
query95	413	316	311	311
query96	578	276	259	259
query97	3236	3034	3036	3034
query98	217	213	194	194
query99	1554	1267	1264	1264
Total cold run time: 277410 ms
Total hot run time: 173323 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c69d162c7fca2a5a5b2b9a8e328bc46a9090c80c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.48	0.49
query6	1.14	0.73	0.73
query7	0.03	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.48	0.48
query10	0.53	0.53	0.54
query11	0.17	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.58	0.59
query14	0.76	0.79	0.78
query15	0.85	0.80	0.82
query16	0.37	0.37	0.36
query17	1.03	1.02	1.06
query18	0.23	0.22	0.22
query19	1.89	1.68	1.75
query20	0.01	0.01	0.02
query21	15.39	0.77	0.67
query22	4.36	7.88	1.66
query23	18.64	1.34	1.32
query24	2.09	0.25	0.21
query25	0.15	0.10	0.09
query26	0.30	0.22	0.21
query27	0.45	0.22	0.23
query28	13.29	1.02	1.02
query29	12.62	3.24	3.29
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.29	0.48	0.47
query33	2.88	2.98	2.91
query34	16.90	4.31	4.35
query35	4.42	4.46	4.38
query36	0.66	0.45	0.46
query37	0.18	0.15	0.15
query38	0.14	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.13
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.1 s
Total hot run time: 30.32 s

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40250 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f2248c34a7d0012c49d2c214d7e1603011972ca, data reload: false

------ Round 1 ----------------------------------
q1	18119	4492	4482	4482
q2	2246	193	196	193
q3	10463	1150	1249	1150
q4	10194	800	727	727
q5	7491	2718	2639	2639
q6	220	135	138	135
q7	978	604	596	596
q8	9242	2079	2105	2079
q9	9077	6534	6530	6530
q10	8871	3721	3700	3700
q11	442	247	245	245
q12	400	226	233	226
q13	17765	2975	3004	2975
q14	270	240	216	216
q15	536	497	479	479
q16	490	370	369	369
q17	985	692	825	692
q18	8135	7460	7591	7460
q19	1692	1529	1476	1476
q20	696	314	323	314
q21	4976	3236	4019	3236
q22	407	344	331	331
Total cold run time: 113695 ms
Total hot run time: 40250 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4353	4245	4451	4245
q2	369	259	275	259
q3	2990	2764	2749	2749
q4	1903	1635	1581	1581
q5	5311	5318	5302	5302
q6	217	129	131	129
q7	2089	1788	1752	1752
q8	3210	3392	3328	3328
q9	8427	8406	8290	8290
q10	3883	3673	3653	3653
q11	581	491	475	475
q12	775	611	616	611
q13	16570	3016	3024	3016
q14	287	276	261	261
q15	515	486	483	483
q16	454	426	412	412
q17	1777	1475	1466	1466
q18	7667	7769	7396	7396
q19	1724	1616	1565	1565
q20	1998	1814	1781	1781
q21	4941	4666	4817	4666
q22	624	520	541	520
Total cold run time: 70665 ms
Total hot run time: 53940 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172931 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f2248c34a7d0012c49d2c214d7e1603011972ca, data reload: false

query1	920	374	364	364
query2	6447	2542	2393	2393
query3	6653	216	225	216
query4	24762	17437	17435	17435
query5	4178	496	485	485
query6	276	189	196	189
query7	4581	295	293	293
query8	321	287	288	287
query9	8557	2410	2406	2406
query10	441	306	283	283
query11	11628	10027	10124	10027
query12	141	88	86	86
query13	1650	373	368	368
query14	9498	6971	7557	6971
query15	236	193	177	177
query16	7807	307	292	292
query17	1561	558	520	520
query18	1959	270	278	270
query19	188	145	146	145
query20	87	83	83	83
query21	211	129	131	129
query22	4495	4264	4013	4013
query23	33862	33234	33097	33097
query24	12262	2970	2855	2855
query25	690	355	359	355
query26	1833	147	145	145
query27	3012	264	268	264
query28	7744	2047	2034	2034
query29	1179	625	630	625
query30	288	146	147	146
query31	953	737	771	737
query32	94	53	53	53
query33	764	294	289	289
query34	1028	478	495	478
query35	657	582	584	582
query36	1088	884	962	884
query37	296	80	81	80
query38	2859	2715	2768	2715
query39	855	812	819	812
query40	282	119	119	119
query41	52	52	51	51
query42	120	94	104	94
query43	605	523	535	523
query44	1193	728	716	716
query45	190	159	156	156
query46	1077	722	714	714
query47	1880	1793	1789	1789
query48	362	297	292	292
query49	1183	416	412	412
query50	780	394	388	388
query51	6866	6892	6765	6765
query52	149	87	96	87
query53	357	293	299	293
query54	1041	440	435	435
query55	78	78	73	73
query56	293	285	273	273
query57	1175	1028	1052	1028
query58	258	269	283	269
query59	3660	3212	3238	3212
query60	306	276	274	274
query61	97	94	94	94
query62	847	643	666	643
query63	327	293	293	293
query64	10482	2163	1625	1625
query65	3203	3104	3120	3104
query66	1386	341	330	330
query67	15542	15036	14922	14922
query68	4547	526	527	526
query69	480	326	330	326
query70	1182	1138	1099	1099
query71	397	281	278	278
query72	7064	5482	5432	5432
query73	747	321	326	321
query74	6152	5545	5534	5534
query75	3371	2688	2674	2674
query76	2700	955	893	893
query77	450	307	296	296
query78	9700	9614	8874	8874
query79	2839	507	533	507
query80	2144	469	527	469
query81	582	219	222	219
query82	756	139	132	132
query83	281	167	175	167
query84	283	85	95	85
query85	2134	308	299	299
query86	504	330	285	285
query87	3321	3152	3131	3131
query88	3830	2372	2380	2372
query89	468	372	380	372
query90	1881	197	190	190
query91	131	105	107	105
query92	75	50	49	49
query93	3985	497	499	497
query94	1265	209	215	209
query95	412	312	315	312
query96	606	266	270	266
query97	3200	3007	3009	3007
query98	221	193	196	193
query99	1592	1260	1255	1255
Total cold run time: 287152 ms
Total hot run time: 172931 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f2248c34a7d0012c49d2c214d7e1603011972ca, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.09	0.07
query5	0.49	0.49	0.49
query6	1.15	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.56	0.55	0.55
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.61	0.60	0.58
query14	0.78	0.77	0.78
query15	0.87	0.82	0.82
query16	0.36	0.37	0.37
query17	1.02	0.96	1.02
query18	0.23	0.22	0.22
query19	1.85	1.83	1.78
query20	0.01	0.01	0.02
query21	15.40	0.78	0.67
query22	4.70	7.27	1.73
query23	18.28	1.43	1.35
query24	2.14	0.23	0.24
query25	0.15	0.08	0.09
query26	0.30	0.21	0.21
query27	0.45	0.24	0.23
query28	13.21	1.01	1.01
query29	12.64	3.32	3.27
query30	0.27	0.06	0.05
query31	2.89	0.39	0.38
query32	3.28	0.48	0.47
query33	2.86	2.87	2.84
query34	16.96	4.34	4.36
query35	4.45	4.46	4.41
query36	0.66	0.48	0.46
query37	0.19	0.15	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.22 s
Total hot run time: 30.56 s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 57f7c86 into apache:master Jul 11, 2024
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Jul 11, 2024
…e#37525)

On 64G machine, warning water mark is 12.8G. when system available
memory reaches the warning water mark, load memtable will flush, this is
not a good idea.

`mem_limit` default 10%, except for the memory used by the Linux kernel
, buffer may be around 5%, so low water mark=min(6.4G, MemTotal * 5%)
looks better.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Jul 12, 2024
…e#37525)

On 64G machine, warning water mark is 12.8G. when system available
memory reaches the warning water mark, load memtable will flush, this is
not a good idea.

`mem_limit` default 10%, except for the memory used by the Linux kernel
, buffer may be around 5%, so low water mark=min(6.4G, MemTotal * 5%)
looks better.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
…e#37525)

On 64G machine, warning water mark is 12.8G. when system available
memory reaches the warning water mark, load memtable will flush, this is
not a good idea.

`mem_limit` default 10%, except for the memory used by the Linux kernel
, buffer may be around 5%, so low water mark=min(6.4G, MemTotal * 5%)
looks better.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
On 64G machine, warning water mark is 12.8G. when system available
memory reaches the warning water mark, load memtable will flush, this is
not a good idea.

`mem_limit` default 10%, except for the memory used by the Linux kernel
, buffer may be around 5%, so low water mark=min(6.4G, MemTotal * 5%)
looks better.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.5-merged dev/3.0.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants