Skip to content

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Jul 16, 2024

pick #37760

MoW will update delete bitmap during load, and the page cache could be modified by segcompaction. Disable page cache touchs when doing segcompaction could solve this problem.

Proposed changes

Issue Number: close #xxx

…(pick apache#37760)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@freemandealer
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49696 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0badca9e1448bcf80c5dcb411c28f370138bf9a, data reload: false

------ Round 1 ----------------------------------
q1	17765	4437	4320	4320
q2	2090	159	146	146
q3	10510	1901	1929	1901
q4	10319	1231	1327	1231
q5	8490	3897	3864	3864
q6	230	149	127	127
q7	2031	1620	1620	1620
q8	9484	2738	2690	2690
q9	13885	10243	10283	10243
q10	8661	3518	3519	3518
q11	420	252	251	251
q12	468	297	309	297
q13	18351	3982	4072	3982
q14	358	319	318	318
q15	500	454	460	454
q16	664	571	578	571
q17	1109	916	926	916
q18	7271	6783	6872	6783
q19	1783	1641	1640	1640
q20	531	284	313	284
q21	4387	4105	4099	4099
q22	532	441	444	441
Total cold run time: 119839 ms
Total hot run time: 49696 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4280	4323	4280
q2	323	220	230	220
q3	4155	4166	4160	4160
q4	2749	2751	2722	2722
q5	7167	7128	7103	7103
q6	238	119	122	119
q7	3244	2924	2806	2806
q8	4335	4458	4473	4458
q9	16902	16650	16850	16650
q10	4225	4234	4291	4234
q11	763	675	685	675
q12	1045	856	849	849
q13	6462	3780	3699	3699
q14	454	430	434	430
q15	496	456	473	456
q16	725	684	683	683
q17	3867	3861	3873	3861
q18	8755	8756	8833	8756
q19	1717	1637	1674	1637
q20	2356	2146	2101	2101
q21	8417	8427	8442	8427
q22	1069	1008	996	996
Total cold run time: 83821 ms
Total hot run time: 79322 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8115/21466)
Line Coverage: 29.47% (66510/225671)
Region Coverage: 28.94% (34275/118428)
Branch Coverage: 24.83% (17612/70940)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b0badca9e1448bcf80c5dcb411c28f370138bf9a_b0badca9e1448bcf80c5dcb411c28f370138bf9a/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 204087 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0badca9e1448bcf80c5dcb411c28f370138bf9a, data reload: false

query1	916	432	377	377
query2	6531	2720	2665	2665
query3	6924	211	206	206
query4	20172	18032	18054	18032
query5	19762	6452	6533	6452
query6	296	218	242	218
query7	4165	304	298	298
query8	448	434	420	420
query9	3124	2650	2584	2584
query10	406	314	298	298
query11	11274	10815	10644	10644
query12	130	72	76	72
query13	5606	703	690	690
query14	17652	13831	13620	13620
query15	365	245	240	240
query16	6471	297	275	275
query17	1672	1471	882	882
query18	2317	413	417	413
query19	205	154	156	154
query20	84	75	82	75
query21	192	98	96	96
query22	5241	5028	5061	5028
query23	32625	31906	31871	31871
query24	7692	6579	6496	6496
query25	531	424	434	424
query26	709	164	163	163
query27	2142	293	303	293
query28	6154	2334	2290	2290
query29	2908	2624	2652	2624
query30	237	168	173	168
query31	906	746	749	746
query32	66	64	55	55
query33	404	264	266	264
query34	848	479	493	479
query35	1148	960	890	890
query36	1410	1105	1152	1105
query37	92	62	64	62
query38	3047	2919	2969	2919
query39	1363	1338	1322	1322
query40	308	92	93	92
query41	48	45	44	44
query42	84	85	82	82
query43	768	686	694	686
query44	1151	713	719	713
query45	248	240	235	235
query46	1238	957	986	957
query47	1780	1942	1848	1848
query48	1042	736	734	734
query49	645	384	378	378
query50	878	652	643	643
query51	4746	4670	4594	4594
query52	102	80	85	80
query53	458	329	326	326
query54	2642	2486	2455	2455
query55	90	89	85	85
query56	232	227	189	189
query57	1327	1049	1048	1048
query58	219	203	203	203
query59	4110	3988	4001	3988
query60	209	214	217	214
query61	98	93	95	93
query62	871	491	470	470
query63	490	355	346	346
query64	2799	1583	1489	1489
query65	3635	3574	3580	3574
query66	812	386	382	382
query67	16038	15518	15490	15490
query68	11468	644	661	644
query69	582	368	353	353
query70	2041	1332	1717	1332
query71	412	312	316	312
query72	6425	3503	3504	3503
query73	1605	326	319	319
query74	6342	5880	5835	5835
query75	5422	3661	3773	3661
query76	6967	1180	1166	1166
query77	1188	270	262	262
query78	12608	11847	11507	11507
query79	9671	647	658	647
query80	847	401	405	401
query81	458	239	234	234
query82	823	111	102	102
query83	160	131	136	131
query84	254	72	73	72
query85	868	328	332	328
query86	338	297	308	297
query87	3201	3026	3025	3025
query88	4247	2326	2307	2307
query89	487	302	290	290
query90	1935	216	206	206
query91	186	139	147	139
query92	63	57	56	56
query93	6917	594	600	594
query94	694	212	210	210
query95	1094	1067	1062	1062
query96	651	316	332	316
query97	6532	6373	6422	6373
query98	192	184	189	184
query99	2996	871	888	871
Total cold run time: 320977 ms
Total hot run time: 204087 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0badca9e1448bcf80c5dcb411c28f370138bf9a, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.04
query4	1.80	0.06	0.07
query5	0.53	0.51	0.51
query6	1.23	0.63	0.63
query7	0.01	0.01	0.02
query8	0.04	0.02	0.02
query9	0.52	0.49	0.48
query10	0.53	0.53	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.61	0.62	0.61
query14	0.79	0.78	0.78
query15	0.79	0.76	0.76
query16	0.38	0.38	0.37
query17	1.04	1.03	1.01
query18	0.23	0.28	0.25
query19	1.90	1.90	1.84
query20	0.01	0.01	0.01
query21	15.49	0.54	0.55
query22	2.42	2.74	1.54
query23	17.40	1.02	1.01
query24	5.49	1.31	1.53
query25	0.38	0.07	0.06
query26	0.69	0.17	0.16
query27	0.05	0.04	0.04
query28	6.82	0.77	0.74
query29	12.66	2.36	2.23
query30	0.52	0.50	0.54
query31	2.79	0.38	0.38
query32	3.40	0.51	0.50
query33	3.07	3.05	3.06
query34	15.26	4.81	4.80
query35	4.87	4.83	4.82
query36	1.03	1.01	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.15	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.68 s
Total hot run time: 31.07 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b0badca9e1448bcf80c5dcb411c28f370138bf9a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@dataroaring dataroaring merged commit c2a950b into apache:branch-2.0 Jul 18, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…(pick apache#37760) (apache#37938)

MoW will update delete bitmap during load, and the page cache could be
modified by segcompaction. Disable page cache touchs when doing
segcompaction could solve this problem.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants