Skip to content

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Jun 20, 2024

Proposed changes

#35357

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49897 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 271a166e571120acaff73d61c75c81fd04573667, data reload: false

------ Round 1 ----------------------------------
q1	17669	4354	4325	4325
q2	2090	153	144	144
q3	10452	1922	1949	1922
q4	10335	1223	1323	1223
q5	8482	3933	3918	3918
q6	235	152	129	129
q7	2052	1617	1602	1602
q8	9554	2717	2698	2698
q9	13939	10483	10586	10483
q10	8603	3464	3468	3464
q11	408	251	243	243
q12	471	311	305	305
q13	18338	3983	4009	3983
q14	372	334	320	320
q15	508	460	457	457
q16	671	571	572	571
q17	1130	964	918	918
q18	7180	6839	6931	6839
q19	1763	1597	1569	1569
q20	554	294	315	294
q21	4385	4110	4061	4061
q22	522	429	430	429
Total cold run time: 119713 ms
Total hot run time: 49897 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4300	4273	4276	4273
q2	317	224	225	224
q3	4172	4116	4136	4116
q4	2740	2724	2718	2718
q5	7175	7059	7094	7059
q6	239	117	119	117
q7	3238	2776	2874	2776
q8	4357	4420	4451	4420
q9	17367	17113	17167	17113
q10	4261	4216	4216	4216
q11	750	717	693	693
q12	1036	837	870	837
q13	7015	3675	3695	3675
q14	447	432	426	426
q15	496	470	450	450
q16	720	677	678	677
q17	3733	3780	3829	3780
q18	8780	8745	8766	8745
q19	1724	1661	1698	1661
q20	2372	2079	2112	2079
q21	8382	8336	8401	8336
q22	1029	968	957	957
Total cold run time: 84650 ms
Total hot run time: 79348 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.79% (8094/21416)
Line Coverage: 29.44% (66143/224669)
Region Coverage: 28.91% (34090/117906)
Branch Coverage: 24.79% (17500/70596)
Coverage Report: http://coverage.selectdb-in.cc/coverage/271a166e571120acaff73d61c75c81fd04573667_271a166e571120acaff73d61c75c81fd04573667/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 203329 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 271a166e571120acaff73d61c75c81fd04573667, data reload: false

query1	940	424	375	375
query2	6546	2767	2684	2684
query3	6914	205	199	199
query4	20880	17914	18034	17914
query5	19730	6496	6496	6496
query6	284	220	223	220
query7	4157	299	306	299
query8	390	390	398	390
query9	3138	2683	2619	2619
query10	411	300	413	300
query11	11278	10680	10595	10595
query12	125	80	77	77
query13	5597	709	691	691
query14	17551	13437	13043	13043
query15	355	242	229	229
query16	6471	294	268	268
query17	1684	1461	868	868
query18	2323	411	411	411
query19	213	149	151	149
query20	79	77	81	77
query21	199	96	96	96
query22	5202	4998	4914	4914
query23	32567	31889	31973	31889
query24	7728	6688	6508	6508
query25	531	430	430	430
query26	895	159	161	159
query27	2221	298	296	296
query28	6188	2387	2350	2350
query29	2891	2737	2759	2737
query30	240	164	173	164
query31	908	740	734	734
query32	70	65	62	62
query33	415	262	257	257
query34	851	478	486	478
query35	1130	933	972	933
query36	1206	1188	1211	1188
query37	90	57	60	57
query38	3099	2933	2947	2933
query39	1361	1329	1339	1329
query40	299	96	92	92
query41	47	44	41	41
query42	79	81	82	81
query43	658	714	660	660
query44	1116	715	715	715
query45	252	241	234	234
query46	1240	967	972	967
query47	1885	1672	1601	1601
query48	1006	728	706	706
query49	654	367	369	367
query50	855	593	619	593
query51	4687	4721	4677	4677
query52	96	78	85	78
query53	447	334	324	324
query54	2667	2478	2457	2457
query55	100	81	87	81
query56	232	220	213	213
query57	1156	1098	1061	1061
query58	211	213	184	184
query59	4038	3951	3960	3951
query60	217	195	216	195
query61	98	96	98	96
query62	766	553	425	425
query63	488	338	339	338
query64	2828	1583	1347	1347
query65	3617	3554	3550	3550
query66	804	380	389	380
query67	15581	15462	17226	15462
query68	8325	671	667	667
query69	573	370	348	348
query70	1661	1474	1507	1474
query71	396	308	327	308
query72	6192	3439	3496	3439
query73	738	331	320	320
query74	6226	5813	5817	5813
query75	4749	3755	3654	3654
query76	4746	1164	1224	1164
query77	666	257	273	257
query78	13043	11794	16444	11794
query79	7130	631	626	626
query80	1295	403	420	403
query81	496	231	236	231
query82	479	102	94	94
query83	167	144	131	131
query84	264	77	71	71
query85	911	324	326	324
query86	329	304	287	287
query87	3247	3017	3083	3017
query88	4578	2343	2358	2343
query89	352	318	285	285
query90	1869	204	212	204
query91	169	149	136	136
query92	62	54	50	50
query93	1631	565	560	560
query94	728	208	217	208
query95	1125	1064	1066	1064
query96	638	338	320	320
query97	6621	6401	6421	6401
query98	191	177	179	177
query99	2910	865	868	865
Total cold run time: 305489 ms
Total hot run time: 203329 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 271a166e571120acaff73d61c75c81fd04573667, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.06	0.06
query5	0.54	0.52	0.52
query6	1.30	0.62	0.62
query7	0.01	0.00	0.01
query8	0.03	0.02	0.02
query9	0.52	0.50	0.50
query10	0.53	0.54	0.51
query11	0.13	0.08	0.08
query12	0.12	0.09	0.08
query13	0.61	0.62	0.60
query14	0.79	0.78	0.78
query15	0.77	0.76	0.77
query16	0.36	0.38	0.36
query17	1.03	1.00	0.99
query18	0.22	0.25	0.25
query19	1.92	1.75	1.79
query20	0.02	0.01	0.01
query21	15.48	0.56	0.56
query22	2.16	2.73	1.58
query23	17.04	1.11	1.02
query24	6.27	0.74	1.97
query25	0.36	0.10	0.06
query26	0.73	0.17	0.17
query27	0.04	0.04	0.04
query28	6.52	0.73	0.75
query29	12.66	2.28	2.32
query30	0.61	0.54	0.54
query31	2.81	0.39	0.38
query32	3.39	0.50	0.49
query33	3.11	3.09	3.03
query34	15.24	4.82	4.80
query35	4.87	4.87	4.85
query36	1.09	1.00	1.00
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.81 s
Total hot run time: 30.47 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 271a166e571120acaff73d61c75c81fd04573667 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@xiaokang xiaokang changed the title [opt](inverted index) reduce generation of the rowid_result if not ne… [opt](inverted index) reduce generation of the rowid_result if not necessary Jun 20, 2024
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xiaokang xiaokang marked this pull request as draft June 20, 2024 09:49
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@xiaokang xiaokang marked this pull request as ready for review July 6, 2024 13:31
@xiaokang
Copy link
Contributor

xiaokang commented Jul 6, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.90% (8116/21417)
Line Coverage: 29.57% (66499/224911)
Region Coverage: 29.04% (34276/118047)
Branch Coverage: 24.91% (17605/70688)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a66e5a79cfbbcd9d1755fc4cbc6c8ba12a1db3be_a66e5a79cfbbcd9d1755fc4cbc6c8ba12a1db3be/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 49454 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a66e5a79cfbbcd9d1755fc4cbc6c8ba12a1db3be, data reload: false

------ Round 1 ----------------------------------
q1	18133	4363	4328	4328
q2	2070	153	149	149
q3	10435	1843	1932	1843
q4	10322	1219	1293	1219
q5	9008	3807	3883	3807
q6	265	141	128	128
q7	2021	1627	1589	1589
q8	9340	2719	2692	2692
q9	10511	10386	10207	10207
q10	8624	3505	3508	3505
q11	426	237	251	237
q12	467	304	303	303
q13	18335	3962	4029	3962
q14	354	327	339	327
q15	504	464	460	460
q16	689	573	574	573
q17	1136	954	936	936
q18	7319	6808	6882	6808
q19	1783	1639	1591	1591
q20	548	309	300	300
q21	4460	4103	4054	4054
q22	554	439	436	436
Total cold run time: 117304 ms
Total hot run time: 49454 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4297	4291	4291
q2	320	230	219	219
q3	4173	4119	4171	4119
q4	2786	2758	2751	2751
q5	7171	7119	7200	7119
q6	236	120	123	120
q7	3260	2861	2900	2861
q8	4419	4553	4560	4553
q9	16804	16906	16678	16678
q10	4264	4290	4344	4290
q11	800	680	718	680
q12	1086	916	922	916
q13	9467	3847	3879	3847
q14	485	456	456	456
q15	510	473	472	472
q16	778	737	729	729
q17	3913	3973	3857	3857
q18	9028	8873	8942	8873
q19	1745	1716	1638	1638
q20	2398	2146	2116	2116
q21	8623	8484	8442	8442
q22	1035	942	1000	942
Total cold run time: 87665 ms
Total hot run time: 79969 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203586 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a66e5a79cfbbcd9d1755fc4cbc6c8ba12a1db3be, data reload: false

query1	929	425	373	373
query2	6551	2809	2410	2410
query3	6919	204	201	201
query4	21223	18073	17960	17960
query5	19729	6541	6503	6503
query6	409	214	222	214
query7	5116	302	313	302
query8	429	417	435	417
query9	3107	2654	2615	2615
query10	433	317	293	293
query11	11344	10708	10635	10635
query12	126	74	75	74
query13	5610	720	707	707
query14	19296	13404	14094	13404
query15	370	242	251	242
query16	6444	285	265	265
query17	1394	1447	885	885
query18	2266	415	413	413
query19	209	155	146	146
query20	83	78	80	78
query21	189	94	102	94
query22	5026	4961	4984	4961
query23	32395	31947	31916	31916
query24	6864	6548	6572	6548
query25	525	448	431	431
query26	508	163	159	159
query27	1723	294	290	290
query28	6159	2353	2323	2323
query29	2919	2739	2641	2641
query30	241	165	172	165
query31	919	720	772	720
query32	69	64	64	64
query33	400	244	264	244
query34	843	473	479	473
query35	1113	912	907	907
query36	1191	1265	1099	1099
query37	87	61	62	61
query38	3052	2901	2943	2901
query39	1380	1320	1320	1320
query40	201	100	95	95
query41	47	45	44	44
query42	78	80	81	80
query43	654	744	735	735
query44	1116	704	721	704
query45	248	241	237	237
query46	1229	953	961	953
query47	1915	1721	1986	1721
query48	1005	724	729	724
query49	621	383	370	370
query50	844	552	608	552
query51	4744	4619	4684	4619
query52	95	79	96	79
query53	441	324	318	318
query54	2656	2433	2449	2433
query55	93	82	84	82
query56	234	237	233	233
query57	1170	1063	1185	1063
query58	221	192	217	192
query59	4231	4111	4194	4111
query60	218	204	205	204
query61	97	95	95	95
query62	817	494	509	494
query63	484	353	340	340
query64	2528	1575	1413	1413
query65	3643	3525	3518	3518
query66	777	380	378	378
query67	17008	15433	15503	15433
query68	9198	654	651	651
query69	575	347	347	347
query70	1828	1466	1390	1390
query71	428	315	315	315
query72	6527	3461	3477	3461
query73	731	323	317	317
query74	6271	5843	5861	5843
query75	5493	3709	3731	3709
query76	5647	1113	1140	1113
query77	980	269	260	260
query78	12556	11883	12539	11883
query79	8867	655	628	628
query80	1090	401	405	401
query81	484	235	233	233
query82	1001	98	99	98
query83	170	133	127	127
query84	259	74	70	70
query85	883	335	341	335
query86	330	287	298	287
query87	3279	3006	3042	3006
query88	4513	2300	2292	2292
query89	419	281	314	281
query90	1945	219	221	219
query91	171	138	149	138
query92	61	55	51	51
query93	4617	546	555	546
query94	685	212	205	205
query95	1106	1076	1068	1068
query96	652	326	325	325
query97	6527	6285	6386	6285
query98	183	161	169	161
query99	3121	868	867	867
Total cold run time: 315602 ms
Total hot run time: 203586 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a66e5a79cfbbcd9d1755fc4cbc6c8ba12a1db3be, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.78	0.07	0.06
query5	0.53	0.52	0.52
query6	1.26	0.61	0.66
query7	0.02	0.01	0.02
query8	0.03	0.02	0.03
query9	0.53	0.49	0.49
query10	0.53	0.54	0.54
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.60	0.60
query14	0.77	0.79	0.79
query15	0.77	0.76	0.76
query16	0.38	0.36	0.39
query17	1.02	1.03	0.99
query18	0.22	0.27	0.26
query19	1.96	1.87	1.86
query20	0.02	0.01	0.01
query21	15.45	0.54	0.53
query22	1.91	2.38	1.32
query23	16.98	0.96	0.85
query24	7.14	2.08	0.64
query25	0.34	0.12	0.06
query26	0.82	0.14	0.15
query27	0.04	0.04	0.04
query28	5.41	0.75	0.76
query29	12.64	2.34	2.26
query30	0.62	0.54	0.50
query31	2.81	0.38	0.37
query32	3.39	0.50	0.50
query33	3.05	3.06	3.03
query34	15.29	4.80	4.75
query35	4.85	4.86	4.87
query36	1.05	1.02	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.01	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.19 s
Total hot run time: 30.01 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a66e5a79cfbbcd9d1755fc4cbc6c8ba12a1db3be with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@xiaokang xiaokang merged commit a420cf6 into apache:branch-2.0 Jul 7, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants