Skip to content

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Feb 20, 2024

Proposed changes

The fix in #31102 can lead to index files loss.
Add a restriction to skip_inverted_index build process to only allow dup_keys and unique_keys with mow enabled to do index compaction.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@qidaye
Copy link
Contributor Author

qidaye commented Feb 20, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8035/21247)
Line Coverage: 29.49% (65512/222158)
Region Coverage: 28.96% (33708/116398)
Branch Coverage: 24.82% (17305/69722)
Coverage Report: http://coverage.selectdb-in.cc/coverage/364fcbd92b5e509843bcded571d75eba06660dbd_364fcbd92b5e509843bcded571d75eba06660dbd/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50018 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 364fcbd92b5e509843bcded571d75eba06660dbd, data reload: false

------ Round 1 ----------------------------------
q1	17677	4357	4348	4348
q2	2048	147	146	146
q3	10437	1868	1859	1859
q4	10353	1234	1301	1234
q5	8495	3947	3961	3947
q6	226	124	123	123
q7	1980	1593	1606	1593
q8	9304	2693	2711	2693
q9	10954	10714	10642	10642
q10	8632	3521	3469	3469
q11	426	244	233	233
q12	462	305	303	303
q13	18365	3985	3993	3985
q14	349	329	315	315
q15	510	459	449	449
q16	701	584	586	584
q17	1108	957	983	957
q18	7246	6828	6962	6828
q19	1673	1570	1528	1528
q20	499	303	295	295
q21	4397	4100	4094	4094
q22	508	401	393	393
Total cold run time: 116350 ms
Total hot run time: 50018 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4315	4312	4317	4312
q2	319	219	226	219
q3	4152	4164	4119	4119
q4	2728	2745	2729	2729
q5	7354	7212	7210	7210
q6	234	118	119	118
q7	3147	2790	2799	2790
q8	4377	4477	4448	4448
q9	17630	17361	17359	17359
q10	4327	4273	4260	4260
q11	736	681	688	681
q12	1029	859	855	855
q13	7200	3788	3806	3788
q14	459	432	426	426
q15	499	457	460	457
q16	772	700	697	697
q17	3767	3893	3660	3660
q18	9287	9280	9075	9075
q19	1712	1694	1658	1658
q20	2382	2196	2086	2086
q21	8456	8446	8723	8446
q22	1066	960	938	938
Total cold run time: 85948 ms
Total hot run time: 80331 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 244655 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 364fcbd92b5e509843bcded571d75eba06660dbd, data reload: false

query1	905	395	374	374
query2	6535	2166	1977	1977
query3	6911	207	205	205
query4	20071	17912	17837	17837
query5	19714	6553	6478	6478
query6	387	208	231	208
query7	4864	302	292	292
query8	271	234	227	227
query9	3120	2685	2631	2631
query10	427	293	314	293
query11	11385	10599	10750	10599
query12	120	71	69	69
query13	5570	644	604	604
query14	18977	13326	13274	13274
query15	389	236	241	236
query16	6391	258	267	258
query17	1627	1461	855	855
query18	2315	397	392	392
query19	205	146	144	144
query20	76	77	75	75
query21	190	92	97	92
query22	5415	5173	5264	5173
query23	32413	31772	31970	31772
query24	6819	6509	6519	6509
query25	521	435	416	416
query26	500	156	157	156
query27	1732	287	293	287
query28	6083	2231	2176	2176
query29	2881	2709	2804	2709
query30	240	158	162	158
query31	933	706	730	706
query32	65	58	58	58
query33	395	240	250	240
query34	845	468	483	468
query35	1126	931	942	931
query36	1502	1606	1722	1606
query37	91	61	60	60
query38	3100	2927	2907	2907
query39	1373	1317	1313	1313
query40	195	96	94	94
query41	35	35	30	30
query42	82	83	78	78
query43	595	597	545	545
query44	1121	716	718	716
query45	242	222	223	222
query46	1241	973	991	973
query47	1824	1672	1708	1672
query48	982	672	661	661
query49	605	384	360	360
query50	853	637	612	612
query51	5526	5461	5433	5433
query52	79	87	85	85
query53	436	307	322	307
query54	2644	2475	2457	2457
query55	82	80	84	80
query56	208	206	197	197
query57	1200	1119	1043	1043
query58	208	202	193	193
query59	3416	3160	3016	3016
query60	220	183	186	183
query61	83	87	83	83
query62	798	547	505	505
query63	472	330	337	330
query64	2480	1450	1430	1430
query65	3636	4032	3566	3566
query66	767	364	363	363
query67	16911	16204	15723	15723
query68	7554	641	686	641
query69	549	346	344	344
query70	1800	1607	1689	1607
query71	399	298	310	298
query72	6460	3410	3409	3409
query73	727	325	320	320
query74	6326	5788	5803	5788
query75	4602	3770	3678	3678
query76	4469	1127	1213	1127
query77	560	242	247	242
query78	31842	53884	55611	53884
query79	15690	614	635	614
query80	5207	385	379	379
query81	552	228	228	228
query82	1946	94	93	93
query83	322	138	136	136
query84	257	68	74	68
query85	1993	283	270	270
query86	482	378	374	374
query87	3295	2997	2967	2967
query88	7309	2343	2303	2303
query89	470	291	274	274
query90	2538	212	200	200
query91	151	124	115	115
query92	57	49	51	49
query93	6732	562	574	562
query94	1842	205	210	205
query95	1125	1066	1052	1052
query96	640	331	330	330
query97	6502	6401	6463	6401
query98	185	170	170	170
query99	3813	805	937	805
Total cold run time: 349821 ms
Total hot run time: 244655 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 364fcbd92b5e509843bcded571d75eba06660dbd, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.24	0.04	0.05
query4	1.83	0.07	0.07
query5	0.53	0.53	0.52
query6	1.26	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.46
query10	0.54	0.54	0.53
query11	0.11	0.09	0.08
query12	0.11	0.09	0.09
query13	0.63	0.61	0.62
query14	0.76	0.80	0.77
query15	0.78	0.76	0.75
query16	0.38	0.38	0.36
query17	0.98	1.01	1.01
query18	0.23	0.25	0.26
query19	1.91	1.84	1.85
query20	0.02	0.01	0.01
query21	15.47	0.54	0.58
query22	1.89	1.60	1.95
query23	17.69	0.98	0.88
query24	5.08	4.08	1.54
query25	1.74	0.12	0.12
query26	0.40	0.14	0.13
query27	0.11	0.10	0.10
query28	4.88	0.71	0.72
query29	12.68	2.37	2.18
query30	0.59	0.58	0.53
query31	2.81	0.38	0.38
query32	3.41	0.48	0.49
query33	3.05	3.07	3.10
query34	15.25	4.81	4.83
query35	4.90	4.85	4.85
query36	1.09	1.01	1.04
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 102.38 s
Total hot run time: 31.26 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 364fcbd92b5e509843bcded571d75eba06660dbd with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@xiaokang xiaokang merged commit ccb592c into apache:branch-2.0 Feb 20, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants