[fix](index compaction) forbid mor table to do index compaction #31102

qidaye · 2024-02-19T07:54:02Z

Proposed changes

The rowid_conversion correctness check is designed for MOW table.
Since we allow MOR table to create inverted index, we should add limitation for the check.

If it's MOR table, we do not allow it to do index compaction temporarily, before we implement the rowid_conversion correctness check for MOR table.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-19T07:54:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

qidaye · 2024-02-19T07:56:30Z

run buildall

github-actions · 2024-02-19T08:00:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-19T08:15:59Z

TPC-H: Total hot run time: 50209 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f382a4a3304f77ab65c551ba97158af14fbb5cb, data reload: false

------ Round 1 ----------------------------------
q1	17603	4406	4381	4381
q2	2070	152	142	142
q3	10304	1892	1949	1892
q4	10173	1251	1336	1251
q5	8494	3936	3993	3936
q6	237	125	126	125
q7	2080	1624	1614	1614
q8	9305	2751	2752	2751
q9	11052	10667	10526	10526
q10	8649	3574	3537	3537
q11	426	246	252	246
q12	467	299	302	299
q13	18364	3982	4082	3982
q14	358	326	326	326
q15	496	457	451	451
q16	679	596	593	593
q17	1165	975	966	966
q18	7360	6930	6821	6821
q19	1711	1551	1501	1501
q20	547	322	316	316
q21	4480	4154	4163	4154
q22	508	399	403	399
Total cold run time: 116528 ms
Total hot run time: 50209 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4315	4312	4312
q2	317	225	231	225
q3	4187	4144	4111	4111
q4	2766	2765	2768	2765
q5	7348	7254	7183	7183
q6	246	122	121	121
q7	3219	2833	2863	2833
q8	4354	4514	4498	4498
q9	17077	17017	17164	17017
q10	4290	4280	4327	4280
q11	755	684	694	684
q12	1026	859	851	851
q13	6468	3774	3761	3761
q14	445	432	435	432
q15	498	457	460	457
q16	753	689	708	689
q17	3806	3818	3834	3818
q18	8861	8751	8883	8751
q19	1745	1729	1658	1658
q20	2406	2120	2119	2119
q21	8610	8617	8588	8588
q22	983	984	946	946
Total cold run time: 84501 ms
Total hot run time: 80099 ms

doris-robot · 2024-02-19T08:16:53Z

TeamCity be ut coverage result:
Function Coverage: 37.83% (8035/21240)
Line Coverage: 29.49% (65505/222121)
Region Coverage: 28.96% (33706/116380)
Branch Coverage: 24.81% (17300/69720)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3f382a4a3304f77ab65c551ba97158af14fbb5cb_3f382a4a3304f77ab65c551ba97158af14fbb5cb/report/index.html

doris-robot · 2024-02-19T08:30:34Z

TPC-DS: Total hot run time: 240824 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f382a4a3304f77ab65c551ba97158af14fbb5cb, data reload: false

query1	927	389	382	382
query2	6520	2228	1998	1998
query3	6919	205	203	203
query4	19883	17938	18000	17938
query5	19729	6493	6538	6493
query6	271	216	231	216
query7	4162	312	307	307
query8	253	227	234	227
query9	3141	2736	2657	2657
query10	411	280	299	280
query11	11229	10670	10825	10670
query12	115	80	71	71
query13	5571	636	644	636
query14	17751	13716	13518	13518
query15	377	223	250	223
query16	6442	269	263	263
query17	1722	1447	868	868
query18	2326	406	407	406
query19	199	139	145	139
query20	76	77	75	75
query21	185	97	95	95
query22	5264	5086	5082	5082
query23	32766	32055	32020	32020
query24	7145	6518	6517	6517
query25	517	425	407	407
query26	601	174	160	160
query27	2027	296	287	287
query28	6164	2244	2189	2189
query29	2859	2864	2698	2698
query30	234	164	163	163
query31	892	720	729	720
query32	55	56	59	56
query33	389	250	257	250
query34	828	464	498	464
query35	1118	923	876	876
query36	1826	1518	1610	1518
query37	88	59	56	56
query38	3081	2927	2904	2904
query39	1352	1317	1343	1317
query40	217	101	97	97
query41	34	31	31	31
query42	92	79	90	79
query43	675	679	647	647
query44	1138	717	728	717
query45	240	228	226	226
query46	1237	976	970	970
query47	1808	1664	1970	1664
query48	983	690	689	689
query49	608	381	383	381
query50	873	618	611	611
query51	5612	5488	5465	5465
query52	84	86	76	76
query53	446	321	315	315
query54	2672	2507	2481	2481
query55	92	70	88	70
query56	201	209	200	200
query57	1208	1086	1050	1050
query58	217	204	205	204
query59	3624	3230	3350	3230
query60	195	188	215	188
query61	80	86	82	82
query62	803	444	490	444
query63	487	347	343	343
query64	2427	1519	1373	1373
query65	3665	3578	3589	3578
query66	787	375	371	371
query67	16297	15355	15514	15355
query68	8953	661	672	661
query69	565	328	329	328
query70	1798	1664	1614	1614
query71	403	304	315	304
query72	6471	3367	3391	3367
query73	731	332	325	325
query74	6362	5815	5883	5815
query75	5144	3800	3655	3655
query76	5376	1106	1128	1106
query77	908	250	248	248
query78	33059	49246	49382	49246
query79	16231	654	643	643
query80	5370	374	377	374
query81	559	227	233	227
query82	965	96	93	93
query83	415	142	139	139
query84	252	69	67	67
query85	2641	288	273	273
query86	481	372	431	372
query87	3271	3003	3001	3001
query88	7064	2345	2383	2345
query89	460	274	318	274
query90	2500	204	204	204
query91	152	118	122	118
query92	65	51	51	51
query93	6644	567	559	559
query94	2052	210	212	210
query95	1115	1080	1082	1080
query96	656	337	328	328
query97	6402	6367	6419	6367
query98	194	180	167	167
query99	3886	977	993	977
Total cold run time: 353362 ms
Total hot run time: 240824 ms

doris-robot · 2024-02-19T08:36:00Z

ClickBench: Total hot run time: 31.54 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f382a4a3304f77ab65c551ba97158af14fbb5cb, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.05
query4	1.83	0.06	0.07
query5	0.53	0.52	0.52
query6	1.23	0.62	0.61
query7	0.02	0.01	0.02
query8	0.04	0.02	0.02
query9	0.53	0.49	0.48
query10	0.54	0.53	0.54
query11	0.12	0.09	0.09
query12	0.11	0.08	0.09
query13	0.62	0.61	0.62
query14	0.80	0.79	0.80
query15	0.77	0.76	0.76
query16	0.39	0.37	0.37
query17	0.98	1.00	1.03
query18	0.23	0.24	0.25
query19	1.87	1.84	1.80
query20	0.01	0.01	0.01
query21	15.46	0.56	0.56
query22	1.83	2.10	1.55
query23	17.21	0.93	0.84
query24	4.66	1.75	4.52
query25	1.58	0.12	0.11
query26	0.40	0.15	0.14
query27	0.12	0.12	0.12
query28	5.07	0.75	0.70
query29	12.75	2.34	2.37
query30	0.56	0.54	0.55
query31	2.82	0.38	0.38
query32	3.43	0.48	0.50
query33	3.03	3.05	3.08
query34	15.26	4.82	4.81
query35	4.83	4.85	4.84
query36	1.07	1.00	1.01
query37	0.05	0.05	0.05
query38	0.03	0.02	0.02
query39	0.01	0.02	0.01
query40	0.15	0.14	0.14
query41	0.07	0.02	0.01
query42	0.01	0.02	0.02
query43	0.02	0.02	0.01
Total cold run time: 101.37 s
Total hot run time: 31.54 s

doris-robot · 2024-02-19T08:39:11Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3f382a4a3304f77ab65c551ba97158af14fbb5cb with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

qidaye · 2024-02-19T09:00:45Z

run p0

…now.

qidaye · 2024-02-19T10:24:09Z

run buildall

github-actions · 2024-02-19T10:25:04Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-19T10:47:59Z

TPC-H: Total hot run time: 49932 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9439187d74f0c5e43e51c72451887a331e3db7a, data reload: false

------ Round 1 ----------------------------------
q1	17685	4370	4404	4370
q2	2034	161	135	135
q3	10453	1886	1925	1886
q4	10382	1245	1319	1245
q5	8457	3907	3991	3907
q6	229	126	123	123
q7	2040	1593	1608	1593
q8	9313	2726	2734	2726
q9	10705	10505	10426	10426
q10	8629	3543	3516	3516
q11	424	240	240	240
q12	460	299	304	299
q13	18340	3970	4039	3970
q14	347	330	340	330
q15	513	461	459	459
q16	701	583	599	583
q17	1122	954	965	954
q18	7363	6910	6839	6839
q19	1683	1549	1534	1534
q20	541	305	305	305
q21	4507	4107	4124	4107
q22	501	385	406	385
Total cold run time: 116429 ms
Total hot run time: 49932 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4342	4290	4290
q2	315	224	221	221
q3	4229	4178	4184	4178
q4	2752	2742	2753	2742
q5	7283	7232	7229	7229
q6	231	122	116	116
q7	3181	2860	2867	2860
q8	4385	4461	4489	4461
q9	17037	16935	16956	16935
q10	4279	4194	4243	4194
q11	775	695	693	693
q12	1018	856	857	856
q13	6638	3709	3762	3709
q14	456	418	427	418
q15	493	458	444	444
q16	762	720	695	695
q17	3910	3793	3888	3793
q18	8805	8665	8690	8665
q19	1718	1695	1627	1627
q20	2445	2177	2149	2149
q21	8514	8455	8554	8455
q22	1058	1001	957	957
Total cold run time: 84641 ms
Total hot run time: 79687 ms

doris-robot · 2024-02-19T10:52:21Z

TeamCity be ut coverage result:
Function Coverage: 37.84% (8037/21240)
Line Coverage: 29.50% (65518/222120)
Region Coverage: 28.97% (33714/116380)
Branch Coverage: 24.82% (17304/69720)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f9439187d74f0c5e43e51c72451887a331e3db7a_f9439187d74f0c5e43e51c72451887a331e3db7a/report/index.html

doris-robot · 2024-02-19T11:02:40Z

TPC-DS: Total hot run time: 237136 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9439187d74f0c5e43e51c72451887a331e3db7a, data reload: false

query1	899	383	378	378
query2	6533	2280	1953	1953
query3	6909	201	197	197
query4	20163	17901	18093	17901
query5	19702	6478	6472	6472
query6	277	213	224	213
query7	4148	297	285	285
query8	279	245	244	244
query9	3075	2657	2608	2608
query10	394	286	307	286
query11	11315	10652	11666	10652
query12	121	73	74	73
query13	5572	637	629	629
query14	17222	13606	13312	13312
query15	381	237	230	230
query16	6441	265	256	256
query17	1771	1459	844	844
query18	2352	416	407	407
query19	198	140	142	140
query20	77	79	72	72
query21	183	97	93	93
query22	5304	5112	5090	5090
query23	32642	32004	31850	31850
query24	7049	6566	6568	6566
query25	527	424	416	416
query26	529	163	153	153
query27	1892	293	289	289
query28	6062	2174	2131	2131
query29	2936	2841	2589	2589
query30	237	163	159	159
query31	889	708	706	706
query32	63	60	57	57
query33	386	251	247	247
query34	843	479	484	479
query35	1118	885	950	885
query36	1544	1518	1551	1518
query37	91	60	64	60
query38	3078	2936	2877	2877
query39	1358	1343	1317	1317
query40	200	94	92	92
query41	34	36	31	31
query42	91	83	76	76
query43	622	730	572	572
query44	1117	702	712	702
query45	238	228	224	224
query46	1257	973	955	955
query47	1719	1745	1750	1745
query48	973	663	646	646
query49	612	364	357	357
query50	849	633	609	609
query51	5505	5421	5441	5421
query52	87	88	81	81
query53	436	317	319	317
query54	2657	2470	2449	2449
query55	81	80	82	80
query56	206	200	186	186
query57	1208	1224	1052	1052
query58	201	205	206	205
query59	3537	3220	3460	3220
query60	197	202	175	175
query61	81	85	82	82
query62	836	458	472	458
query63	474	333	331	331
query64	2525	1446	1319	1319
query65	3591	3565	3607	3565
query66	770	372	370	370
query67	16061	15374	16619	15374
query68	8478	653	667	653
query69	552	340	338	338
query70	2011	1613	1699	1613
query71	399	312	330	312
query72	6456	3388	3444	3388
query73	739	319	313	313
query74	6312	5840	5881	5840
query75	4620	3715	3684	3684
query76	4741	1146	1210	1146
query77	622	239	247	239
query78	32191	52496	46059	46059
query79	17014	718	648	648
query80	5508	378	386	378
query81	578	232	246	232
query82	2617	97	107	97
query83	816	149	149	149
query84	766	69	78	69
query85	2861	288	277	277
query86	491	405	377	377
query87	3283	3028	3008	3008
query88	7382	2303	2293	2293
query89	503	296	298	296
query90	2550	220	203	203
query91	153	116	130	116
query92	66	54	53	53
query93	8510	620	558	558
query94	3673	237	226	226
query95	1181	1133	1145	1133
query96	681	335	336	335
query97	6609	6591	6604	6591
query98	216	199	198	198
query99	3986	986	988	986
Total cold run time: 357290 ms
Total hot run time: 237136 ms

doris-robot · 2024-02-19T11:08:09Z

ClickBench: Total hot run time: 30.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f9439187d74f0c5e43e51c72451887a331e3db7a, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.04	0.05
query4	1.83	0.07	0.07
query5	0.52	0.52	0.52
query6	1.27	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.47
query10	0.53	0.53	0.53
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.61	0.63	0.60
query14	0.77	0.79	0.80
query15	0.77	0.75	0.76
query16	0.36	0.38	0.36
query17	1.00	0.99	1.03
query18	0.23	0.26	0.25
query19	1.95	1.85	1.87
query20	0.02	0.02	0.01
query21	15.45	0.57	0.57
query22	1.72	1.84	1.74
query23	17.31	1.15	0.94
query24	5.08	0.78	4.29
query25	1.70	0.12	0.11
query26	0.38	0.15	0.14
query27	0.10	0.11	0.11
query28	4.94	0.75	0.78
query29	12.59	2.14	2.29
query30	0.59	0.52	0.52
query31	2.81	0.37	0.39
query32	3.41	0.50	0.49
query33	3.06	3.05	3.12
query34	15.28	4.82	4.83
query35	4.84	4.84	4.86
query36	1.05	1.00	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.14
query41	0.07	0.02	0.01
query42	0.01	0.02	0.02
query43	0.02	0.02	0.01
Total cold run time: 101.67 s
Total hot run time: 30.71 s

doris-robot · 2024-02-19T11:11:21Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f9439187d74f0c5e43e51c72451887a331e3db7a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

xiaokang

LGTM

…now. (apache#31102)

[fix](index compaction) Only check rowid_coversion for MOW table for …

f943918

…now.

qidaye force-pushed the compaction-core branch from 3f382a4 to f943918 Compare February 19, 2024 10:18

qidaye changed the title ~~[fix](index compaction) Only check rowid_coversion for MOW table for …~~ [fix](index compaction) forbid mor table to do index compaction Feb 19, 2024

xiaokang approved these changes Feb 19, 2024

View reviewed changes

xiaokang merged commit 39c1d4c into apache:branch-2.0 Feb 19, 2024

xiaokang added the dev/2.0.5-merged label Feb 19, 2024

qidaye deleted the compaction-core branch February 19, 2024 13:24

qidaye mentioned this pull request Feb 20, 2024

[fix](index compaction) Only do index compaction for dup_keys and unique_keys with mow enabled #31136

Merged

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[fix](index compaction) Only check rowid_coversion for MOW table for …

e1bf11d

…now. (apache#31102)

[fix](index compaction) forbid mor table to do index compaction #31102

[fix](index compaction) forbid mor table to do index compaction #31102

Uh oh!

Conversation

qidaye commented Feb 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

qidaye commented Feb 19, 2024

Uh oh!

github-actions bot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

qidaye commented Feb 19, 2024

Uh oh!

qidaye commented Feb 19, 2024

Uh oh!

github-actions bot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

doris-robot commented Feb 19, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qidaye commented Feb 19, 2024 •

edited

Loading