[fix](index) Fix index for none key column of unique mor table #31035

xiaokang · 2024-02-17T13:26:59Z

Proposed changes

related to #31003

Add back support index for none key column of unique mor table.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

xiaokang · 2024-02-17T13:27:12Z

run buildall

doris-robot · 2024-02-17T13:27:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

xiaokang · 2024-02-17T13:31:09Z

run buildall

github-actions · 2024-02-17T13:33:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-02-17T13:37:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-17T13:55:46Z

TPC-H: Total hot run time: 49922 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6d73b97ae3350727cbb9d6f87a08af72defdb72, data reload: false

------ Round 1 ----------------------------------
q1	17746	4440	4339	4339
q2	2050	154	142	142
q3	10435	1928	1950	1928
q4	10334	1246	1329	1246
q5	8395	4062	3985	3985
q6	232	127	131	127
q7	2040	1618	1579	1579
q8	9539	2722	2715	2715
q9	11538	10499	10380	10380
q10	8656	3545	3525	3525
q11	423	244	236	236
q12	464	303	304	303
q13	18371	3973	4032	3973
q14	354	332	326	326
q15	508	447	452	447
q16	679	599	602	599
q17	1121	963	958	958
q18	7329	6849	6809	6809
q19	1670	1565	1550	1550
q20	537	298	325	298
q21	4473	4126	4067	4067
q22	484	406	390	390
Total cold run time: 117378 ms
Total hot run time: 49922 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4293	4319	4293
q2	316	247	224	224
q3	4160	4171	4143	4143
q4	2748	2742	2741	2741
q5	7236	7251	7224	7224
q6	237	127	122	122
q7	3247	2789	2842	2789
q8	4353	4467	4466	4466
q9	17132	17067	17064	17064
q10	4204	4277	4295	4277
q11	785	704	673	673
q12	1012	832	838	832
q13	6907	3750	3770	3750
q14	443	423	415	415
q15	492	448	451	448
q16	756	713	682	682
q17	3801	3904	3826	3826
q18	8902	8742	8757	8742
q19	1706	1677	1646	1646
q20	2390	2158	2080	2080
q21	8617	8500	8391	8391
q22	1073	981	963	963
Total cold run time: 84837 ms
Total hot run time: 79791 ms

doris-robot · 2024-02-17T13:58:52Z

TeamCity be ut coverage result:
Function Coverage: 37.83% (8035/21240)
Line Coverage: 29.49% (65497/222118)
Region Coverage: 28.96% (33701/116376)
Branch Coverage: 24.81% (17298/69716)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b6d73b97ae3350727cbb9d6f87a08af72defdb72_b6d73b97ae3350727cbb9d6f87a08af72defdb72/report/index.html

doris-robot · 2024-02-17T14:10:16Z

TPC-DS: Total hot run time: 239019 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6d73b97ae3350727cbb9d6f87a08af72defdb72, data reload: false

query1	913	386	388	386
query2	6517	2124	2234	2124
query3	6929	207	206	206
query4	21527	18077	17974	17974
query5	19722	6535	6504	6504
query6	291	218	228	218
query7	4147	311	311	311
query8	273	241	238	238
query9	3157	2697	2634	2634
query10	429	292	292	292
query11	11331	10579	10717	10579
query12	117	69	67	67
query13	5579	649	633	633
query14	17999	13438	13063	13063
query15	372	226	230	226
query16	6397	260	260	260
query17	1906	1550	876	876
query18	2323	406	410	406
query19	201	144	152	144
query20	78	77	75	75
query21	185	94	98	94
query22	5289	5031	5085	5031
query23	32822	32042	32607	32042
query24	7652	6605	6685	6605
query25	600	475	449	449
query26	533	175	170	170
query27	1855	316	316	316
query28	6228	2232	2210	2210
query29	2922	2951	2762	2762
query30	243	157	156	156
query31	927	744	732	732
query32	70	56	61	56
query33	397	230	246	230
query34	857	465	483	465
query35	1138	948	968	948
query36	1608	1483	1752	1483
query37	86	62	60	60
query38	3137	2944	2875	2875
query39	1392	1325	1317	1317
query40	208	94	96	94
query41	35	32	32	32
query42	84	84	87	84
query43	644	574	559	559
query44	1141	706	702	702
query45	241	236	231	231
query46	1259	977	973	973
query47	1827	1852	2060	1852
query48	989	702	676	676
query49	616	386	380	380
query50	863	617	615	615
query51	5531	5436	5463	5436
query52	88	84	74	74
query53	438	312	322	312
query54	2611	2471	2479	2471
query55	95	82	86	82
query56	208	207	195	195
query57	1210	1120	1119	1119
query58	212	205	205	205
query59	3504	3193	3230	3193
query60	206	188	218	188
query61	87	82	81	81
query62	818	533	482	482
query63	475	331	343	331
query64	2563	1505	1474	1474
query65	3620	3623	3571	3571
query66	795	374	382	374
query67	15739	15478	15700	15478
query68	8405	690	675	675
query69	576	354	349	349
query70	1948	1759	1730	1730
query71	413	305	309	305
query72	6398	3394	3438	3394
query73	738	327	316	316
query74	6304	5855	5873	5855
query75	4561	3766	3747	3747
query76	4752	1178	1211	1178
query77	568	251	258	251
query78	32218	51534	47041	47041
query79	14672	652	653	652
query80	4752	397	382	382
query81	570	227	231	227
query82	670	99	94	94
query83	330	140	138	138
query84	253	68	69	68
query85	2494	285	279	279
query86	454	349	361	349
query87	3281	3013	3002	3002
query88	6511	2352	2348	2348
query89	385	280	285	280
query90	2498	213	211	211
query91	163	120	115	115
query92	65	52	50	50
query93	4433	567	555	555
query94	1769	207	205	205
query95	1112	1052	1076	1052
query96	653	337	326	326
query97	6520	6426	6322	6322
query98	201	172	161	161
query99	3880	906	881	881
Total cold run time: 346803 ms
Total hot run time: 239019 ms

doris-robot · 2024-02-17T14:15:44Z

ClickBench: Total hot run time: 30.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6d73b97ae3350727cbb9d6f87a08af72defdb72, data reload: false

query1	0.03	0.02	0.02
query2	0.05	0.02	0.02
query3	0.24	0.05	0.04
query4	1.84	0.06	0.07
query5	0.52	0.52	0.51
query6	1.32	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.48
query10	0.54	0.53	0.54
query11	0.11	0.09	0.08
query12	0.11	0.09	0.09
query13	0.61	0.61	0.62
query14	0.78	0.79	0.78
query15	0.79	0.76	0.75
query16	0.35	0.39	0.36
query17	1.03	1.01	1.05
query18	0.20	0.27	0.27
query19	1.91	1.81	1.84
query20	0.01	0.01	0.01
query21	15.50	0.57	0.57
query22	2.15	2.18	1.69
query23	17.52	1.05	0.94
query24	6.31	0.85	1.30
query25	1.59	0.12	0.10
query26	0.32	0.14	0.13
query27	0.10	0.11	0.12
query28	6.60	0.72	0.74
query29	12.67	2.28	2.26
query30	0.57	0.55	0.50
query31	2.82	0.38	0.38
query32	3.40	0.50	0.50
query33	3.07	3.02	3.06
query34	15.26	4.82	4.78
query35	4.83	4.85	4.84
query36	1.04	1.03	1.03
query37	0.05	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.14
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.01
Total cold run time: 105.12 s
Total hot run time: 30.73 s

doris-robot · 2024-02-17T14:18:55Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b6d73b97ae3350727cbb9d6f87a08af72defdb72 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

…e#31035)

* [fix](index) Fix index for none key column of unique mor table (#31035) * disable INVERTED index with parser on value columns of MOR unique table * add debug log for test_build_index * add debug log * only do index compaction for dup and mow

…#31051) * [fix](index) Fix index for none key column of unique mor table (apache#31035) * disable INVERTED index with parser on value columns of MOR unique table * add debug log for test_build_index * add debug log * only do index compaction for dup and mow

…e#31035)

xiaokang added 4 commits February 17, 2024 21:21

allow index for non key columns of mor unique table

2e0a3fd

add mor index testcase

dc8e074

add back mor bitmap index testcase

ab06f41

change index error message

660dbdb

github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Feb 17, 2024

clang format

b6d73b9

xiaokang changed the title ~~(fix)[index] Fix index for none key column of unique mor table~~ [fix](index) Fix index for none key column of unique mor table Feb 17, 2024

xiaokang merged commit e069e43 into apache:branch-2.0 Feb 17, 2024

xiaokang added a commit to xiaokang/doris that referenced this pull request Feb 18, 2024

[fix](index) Fix index for none key column of unique mor table (apach…

52b3f3f

…e#31035)

xiaokang mentioned this pull request Feb 18, 2024

[fix](inverted index) disallow inverted index with parser for value columns of MOR unique table #31054

Merged

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[fix](index) Fix index for none key column of unique mor table (apach…

fc77603

…e#31035)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](index) Fix index for none key column of unique mor table #31035

[fix](index) Fix index for none key column of unique mor table #31035

Uh oh!

xiaokang commented Feb 17, 2024 •

edited

Loading

Uh oh!

xiaokang commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

xiaokang commented Feb 17, 2024

Uh oh!

github-actions bot commented Feb 17, 2024

Uh oh!

github-actions bot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[fix](index) Fix index for none key column of unique mor table #31035

[fix](index) Fix index for none key column of unique mor table #31035

Uh oh!

Conversation

xiaokang commented Feb 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

xiaokang commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

xiaokang commented Feb 17, 2024

Uh oh!

github-actions bot commented Feb 17, 2024

Uh oh!

github-actions bot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

doris-robot commented Feb 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xiaokang commented Feb 17, 2024 •

edited

Loading