Skip to content

[opt](index compaction)Optimize logic of picking columns for index compaction #42051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qidaye merged 1 commit into from
Oct 18, 2024
Merged

[opt](index compaction)Optimize logic of picking columns for index compaction #42051

qidaye merged 1 commit into from
Oct 18, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Oct 17, 2024

Proposed changes

  1. Rename skip_inverted_index to columns_to_do_index_compaction to make it more clear.
  2. Remove redundant rowid_conversion check in index compaction
  3. Rename function construct_skip_inverted_index to construct_index_compaction_columns
  4. Check index.col_unique_ids empty before using to avoid crash and log tablet schema for debug.
  5. Avoid unnecessary bkd and variant index checks by advancing column type checking before has_inverted_index

@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
4365bf4 
…mpaction

1. Rename `skip_inverted_index` to `columns_to_do_index_compaction` to make it more clear.
2. Remove redundant rowid_conversion check in index compaction
3. Rename function `construct_skip_inverted_index` to `construct_index_compaction_columns`
4. Check `index.col_unique_ids` empty before using to avoid crash and log tablet schema for debug.
5. Avoid unnecessary bkd and variant index checks by advancing column type checking before `has_inverted_index`。
@doris-robot
Copy link

doris-robot commented Oct 17, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Oct 17, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Oct 17, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@doris-robot
Copy link

doris-robot commented Oct 17, 2024

TeamCity be ut coverage result:
Function Coverage: 37.46% (9710/25919)
Line Coverage: 28.72% (80596/280597)
Region Coverage: 28.17% (41713/148075)
Branch Coverage: 24.75% (21205/85688)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4365bf4749ddf2f0b1fa2fe3f5710e260ad426f4_4365bf4749ddf2f0b1fa2fe3f5710e260ad426f4/report/index.html

@doris-robot
Copy link

doris-robot commented Oct 17, 2024

TPC-H: Total hot run time: 41453 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4365bf4749ddf2f0b1fa2fe3f5710e260ad426f4, data reload: false

------ Round 1 ----------------------------------
q1	17565	7930	7291	7291
q2	2040	275	275	275
q3	12386	1076	1226	1076
q4	10568	818	818	818
q5	7777	3059	3016	3016
q6	235	155	149	149
q7	1034	625	603	603
q8	9360	1972	2002	1972
q9	6676	6471	6442	6442
q10	7072	2419	2403	2403
q11	445	250	254	250
q12	403	224	225	224
q13	17776	2975	3009	2975
q14	242	215	217	215
q15	574	516	519	516
q16	646	582	601	582
q17	977	537	563	537
q18	7377	6720	6774	6720
q19	1372	1022	976	976
q20	465	187	187	187
q21	4121	3325	3213	3213
q22	1120	1013	1024	1013
Total cold run time: 110231 ms
Total hot run time: 41453 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7428	7229	7289	7229
q2	353	238	226	226
q3	3131	2987	3066	2987
q4	2054	1806	1796	1796
q5	5711	5760	5854	5760
q6	238	152	155	152
q7	2247	1845	1836	1836
q8	3305	3525	3504	3504
q9	8921	8929	8946	8929
q10	3573	3611	3572	3572
q11	596	495	494	494
q12	816	663	606	606
q13	9198	3169	3220	3169
q14	298	271	274	271
q15	574	527	517	517
q16	699	653	640	640
q17	1877	1624	1596	1596
q18	8428	7785	7640	7640
q19	1711	1453	1391	1391
q20	2141	1863	1856	1856
q21	5388	5436	5245	5245
q22	1176	1099	1063	1063
Total cold run time: 69863 ms
Total hot run time: 60479 ms

@doris-robot
Copy link

doris-robot commented Oct 17, 2024

TPC-DS: Total hot run time: 192331 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4365bf4749ddf2f0b1fa2fe3f5710e260ad426f4, data reload: false

query1	905	410	404	404
query2	6251	2154	1981	1981
query3	8677	190	205	190
query4	34314	23755	23626	23626
query5	3502	481	463	463
query6	274	159	159	159
query7	4184	291	290	290
query8	283	222	246	222
query9	9444	2758	2761	2758
query10	470	276	258	258
query11	18073	15406	15310	15310
query12	161	99	98	98
query13	1567	427	417	417
query14	9867	7575	6752	6752
query15	253	173	189	173
query16	8045	494	481	481
query17	1624	619	582	582
query18	2145	306	320	306
query19	355	188	159	159
query20	123	117	115	115
query21	213	106	106	106
query22	4643	4511	4562	4511
query23	34997	34398	34036	34036
query24	11057	2788	2783	2783
query25	633	411	411	411
query26	1357	161	163	161
query27	2641	284	291	284
query28	7820	2439	2430	2430
query29	890	428	424	424
query30	258	154	149	149
query31	1070	796	819	796
query32	98	64	60	60
query33	758	300	304	300
query34	931	529	532	529
query35	926	769	757	757
query36	1119	957	956	956
query37	160	103	98	98
query38	4018	3933	3861	3861
query39	1474	1419	1414	1414
query40	253	98	99	98
query41	50	46	46	46
query42	127	103	108	103
query43	540	492	504	492
query44	1211	828	810	810
query45	197	164	170	164
query46	1163	725	702	702
query47	1961	1814	1849	1814
query48	429	319	333	319
query49	941	429	446	429
query50	827	410	396	396
query51	7091	6910	6924	6910
query52	96	91	91	91
query53	266	181	182	181
query54	1145	431	432	431
query55	82	76	78	76
query56	296	272	289	272
query57	1312	1179	1191	1179
query58	240	236	225	225
query59	3232	3188	3037	3037
query60	286	262	263	262
query61	105	101	100	100
query62	879	666	675	666
query63	233	192	193	192
query64	4998	625	606	606
query65	3292	3306	3199	3199
query66	1382	309	313	309
query67	16272	15935	16233	15935
query68	4283	584	568	568
query69	494	299	287	287
query70	1195	1137	1135	1135
query71	402	289	271	271
query72	7210	4005	4000	4000
query73	792	363	368	363
query74	9840	8934	8993	8934
query75	3434	2701	2686	2686
query76	2996	980	934	934
query77	586	300	295	295
query78	11818	10363	9659	9659
query79	2249	608	619	608
query80	2515	452	452	452
query81	582	243	243	243
query82	561	139	147	139
query83	312	137	137	137
query84	282	73	70	70
query85	809	305	284	284
query86	491	306	288	288
query87	4515	4298	4328	4298
query88	3610	2219	2190	2190
query89	405	297	294	294
query90	2083	191	189	189
query91	146	99	100	99
query92	74	46	49	46
query93	1690	552	557	552
query94	1113	299	300	299
query95	354	254	252	252
query96	610	285	277	277
query97	3261	3141	3164	3141
query98	217	200	190	190
query99	1575	1325	1297	1297
Total cold run time: 304990 ms
Total hot run time: 192331 ms

@doris-robot
Copy link

doris-robot commented Oct 17, 2024

ClickBench: Total hot run time: 33.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4365bf4749ddf2f0b1fa2fe3f5710e260ad426f4, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.52	0.51	0.52
query6	1.14	0.71	0.73
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.56	0.55
query11	0.15	0.12	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.83	2.74	2.75
query15	0.89	0.83	0.84
query16	0.38	0.40	0.36
query17	1.04	1.02	1.04
query18	0.20	0.20	0.19
query19	1.99	1.83	1.92
query20	0.01	0.01	0.01
query21	15.37	0.58	0.62
query22	2.49	2.55	2.68
query23	16.87	1.12	0.84
query24	3.09	0.60	1.64
query25	0.32	0.07	0.05
query26	0.55	0.14	0.14
query27	0.04	0.04	0.03
query28	10.52	1.10	1.06
query29	12.55	3.29	3.30
query30	0.25	0.06	0.06
query31	2.89	0.38	0.39
query32	3.26	0.46	0.46
query33	3.02	2.97	3.04
query34	16.97	4.49	4.58
query35	4.57	4.52	4.57
query36	0.68	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 106.92 s
Total hot run time: 33.11 s

airborne12
airborne12 approved these changes Oct 17, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2024

PR approved by anyone and no changes requested.

csun5285
csun5285 approved these changes Oct 18, 2024
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 0512983 into apache:master Oct 18, 2024
@qidaye qidaye deleted the opt_pick_index_compaction_columns branch October 18, 2024 11:43
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Oct 22, 2024
@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
9592458 
…mpaction (apache#42051)

1. Rename `skip_inverted_index` to `columns_to_do_index_compaction` to
make it more clear.
2. Remove redundant rowid_conversion check in index compaction
3. Rename function `construct_skip_inverted_index` to
`construct_index_compaction_columns`
4. Check `index.col_unique_ids` empty before using to avoid crash and
log tablet schema for debug.
5. Avoid unnecessary bkd and variant index checks by advancing column
type checking before `has_inverted_index`
@qidaye qidaye mentioned this pull request Oct 22, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Oct 22, 2024
@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
5dcf4b4 
…mpaction(apache#42051)

bp apache#42051
@qidaye qidaye mentioned this pull request Oct 22, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Oct 22, 2024
@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
e163bee 
…mpaction(apache#42051)

bp apache#42051
airborne12 pushed a commit that referenced this pull request Oct 23, 2024
@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
ae5535e 
…mpaction (#42051) (#42285)

## Proposed changes
bp #42051
airborne12 pushed a commit that referenced this pull request Oct 23, 2024
@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
bc84734 
…mpaction(#42051) (#42287)

## Proposed changes

bp #42051
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@airborne12 airborne12 airborne12 approved these changes

+1 more reviewer

@csun5285 csun5285 csun5285 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@qidaye @doris-robot @airborne12 @csun5285 @wm1581066