Skip to content

[fix] (compaction) fix time series compaction policy #39170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Aug 12, 2024
Merged

[fix] (compaction) fix time series compaction policy #39170

airborne12 merged 1 commit into from
Aug 12, 2024

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Aug 9, 2024 •
edited
Loading

Proposed changes

Fix the issue introduced by #38220

BUG

The permits requested for compaction cannot be 0, 0 indicates failure.

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

csun5285 commented Aug 9, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

TPC-H: Total hot run time: 39928 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06a84da44a1b8149248eb13f04f020104c2d4418, data reload: false

------ Round 1 ----------------------------------
q1	17615	4389	4276	4276
q2	2026	193	173	173
q3	11043	1281	1147	1147
q4	12265	684	719	684
q5	7886	2544	2540	2540
q6	232	142	144	142
q7	994	620	623	620
q8	9480	1954	1964	1954
q9	9055	6680	6589	6589
q10	7076	2141	2146	2141
q11	502	256	261	256
q12	387	228	226	226
q13	17766	2988	3021	2988
q14	297	243	241	241
q15	522	496	503	496
q16	528	383	391	383
q17	969	704	706	704
q18	7997	7428	7415	7415
q19	1604	1134	1059	1059
q20	686	323	338	323
q21	5986	4554	4639	4554
q22	1138	1058	1017	1017
Total cold run time: 116054 ms
Total hot run time: 39928 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4699	4288	4288
q2	377	278	266	266
q3	2852	2713	2730	2713
q4	2053	1733	1772	1733
q5	5543	5495	5427	5427
q6	228	136	136	136
q7	2178	1763	1775	1763
q8	3283	3434	3457	3434
q9	8972	8800	8590	8590
q10	3511	3287	3265	3265
q11	612	499	517	499
q12	809	633	648	633
q13	16243	3146	3179	3146
q14	327	286	300	286
q15	523	505	496	496
q16	480	454	471	454
q17	1813	1525	1531	1525
q18	8327	7999	7872	7872
q19	5378	1618	1604	1604
q20	2159	1902	1882	1882
q21	5405	5269	5370	5269
q22	1137	1052	1033	1033
Total cold run time: 76599 ms
Total hot run time: 56314 ms

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

TPC-DS: Total hot run time: 202216 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06a84da44a1b8149248eb13f04f020104c2d4418, data reload: false

query1	898	362	362	362
query2	6278	1905	1804	1804
query3	6634	204	221	204
query4	33843	23323	23062	23062
query5	3741	507	485	485
query6	270	167	166	166
query7	4591	291	291	291
query8	224	193	193	193
query9	8536	2483	2468	2468
query10	559	439	490	439
query11	15555	15062	15084	15062
query12	130	98	101	98
query13	1654	384	372	372
query14	9524	7695	8245	7695
query15	260	228	237	228
query16	7006	490	501	490
query17	1648	585	559	559
query18	1154	290	292	290
query19	201	152	146	146
query20	118	108	107	107
query21	209	103	102	102
query22	4659	4592	4376	4376
query23	34667	33806	33928	33806
query24	11849	2843	2622	2622
query25	658	403	406	403
query26	1623	159	159	159
query27	2438	290	286	286
query28	7053	2057	2039	2039
query29	964	429	438	429
query30	260	155	151	151
query31	951	749	747	747
query32	100	55	55	55
query33	720	293	280	280
query34	843	463	478	463
query35	951	814	855	814
query36	1069	925	910	910
query37	146	81	87	81
query38	4288	4126	4076	4076
query39	1414	1366	1417	1366
query40	251	118	118	118
query41	49	45	47	45
query42	116	96	95	95
query43	490	464	480	464
query44	1191	735	739	735
query45	237	218	207	207
query46	1103	758	757	757
query47	1851	1762	1803	1762
query48	386	302	302	302
query49	901	430	442	430
query50	815	416	414	414
query51	6941	6708	6595	6595
query52	107	90	90	90
query53	257	193	182	182
query54	1012	462	463	462
query55	77	179	72	72
query56	257	236	244	236
query57	1158	1053	1037	1037
query58	234	223	250	223
query59	2946	2687	2882	2687
query60	277	276	267	267
query61	96	96	117	96
query62	787	657	652	652
query63	205	178	177	177
query64	10411	2275	1743	1743
query65	3466	3160	3146	3146
query66	1100	338	334	334
query67	15486	14735	14737	14735
query68	8714	555	562	555
query69	481	414	418	414
query70	1398	1154	1124	1124
query71	586	271	261	261
query72	20190	16787	16568	16568
query73	2230	332	328	328
query74	9089	8809	8927	8809
query75	5262	2654	2712	2654
query76	5488	1074	1038	1038
query77	822	315	313	313
query78	9859	8952	8835	8835
query79	10338	519	522	519
query80	1321	533	489	489
query81	597	232	218	218
query82	394	135	129	129
query83	327	144	144	144
query84	269	79	75	75
query85	1018	280	277	277
query86	338	294	306	294
query87	4819	4478	4514	4478
query88	4476	2486	2496	2486
query89	499	280	283	280
query90	2173	228	190	190
query91	119	93	96	93
query92	61	47	48	47
query93	7139	541	539	539
query94	1129	293	273	273
query95	358	260	264	260
query96	624	277	281	277
query97	3171	3054	3067	3054
query98	215	199	198	198
query99	1517	1248	1249	1248
Total cold run time: 334914 ms
Total hot run time: 202216 ms

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06a84da44a1b8149248eb13f04f020104c2d4418, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.67	0.07	0.08
query5	0.50	0.48	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.48
query10	0.54	0.53	0.53
query11	0.16	0.12	0.12
query12	0.14	0.12	0.13
query13	0.61	0.60	0.58
query14	0.76	0.77	0.79
query15	0.85	0.84	0.82
query16	0.36	0.37	0.37
query17	1.04	1.04	0.97
query18	0.23	0.22	0.21
query19	1.83	1.67	1.68
query20	0.01	0.03	0.01
query21	15.41	0.76	0.66
query22	4.11	7.86	2.11
query23	18.28	1.40	1.27
query24	2.09	0.21	0.22
query25	0.14	0.08	0.08
query26	0.29	0.21	0.21
query27	0.45	0.22	0.23
query28	13.34	1.01	0.99
query29	12.65	3.28	3.26
query30	0.23	0.05	0.04
query31	2.89	0.40	0.38
query32	3.28	0.47	0.48
query33	2.90	3.03	2.93
query34	17.11	4.34	4.37
query35	4.39	4.44	4.40
query36	0.65	0.47	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.14	0.13	0.12
query41	0.10	0.04	0.06
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.73 s
Total hot run time: 30.72 s

@csun5285 csun5285 mentioned this pull request Aug 9, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 10, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (#39170) (#39176)
a79b297 
## Proposed changes


pick from master #39170
Issue Number: close #xxx

<!--Describe your changes.-->
zzzxl1993
zzzxl1993 approved these changes Aug 12, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 12, 2024

PR approved by anyone and no changes requested.

airborne12
airborne12 approved these changes Aug 12, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 12, 2024

PR approved by at least one committer and no changes requested.

@airborne12 airborne12 merged commit 191371d into apache:master Aug 12, 2024
This was referenced Aug 12, 2024
Merged
Merged
airborne12 pushed a commit that referenced this pull request Aug 13, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (#39170) (#39227)
f3d0276 
## Proposed changes

pick from master #39170
airborne12 pushed a commit that referenced this pull request Aug 13, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (#39170) (#39228)
beab6a8 
## Proposed changes

pick from master #39170
@airborne12 airborne12 removed the doing label Aug 13, 2024
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@csun5285 @wyxxxcat
[fix] (compaction) fix time series compaction policy (apache#39170)
25c2fb7 
## Proposed changes

Fix the issue introduced by apache#38220 

### BUG
The permits requested for compaction cannot be 0, 0 indicates failure.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@csun5285
[fix] (compaction) fix time series compaction policy (apache#39170) (a…
9de63f8 
…pache#39227)

## Proposed changes

pick from master apache#39170
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@csun5285 @doris-robot @airborne12 @zzzxl1993 @dataroaring