Skip to content

[fix](segcompaction) enable segcompaction by default #30810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Feb 5, 2024
Merged

[fix](segcompaction) enable segcompaction by default #30810

yiguolei merged 1 commit into from
Feb 5, 2024

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Feb 4, 2024

Proposed changes

Issue Number: close #xxx

PR #28901 has been merged, so we enable segment compaction by default.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Feb 4, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Feb 4, 2024

TeamCity be ut coverage result:
Function Coverage: 36.16% (8627/23858)
Line Coverage: 28.22% (70546/249999)
Region Coverage: 27.23% (36405/133678)
Branch Coverage: 24.02% (18648/77620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/16185b72162df5a789b2309bbf4d6878172888e8_16185b72162df5a789b2309bbf4d6878172888e8/report/index.html

@doris-robot
Copy link

doris-robot commented Feb 4, 2024

TPC-H: Total hot run time: 36913 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 16185b72162df5a789b2309bbf4d6878172888e8, data reload: false

------ Round 1 ----------------------------------
q1	17661	4439	4403	4403
q2	2042	143	134	134
q3	10596	908	912	908
q4	4667	731	737	731
q5	7674	2931	2953	2931
q6	182	117	117	117
q7	1118	721	706	706
q8	9316	2017	1980	1980
q9	7213	6363	6368	6363
q10	8085	2405	2412	2405
q11	423	211	203	203
q12	747	270	275	270
q13	18008	3304	3297	3297
q14	270	247	245	245
q15	522	498	489	489
q16	461	392	402	392
q17	926	514	483	483
q18	6728	5968	6027	5968
q19	1573	1346	1344	1344
q20	652	344	319	319
q21	7035	3108	2925	2925
q22	792	307	300	300
Total cold run time: 106691 ms
Total hot run time: 36913 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4476	4407	4413	4407
q2	323	240	242	240
q3	2983	2825	2844	2825
q4	1824	1575	1578	1575
q5	5203	5219	5132	5132
q6	192	114	114	114
q7	2127	1792	1794	1792
q8	3085	3226	3258	3226
q9	8334	8272	8231	8231
q10	6004	3600	3560	3560
q11	550	453	472	453
q12	729	594	571	571
q13	15865	3105	3077	3077
q14	270	263	264	263
q15	535	497	482	482
q16	523	449	462	449
q17	1832	1664	1679	1664
q18	7902	7664	7470	7470
q19	8953	1503	1503	1503
q20	2114	1934	1895	1895
q21	4837	4436	4577	4436
q22	564	467	455	455
Total cold run time: 79225 ms
Total hot run time: 53820 ms

@doris-robot
Copy link

doris-robot commented Feb 4, 2024

TPC-DS: Total hot run time: 173605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16185b72162df5a789b2309bbf4d6878172888e8, data reload: false

query1	940	337	333	333
query2	6555	2003	1951	1951
query3	6707	206	201	201
query4	31855	21853	21929	21853
query5	4285	434	341	341
query6	258	165	176	165
query7	4606	278	276	276
query8	246	181	183	181
query9	8943	2322	2297	2297
query10	418	201	210	201
query11	17122	15320	15379	15320
query12	139	76	73	73
query13	1630	428	411	411
query14	9426	7085	6760	6760
query15	259	179	186	179
query16	8150	253	240	240
query17	1912	530	503	503
query18	2100	259	262	259
query19	360	139	135	135
query20	81	75	77	75
query21	189	129	121	121
query22	4886	4700	4557	4557
query23	30746	29881	29956	29881
query24	10472	2766	2769	2766
query25	544	329	330	329
query26	719	146	149	146
query27	2155	289	294	289
query28	5862	1858	1861	1858
query29	890	628	595	595
query30	278	132	141	132
query31	925	686	723	686
query32	94	49	47	47
query33	624	216	213	213
query34	838	440	447	440
query35	861	778	775	775
query36	1049	916	918	916
query37	95	65	57	57
query38	3258	3126	3119	3119
query39	1309	1242	1240	1240
query40	183	94	92	92
query41	38	36	33	33
query42	95	92	89	89
query43	523	479	499	479
query44	1048	683	719	683
query45	193	176	168	168
query46	1050	660	639	639
query47	1646	1502	1526	1502
query48	434	346	362	346
query49	1044	285	291	285
query50	765	368	387	368
query51	5255	5110	5206	5110
query52	105	84	88	84
query53	352	271	263	263
query54	279	208	216	208
query55	78	81	78	78
query56	219	197	201	197
query57	938	881	901	881
query58	206	170	176	170
query59	2392	2384	2177	2177
query60	236	206	210	206
query61	81	83	82	82
query62	620	378	369	369
query63	299	262	259	259
query64	4746	3676	3532	3532
query65	3251	3217	3190	3190
query66	821	329	297	297
query67	14343	14127	14040	14040
query68	4151	537	545	537
query69	466	319	307	307
query70	1246	1232	1185	1185
query71	320	241	260	241
query72	5972	2821	2663	2663
query73	709	323	313	313
query74	6626	6243	6250	6243
query75	3020	2353	2331	2331
query76	2558	945	955	945
query77	363	226	220	220
query78	9125	8826	8529	8529
query79	2654	499	490	490
query80	1976	353	331	331
query81	542	192	196	192
query82	779	85	75	75
query83	253	123	119	119
query84	282	83	79	79
query85	2172	330	317	317
query86	488	305	314	305
query87	3372	3199	3176	3176
query88	3827	2328	2311	2311
query89	445	375	365	365
query90	1986	164	163	163
query91	164	119	119	119
query92	54	42	42	42
query93	2638	473	490	473
query94	1301	173	175	173
query95	464	370	362	362
query96	606	273	266	266
query97	4244	4103	4136	4103
query98	207	197	190	190
query99	1149	692	696	692
Total cold run time: 277185 ms
Total hot run time: 173605 ms

@doris-robot
Copy link

doris-robot commented Feb 4, 2024

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16185b72162df5a789b2309bbf4d6878172888e8, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.07
query4	1.65	0.09	0.10
query5	0.52	0.52	0.53
query6	1.20	0.64	0.68
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.50	0.48
query10	0.55	0.56	0.54
query11	0.13	0.08	0.09
query12	0.11	0.08	0.09
query13	0.61	0.61	0.61
query14	0.78	0.80	0.81
query15	0.80	0.77	0.77
query16	0.39	0.38	0.39
query17	1.04	1.03	1.00
query18	0.24	0.23	0.25
query19	1.92	1.75	1.73
query20	0.01	0.01	0.01
query21	15.40	0.61	0.59
query22	3.16	2.92	1.98
query23	17.22	0.91	0.81
query24	2.80	1.47	0.99
query25	0.23	0.20	0.04
query26	0.60	0.13	0.14
query27	0.06	0.05	0.06
query28	11.10	0.86	0.83
query29	12.49	3.26	3.33
query30	0.60	0.55	0.53
query31	2.78	0.35	0.36
query32	3.32	0.48	0.48
query33	3.29	3.23	3.25
query34	15.71	4.32	4.28
query35	4.31	4.30	4.31
query36	1.10	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.16
query41	0.08	0.01	0.02
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.48 s
Total hot run time: 31.27 s

@doris-robot
Copy link

doris-robot commented Feb 4, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 16185b72162df5a789b2309bbf4d6878172888e8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.7 seconds inserted 10000000 Rows, about 680K ops/s

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Feb 4, 2024

run pipelinex_p0

1 similar comment
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Feb 4, 2024

run pipelinex_p0

dataroaring
dataroaring approved these changes Feb 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 5, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

PR approved by anyone and no changes requested.

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Feb 5, 2024

run pipelinex_p0

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Feb 5, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Feb 5, 2024

TeamCity be ut coverage result:
Function Coverage: 36.16% (8628/23858)
Line Coverage: 28.22% (70572/250068)
Region Coverage: 27.23% (36428/133780)
Branch Coverage: 24.02% (18660/77688)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7225f0399f4e531fc29d4e5f0070ffb7f1a89f33_7225f0399f4e531fc29d4e5f0070ffb7f1a89f33/report/index.html

@yiguolei yiguolei merged commit cc6b0a0 into apache:master Feb 5, 2024
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Feb 6, 2024
@liaoxin01 @starocean999
[fix](segcompaction) enable segcompaction by default (apache#30810)
0b8be47
@liaoxin01 liaoxin01 deleted the enable_segcom branch February 6, 2024 12:26
yiguolei pushed a commit that referenced this pull request Feb 19, 2024
@liaoxin01 @Doris-Extras
[fix](segcompaction) enable segcompaction by default (#30810)
9a70880
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liaoxin01 @doris-robot @yiguolei @dataroaring