Skip to content

[enhancement](merge-on-write) consider version count on size-based cloud cu compaction policy #33244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 4, 2024
Merged

[enhancement](merge-on-write) consider version count on size-based cloud cu compaction policy #33244

dataroaring merged 1 commit into from
Apr 4, 2024

Conversation

@hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Apr 3, 2024

this feature introduce from #25352

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Apr 3, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

TPC-H: Total hot run time: 38695 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce05f1cfb4f10f13fcfa9807de8751a9f30987ac, data reload: false

------ Round 1 ----------------------------------
q1	17686	4098	4079	4079
q2	2016	195	191	191
q3	10470	1185	1309	1185
q4	10202	812	924	812
q5	7538	2963	2946	2946
q6	222	140	135	135
q7	1120	620	604	604
q8	9406	2021	2056	2021
q9	6735	6244	6157	6157
q10	8403	3552	3513	3513
q11	428	235	235	235
q12	375	206	202	202
q13	17770	2873	2924	2873
q14	275	233	237	233
q15	528	486	481	481
q16	478	393	369	369
q17	955	913	923	913
q18	7538	6459	6525	6459
q19	1609	1543	1530	1530
q20	562	308	304	304
q21	3516	3155	3163	3155
q22	356	298	300	298
Total cold run time: 108188 ms
Total hot run time: 38695 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4091	4051	4032	4032
q2	327	214	220	214
q3	2986	2934	2944	2934
q4	1884	1875	1853	1853
q5	5250	5236	5224	5224
q6	211	124	124	124
q7	2247	1792	1832	1792
q8	3223	3296	3291	3291
q9	8476	8438	8478	8438
q10	3741	4002	3994	3994
q11	568	476	474	474
q12	748	585	591	585
q13	16262	3131	3128	3128
q14	311	274	278	274
q15	526	478	508	478
q16	503	450	456	450
q17	1796	1756	1713	1713
q18	8401	7749	7541	7541
q19	1697	1686	1685	1685
q20	2062	1833	1841	1833
q21	5283	4946	5007	4946
q22	493	428	432	428
Total cold run time: 71086 ms
Total hot run time: 55431 ms

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

TeamCity be ut coverage result:
Function Coverage: 35.64% (8881/24922)
Line Coverage: 27.37% (72919/266407)
Region Coverage: 26.55% (37713/142020)
Branch Coverage: 23.35% (19216/82284)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ce05f1cfb4f10f13fcfa9807de8751a9f30987ac_ce05f1cfb4f10f13fcfa9807de8751a9f30987ac/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

TPC-DS: Total hot run time: 181204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce05f1cfb4f10f13fcfa9807de8751a9f30987ac, data reload: false

query1	1237	1121	360	360
query2	6329	1949	2002	1949
query3	6665	214	220	214
query4	25841	21481	21399	21399
query5	4200	393	411	393
query6	284	196	182	182
query7	4612	300	293	293
query8	238	180	174	174
query9	8474	2236	2229	2229
query10	441	247	252	247
query11	15043	14480	14436	14436
query12	135	88	88	88
query13	1641	380	380	380
query14	8526	6924	6999	6924
query15	212	169	173	169
query16	6786	265	254	254
query17	1010	570	561	561
query18	1821	278	263	263
query19	189	155	148	148
query20	96	85	86	85
query21	193	124	120	120
query22	5016	4811	4863	4811
query23	33561	32940	32800	32800
query24	10993	3111	3118	3111
query25	699	395	403	395
query26	1916	158	155	155
query27	3205	363	380	363
query28	7401	1894	1888	1888
query29	1256	628	633	628
query30	310	178	176	176
query31	990	737	770	737
query32	104	56	55	55
query33	661	248	251	248
query34	1230	503	505	503
query35	839	714	721	714
query36	999	882	863	863
query37	266	74	73	73
query38	3616	3613	3553	3553
query39	1644	1591	1627	1591
query40	235	133	132	132
query41	51	46	48	46
query42	111	110	106	106
query43	473	422	447	422
query44	1148	744	747	744
query45	303	266	270	266
query46	1119	879	804	804
query47	2013	1894	1841	1841
query48	369	313	306	306
query49	949	371	372	371
query50	823	400	417	400
query51	6874	6794	6826	6794
query52	108	94	96	94
query53	352	290	296	290
query54	278	224	232	224
query55	84	79	73	73
query56	250	238	238	238
query57	1274	1176	1186	1176
query58	243	231	214	214
query59	2786	2632	2518	2518
query60	269	238	235	235
query61	113	104	105	104
query62	672	457	436	436
query63	306	279	276	276
query64	5838	3128	3115	3115
query65	3096	3013	3011	3011
query66	1302	320	309	309
query67	15353	15049	14868	14868
query68	9213	568	562	562
query69	563	309	321	309
query70	1442	1150	1132	1132
query71	502	272	269	269
query72	6264	2582	2385	2385
query73	1504	327	322	322
query74	6669	6262	6354	6262
query75	3702	2347	2288	2288
query76	5865	1133	1234	1133
query77	629	248	246	246
query78	10927	10209	10121	10121
query79	9549	531	526	526
query80	1575	423	408	408
query81	486	234	232	232
query82	333	94	89	89
query83	209	165	164	164
query84	274	83	87	83
query85	949	276	275	275
query86	367	289	278	278
query87	3749	3442	3452	3442
query88	3574	2271	2280	2271
query89	536	362	375	362
query90	2046	176	174	174
query91	130	104	100	100
query92	62	46	46	46
query93	6058	524	525	524
query94	1280	178	177	177
query95	425	314	309	309
query96	626	270	265	265
query97	2639	2467	2466	2466
query98	229	215	214	214
query99	1197	857	825	825
Total cold run time: 299044 ms
Total hot run time: 181204 ms

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

ClickBench: Total hot run time: 29.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce05f1cfb4f10f13fcfa9807de8751a9f30987ac, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.24	0.04	0.04
query4	1.67	0.06	0.06
query5	0.48	0.48	0.48
query6	1.15	0.65	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.50
query10	0.56	0.57	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.79	0.80
query15	0.86	0.83	0.85
query16	0.35	0.35	0.36
query17	1.00	0.98	0.96
query18	0.26	0.25	0.26
query19	1.78	1.71	1.79
query20	0.02	0.01	0.01
query21	15.40	0.64	0.64
query22	4.19	8.84	1.02
query23	17.90	1.24	1.24
query24	1.47	0.19	0.20
query25	0.16	0.08	0.08
query26	0.26	0.16	0.16
query27	0.07	0.08	0.07
query28	13.93	0.96	0.93
query29	12.54	3.24	3.23
query30	0.26	0.07	0.06
query31	2.90	0.38	0.38
query32	3.23	0.48	0.47
query33	2.86	2.93	2.86
query34	15.50	4.31	4.34
query35	4.36	4.38	4.35
query36	0.68	0.47	0.48
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.15	0.16
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.3 s
Total hot run time: 29.2 s

@doris-robot
Copy link

doris-robot commented Apr 3, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ce05f1cfb4f10f13fcfa9807de8751a9f30987ac with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       15.9 seconds inserted 10000000 Rows, about 628K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Apr 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 82f86a8 into apache:master Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@luwei16 luwei16 luwei16 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hust-hhb @doris-robot @luwei16 @dataroaring @morningman @gavinchou