Skip to content

[fix](index compaction)Fix core when having multiple dest segments #40254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qidaye merged 1 commit into from
Sep 3, 2024
Merged

[fix](index compaction)Fix core when having multiple dest segments #40254

qidaye merged 1 commit into from
Sep 3, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Sep 2, 2024

Proposed changes

introduced by: #39076

@doris-robot
Copy link

doris-robot commented Sep 2, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Sep 2, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@qidaye qidaye force-pushed the fix_index_compaction_core branch from fa19fcc to ccc5019 Compare September 2, 2024 08:01
@qidaye
Copy link
Contributor Author

qidaye commented Sep 2, 2024

run buildall

zzzxl1993
zzzxl1993 approved these changes Sep 2, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

PR approved by anyone and no changes requested.

@airborne12
Copy link
Member

airborne12 commented Sep 2, 2024

pls paste coredump stack.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

csun5285
csun5285 approved these changes Sep 2, 2024
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Sep 2, 2024

TPC-H: Total hot run time: 38419 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccc5019e102e2eb9ca53100b02f2b3758633c919, data reload: false

------ Round 1 ----------------------------------
q1	17043	4523	4297	4297
q2	1272	183	179	179
q3	7761	1096	1137	1096
q4	8050	755	765	755
q5	6134	2893	2842	2842
q6	226	141	142	141
q7	972	637	603	603
q8	7393	2071	2060	2060
q9	6727	6574	6541	6541
q10	4607	2210	2212	2210
q11	424	231	240	231
q12	527	220	223	220
q13	17286	3129	3108	3108
q14	286	235	231	231
q15	513	493	475	475
q16	581	524	512	512
q17	956	738	708	708
q18	7330	6905	6848	6848
q19	1399	1080	1076	1076
q20	660	361	339	339
q21	4206	3154	2920	2920
q22	1153	1027	1032	1027
Total cold run time: 95506 ms
Total hot run time: 38419 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4302	4293	4293
q2	381	266	264	264
q3	2904	2671	2686	2671
q4	1944	1699	1697	1697
q5	5400	5400	5468	5400
q6	225	133	134	133
q7	2105	1716	1733	1716
q8	3213	3338	3329	3329
q9	8412	8439	8417	8417
q10	3449	3206	3193	3193
q11	597	519	487	487
q12	792	628	645	628
q13	8239	3083	3072	3072
q14	321	272	279	272
q15	535	483	467	467
q16	613	581	558	558
q17	1778	1522	1467	1467
q18	7932	7425	7488	7425
q19	1667	1447	1389	1389
q20	2058	1835	1831	1831
q21	5485	5277	5147	5147
q22	1135	1079	1055	1055
Total cold run time: 63544 ms
Total hot run time: 54911 ms

@doris-robot
Copy link

doris-robot commented Sep 2, 2024

TPC-DS: Total hot run time: 187816 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccc5019e102e2eb9ca53100b02f2b3758633c919, data reload: false

query1	902	381	368	368
query2	6436	2095	1976	1976
query3	6647	210	214	210
query4	34196	23219	23315	23219
query5	4170	507	479	479
query6	249	161	175	161
query7	4584	306	305	305
query8	251	204	203	203
query9	8697	2443	2442	2442
query10	427	262	264	262
query11	17579	15070	15029	15029
query12	154	94	96	94
query13	1633	381	371	371
query14	9825	7005	7439	7005
query15	253	173	179	173
query16	8058	482	457	457
query17	1607	570	569	569
query18	2124	296	308	296
query19	260	142	140	140
query20	120	104	106	104
query21	210	102	107	102
query22	4394	3998	4006	3998
query23	34023	34497	33352	33352
query24	11215	2834	2901	2834
query25	553	385	395	385
query26	714	153	154	153
query27	2269	282	286	282
query28	6988	2077	2046	2046
query29	659	410	402	402
query30	284	162	151	151
query31	971	784	762	762
query32	100	57	78	57
query33	755	277	291	277
query34	1014	475	484	475
query35	891	742	715	715
query36	1103	891	928	891
query37	145	89	91	89
query38	3961	3807	3913	3807
query39	1455	1402	1399	1399
query40	202	118	115	115
query41	49	45	46	45
query42	109	95	97	95
query43	547	488	488	488
query44	1234	752	745	745
query45	196	177	175	175
query46	1097	762	741	741
query47	1928	1796	1763	1763
query48	392	310	294	294
query49	1086	418	443	418
query50	820	405	421	405
query51	7218	7044	7078	7044
query52	100	89	86	86
query53	253	181	180	180
query54	904	453	463	453
query55	76	76	77	76
query56	290	253	258	253
query57	1178	1077	1110	1077
query58	238	224	236	224
query59	3143	2914	2924	2914
query60	294	274	262	262
query61	103	98	100	98
query62	839	658	673	658
query63	229	188	182	182
query64	4091	660	647	647
query65	3252	3173	3201	3173
query66	853	330	342	330
query67	15883	15275	15342	15275
query68	3272	592	566	566
query69	392	273	273	273
query70	1222	1102	1120	1102
query71	344	274	278	274
query72	6202	3941	3997	3941
query73	753	336	352	336
query74	9203	8838	8915	8838
query75	3405	2695	2732	2695
query76	1861	980	1041	980
query77	475	314	318	314
query78	9630	9059	9149	9059
query79	1064	530	544	530
query80	693	496	500	496
query81	466	237	234	234
query82	242	145	145	145
query83	169	150	148	148
query84	224	79	75	75
query85	677	283	273	273
query86	294	301	298	298
query87	4324	4375	4231	4231
query88	3033	2354	2316	2316
query89	398	295	285	285
query90	1769	190	189	189
query91	135	100	101	100
query92	63	49	50	49
query93	1039	555	529	529
query94	826	299	288	288
query95	346	250	255	250
query96	591	266	263	263
query97	3189	3070	3021	3021
query98	216	203	198	198
query99	1547	1290	1264	1264
Total cold run time: 284544 ms
Total hot run time: 187816 ms

@doris-robot
Copy link

doris-robot commented Sep 2, 2024

ClickBench: Total hot run time: 31.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccc5019e102e2eb9ca53100b02f2b3758633c919, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.09
query5	0.50	0.49	0.51
query6	1.12	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.56	0.55	0.52
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.61	0.59	0.58
query14	2.04	2.06	2.05
query15	0.90	0.84	0.82
query16	0.38	0.37	0.37
query17	1.03	1.05	1.05
query18	0.21	0.19	0.20
query19	1.88	1.80	1.87
query20	0.01	0.01	0.02
query21	15.38	0.68	0.65
query22	4.46	7.30	1.50
query23	18.30	1.36	1.32
query24	2.14	0.22	0.22
query25	0.14	0.08	0.08
query26	0.25	0.18	0.17
query27	0.08	0.07	0.07
query28	13.24	1.00	1.00
query29	12.61	3.39	3.42
query30	0.24	0.05	0.05
query31	2.87	0.40	0.39
query32	3.27	0.49	0.47
query33	2.94	2.99	3.03
query34	16.90	4.42	4.44
query35	4.51	4.50	4.47
query36	0.66	0.47	0.47
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.86 s
Total hot run time: 31.86 s

eldenmoon
eldenmoon approved these changes Sep 3, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 3, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

PR approved by at least one committer and no changes requested.

@qidaye qidaye merged commit ee3be4f into apache:master Sep 3, 2024
@qidaye qidaye deleted the fix_index_compaction_core branch September 3, 2024 02:35
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 3, 2024
@qidaye
[fix](index compaction)Fix core when having multiple dest segments (a…
58484a5 
…pache#40254)

introduced by: apache#39076
@qidaye qidaye mentioned this pull request Sep 3, 2024
Merged
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@qidaye
[fix](index compaction)Fix core when having multiple dest segments (#…
959cac4 
…40254) (#40294)

bp #40254
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

+1 more reviewer

@csun5285 csun5285 csun5285 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@qidaye @doris-robot @airborne12 @zzzxl1993 @eldenmoon @csun5285 @dataroaring