Skip to content

[fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction #38224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 23, 2024
Merged

[fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction #38224

dataroaring merged 1 commit into from
Jul 23, 2024

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Jul 22, 2024

No description provided.

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

dataroaring
dataroaring approved these changes Jul 22, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 22, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 22, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 22, 2024

PR approved by anyone and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 22, 2024

clang-tidy review says "All clean, LGTM! 👍"

@luwei16
Copy link
Contributor Author

luwei16 commented Jul 22, 2024

run buildall

@zhannngchen
Copy link
Contributor

zhannngchen commented Jul 22, 2024

we'd better add a case?

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

TPC-H: Total hot run time: 39964 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d28230d9992e85f5a35bbe4fb41f64965faf421d, data reload: false

------ Round 1 ----------------------------------
q1	17964	4620	4437	4437
q2	2560	196	204	196
q3	11496	1188	1165	1165
q4	10776	799	798	798
q5	7611	2740	2684	2684
q6	226	138	139	138
q7	971	612	604	604
q8	9277	2089	2084	2084
q9	8820	6513	6580	6513
q10	8746	3790	3755	3755
q11	481	238	239	238
q12	394	223	224	223
q13	18742	2996	2990	2990
q14	294	238	234	234
q15	539	486	482	482
q16	499	373	383	373
q17	954	638	673	638
q18	7890	7545	7329	7329
q19	6426	1409	1342	1342
q20	702	316	336	316
q21	4872	3186	3142	3142
q22	344	287	283	283
Total cold run time: 120584 ms
Total hot run time: 39964 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4297	4296	4296
q2	381	276	274	274
q3	2982	2717	2717	2717
q4	1917	1612	1581	1581
q5	5281	5292	5303	5292
q6	218	131	130	130
q7	2098	1678	1728	1678
q8	3218	3355	3309	3309
q9	8471	8418	8435	8418
q10	3882	3747	3699	3699
q11	593	528	503	503
q12	765	616	609	609
q13	17465	2977	3016	2977
q14	306	262	271	262
q15	523	484	467	467
q16	494	419	419	419
q17	1801	1486	1478	1478
q18	7656	7445	7428	7428
q19	1660	1544	1567	1544
q20	1970	1758	1815	1758
q21	4922	4768	4741	4741
q22	579	488	497	488
Total cold run time: 71587 ms
Total hot run time: 54068 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

TPC-DS: Total hot run time: 171772 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d28230d9992e85f5a35bbe4fb41f64965faf421d, data reload: false

query1	909	370	360	360
query2	6473	1969	1857	1857
query3	6750	204	210	204
query4	28329	17500	17205	17205
query5	4204	485	484	484
query6	294	168	154	154
query7	4590	292	282	282
query8	244	204	197	197
query9	8439	2430	2397	2397
query10	454	289	267	267
query11	11192	9927	9980	9927
query12	135	86	82	82
query13	1640	368	371	368
query14	10271	7019	8084	7019
query15	215	173	168	168
query16	7515	488	422	422
query17	1593	567	544	544
query18	1358	276	284	276
query19	209	165	158	158
query20	94	85	82	82
query21	212	126	128	126
query22	4480	4185	3943	3943
query23	33799	33127	33118	33118
query24	11918	2884	2855	2855
query25	686	386	409	386
query26	1773	149	156	149
query27	2765	271	269	269
query28	7260	1996	1973	1973
query29	1070	627	617	617
query30	288	151	151	151
query31	970	741	766	741
query32	102	55	56	55
query33	786	412	335	335
query34	933	465	483	465
query35	858	731	764	731
query36	1122	910	929	910
query37	280	78	76	76
query38	2880	2735	2738	2735
query39	871	797	814	797
query40	284	116	116	116
query41	50	46	43	43
query42	114	103	107	103
query43	502	473	461	461
query44	1208	735	728	728
query45	191	162	158	158
query46	1098	713	728	713
query47	1859	1760	1792	1760
query48	361	286	289	286
query49	1178	415	417	415
query50	789	385	386	385
query51	6717	6674	6626	6626
query52	108	88	97	88
query53	358	292	293	292
query54	1106	448	441	441
query55	78	74	91	74
query56	280	269	274	269
query57	1136	1065	1084	1065
query58	259	253	246	246
query59	2906	2568	2674	2568
query60	312	273	283	273
query61	94	92	120	92
query62	836	652	649	649
query63	319	291	288	288
query64	10497	2225	1689	1689
query65	3146	3164	3109	3109
query66	1266	333	327	327
query67	15470	14882	15024	14882
query68	5148	547	557	547
query69	688	430	358	358
query70	1228	1126	1173	1126
query71	448	279	283	279
query72	8444	5797	5470	5470
query73	760	324	317	317
query74	6104	5639	5598	5598
query75	3517	2638	2738	2638
query76	3458	915	934	915
query77	703	311	319	311
query78	9817	9007	9114	9007
query79	3924	523	536	523
query80	1899	523	484	484
query81	590	226	218	218
query82	1174	139	142	139
query83	327	177	171	171
query84	276	85	96	85
query85	1558	333	321	321
query86	464	322	302	302
query87	3277	3084	3056	3056
query88	4929	2367	2372	2367
query89	513	378	383	378
query90	2008	196	194	194
query91	135	100	102	100
query92	70	51	50	50
query93	4965	504	502	502
query94	1372	282	298	282
query95	408	328	324	324
query96	603	269	265	265
query97	3292	2971	3039	2971
query98	213	206	200	200
query99	1567	1258	1243	1243
Total cold run time: 294362 ms
Total hot run time: 171772 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d28230d9992e85f5a35bbe4fb41f64965faf421d, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.06	0.06
query4	1.65	0.09	0.08
query5	0.49	0.47	0.50
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.57
query14	0.74	0.79	0.78
query15	0.86	0.81	0.81
query16	0.35	0.36	0.36
query17	1.02	0.96	0.98
query18	0.22	0.21	0.23
query19	1.90	1.81	1.74
query20	0.01	0.01	0.02
query21	15.55	0.76	0.66
query22	4.50	7.24	2.09
query23	18.28	1.36	1.21
query24	2.13	0.24	0.23
query25	0.14	0.08	0.09
query26	0.30	0.21	0.22
query27	0.45	0.23	0.23
query28	13.19	1.01	0.99
query29	12.55	3.26	3.25
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.31	0.48	0.47
query33	2.91	2.92	2.88
query34	17.02	4.37	4.34
query35	4.39	4.41	4.45
query36	0.65	0.46	0.49
query37	0.20	0.16	0.16
query38	0.16	0.14	0.16
query39	0.04	0.03	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.01 s
Total hot run time: 30.67 s

@wm1581066 wm1581066 added the p0_w label Jul 23, 2024
@luwei16
Copy link
Contributor Author

luwei16 commented Jul 23, 2024

run external

@luwei16
Copy link
Contributor Author

luwei16 commented Jul 23, 2024

run p0

@dataroaring dataroaring merged commit 34ccf3a into apache:master Jul 23, 2024
luwei16 added a commit to luwei16/Doris that referenced this pull request Jul 24, 2024
@luwei16
[fix](compaction) fix duplicate key in agg/mor table caused by ordere…
26c63c8 
…d data compaction (apache#38224)
@luwei16 luwei16 mentioned this pull request Jul 24, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@luwei16
[fix](compaction) fix duplicate key in agg/mor table caused by ordere…
9a40cd5 
…d data compaction (#38224) (#38299)

pick master #38224
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@luwei16 @dataroaring
[fix](compaction) fix duplicate key in agg/mor table caused by ordere…
f58a1ab 
…d data compaction (#38224)
luwei16 added a commit to luwei16/Doris that referenced this pull request Jul 31, 2024
@luwei16
[fix](compaction) fix duplicate key in agg/mor table caused by ordere…
726d807 
…d data compaction (apache#38224)
@luwei16 luwei16 mentioned this pull request Jul 31, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 1, 2024
@luwei16
[fix](compaction) fix duplicate key in agg/mor table caused by ordere…
6c0fba4 
…d data compaction (#38224) (#38642)

pick master #38224
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@luwei16
[fix](compaction) fix duplicate key in agg/mor table caused by ordere…
f28a111 
…d data compaction (apache#38224) (apache#38642)

pick master apache#38224
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.6-merged dev/3.0.1-merged p0_w reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@luwei16 @doris-robot @zhannngchen @gavinchou @dataroaring @wm1581066