Skip to content

[enhance](catalog)Allow parallel running of insert overwrite on the external table #41575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Oct 11, 2024
Merged

[enhance](catalog)Allow parallel running of insert overwrite on the external table #41575

morningman merged 2 commits into from
Oct 11, 2024

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Oct 9, 2024 •
edited
Loading

pre pr: #40558

Allow parallel running of insert overwrite on the external table

@doris-robot
Copy link

doris-robot commented Oct 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Oct 9, 2024

run buildall

@zddr
Copy link
Contributor Author

zddr commented Oct 9, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 9, 2024

TPC-H: Total hot run time: 40777 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a303fa158fef91c3caae43ec6c97beaa1baf9a1c, data reload: false

------ Round 1 ----------------------------------
q1	17577	7425	7285	7285
q2	2042	272	267	267
q3	12256	1096	1261	1096
q4	10554	803	764	764
q5	7749	2898	2781	2781
q6	237	150	144	144
q7	993	616	603	603
q8	9356	1972	1944	1944
q9	6566	6412	6362	6362
q10	6966	2308	2324	2308
q11	444	251	241	241
q12	414	215	215	215
q13	17756	2986	2993	2986
q14	248	217	218	217
q15	570	516	523	516
q16	664	608	585	585
q17	988	547	524	524
q18	7345	6637	6806	6637
q19	1341	960	953	953
q20	487	214	202	202
q21	3941	3175	3165	3165
q22	1120	982	991	982
Total cold run time: 109614 ms
Total hot run time: 40777 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7212	7287	7210	7210
q2	334	226	235	226
q3	2988	2952	3014	2952
q4	2092	1797	1828	1797
q5	5773	5715	5753	5715
q6	237	147	147	147
q7	2230	1864	1842	1842
q8	3355	3569	3435	3435
q9	8932	8911	8809	8809
q10	3565	3573	3509	3509
q11	579	496	487	487
q12	867	654	647	647
q13	9412	3218	3207	3207
q14	306	271	270	270
q15	600	550	535	535
q16	683	643	658	643
q17	1836	1593	1598	1593
q18	8289	7708	7739	7708
q19	1750	1528	1456	1456
q20	2119	1852	1898	1852
q21	5606	5394	5386	5386
q22	1150	1048	1061	1048
Total cold run time: 69915 ms
Total hot run time: 60474 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2024

TPC-DS: Total hot run time: 192780 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a303fa158fef91c3caae43ec6c97beaa1baf9a1c, data reload: false

query1	933	394	399	394
query2	6254	1996	2045	1996
query3	8699	192	204	192
query4	33992	23620	23638	23620
query5	3347	476	475	475
query6	265	172	159	159
query7	4194	311	297	297
query8	278	208	223	208
query9	9280	2699	2686	2686
query10	482	280	279	279
query11	17722	15352	15377	15352
query12	154	106	98	98
query13	1577	462	446	446
query14	10549	7716	7540	7540
query15	255	172	174	172
query16	7993	502	486	486
query17	1610	616	594	594
query18	2244	315	318	315
query19	341	163	160	160
query20	123	114	111	111
query21	225	109	111	109
query22	4849	4542	4679	4542
query23	34886	34093	34002	34002
query24	12066	2831	2808	2808
query25	657	402	406	402
query26	1672	166	169	166
query27	2654	295	299	295
query28	7767	2454	2469	2454
query29	1000	439	436	436
query30	267	154	150	150
query31	1062	801	790	790
query32	100	54	54	54
query33	768	304	307	304
query34	965	536	501	501
query35	909	753	756	753
query36	1116	928	948	928
query37	287	87	89	87
query38	4051	3958	3920	3920
query39	1504	1421	1414	1414
query40	260	102	99	99
query41	50	45	42	42
query42	118	100	102	100
query43	533	510	478	478
query44	1253	825	820	820
query45	198	168	164	164
query46	1125	707	727	707
query47	1927	1839	1851	1839
query48	439	349	346	346
query49	1083	413	396	396
query50	828	420	417	417
query51	7218	6991	6991	6991
query52	97	87	88	87
query53	255	186	182	182
query54	1258	488	470	470
query55	80	77	80	77
query56	302	260	279	260
query57	1233	1136	1160	1136
query58	242	250	240	240
query59	3297	3004	2822	2822
query60	288	271	275	271
query61	104	98	100	98
query62	889	684	685	684
query63	217	196	185	185
query64	4939	640	607	607
query65	3250	3199	3154	3154
query66	1365	297	296	296
query67	16007	15750	15947	15750
query68	4345	592	560	560
query69	521	284	335	284
query70	1199	1078	1149	1078
query71	387	269	286	269
query72	7278	3988	3943	3943
query73	772	347	348	347
query74	10034	9008	8901	8901
query75	3422	2715	2704	2704
query76	3002	972	968	968
query77	450	301	314	301
query78	10513	9555	9602	9555
query79	2088	599	614	599
query80	916	447	454	447
query81	590	239	243	239
query82	757	144	141	141
query83	269	142	136	136
query84	248	75	75	75
query85	1200	301	286	286
query86	320	299	308	299
query87	4463	4278	4337	4278
query88	3254	2454	2390	2390
query89	399	288	290	288
query90	2090	186	186	186
query91	142	106	107	106
query92	59	47	48	47
query93	1142	542	548	542
query94	957	302	300	300
query95	357	256	253	253
query96	605	284	286	284
query97	3246	3084	3084	3084
query98	223	202	196	196
query99	1634	1316	1281	1281
Total cold run time: 302834 ms
Total hot run time: 192780 ms

@doris-robot
Copy link

doris-robot commented Oct 9, 2024

ClickBench: Total hot run time: 33.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a303fa158fef91c3caae43ec6c97beaa1baf9a1c, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.52	0.53	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.54	0.53
query11	0.14	0.11	0.11
query12	0.13	0.10	0.11
query13	0.60	0.60	0.59
query14	2.70	2.71	2.72
query15	0.90	0.82	0.83
query16	0.39	0.38	0.40
query17	1.03	1.01	1.04
query18	0.21	0.20	0.20
query19	1.97	1.84	2.00
query20	0.01	0.01	0.02
query21	15.37	0.59	0.60
query22	2.47	1.90	1.95
query23	17.04	0.89	0.86
query24	2.74	2.17	1.19
query25	0.26	0.12	0.08
query26	0.54	0.14	0.15
query27	0.04	0.04	0.04
query28	9.86	1.09	1.06
query29	12.58	3.23	3.23
query30	0.25	0.06	0.06
query31	2.89	0.39	0.37
query32	3.29	0.46	0.45
query33	2.94	3.00	3.04
query34	16.90	4.53	4.50
query35	4.49	4.46	4.53
query36	0.65	0.50	0.51
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.15	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.64 s
Total hot run time: 33.01 s

@zddr
Copy link
Contributor Author

zddr commented Oct 9, 2024

run external

@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 10, 2024

PR approved by anyone and no changes requested.

morningman
morningman approved these changes Oct 11, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 5d4ab4d into apache:master Oct 11, 2024
zddr added a commit to zddr/incubator-doris that referenced this pull request Oct 12, 2024
@zddr
[enhance](catalog)Allow parallel running of insert overwrite on the e…
d971b7a 
…xternal table (apache#41575)

pre pr: apache#40558

Allow parallel running of insert overwrite on the external table
@zddr zddr mentioned this pull request Oct 12, 2024
Merged
yiguolei pushed a commit that referenced this pull request Oct 12, 2024
@zddr
[enhance](catalog)Allow parallel running of insert overwrite on the e… (
cb5a5a5 
#41742)

…xternal table (#41575)

pick: #41575
zddr added a commit to zddr/incubator-doris that referenced this pull request Oct 15, 2024
@zddr
[enhance](catalog)Allow parallel running of insert overwrite on the e…
ad91575 
…xternal table (apache#41575)

pre pr: apache#40558

Allow parallel running of insert overwrite on the external table
@zddr zddr mentioned this pull request Oct 15, 2024
Merged
morningman pushed a commit that referenced this pull request Oct 17, 2024
@zddr
[enhance](catalog)Allow parallel running of insert overwrite on the e…
ea0fb5e 
…xternal table (#41575) (#41842)

pick: #41575
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zddr @doris-robot @morningman @morrySnow @yiguolei