Skip to content

[fix](multi-catalog)fix hive partition insert regression case #35821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 3 commits into from
Jun 4, 2024
Merged

[fix](multi-catalog)fix hive partition insert regression case #35821

morningman merged 3 commits into from
Jun 4, 2024

Conversation

@wsjz
Copy link
Contributor

@wsjz wsjz commented Jun 3, 2024

Proposed changes

forbid hive external table insert with partition spec and modify regresion case

@doris-robot
Copy link

doris-robot commented Jun 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wsjz
Copy link
Contributor Author

wsjz commented Jun 4, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 4, 2024

TPC-H: Total hot run time: 40267 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a6c649873e215ba27d4166e02ee433360f53526, data reload: false

------ Round 1 ----------------------------------
q1	17827	4486	4342	4342
q2	2422	197	198	197
q3	11461	1210	1188	1188
q4	10390	775	812	775
q5	7570	2727	2778	2727
q6	223	141	145	141
q7	983	612	612	612
q8	9654	2095	2090	2090
q9	8831	6447	6456	6447
q10	9006	3692	3698	3692
q11	473	256	242	242
q12	529	233	223	223
q13	17772	2991	2942	2942
q14	261	220	219	219
q15	514	476	465	465
q16	499	398	387	387
q17	965	752	703	703
q18	7975	7451	7437	7437
q19	2502	1506	1499	1499
q20	659	319	322	319
q21	5014	3282	3863	3282
q22	390	348	338	338
Total cold run time: 115920 ms
Total hot run time: 40267 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4246	4219	4219
q2	370	273	266	266
q3	2930	2793	2725	2725
q4	1899	1590	1575	1575
q5	5259	5326	5276	5276
q6	219	128	131	128
q7	2140	1736	1750	1736
q8	3208	3309	3328	3309
q9	8281	8300	8295	8295
q10	3885	3657	3714	3657
q11	591	495	519	495
q12	782	589	631	589
q13	17617	2979	2978	2978
q14	301	279	266	266
q15	510	475	472	472
q16	482	410	413	410
q17	1743	1454	1464	1454
q18	7697	7577	7324	7324
q19	1721	1664	1438	1438
q20	1951	1782	1772	1772
q21	4901	4766	4756	4756
q22	650	528	535	528
Total cold run time: 71504 ms
Total hot run time: 53668 ms

@doris-robot
Copy link

doris-robot commented Jun 4, 2024

TPC-DS: Total hot run time: 169207 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a6c649873e215ba27d4166e02ee433360f53526, data reload: false

query1	925	378	372	372
query2	6455	2469	2251	2251
query3	6661	209	208	208
query4	19818	17226	17436	17226
query5	4169	465	459	459
query6	281	153	160	153
query7	4587	298	294	294
query8	310	292	292	292
query9	8506	2391	2373	2373
query10	452	289	279	279
query11	10541	9944	10000	9944
query12	141	94	86	86
query13	1635	369	366	366
query14	9441	7003	6477	6477
query15	236	193	183	183
query16	7560	265	262	262
query17	1336	537	531	531
query18	1896	268	265	265
query19	192	154	151	151
query20	95	89	92	89
query21	207	136	128	128
query22	4417	3935	4067	3935
query23	33497	33154	32927	32927
query24	12077	2833	2842	2833
query25	668	361	359	359
query26	1807	163	156	156
query27	3112	323	325	323
query28	7783	2078	2086	2078
query29	1146	614	589	589
query30	289	147	151	147
query31	955	755	751	751
query32	85	53	53	53
query33	767	281	277	277
query34	1001	476	476	476
query35	747	622	602	602
query36	1083	948	892	892
query37	195	67	69	67
query38	2863	2738	2723	2723
query39	846	791	777	777
query40	284	126	123	123
query41	53	51	54	51
query42	120	103	97	97
query43	600	564	573	564
query44	1218	749	788	749
query45	204	162	170	162
query46	1075	723	742	723
query47	1847	1795	1760	1760
query48	405	307	309	307
query49	1185	409	412	409
query50	778	384	390	384
query51	6910	6748	6849	6748
query52	95	101	88	88
query53	364	290	297	290
query54	923	448	440	440
query55	74	71	71	71
query56	275	258	259	258
query57	1130	1058	1037	1037
query58	266	248	236	236
query59	3579	3253	3088	3088
query60	283	273	272	272
query61	88	87	98	87
query62	663	445	437	437
query63	324	296	287	287
query64	9931	2195	1674	1674
query65	3206	3069	3142	3069
query66	1377	329	323	323
query67	15290	14879	14960	14879
query68	4601	562	531	531
query69	461	299	292	292
query70	1179	1158	1129	1129
query71	401	290	287	287
query72	7164	5597	2831	2831
query73	763	334	323	323
query74	5954	5555	5533	5533
query75	3393	2670	2679	2670
query76	2552	994	904	904
query77	484	308	296	296
query78	10485	9727	9718	9718
query79	2880	511	523	511
query80	1905	472	473	472
query81	598	222	222	222
query82	769	105	102	102
query83	401	214	174	174
query84	271	86	87	86
query85	1917	269	268	268
query86	514	309	323	309
query87	3289	3129	3101	3101
query88	3778	2455	2460	2455
query89	475	388	384	384
query90	1802	199	194	194
query91	126	103	93	93
query92	64	50	50	50
query93	2515	501	495	495
query94	1276	192	183	183
query95	403	313	311	311
query96	600	272	275	272
query97	3193	3003	3036	3003
query98	248	211	215	211
query99	1272	830	876	830
Total cold run time: 277062 ms
Total hot run time: 169207 ms

@doris-robot
Copy link

doris-robot commented Jun 4, 2024

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a6c649873e215ba27d4166e02ee433360f53526, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.04
query4	1.66	0.07	0.08
query5	0.49	0.48	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.60
query14	0.77	0.78	0.78
query15	0.84	0.81	0.81
query16	0.37	0.37	0.38
query17	1.02	1.04	0.99
query18	0.23	0.21	0.27
query19	1.88	1.67	1.68
query20	0.02	0.01	0.02
query21	15.73	0.69	0.68
query22	4.30	7.18	2.03
query23	18.19	1.39	1.19
query24	1.66	0.26	0.25
query25	0.14	0.09	0.08
query26	0.25	0.17	0.17
query27	0.08	0.08	0.09
query28	13.37	1.01	1.00
query29	13.25	3.31	3.31
query30	0.25	0.05	0.06
query31	2.84	0.39	0.38
query32	3.30	0.48	0.47
query33	2.87	2.92	2.80
query34	17.13	4.49	4.44
query35	4.47	4.54	4.54
query36	0.64	0.46	0.46
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.05 s
Total hot run time: 30.65 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

PR approved by anyone and no changes requested.

morningman
morningman approved these changes Jun 4, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit b33a161 into apache:master Jun 4, 2024
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@wsjz @dataroaring
[fix](multi-catalog)fix hive partition insert regression case (#35821)
a51f768 
forbid hive external table insert with partition spec and modify
regression case
@wsjz wsjz mentioned this pull request Jun 4, 2024
Merged
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@wsjz @seawinde
[fix](multi-catalog)fix hive partition insert regression case (apache…
3bbcfe9 
…#35821)

forbid hive external table insert with partition spec and modify
regression case
morningman pushed a commit that referenced this pull request Jun 6, 2024
@wsjz
[fix](multi-catalog)fix hive partition insert regression case (#35846)
b6ab0c4 
from #35821
@wsjz wsjz deleted the fix_re1 branch July 4, 2024 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wsjz @doris-robot @morningman @morrySnow @dataroaring