Skip to content

[Configuration](transactional-hive) Add skip_checking_acid_version_file session var to skip checking acid version file in some hive envs. #42111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 21, 2024
Merged

[Configuration](transactional-hive) Add skip_checking_acid_version_file session var to skip checking acid version file in some hive envs. #42111

morningman merged 1 commit into from
Oct 21, 2024

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Oct 18, 2024

Proposed changes

[Configuration] (transactional-hive) Add skip_checking_acid_version_file session var to skip checking acid version file in some hive envs.

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Oct 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

TPC-H: Total hot run time: 41652 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 486aff852e448418d1b1acbcb2b893018f2a0235, data reload: false

------ Round 1 ----------------------------------
q1	17573	7514	7369	7369
q2	2046	160	155	155
q3	10745	1161	1176	1161
q4	10489	884	825	825
q5	7743	3119	3090	3090
q6	238	153	150	150
q7	1048	606	606	606
q8	9372	1973	1959	1959
q9	6726	6444	6404	6404
q10	7065	2430	2472	2430
q11	452	238	241	238
q12	450	222	227	222
q13	17768	3033	2996	2996
q14	265	214	221	214
q15	566	523	515	515
q16	674	586	577	577
q17	996	558	521	521
q18	7431	6728	6818	6728
q19	1352	1066	1005	1005
q20	486	192	189	189
q21	4032	3322	3270	3270
q22	1124	1030	1028	1028
Total cold run time: 108641 ms
Total hot run time: 41652 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7354	7297	7285	7285
q2	334	229	229	229
q3	2909	2812	2834	2812
q4	1929	1690	1699	1690
q5	5512	5496	5512	5496
q6	224	144	144	144
q7	2125	1726	1733	1726
q8	3236	3448	3461	3448
q9	8656	8593	8598	8593
q10	3495	3499	3443	3443
q11	580	487	492	487
q12	785	591	607	591
q13	12064	3060	3036	3036
q14	293	263	267	263
q15	562	512	510	510
q16	653	661	626	626
q17	1824	1581	1565	1565
q18	7877	7411	7441	7411
q19	1677	1427	1481	1427
q20	2068	1830	1810	1810
q21	5490	5218	5256	5218
q22	1122	1032	1033	1032
Total cold run time: 70769 ms
Total hot run time: 58842 ms

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

TPC-DS: Total hot run time: 191908 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 486aff852e448418d1b1acbcb2b893018f2a0235, data reload: false

query1	964	396	374	374
query2	5946	2064	1984	1984
query3	5875	213	221	213
query4	33912	23444	23370	23370
query5	3970	466	460	460
query6	246	165	164	164
query7	4316	292	288	288
query8	261	220	230	220
query9	9033	2744	2744	2744
query10	466	271	270	270
query11	18038	15313	15085	15085
query12	149	102	98	98
query13	1519	443	434	434
query14	10105	7409	7433	7409
query15	290	175	175	175
query16	7957	460	484	460
query17	1496	573	556	556
query18	1973	313	329	313
query19	358	151	149	149
query20	121	111	106	106
query21	214	106	105	105
query22	4305	4392	4114	4114
query23	35153	34194	34124	34124
query24	11249	2786	2855	2786
query25	652	401	419	401
query26	1631	170	159	159
query27	2672	285	290	285
query28	7662	2450	2471	2450
query29	999	423	428	423
query30	324	166	152	152
query31	1032	821	828	821
query32	101	57	58	57
query33	768	301	300	300
query34	935	509	506	506
query35	950	749	768	749
query36	1095	956	957	956
query37	164	92	90	90
query38	4030	3950	4066	3950
query39	1484	1416	1434	1416
query40	281	102	102	102
query41	50	48	48	48
query42	120	104	104	104
query43	523	468	460	460
query44	1246	805	811	805
query45	198	168	170	168
query46	1152	698	698	698
query47	1941	1843	1841	1841
query48	424	312	328	312
query49	1244	439	436	436
query50	819	379	385	379
query51	7087	7160	7059	7059
query52	103	88	87	87
query53	251	181	182	181
query54	1049	471	425	425
query55	80	78	80	78
query56	286	293	263	263
query57	1376	1165	1192	1165
query58	245	240	247	240
query59	3347	3039	2975	2975
query60	297	269	253	253
query61	105	104	102	102
query62	864	693	674	674
query63	223	187	182	182
query64	4943	655	600	600
query65	3331	3237	3213	3213
query66	1404	309	305	305
query67	16048	15693	15646	15646
query68	4220	563	558	558
query69	466	285	285	285
query70	1215	1126	1154	1126
query71	341	309	280	280
query72	7181	4030	3971	3971
query73	772	352	368	352
query74	10024	9082	9073	9073
query75	3430	2695	2706	2695
query76	3010	963	883	883
query77	438	297	297	297
query78	10564	9660	9557	9557
query79	1296	598	588	588
query80	958	450	448	448
query81	571	241	242	241
query82	887	147	140	140
query83	247	136	134	134
query84	249	71	68	68
query85	1271	291	286	286
query86	375	302	309	302
query87	4478	4327	4376	4327
query88	3271	2219	2184	2184
query89	400	290	296	290
query90	1971	190	184	184
query91	138	102	104	102
query92	68	49	47	47
query93	1071	539	534	534
query94	1065	301	291	291
query95	353	249	247	247
query96	617	280	284	280
query97	3242	3179	3139	3139
query98	217	196	196	196
query99	1532	1286	1288	1286
Total cold run time: 297390 ms
Total hot run time: 191908 ms

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

ClickBench: Total hot run time: 32.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 486aff852e448418d1b1acbcb2b893018f2a0235, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.52	0.53	0.53
query6	1.13	0.74	0.72
query7	0.03	0.02	0.02
query8	0.03	0.03	0.03
query9	0.56	0.51	0.54
query10	0.54	0.53	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.10
query13	0.62	0.60	0.60
query14	2.72	2.76	2.81
query15	0.91	0.83	0.84
query16	0.40	0.38	0.38
query17	1.02	1.03	1.06
query18	0.21	0.19	0.20
query19	1.88	1.90	1.96
query20	0.01	0.01	0.01
query21	15.37	0.60	0.59
query22	2.53	1.88	1.85
query23	17.01	1.10	0.82
query24	2.62	0.57	1.47
query25	0.12	0.15	0.06
query26	0.52	0.14	0.14
query27	0.04	0.06	0.04
query28	11.05	1.10	1.06
query29	12.65	3.19	3.21
query30	0.24	0.06	0.06
query31	2.88	0.38	0.39
query32	3.25	0.46	0.45
query33	3.05	3.00	3.01
query34	17.23	4.46	4.47
query35	4.51	4.49	4.59
query36	0.68	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.06 s
Total hot run time: 32.42 s

@kaka11chen kaka11chen force-pushed the add_skip_acid_file_transactional_hive branch from 486aff8 to 8c06dab Compare October 21, 2024 12:42
@kaka11chen
[Configuration](transactional-hive) Add `skip_checking_acid_version_f…
4744029 
…ile` session var to skip checking acid version file in some hive envs.
@kaka11chen kaka11chen force-pushed the add_skip_acid_file_transactional_hive branch from 8c06dab to 4744029 Compare October 21, 2024 12:48
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Oct 21, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 21, 2024

TPC-H: Total hot run time: 41202 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4744029204bd7ad0bb91b675c438cb5a4dd65513, data reload: false

------ Round 1 ----------------------------------
q1	17574	7486	7310	7310
q2	2056	154	153	153
q3	10585	1139	1161	1139
q4	10235	877	857	857
q5	7738	3091	3081	3081
q6	234	151	150	150
q7	998	616	600	600
q8	9367	1882	1993	1882
q9	6552	6386	6399	6386
q10	7009	2442	2454	2442
q11	453	246	248	246
q12	403	215	224	215
q13	17788	3037	3011	3011
q14	255	231	229	229
q15	568	529	524	524
q16	644	582	588	582
q17	978	546	610	546
q18	7294	6689	6758	6689
q19	1337	918	915	915
q20	464	186	184	184
q21	3955	3080	3180	3080
q22	1131	981	1023	981
Total cold run time: 107618 ms
Total hot run time: 41202 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7302	7247	7266	7247
q2	325	238	226	226
q3	2907	2790	2805	2790
q4	1940	1725	1737	1725
q5	5402	5475	5527	5475
q6	226	142	142	142
q7	2107	1714	1760	1714
q8	3276	3415	3414	3414
q9	8607	8615	8566	8566
q10	3499	3486	3446	3446
q11	580	480	483	480
q12	798	650	588	588
q13	9910	3063	3022	3022
q14	290	273	262	262
q15	556	518	518	518
q16	673	624	611	611
q17	1820	1613	1530	1530
q18	7797	7521	7326	7326
q19	1685	1485	1470	1470
q20	2040	1838	1828	1828
q21	5385	5166	5219	5166
q22	1111	1019	1013	1013
Total cold run time: 68236 ms
Total hot run time: 58559 ms

@doris-robot
Copy link

doris-robot commented Oct 21, 2024

TPC-DS: Total hot run time: 192521 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4744029204bd7ad0bb91b675c438cb5a4dd65513, data reload: false

query1	978	370	401	370
query2	6527	2150	2131	2131
query3	6783	212	220	212
query4	34048	23640	23574	23574
query5	4297	466	453	453
query6	290	179	182	179
query7	4595	296	293	293
query8	291	257	238	238
query9	9537	2754	2749	2749
query10	450	284	290	284
query11	18370	15459	15282	15282
query12	158	102	101	101
query13	1652	443	428	428
query14	11815	7507	7492	7492
query15	314	170	172	170
query16	7968	441	460	441
query17	1719	581	555	555
query18	2040	298	315	298
query19	362	157	150	150
query20	119	110	110	110
query21	215	109	101	101
query22	4543	4339	4372	4339
query23	35036	34129	34199	34129
query24	11156	2769	2777	2769
query25	673	409	402	402
query26	1200	161	162	161
query27	2436	289	295	289
query28	7839	2443	2441	2441
query29	848	426	425	425
query30	331	165	157	157
query31	1039	809	815	809
query32	100	56	58	56
query33	766	302	303	302
query34	923	513	518	513
query35	923	764	748	748
query36	1098	943	948	943
query37	159	94	95	94
query38	4121	4009	4039	4009
query39	1481	1410	1417	1410
query40	215	103	102	102
query41	50	46	48	46
query42	120	102	102	102
query43	547	510	508	508
query44	1283	808	798	798
query45	197	165	170	165
query46	1145	711	708	708
query47	1914	1816	1814	1814
query48	420	315	334	315
query49	1187	441	419	419
query50	828	392	412	392
query51	7383	7152	6893	6893
query52	102	88	90	88
query53	259	184	179	179
query54	1114	427	448	427
query55	81	80	74	74
query56	270	310	257	257
query57	1312	1172	1154	1154
query58	250	240	244	240
query59	3273	3034	3088	3034
query60	302	271	258	258
query61	113	103	108	103
query62	849	640	683	640
query63	213	196	187	187
query64	4143	649	618	618
query65	3296	3242	3187	3187
query66	899	324	304	304
query67	16131	15972	15430	15430
query68	4893	570	582	570
query69	468	286	286	286
query70	1230	1108	1114	1108
query71	356	276	279	276
query72	7076	4030	3984	3984
query73	797	366	363	363
query74	10026	9114	9013	9013
query75	3413	2719	2675	2675
query76	3155	910	916	910
query77	447	296	296	296
query78	10598	9639	9610	9610
query79	1152	602	598	598
query80	812	460	467	460
query81	587	247	245	245
query82	649	146	135	135
query83	229	139	137	137
query84	258	72	70	70
query85	1629	303	292	292
query86	480	291	298	291
query87	4612	4404	4358	4358
query88	3775	2208	2174	2174
query89	399	292	291	291
query90	1999	184	187	184
query91	143	105	108	105
query92	63	47	49	47
query93	1713	551	543	543
query94	1033	287	280	280
query95	360	249	250	249
query96	614	278	278	278
query97	3296	3141	3144	3141
query98	216	208	196	196
query99	1684	1314	1307	1307
Total cold run time: 303258 ms
Total hot run time: 192521 ms

@doris-robot
Copy link

doris-robot commented Oct 21, 2024

ClickBench: Total hot run time: 32.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4744029204bd7ad0bb91b675c438cb5a4dd65513, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.52	0.51	0.52
query6	1.14	0.71	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.05
query9	0.58	0.50	0.50
query10	0.53	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.10
query13	0.60	0.60	0.60
query14	2.85	2.84	2.73
query15	0.91	0.82	0.82
query16	0.40	0.37	0.38
query17	1.06	1.00	0.99
query18	0.24	0.22	0.22
query19	1.94	1.84	1.98
query20	0.01	0.01	0.02
query21	15.38	0.60	0.57
query22	2.70	1.68	1.50
query23	17.19	0.92	0.81
query24	3.01	1.60	2.08
query25	0.12	0.20	0.08
query26	0.60	0.13	0.14
query27	0.04	0.05	0.04
query28	9.53	1.11	1.08
query29	12.53	3.29	3.26
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.28	0.45	0.46
query33	2.97	3.05	3.02
query34	17.20	4.47	4.44
query35	4.52	4.48	4.49
query36	0.65	0.49	0.48
query37	0.08	0.06	0.05
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.41 s
Total hot run time: 32.99 s

morningman
morningman approved these changes Oct 21, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 21, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2024

PR approved by anyone and no changes requested.

@morningman morningman merged commit f46946e into apache:master Oct 21, 2024
@morningman morningman mentioned this pull request Oct 21, 2024
Merged
morningman pushed a commit to morningman/doris that referenced this pull request Oct 21, 2024
@kaka11chen @morningman
[Configuration](transactional-hive) Add `skip_checking_acid_version_f…
35906a8 
…ile` session var to skip checking acid version file in some hive envs. (apache#42111)

[Configuration] (transactional-hive) Add
`skip_checking_acid_version_file` session var to skip checking acid
version file in some hive envs.
@morningman morningman mentioned this pull request Oct 21, 2024
Merged
morningman added a commit that referenced this pull request Oct 21, 2024
@morningman @kaka11chen
[Configuration](transactional-hive) Add `skip_checking_acid_version_f…
701ebe5 
…ile` session var to skip checking acid version file in some hive envs. #42111 #42225 (#42226)

bp #42111 #42225

---------

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
morningman added a commit that referenced this pull request Oct 22, 2024
@morningman
[fix](hive-transaction) skip_checking_acid_version_file need to be fo…
bd2d846 
…rwarded (#42225)

followup #42111
the skip_checking_acid_version_file need to be forwarded
morningman added a commit to morningman/doris that referenced this pull request Oct 30, 2024
@morningman
[fix](hive-transaction) skip_checking_acid_version_file need to be fo…
f81d7bd 
…rwarded (apache#42225)

followup apache#42111
the skip_checking_acid_version_file need to be forwarded
morningman pushed a commit to morningman/doris that referenced this pull request Oct 30, 2024
@kaka11chen @morningman
[Configuration](transactional-hive) Add `skip_checking_acid_version_f…
e142a2d 
…ile` session var to skip checking acid version file in some hive envs. (apache#42111)

[Configuration] (transactional-hive) Add
`skip_checking_acid_version_file` session var to skip checking acid
version file in some hive envs.
@morningman morningman mentioned this pull request Oct 30, 2024
Merged
morningman added a commit that referenced this pull request Oct 31, 2024
@morningman @kaka11chen
[Configuration](transactional-hive) Add `skip_checking_acid_version_f…
fce4695 
…ile` session var to skip checking acid version file in some hive envs. (#42111)(#42225) (#42939)

cherry-pick (#42111)(#42225)

---------

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
hubgeter pushed a commit to hubgeter/doris that referenced this pull request Nov 4, 2024
@kaka11chen @hubgeter
[Configuration](transactional-hive) Add `skip_checking_acid_version_f…
c2d0ffd 
…ile` session var to skip checking acid version file in some hive envs. (apache#42111)

[Configuration] (transactional-hive) Add
`skip_checking_acid_version_file` session var to skip checking acid
version file in some hive envs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+1 more reviewer

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kaka11chen @doris-robot @morningman @hubgeter