Skip to content

[improvement](statistics)Disable fetch stats for iceberg table through Iceberg api by default. #36931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jibing-Li merged 1 commit into from
Jun 28, 2024
Merged

[improvement](statistics)Disable fetch stats for iceberg table through Iceberg api by default. #36931

Jibing-Li merged 1 commit into from
Jun 28, 2024

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Jun 27, 2024

Disable fetch iceberg table stats through iceberg api by default. Because it could be very heavy and external table performs fine without column stats.
Could enable it by : set global enable_fetch_iceberg_stats=true

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review June 27, 2024 09:36
morningman
morningman approved these changes Jun 27, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 27, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 27, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

TPC-H: Total hot run time: 40076 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e39ab9d1ba056d749db9fbbc75b450be56a6c48e, data reload: false

------ Round 1 ----------------------------------
q1	17866	4682	4363	4363
q2	2938	194	200	194
q3	11112	1248	1203	1203
q4	10555	845	763	763
q5	7618	2691	2644	2644
q6	224	138	139	138
q7	1059	617	620	617
q8	9557	2092	2119	2092
q9	8771	6491	6460	6460
q10	9005	3710	3725	3710
q11	447	244	238	238
q12	445	234	226	226
q13	18811	2997	3049	2997
q14	255	222	228	222
q15	523	483	479	479
q16	535	391	378	378
q17	952	667	642	642
q18	8051	7472	7373	7373
q19	6087	1497	1486	1486
q20	664	327	327	327
q21	5022	3191	3283	3191
q22	391	335	333	333
Total cold run time: 120888 ms
Total hot run time: 40076 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4248	4247	4247
q2	375	270	268	268
q3	2990	2747	2718	2718
q4	1841	1636	1619	1619
q5	5250	5287	5281	5281
q6	218	127	130	127
q7	2120	1721	1732	1721
q8	3175	3339	3307	3307
q9	8290	8302	8317	8302
q10	3842	3707	3637	3637
q11	579	496	477	477
q12	769	590	628	590
q13	16295	3014	3010	3010
q14	300	260	260	260
q15	505	486	472	472
q16	453	417	427	417
q17	1774	1484	1504	1484
q18	7604	7511	7543	7511
q19	1754	1652	1602	1602
q20	1985	1792	1786	1786
q21	4961	4803	4763	4763
q22	624	535	547	535
Total cold run time: 70048 ms
Total hot run time: 54134 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

TPC-DS: Total hot run time: 173433 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e39ab9d1ba056d749db9fbbc75b450be56a6c48e, data reload: false

query1	915	389	372	372
query2	6480	2447	2503	2447
query3	6662	211	222	211
query4	18963	17625	17359	17359
query5	4206	482	486	482
query6	264	169	173	169
query7	4598	289	292	289
query8	304	276	281	276
query9	8456	2342	2312	2312
query10	598	276	290	276
query11	10734	10213	10190	10190
query12	133	88	83	83
query13	1666	373	355	355
query14	8923	6206	7861	6206
query15	241	191	193	191
query16	7876	271	270	270
query17	1901	555	517	517
query18	1986	290	278	278
query19	198	154	145	145
query20	89	82	81	81
query21	209	145	133	133
query22	4553	4237	4131	4131
query23	33837	33333	33402	33333
query24	10897	2794	2878	2794
query25	650	365	359	359
query26	1686	152	147	147
query27	3060	311	311	311
query28	7749	2030	2016	2016
query29	998	638	596	596
query30	280	151	152	151
query31	972	761	770	761
query32	92	51	62	51
query33	767	298	290	290
query34	953	468	466	466
query35	760	636	628	628
query36	1047	957	957	957
query37	168	72	68	68
query38	2896	2780	2810	2780
query39	838	776	811	776
query40	273	123	124	123
query41	50	56	53	53
query42	117	96	102	96
query43	596	530	548	530
query44	1283	730	736	730
query45	201	166	162	162
query46	1068	725	699	699
query47	1873	1808	1795	1795
query48	366	311	318	311
query49	1201	448	403	403
query50	791	385	393	385
query51	6877	6827	6829	6827
query52	103	88	92	88
query53	360	283	282	282
query54	1016	444	436	436
query55	75	73	74	73
query56	277	263	272	263
query57	1204	1050	1062	1050
query58	268	235	260	235
query59	3294	3183	3133	3133
query60	293	275	272	272
query61	92	91	92	91
query62	664	454	465	454
query63	326	321	284	284
query64	9929	2207	1718	1718
query65	3262	3096	3084	3084
query66	1387	337	329	329
query67	15502	14971	15057	14971
query68	4589	535	535	535
query69	479	327	323	323
query70	1203	1136	1088	1088
query71	388	293	284	284
query72	7250	5726	5721	5721
query73	738	327	323	323
query74	6181	5842	5581	5581
query75	3423	2651	2648	2648
query76	2576	949	956	949
query77	467	310	312	310
query78	10570	9930	9817	9817
query79	2524	522	512	512
query80	1156	482	462	462
query81	541	224	218	218
query82	798	102	103	102
query83	234	169	171	169
query84	230	89	87	87
query85	2020	281	272	272
query86	502	315	314	314
query87	3308	3167	3110	3110
query88	4242	2453	2440	2440
query89	497	383	376	376
query90	1811	189	186	186
query91	125	101	99	99
query92	59	49	49	49
query93	2406	503	492	492
query94	1276	187	190	187
query95	406	311	317	311
query96	594	269	285	269
query97	3228	3136	3040	3040
query98	217	205	204	204
query99	1208	821	842	821
Total cold run time: 275599 ms
Total hot run time: 173433 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e39ab9d1ba056d749db9fbbc75b450be56a6c48e, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.08
query5	0.49	0.50	0.50
query6	1.15	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.51
query10	0.54	0.55	0.53
query11	0.15	0.11	0.12
query12	0.16	0.13	0.13
query13	0.60	0.59	0.61
query14	0.78	0.80	0.78
query15	0.83	0.81	0.81
query16	0.36	0.36	0.38
query17	0.98	0.99	1.02
query18	0.21	0.27	0.21
query19	1.87	1.76	1.75
query20	0.01	0.01	0.01
query21	15.44	0.75	0.66
query22	3.89	7.00	2.38
query23	18.28	1.34	1.27
query24	2.15	0.22	0.22
query25	0.16	0.09	0.09
query26	0.26	0.17	0.16
query27	0.08	0.07	0.09
query28	13.23	1.03	1.00
query29	12.63	3.30	3.26
query30	0.25	0.07	0.06
query31	2.86	0.40	0.38
query32	3.27	0.49	0.47
query33	2.86	2.90	2.93
query34	17.02	4.46	4.49
query35	4.50	4.52	4.51
query36	0.65	0.46	0.46
query37	0.17	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.02 s
Total hot run time: 31.21 s

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 27, 2024

run feut

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 28, 2024

run p0

@Jibing-Li Jibing-Li merged commit 89bbad9 into apache:master Jun 28, 2024
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 28, 2024
@Jibing-Li
[improvement](statistics)Disable fetch stats for iceberg table throug…
e773eb7 
…h Iceberg api by default. (apache#36931)

Disable fetch iceberg table stats through iceberg api by default.
Because it could be very heavy and external table performs fine without
column stats.
Could enable it by : set global enable_fetch_iceberg_stats=true
@Jibing-Li Jibing-Li mentioned this pull request Jun 28, 2024
Merged
Jibing-Li added a commit that referenced this pull request Jun 28, 2024
@Jibing-Li
[improvement](statistics)Disable fetch stats for iceberg table throug…
65e8bbf 
…h Iceberg api by default. (#36931) (#36999)

backport #36931
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@Jibing-Li @dataroaring
[improvement](statistics)Disable fetch stats for iceberg table throug…
b3d4066 
…h Iceberg api by default. (#36931)

Disable fetch iceberg table stats through iceberg api by default.
Because it could be very heavy and external table performs fine without
column stats.
Could enable it by : set global enable_fetch_iceberg_stats=true
@Jibing-Li Jibing-Li deleted the iceberg branch June 28, 2024 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Jibing-Li @doris-robot @morningman @wuwenchi @dataroaring