Skip to content

[fix](parquet) prevent parquet page reader print much warning logs #37011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Jul 1, 2024
Merged

[fix](parquet) prevent parquet page reader print much warning logs #37011

morningman merged 1 commit into from
Jul 1, 2024

Conversation

@AshinGau
Copy link
Member

@AshinGau AshinGau commented Jun 28, 2024

Proposed changes

Prevent parquet page reader print much warning logs.

Couldn't deserialize thrift msg:

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@AshinGau
Copy link
Member Author

AshinGau commented Jun 28, 2024

run buidall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

clang-tidy review says "All clean, LGTM! 👍"

morningman
morningman approved these changes Jun 28, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman mentioned this pull request Jun 28, 2024
Merged
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by anyone and no changes requested.

@wm1581066 wm1581066 added the usercase Important user case type label label Jun 29, 2024
@AshinGau
Copy link
Member Author

AshinGau commented Jun 30, 2024

run buidall

1 similar comment
@AshinGau
Copy link
Member Author

AshinGau commented Jul 1, 2024

run buidall

@morningman
Copy link
Contributor

morningman commented Jul 1, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

TPC-H: Total hot run time: 39755 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 208b510d9ba95c108cbcd6fe832b7dec4c3f4c8c, data reload: false

------ Round 1 ----------------------------------
q1	17666	4436	4263	4263
q2	2029	195	195	195
q3	10513	1259	1053	1053
q4	10199	798	716	716
q5	7515	2717	2667	2667
q6	224	139	139	139
q7	954	597	594	594
q8	9240	2080	2080	2080
q9	8904	6504	6480	6480
q10	9053	3726	3741	3726
q11	473	244	233	233
q12	501	233	228	228
q13	17779	2974	2984	2974
q14	258	234	221	221
q15	520	484	485	484
q16	504	375	371	371
q17	964	736	641	641
q18	8122	7472	7476	7472
q19	6422	1500	1437	1437
q20	659	330	317	317
q21	4887	3904	3129	3129
q22	394	335	341	335
Total cold run time: 117780 ms
Total hot run time: 39755 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4223	4271	4223
q2	376	269	268	268
q3	2948	2887	2855	2855
q4	2008	1681	1768	1681
q5	5580	5487	5450	5450
q6	227	144	130	130
q7	2287	1876	1849	1849
q8	3305	3448	3436	3436
q9	8738	8732	8764	8732
q10	4104	3706	3786	3706
q11	611	503	502	502
q12	827	620	633	620
q13	17311	3143	3171	3143
q14	319	278	268	268
q15	538	489	481	481
q16	490	427	415	415
q17	1801	1543	1495	1495
q18	8195	7959	7813	7813
q19	2107	1747	1625	1625
q20	2923	1905	1866	1866
q21	5098	4816	4654	4654
q22	665	530	583	530
Total cold run time: 74871 ms
Total hot run time: 55742 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

TPC-DS: Total hot run time: 173825 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 208b510d9ba95c108cbcd6fe832b7dec4c3f4c8c, data reload: false

query1	932	386	379	379
query2	6485	2440	2244	2244
query3	6631	205	216	205
query4	19286	17340	17238	17238
query5	3741	480	459	459
query6	279	162	172	162
query7	4597	301	296	296
query8	322	315	303	303
query9	8738	2394	2399	2394
query10	576	314	284	284
query11	10593	10144	10059	10059
query12	118	86	85	85
query13	1661	370	371	370
query14	10379	7839	7437	7437
query15	243	182	189	182
query16	7825	278	276	276
query17	1878	566	545	545
query18	1675	289	274	274
query19	215	156	158	156
query20	95	85	83	83
query21	220	135	123	123
query22	4445	4063	4024	4024
query23	33842	33511	33784	33511
query24	10835	2970	2895	2895
query25	612	418	389	389
query26	728	160	157	157
query27	2283	326	320	320
query28	5619	2163	2138	2138
query29	891	668	648	648
query30	274	169	167	167
query31	994	757	757	757
query32	99	55	57	55
query33	678	311	297	297
query34	869	485	503	485
query35	756	660	650	650
query36	1127	974	980	974
query37	138	78	81	78
query38	2986	2806	2806	2806
query39	880	858	827	827
query40	216	138	131	131
query41	62	54	62	54
query42	120	105	114	105
query43	594	541	540	540
query44	1086	741	731	731
query45	195	169	173	169
query46	1083	730	712	712
query47	1832	1798	1779	1779
query48	378	307	319	307
query49	891	513	417	417
query50	766	381	382	381
query51	6889	6769	6805	6769
query52	106	96	96	96
query53	363	285	291	285
query54	914	439	450	439
query55	74	75	72	72
query56	292	259	259	259
query57	1108	1028	1043	1028
query58	239	242	259	242
query59	3260	3165	3239	3165
query60	313	287	287	287
query61	89	91	90	90
query62	603	439	443	439
query63	322	283	288	283
query64	8638	2271	1759	1759
query65	3154	3131	3093	3093
query66	736	341	331	331
query67	15681	14855	14908	14855
query68	8632	529	542	529
query69	762	440	351	351
query70	1392	1052	1133	1052
query71	519	283	272	272
query72	8960	5604	5243	5243
query73	2256	319	324	319
query74	5917	5406	5483	5406
query75	4939	2633	2682	2633
query76	5163	1024	930	930
query77	772	321	302	302
query78	10508	9714	9779	9714
query79	8716	510	515	510
query80	1090	465	453	453
query81	547	220	213	213
query82	425	109	103	103
query83	328	170	170	170
query84	275	82	84	82
query85	886	273	274	273
query86	357	296	322	296
query87	3270	3048	3111	3048
query88	4706	2359	2334	2334
query89	532	390	384	384
query90	2047	186	187	186
query91	133	115	133	115
query92	61	47	49	47
query93	6747	502	495	495
query94	1350	189	184	184
query95	412	309	305	305
query96	611	265	259	259
query97	3160	3031	3057	3031
query98	224	203	200	200
query99	1151	836	857	836
Total cold run time: 291397 ms
Total hot run time: 173825 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

ClickBench: Total hot run time: 31.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 208b510d9ba95c108cbcd6fe832b7dec4c3f4c8c, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.48	0.49
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.55	0.54	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.60	0.60
query14	0.77	0.77	0.77
query15	0.85	0.81	0.81
query16	0.36	0.37	0.35
query17	1.00	0.98	1.00
query18	0.22	0.25	0.26
query19	1.78	1.70	1.71
query20	0.01	0.01	0.00
query21	15.43	0.74	0.66
query22	4.39	6.55	2.44
query23	18.29	1.38	1.28
query24	2.10	0.22	0.25
query25	0.17	0.08	0.08
query26	0.25	0.19	0.18
query27	0.08	0.08	0.08
query28	13.19	1.02	0.99
query29	12.68	3.32	3.31
query30	0.26	0.06	0.05
query31	2.88	0.39	0.40
query32	3.26	0.48	0.47
query33	2.91	2.85	2.89
query34	17.19	4.34	4.44
query35	4.47	4.46	4.43
query36	0.65	0.47	0.47
query37	0.19	0.15	0.17
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.63 s
Total hot run time: 31.01 s

@morningman morningman merged commit a2534a6 into apache:master Jul 1, 2024
morningman pushed a commit that referenced this pull request Jul 2, 2024
@AshinGau
[fix](parquet) prevent parquet page reader print much warning logs (#…
859a7c8 
…37012)

bp #37011
dataroaring pushed a commit that referenced this pull request Jul 2, 2024
@AshinGau @dataroaring
[fix](parquet) prevent parquet page reader print much warning logs (#…
2f8298e 
…37011)

Prevent parquet page reader print much warning logs.
```
Couldn't deserialize thrift msg:
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@AshinGau @doris-robot @morningman @zy-kkk @wm1581066 @dataroaring