Skip to content

[Fix](parquet-reader) Fix Timestamp Int96 min-max statistics is incorrect when was written by some old parquet writers by disable it. #35160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 22, 2024
Merged

[Fix](parquet-reader) Fix Timestamp Int96 min-max statistics is incorrect when was written by some old parquet writers by disable it. #35160

morningman merged 1 commit into from
May 22, 2024

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented May 21, 2024

Proposed changes

backport #35041

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaka11chen
[Fix](parquet-reader) Fix Timestamp Int96 min-max statistics is incor…
360ffad 
…rect when was written by some old parquet writers by disable it. (apache#35041)

Parquet INT96 timestamp values were compared incorrectly for the purposes of producing statistics
by older parquet writers, so PARQUET-1065 deprecated them. The result is that any writer that produced
stats was producing unusable incorrect values, except the special case where min == max and an incorrect
ordering would not be material to the result. PARQUET-1026 made binary stats available and valid in that special case.
@doris-robot
Copy link

doris-robot commented May 21, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaka11chen
Copy link
Contributor Author

kaka11chen commented May 21, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 21, 2024

TPC-H: Total hot run time: 49329 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 360ffade3b7b657c87aa5d1f0a143b5f176331cb, data reload: false

------ Round 1 ----------------------------------
q1	18185	4389	4328	4328
q2	2029	159	146	146
q3	10457	1899	1898	1898
q4	10320	1243	1320	1243
q5	8571	3839	3885	3839
q6	228	125	126	125
q7	2008	1579	1592	1579
q8	9547	2727	2687	2687
q9	13634	10210	10158	10158
q10	8618	3511	3509	3509
q11	413	233	248	233
q12	470	297	302	297
q13	18350	3962	4022	3962
q14	350	332	325	325
q15	512	475	448	448
q16	680	575	588	575
q17	1130	970	881	881
q18	7156	6795	6827	6795
q19	1712	1635	1546	1546
q20	554	312	307	307
q21	4406	4057	4074	4057
q22	492	397	391	391
Total cold run time: 119822 ms
Total hot run time: 49329 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4294	4279	4279
q2	321	218	225	218
q3	4172	4146	4143	4143
q4	2729	2730	2723	2723
q5	7186	7033	7068	7033
q6	233	120	118	118
q7	3251	2840	2849	2840
q8	4359	4471	4485	4471
q9	16937	16860	16698	16698
q10	4184	4245	4312	4245
q11	754	698	677	677
q12	1022	861	882	861
q13	7171	3739	3730	3730
q14	450	423	419	419
q15	505	458	453	453
q16	755	686	679	679
q17	3905	3815	3804	3804
q18	8726	8892	8724	8724
q19	1740	1693	1664	1664
q20	2370	2153	2107	2107
q21	8527	8567	8475	8475
q22	1035	960	989	960
Total cold run time: 84698 ms
Total hot run time: 79321 ms

@doris-robot
Copy link

doris-robot commented May 21, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8078/21369)
Line Coverage: 29.46% (65963/223908)
Region Coverage: 28.93% (33963/117391)
Branch Coverage: 24.78% (17427/70318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/360ffade3b7b657c87aa5d1f0a143b5f176331cb_360ffade3b7b657c87aa5d1f0a143b5f176331cb/report/index.html

@doris-robot
Copy link

doris-robot commented May 21, 2024

TPC-DS: Total hot run time: 202810 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 360ffade3b7b657c87aa5d1f0a143b5f176331cb, data reload: false

query1	920	383	373	373
query2	6549	2980	2801	2801
query3	6920	209	211	209
query4	21009	17883	17928	17883
query5	19720	6511	6491	6491
query6	285	224	227	224
query7	4152	296	302	296
query8	283	267	273	267
query9	3105	2657	2603	2603
query10	405	323	298	298
query11	11299	10655	10709	10655
query12	122	75	71	71
query13	5586	678	648	648
query14	17438	13301	13382	13301
query15	367	214	228	214
query16	6461	273	252	252
query17	1739	1449	871	871
query18	2324	413	418	413
query19	201	145	146	145
query20	80	77	82	77
query21	188	93	90	90
query22	5338	5093	5121	5093
query23	32555	31993	31793	31793
query24	6971	6484	6504	6484
query25	517	417	409	409
query26	536	159	158	158
query27	1912	291	300	291
query28	6083	2350	2315	2315
query29	2858	2675	2791	2675
query30	245	163	162	162
query31	918	725	741	725
query32	71	66	60	60
query33	399	256	255	255
query34	835	484	495	484
query35	1124	875	898	875
query36	1292	1457	1267	1267
query37	85	59	61	59
query38	3098	2943	2934	2934
query39	1398	1311	1325	1311
query40	212	92	97	92
query41	37	38	36	36
query42	80	84	80	80
query43	758	765	760	760
query44	1120	718	717	717
query45	243	228	230	228
query46	1222	971	985	971
query47	1806	1696	1716	1696
query48	1014	698	703	698
query49	619	362	370	362
query50	888	612	630	612
query51	4753	4630	4656	4630
query52	83	88	81	81
query53	437	322	320	320
query54	2641	2472	2443	2443
query55	91	85	80	80
query56	238	214	218	214
query57	1172	1167	1104	1104
query58	218	196	180	180
query59	4152	4178	3767	3767
query60	205	216	215	215
query61	88	82	86	82
query62	878	487	463	463
query63	481	346	332	332
query64	2478	1502	1461	1461
query65	3648	3542	3542	3542
query66	805	376	383	376
query67	16641	16653	15221	15221
query68	9001	656	663	656
query69	575	341	347	341
query70	1548	1345	1598	1345
query71	407	307	301	301
query72	6433	3529	3464	3464
query73	738	331	321	321
query74	6250	5786	5869	5786
query75	5292	3592	3702	3592
query76	5612	1129	1195	1129
query77	950	248	252	248
query78	12460	12307	11642	11642
query79	11173	624	642	624
query80	783	390	389	389
query81	496	239	228	228
query82	883	97	92	92
query83	167	133	129	129
query84	259	67	67	67
query85	811	302	299	299
query86	335	292	292	292
query87	3228	3019	3029	3019
query88	4890	2340	2360	2340
query89	428	299	304	299
query90	1954	208	207	207
query91	166	133	136	133
query92	61	51	53	51
query93	5263	617	567	567
query94	730	204	203	203
query95	1082	1062	1053	1053
query96	653	339	331	331
query97	6374	6368	6312	6312
query98	200	189	168	168
query99	3051	913	887	887
Total cold run time: 314649 ms
Total hot run time: 202810 ms

@doris-robot
Copy link

doris-robot commented May 21, 2024

ClickBench: Total hot run time: 30.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 360ffade3b7b657c87aa5d1f0a143b5f176331cb, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.04	0.04
query4	1.80	0.08	0.08
query5	0.53	0.52	0.52
query6	1.23	0.60	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.49	0.47
query10	0.55	0.53	0.52
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.77	0.80	0.79
query15	0.78	0.76	0.76
query16	0.38	0.36	0.37
query17	1.01	0.97	0.97
query18	0.23	0.25	0.25
query19	1.92	1.86	1.84
query20	0.02	0.01	0.01
query21	15.47	0.55	0.55
query22	1.96	2.53	1.60
query23	17.14	1.12	1.11
query24	5.96	0.98	1.26
query25	0.33	0.12	0.06
query26	0.66	0.15	0.16
query27	0.04	0.03	0.04
query28	6.94	0.76	0.71
query29	12.68	2.40	2.28
query30	0.60	0.53	0.48
query31	2.81	0.39	0.38
query32	3.37	0.49	0.50
query33	3.08	3.04	3.07
query34	15.24	4.81	4.78
query35	4.88	4.84	4.84
query36	1.06	1.01	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.59 s
Total hot run time: 30.78 s

@doris-robot
Copy link

doris-robot commented May 21, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 360ffade3b7b657c87aa5d1f0a143b5f176331cb with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s

@morningman morningman merged commit c13f11a into apache:branch-2.0 May 22, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@kaka11chen
[Fix](parquet-reader) Fix Timestamp Int96 min-max statistics is incor…
bbfb2ec 
…rect when was written by some old parquet writers by disable it. (apache#35041) (apache#35160)

backport apache#35041
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kaka11chen @doris-robot @morningman