Skip to content

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Nov 5, 2024

Remove useless stats validation check.
Before, we will not load the column stats when table rowCount > 0 and ndv == 0 and nullCount != rowCount. This is to avoid using invalid stats.
Now, we remove this validation because the planner side added validation to the column stats, (see #41790). Besides, after remove of the validation, it is easier to add regression test using stats injection.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review November 5, 2024 11:30
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41290 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6cdc043417bb002179154f3e7587baa57487b33, data reload: false

------ Round 1 ----------------------------------
q1	17598	7429	7295	7295
q2	2036	164	159	159
q3	10576	1109	1196	1109
q4	10578	865	784	784
q5	7747	3092	3100	3092
q6	235	147	145	145
q7	1011	603	605	603
q8	9343	1992	2010	1992
q9	6595	6437	6463	6437
q10	7069	2427	2468	2427
q11	451	243	263	243
q12	415	214	209	209
q13	17781	3004	2994	2994
q14	248	208	221	208
q15	580	521	516	516
q16	657	571	588	571
q17	969	576	516	516
q18	7286	6729	6631	6631
q19	1324	1035	1002	1002
q20	472	182	184	182
q21	4114	3234	3175	3175
q22	1116	1002	1000	1000
Total cold run time: 108201 ms
Total hot run time: 41290 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7304	7308	7295	7295
q2	323	227	232	227
q3	2970	2959	2973	2959
q4	2102	1922	1863	1863
q5	5717	5771	5845	5771
q6	220	142	140	140
q7	2336	1832	1826	1826
q8	3355	3473	3516	3473
q9	8888	8922	8879	8879
q10	3617	3868	3564	3564
q11	608	518	512	512
q12	814	652	654	652
q13	9527	3225	3248	3225
q14	308	285	291	285
q15	584	527	526	526
q16	668	655	640	640
q17	1857	1635	1646	1635
q18	8192	7799	7538	7538
q19	1701	1670	1408	1408
q20	2157	1860	1910	1860
q21	5556	5379	5413	5379
q22	1142	1057	1061	1057
Total cold run time: 69946 ms
Total hot run time: 60714 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196113 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6cdc043417bb002179154f3e7587baa57487b33, data reload: false

query1	1223	916	895	895
query2	6248	2042	2019	2019
query3	10786	3949	3915	3915
query4	67597	29800	23667	23667
query5	4816	490	441	441
query6	410	178	177	177
query7	5640	300	290	290
query8	314	225	222	222
query9	8795	2650	2641	2641
query10	462	267	261	261
query11	17438	15405	15858	15405
query12	164	104	106	104
query13	1574	433	436	433
query14	10102	6841	7835	6841
query15	212	200	193	193
query16	6971	469	449	449
query17	1009	563	583	563
query18	1788	299	295	295
query19	203	152	155	152
query20	118	118	118	118
query21	202	106	103	103
query22	4632	4453	4428	4428
query23	34672	34273	34246	34246
query24	5993	2777	2728	2728
query25	504	403	395	395
query26	647	162	160	160
query27	1665	288	285	285
query28	4163	2420	2386	2386
query29	687	436	439	436
query30	228	155	152	152
query31	989	815	801	801
query32	73	56	58	56
query33	409	273	264	264
query34	931	517	516	516
query35	862	731	747	731
query36	1081	960	970	960
query37	128	77	80	77
query38	4439	4255	4351	4255
query39	1459	1451	1447	1447
query40	203	102	101	101
query41	50	47	49	47
query42	110	101	98	98
query43	532	487	494	487
query44	1197	825	815	815
query45	188	166	170	166
query46	1139	713	695	695
query47	1954	1845	1882	1845
query48	432	320	330	320
query49	750	422	401	401
query50	831	400	405	400
query51	7192	7166	7120	7120
query52	95	89	85	85
query53	263	182	181	181
query54	540	398	389	389
query55	77	76	74	74
query56	252	234	232	232
query57	1261	1192	1153	1153
query58	219	217	204	204
query59	3266	2831	2998	2831
query60	280	247	245	245
query61	105	107	104	104
query62	823	678	671	671
query63	219	191	186	186
query64	1366	655	643	643
query65	3271	3206	3218	3206
query66	709	310	304	304
query67	15968	15726	15667	15667
query68	3181	569	572	569
query69	429	253	251	251
query70	1215	1156	1153	1153
query71	342	255	270	255
query72	6119	3943	4008	3943
query73	777	358	358	358
query74	10047	9063	9003	9003
query75	3386	2640	2672	2640
query76	1755	966	1122	966
query77	496	289	279	279
query78	10496	9358	9453	9358
query79	1490	606	590	590
query80	903	427	424	424
query81	523	239	243	239
query82	1294	121	111	111
query83	203	136	143	136
query84	282	69	71	69
query85	880	314	307	307
query86	335	305	280	280
query87	4803	4817	4696	4696
query88	3685	2184	2152	2152
query89	417	298	283	283
query90	2026	187	189	187
query91	136	102	103	102
query92	71	48	51	48
query93	1900	539	535	535
query94	851	289	298	289
query95	339	251	252	251
query96	615	278	280	278
query97	2887	2675	2740	2675
query98	221	199	194	194
query99	1565	1319	1315	1315
Total cold run time: 316595 ms
Total hot run time: 196113 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d6cdc043417bb002179154f3e7587baa57487b33, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.22	0.07	0.06
query4	1.64	0.10	0.10
query5	0.42	0.41	0.41
query6	1.15	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.57	0.49	0.49
query10	0.57	0.56	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.73	2.73	2.75
query15	0.89	0.83	0.83
query16	0.39	0.38	0.38
query17	1.00	1.02	1.02
query18	0.20	0.20	0.20
query19	1.96	1.80	1.78
query20	0.02	0.01	0.02
query21	15.38	0.58	0.61
query22	2.75	2.04	1.69
query23	17.02	0.92	0.80
query24	2.64	0.58	2.28
query25	0.18	0.04	0.26
query26	0.51	0.14	0.14
query27	0.07	0.04	0.04
query28	10.52	1.10	1.07
query29	12.54	3.21	3.20
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.29	0.45	0.45
query33	3.01	3.08	3.04
query34	17.01	4.53	4.44
query35	4.47	4.50	4.49
query36	0.70	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.51 s
Total hot run time: 31.89 s

@Jibing-Li
Copy link
Contributor Author

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 7, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2024

PR approved by anyone and no changes requested.

@englefly englefly merged commit 09280f8 into apache:master Nov 8, 2024
github-actions bot pushed a commit that referenced this pull request Nov 8, 2024
Remove useless stats validation check.
Before, we will not load the column stats when table rowCount > 0 and
ndv == 0 and nullCount != rowCount. This is to avoid using invalid
stats.
Now, we remove this validation because the planner side added validation
to the column stats, (see #41790).
Besides, after remove of the validation, it is easier to add regression
test using stats injection.
github-actions bot pushed a commit that referenced this pull request Nov 8, 2024
Remove useless stats validation check.
Before, we will not load the column stats when table rowCount > 0 and
ndv == 0 and nullCount != rowCount. This is to avoid using invalid
stats.
Now, we remove this validation because the planner side added validation
to the column stats, (see #41790).
Besides, after remove of the validation, it is easier to add regression
test using stats injection.
@Jibing-Li Jibing-Li deleted the ndvcheck branch November 11, 2024 02:01
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Nov 11, 2024
…he#43279)

Remove useless stats validation check.
Before, we will not load the column stats when table rowCount > 0 and
ndv == 0 and nullCount != rowCount. This is to avoid using invalid
stats.
Now, we remove this validation because the planner side added validation
to the column stats, (see apache#41790).
Besides, after remove of the validation, it is easier to add regression
test using stats injection.
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Nov 11, 2024
…he#43279)

Remove useless stats validation check.
Before, we will not load the column stats when table rowCount > 0 and
ndv == 0 and nullCount != rowCount. This is to avoid using invalid
stats.
Now, we remove this validation because the planner side added validation
to the column stats, (see apache#41790).
Besides, after remove of the validation, it is easier to add regression
test using stats injection.
Jibing-Li added a commit that referenced this pull request Nov 12, 2024
…check. (#43498)

Cherry-picked from #43279

Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Nov 12, 2024
…he#43279)

Remove useless stats validation check.
Before, we will not load the column stats when table rowCount > 0 and
ndv == 0 and nullCount != rowCount. This is to avoid using invalid
stats.
Now, we remove this validation because the planner side added validation
to the column stats, (see apache#41790).
Besides, after remove of the validation, it is easier to add regression
test using stats injection.
Jibing-Li added a commit that referenced this pull request Nov 12, 2024
…check. (#43499)

Cherry-picked from #43279

Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
englefly added a commit that referenced this pull request Nov 28, 2024
… invalid stats (#43501)

### What problem does this PR solve?
add test case for pr #41790.
this case is blocked by #43279
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants