Skip to content

[fix](nereids) bug: after is-null stats derive, other column stats are dropped #37809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 17, 2024
Merged

[fix](nereids) bug: after is-null stats derive, other column stats are dropped #37809

morrySnow merged 1 commit into from
Jul 17, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Jul 15, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly englefly mentioned this pull request Jul 15, 2024
Merged
@englefly
Copy link
Contributor Author

englefly commented Jul 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

TPC-H: Total hot run time: 40134 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e6a3489ec6ab499ca2fba958d520924d4fb644c, data reload: false

------ Round 1 ----------------------------------
q1	17923	4516	4353	4353
q2	2956	202	205	202
q3	11048	1261	1124	1124
q4	10662	814	800	800
q5	7589	2711	2699	2699
q6	224	144	146	144
q7	986	611	618	611
q8	9382	2054	2084	2054
q9	8594	6503	6501	6501
q10	8725	3796	3776	3776
q11	453	250	255	250
q12	408	239	228	228
q13	17775	2984	3005	2984
q14	295	240	240	240
q15	525	479	487	479
q16	525	405	379	379
q17	960	635	678	635
q18	8121	7491	7428	7428
q19	4490	1422	1503	1422
q20	658	328	325	325
q21	4876	3216	3222	3216
q22	352	294	284	284
Total cold run time: 117527 ms
Total hot run time: 40134 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4424	4239	4246	4239
q2	379	265	266	265
q3	3022	2838	2725	2725
q4	1917	1634	1580	1580
q5	5320	5326	5324	5324
q6	219	130	131	130
q7	2091	1736	1723	1723
q8	3188	3318	3318	3318
q9	8441	8376	8343	8343
q10	3907	3705	3649	3649
q11	580	486	495	486
q12	787	612	623	612
q13	17290	3026	3035	3026
q14	299	262	277	262
q15	516	489	483	483
q16	481	415	429	415
q17	1767	1488	1472	1472
q18	7598	7407	7282	7282
q19	2486	1546	1576	1546
q20	1979	1764	1813	1764
q21	5002	4679	4674	4674
q22	561	516	510	510
Total cold run time: 72254 ms
Total hot run time: 53828 ms

dataroaring pushed a commit that referenced this pull request Jul 15, 2024
@englefly
[fix](nereids) bug: after is-null stats derive, other column stats ar…
a207a79 
…e dropped #37809 (#37811)

## Proposed changes
pick from #37809
Issue Number: close #xxx

<!--Describe your changes.-->
@doris-robot
Copy link

doris-robot commented Jul 15, 2024

TPC-DS: Total hot run time: 172415 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e6a3489ec6ab499ca2fba958d520924d4fb644c, data reload: false

query1	920	364	363	363
query2	6454	1767	1769	1767
query3	6672	203	221	203
query4	28059	17367	17471	17367
query5	4261	481	461	461
query6	260	173	184	173
query7	4592	289	284	284
query8	242	193	194	193
query9	8394	2404	2366	2366
query10	456	305	270	270
query11	10700	9992	10049	9992
query12	130	92	82	82
query13	1652	378	363	363
query14	9686	7744	7647	7647
query15	232	170	164	164
query16	7840	325	322	322
query17	1778	557	541	541
query18	1972	283	284	283
query19	200	155	158	155
query20	91	81	92	81
query21	210	137	136	136
query22	4469	4050	4025	4025
query23	33788	33242	33085	33085
query24	12090	3015	2820	2820
query25	730	390	397	390
query26	1808	156	149	149
query27	2996	277	267	267
query28	7733	1970	1970	1970
query29	1168	632	628	628
query30	294	150	149	149
query31	991	759	761	759
query32	98	54	56	54
query33	783	318	314	314
query34	954	498	477	477
query35	710	606	563	563
query36	1122	957	930	930
query37	288	83	79	79
query38	2867	2747	2755	2747
query39	873	807	803	803
query40	279	119	123	119
query41	48	46	49	46
query42	117	99	102	99
query43	506	465	458	458
query44	1250	749	734	734
query45	198	165	163	163
query46	1097	693	738	693
query47	1846	1768	1758	1758
query48	379	310	294	294
query49	1202	420	437	420
query50	778	401	394	394
query51	6855	6798	6744	6744
query52	104	96	96	96
query53	365	300	302	300
query54	993	456	458	456
query55	78	80	79	79
query56	310	386	273	273
query57	1145	1061	1026	1026
query58	262	239	243	239
query59	3018	2614	2541	2541
query60	298	276	303	276
query61	114	98	94	94
query62	822	631	646	631
query63	320	303	295	295
query64	10496	2232	1684	1684
query65	3180	3126	3115	3115
query66	1377	368	374	368
query67	15479	15003	14832	14832
query68	8502	542	544	542
query69	797	446	370	370
query70	1288	1084	1115	1084
query71	542	287	278	278
query72	8573	5251	5859	5251
query73	1244	322	318	318
query74	6285	5682	5669	5669
query75	5387	2681	2689	2681
query76	4961	964	938	938
query77	816	327	319	319
query78	9693	9035	9884	9035
query79	10104	515	525	515
query80	986	483	481	481
query81	587	225	218	218
query82	429	129	138	129
query83	347	162	162	162
query84	277	84	86	84
query85	1024	364	302	302
query86	342	310	303	303
query87	3294	3163	3120	3120
query88	4435	2456	2455	2455
query89	553	389	371	371
query90	1775	193	194	193
query91	134	101	101	101
query92	60	52	50	50
query93	6888	497	494	494
query94	672	217	211	211
query95	396	321	319	319
query96	623	281	271	271
query97	3166	3031	3075	3031
query98	231	213	213	213
query99	1575	1258	1275	1258
Total cold run time: 307465 ms
Total hot run time: 172415 ms

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

ClickBench: Total hot run time: 30.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7e6a3489ec6ab499ca2fba958d520924d4fb644c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.10	0.09
query5	0.50	0.50	0.48
query6	1.13	0.73	0.72
query7	0.01	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.48	0.47
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.58
query14	0.75	0.79	0.78
query15	0.85	0.81	0.82
query16	0.37	0.37	0.36
query17	1.04	1.05	1.04
query18	0.22	0.22	0.21
query19	1.90	1.70	1.79
query20	0.01	0.01	0.01
query21	15.39	0.74	0.66
query22	4.02	8.25	1.60
query23	18.32	1.42	1.25
query24	2.12	0.22	0.21
query25	0.15	0.09	0.08
query26	0.29	0.20	0.21
query27	0.45	0.23	0.23
query28	13.30	1.01	1.00
query29	12.65	3.34	3.32
query30	0.27	0.06	0.05
query31	2.86	0.39	0.39
query32	3.28	0.47	0.47
query33	2.92	2.87	2.94
query34	16.77	4.39	4.31
query35	4.46	4.36	4.43
query36	0.65	0.47	0.50
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.48 s
Total hot run time: 30.24 s

@englefly englefly force-pushed the is-null-missing-column branch from 7e6a348 to 5298249 Compare July 15, 2024 10:03
@englefly englefly mentioned this pull request Jul 15, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 15, 2024
@englefly
[fix](nereids) bug: after is-null stats derive, other column stats ar…
74691b7 
…e dropped #37809 (#37832)

## Proposed changes
pick #37809

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly
Copy link
Contributor Author

englefly commented Jul 17, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 17, 2024

TPC-H: Total hot run time: 39594 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5298249c659055e87e17e17b812a6232d6d8c1e8, data reload: false

------ Round 1 ----------------------------------
q1	17606	4293	4297	4293
q2	2020	187	184	184
q3	10475	1137	1055	1055
q4	10198	676	905	676
q5	7523	2701	2672	2672
q6	217	137	135	135
q7	960	600	587	587
q8	9214	2053	2094	2053
q9	9006	6521	6519	6519
q10	8837	3786	3749	3749
q11	442	233	235	233
q12	430	215	225	215
q13	17769	2999	2999	2999
q14	279	238	233	233
q15	515	480	478	478
q16	497	379	370	370
q17	962	584	710	584
q18	7990	7562	7403	7403
q19	4527	1384	1458	1384
q20	688	321	326	321
q21	4992	3164	3293	3164
q22	339	297	287	287
Total cold run time: 115486 ms
Total hot run time: 39594 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4229	4221	4221
q2	384	258	267	258
q3	3046	2750	2855	2750
q4	2001	1684	1646	1646
q5	5585	5581	5517	5517
q6	230	144	135	135
q7	2203	1801	1866	1801
q8	3263	3410	3372	3372
q9	8773	8802	8737	8737
q10	4102	3934	3880	3880
q11	586	467	479	467
q12	799	607	636	607
q13	17020	3246	3124	3124
q14	331	276	300	276
q15	532	491	478	478
q16	490	444	426	426
q17	1801	1539	1496	1496
q18	8033	8084	7783	7783
q19	2286	1641	1555	1555
q20	2177	1862	1843	1843
q21	5141	4965	4782	4782
q22	605	495	521	495
Total cold run time: 73715 ms
Total hot run time: 55649 ms

@doris-robot
Copy link

doris-robot commented Jul 17, 2024

TPC-DS: Total hot run time: 174177 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5298249c659055e87e17e17b812a6232d6d8c1e8, data reload: false

query1	907	367	367	367
query2	6450	1864	1893	1864
query3	6785	215	222	215
query4	26054	17728	17312	17312
query5	3622	491	476	476
query6	263	178	167	167
query7	4582	291	296	291
query8	245	196	216	196
query9	8554	2373	2372	2372
query10	424	283	263	263
query11	11766	10220	10184	10184
query12	112	82	79	79
query13	1638	352	347	347
query14	10376	7556	7568	7556
query15	214	167	169	167
query16	7719	313	314	313
query17	1623	586	524	524
query18	1947	275	277	275
query19	195	143	148	143
query20	90	81	80	80
query21	202	125	123	123
query22	4506	4357	4249	4249
query23	33616	33868	33706	33706
query24	10932	2901	2851	2851
query25	589	395	368	368
query26	697	150	147	147
query27	2344	283	278	278
query28	6242	2066	2065	2065
query29	905	660	632	632
query30	252	152	154	152
query31	984	777	773	773
query32	95	55	55	55
query33	653	289	300	289
query34	889	507	501	501
query35	711	603	563	563
query36	1141	961	989	961
query37	148	84	86	84
query38	2982	2875	2873	2873
query39	865	806	811	806
query40	219	123	122	122
query41	47	45	47	45
query42	114	100	100	100
query43	499	463	476	463
query44	1064	719	736	719
query45	200	165	161	161
query46	1078	758	720	720
query47	1912	1848	1806	1806
query48	371	300	299	299
query49	877	416	436	416
query50	774	396	384	384
query51	6868	6867	6754	6754
query52	105	96	90	90
query53	356	287	291	287
query54	868	467	450	450
query55	75	78	78	78
query56	316	292	288	288
query57	1142	1059	1036	1036
query58	273	252	259	252
query59	2845	2926	2660	2660
query60	320	287	295	287
query61	120	115	116	115
query62	774	658	654	654
query63	317	290	288	288
query64	9220	2293	1769	1769
query65	3206	3115	3122	3115
query66	762	342	339	339
query67	15453	14959	15261	14959
query68	4571	548	541	541
query69	693	418	359	359
query70	1162	1155	1072	1072
query71	471	281	284	281
query72	8551	5440	6203	5440
query73	788	326	323	323
query74	6101	5776	5794	5776
query75	3774	2666	2685	2666
query76	3314	916	921	916
query77	628	295	297	295
query78	9720	9057	9202	9057
query79	4025	520	517	517
query80	3161	482	474	474
query81	601	226	221	221
query82	1455	136	134	134
query83	311	170	176	170
query84	280	89	83	83
query85	1475	309	298	298
query86	466	301	282	282
query87	3292	3149	3191	3149
query88	4826	2354	2369	2354
query89	499	377	372	372
query90	1896	190	187	187
query91	137	98	99	98
query92	64	48	50	48
query93	5069	497	504	497
query94	1239	211	211	211
query95	395	322	321	321
query96	613	273	265	265
query97	3256	3003	3003	3003
query98	221	199	192	192
query99	1542	1252	1280	1252
Total cold run time: 287090 ms
Total hot run time: 174177 ms

@doris-robot
Copy link

doris-robot commented Jul 17, 2024

ClickBench: Total hot run time: 30.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5298249c659055e87e17e17b812a6232d6d8c1e8, data reload: false

query1	0.05	0.03	0.03
query2	0.07	0.03	0.04
query3	0.22	0.05	0.06
query4	1.68	0.08	0.08
query5	0.50	0.49	0.49
query6	1.14	0.72	0.74
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.56	0.48	0.49
query10	0.55	0.54	0.54
query11	0.15	0.11	0.12
query12	0.14	0.12	0.13
query13	0.59	0.58	0.58
query14	0.76	0.77	0.80
query15	0.85	0.81	0.81
query16	0.36	0.37	0.38
query17	1.04	0.97	1.03
query18	0.23	0.21	0.22
query19	1.86	1.72	1.73
query20	0.02	0.01	0.01
query21	15.38	0.76	0.66
query22	3.96	7.41	1.70
query23	18.24	1.35	1.26
query24	2.18	0.23	0.22
query25	0.17	0.08	0.10
query26	0.29	0.21	0.21
query27	0.46	0.23	0.23
query28	13.26	1.00	1.00
query29	12.62	3.34	3.28
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.26	0.47	0.48
query33	2.86	2.94	2.93
query34	17.14	4.30	4.39
query35	4.53	4.40	4.47
query36	0.65	0.49	0.48
query37	0.18	0.14	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.13
query41	0.10	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.71 s
Total hot run time: 30.39 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 17, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 8e24d8f into apache:master Jul 17, 2024
englefly added a commit to englefly/incubator-doris that referenced this pull request Jul 17, 2024
@englefly
[fix](nereids) bug: after is-null stats derive, other column stats ar…
c557882 
…e dropped (apache#37809)
@englefly englefly mentioned this pull request Jul 17, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 18, 2024
@englefly
[fix](nereids) bug: after is-null stats derive, other column stats ar…
dcfc72b 
…e dropped (#37809) (#38024)

## Proposed changes
pick from #37809 
Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@englefly @doris-robot @starocean999 @morrySnow @yiguolei @dataroaring