Skip to content

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Jul 3, 2024

Proposed changes

the algorithm for computing stats for "expr1 and expr2" predicate is as following:

  1. compute output stats of expr1 based on input stats. the result stats is denoted by leftStats
  2. compute stats of expr2 based on leftStats
    after step1, leftStats should be normalized to avoid abnormal cases, such as ndv > rowCount or numNulls > rowCount

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

englefly commented Jul 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39904 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3d7ac12e012f362f9d6376b8970d2c1d3fa43b0, data reload: false

------ Round 1 ----------------------------------
q1	18037	4526	4409	4409
q2	2147	195	196	195
q3	10549	1183	1068	1068
q4	10201	892	757	757
q5	7496	2632	2598	2598
q6	219	138	139	138
q7	952	588	605	588
q8	9230	2074	2058	2058
q9	8991	6469	6468	6468
q10	9007	3734	3718	3718
q11	464	239	236	236
q12	402	236	228	228
q13	17776	3007	2962	2962
q14	269	221	230	221
q15	517	492	489	489
q16	515	383	373	373
q17	965	647	659	647
q18	8076	7551	7410	7410
q19	3582	1573	1488	1488
q20	667	324	319	319
q21	4947	3232	3198	3198
q22	400	336	337	336
Total cold run time: 115409 ms
Total hot run time: 39904 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4418	4297	4256	4256
q2	366	282	270	270
q3	2985	2698	2732	2698
q4	1870	1614	1598	1598
q5	5230	5252	5233	5233
q6	217	132	133	132
q7	2159	1737	1721	1721
q8	3205	3409	3334	3334
q9	8265	8354	8352	8352
q10	3854	3695	3668	3668
q11	598	508	477	477
q12	800	622	604	604
q13	17349	2952	3001	2952
q14	308	271	268	268
q15	527	480	477	477
q16	469	417	428	417
q17	1775	1505	1472	1472
q18	7809	7537	7399	7399
q19	3439	1674	1558	1558
q20	1989	1788	1767	1767
q21	4957	4776	4666	4666
q22	634	544	556	544
Total cold run time: 73223 ms
Total hot run time: 53863 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171351 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3d7ac12e012f362f9d6376b8970d2c1d3fa43b0, data reload: false

query1	916	368	368	368
query2	6491	2440	2302	2302
query3	6656	207	213	207
query4	19238	17533	17317	17317
query5	4174	499	484	484
query6	298	170	168	168
query7	4602	302	294	294
query8	320	302	302	302
query9	8544	2396	2363	2363
query10	619	304	305	304
query11	10415	10104	9907	9907
query12	130	85	82	82
query13	1644	354	364	354
query14	9437	6096	6218	6096
query15	231	181	194	181
query16	7244	273	259	259
query17	1441	531	536	531
query18	1923	274	273	273
query19	188	163	144	144
query20	86	79	84	79
query21	211	135	128	128
query22	4284	3964	4208	3964
query23	33634	33010	33146	33010
query24	12281	2775	2798	2775
query25	694	355	356	355
query26	1825	153	152	152
query27	3078	316	315	315
query28	7719	2041	2029	2029
query29	1319	613	601	601
query30	282	158	154	154
query31	957	723	756	723
query32	102	54	53	53
query33	784	312	303	303
query34	953	465	483	465
query35	748	618	607	607
query36	1089	929	947	929
query37	207	76	74	74
query38	2874	2750	2748	2748
query39	848	808	800	800
query40	290	130	129	129
query41	58	109	52	52
query42	120	96	102	96
query43	578	530	549	530
query44	1203	726	721	721
query45	193	161	161	161
query46	1086	753	729	729
query47	1865	1786	1747	1747
query48	359	286	299	286
query49	1196	411	418	411
query50	764	381	381	381
query51	6846	6819	6717	6717
query52	102	93	97	93
query53	361	295	301	295
query54	970	431	434	431
query55	75	75	75	75
query56	281	265	277	265
query57	1131	1104	1027	1027
query58	261	244	249	244
query59	3297	3057	3301	3057
query60	306	279	275	275
query61	92	95	93	93
query62	674	424	439	424
query63	320	291	297	291
query64	10561	2157	1640	1640
query65	3165	3084	3086	3084
query66	1448	338	329	329
query67	15364	14912	15059	14912
query68	4879	522	543	522
query69	617	391	320	320
query70	1168	1135	1157	1135
query71	430	280	279	279
query72	7826	5316	5635	5316
query73	769	322	321	321
query74	5826	5487	5550	5487
query75	3560	2671	2686	2671
query76	3226	1013	886	886
query77	669	293	293	293
query78	10221	9849	9874	9849
query79	2165	504	506	504
query80	1801	484	462	462
query81	585	216	223	216
query82	753	105	103	103
query83	290	167	170	167
query84	259	87	87	87
query85	1571	383	276	276
query86	483	320	297	297
query87	3294	3111	3152	3111
query88	4099	2361	2332	2332
query89	474	376	375	375
query90	1824	180	183	180
query91	126	98	103	98
query92	65	50	51	50
query93	2697	498	508	498
query94	1230	188	187	187
query95	410	312	306	306
query96	606	265	262	262
query97	3187	3045	3017	3017
query98	222	197	195	195
query99	1329	854	861	854
Total cold run time: 278113 ms
Total hot run time: 171351 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3d7ac12e012f362f9d6376b8970d2c1d3fa43b0, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.06
query5	0.50	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.54	0.50	0.49
query10	0.53	0.53	0.55
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.58	0.58
query14	0.81	0.77	0.79
query15	0.82	0.80	0.82
query16	0.36	0.35	0.38
query17	0.98	0.98	1.06
query18	0.20	0.27	0.23
query19	1.75	1.79	1.82
query20	0.01	0.02	0.01
query21	15.46	0.78	0.66
query22	4.22	5.95	2.72
query23	18.30	1.38	1.31
query24	2.07	0.23	0.23
query25	0.16	0.09	0.09
query26	0.28	0.17	0.17
query27	0.08	0.09	0.08
query28	13.27	1.00	0.99
query29	12.62	3.32	3.30
query30	0.26	0.05	0.06
query31	2.87	0.40	0.38
query32	3.27	0.48	0.47
query33	2.89	2.91	2.92
query34	17.30	4.43	4.40
query35	4.49	4.44	4.45
query36	0.64	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.53 s
Total hot run time: 31.46 s

@wm1581066 wm1581066 added the usercase Important user case type label label Jul 4, 2024
@englefly
Copy link
Contributor Author

englefly commented Jul 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39719 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf70e83501b2dea39ddcea6efcd71ef3a6147f53, data reload: false

------ Round 1 ----------------------------------
q1	17609	4710	4267	4267
q2	2019	202	189	189
q3	10459	1154	1057	1057
q4	10195	866	759	759
q5	7492	2616	2609	2609
q6	218	137	137	137
q7	948	599	614	599
q8	9231	2073	2095	2073
q9	9021	6467	6481	6467
q10	8969	3747	3763	3747
q11	466	237	252	237
q12	413	234	231	231
q13	17763	2984	3011	2984
q14	270	219	220	219
q15	518	488	487	487
q16	519	381	374	374
q17	964	663	616	616
q18	8101	7392	7402	7392
q19	7645	1445	1446	1445
q20	664	323	339	323
q21	4792	3168	3820	3168
q22	400	340	339	339
Total cold run time: 118676 ms
Total hot run time: 39719 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4377	4272	4243	4243
q2	370	270	250	250
q3	3037	2886	2859	2859
q4	1999	1756	1751	1751
q5	5532	5509	5427	5427
q6	232	133	128	128
q7	2239	1895	1823	1823
q8	3285	3423	3422	3422
q9	8715	8700	8764	8700
q10	4193	3753	3790	3753
q11	611	495	513	495
q12	819	667	643	643
q13	17152	3161	3147	3147
q14	308	274	288	274
q15	531	491	491	491
q16	480	428	432	428
q17	1811	1515	1508	1508
q18	8078	8145	7616	7616
q19	1760	1612	1561	1561
q20	2148	1861	1859	1859
q21	5238	4741	4973	4741
q22	636	580	559	559
Total cold run time: 73551 ms
Total hot run time: 55678 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173802 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf70e83501b2dea39ddcea6efcd71ef3a6147f53, data reload: false

query1	915	368	373	368
query2	6367	2389	2342	2342
query3	6633	203	207	203
query4	18777	17558	17353	17353
query5	3768	483	481	481
query6	287	167	157	157
query7	4580	304	285	285
query8	312	306	298	298
query9	8488	2470	2453	2453
query10	585	326	280	280
query11	10578	10096	10112	10096
query12	113	91	82	82
query13	1644	385	376	376
query14	9225	6997	7060	6997
query15	224	195	185	185
query16	7763	309	314	309
query17	1439	583	543	543
query18	1983	279	281	279
query19	196	158	163	158
query20	91	83	83	83
query21	218	150	148	148
query22	4571	4264	4289	4264
query23	33717	33554	33576	33554
query24	10976	2984	2960	2960
query25	608	429	399	399
query26	721	159	155	155
query27	2263	326	329	326
query28	5998	2213	2182	2182
query29	908	660	636	636
query30	238	165	165	165
query31	955	819	730	730
query32	99	65	59	59
query33	668	313	309	309
query34	890	487	495	487
query35	754	665	738	665
query36	1104	963	991	963
query37	140	74	78	74
query38	2960	2842	2823	2823
query39	925	838	833	833
query40	217	127	126	126
query41	53	50	57	50
query42	117	104	98	98
query43	597	564	568	564
query44	1080	754	732	732
query45	205	170	163	163
query46	1081	722	714	714
query47	1894	1750	1772	1750
query48	377	302	297	297
query49	844	415	430	415
query50	767	386	380	380
query51	6923	6753	6788	6753
query52	104	98	91	91
query53	371	305	294	294
query54	862	447	438	438
query55	75	73	72	72
query56	281	259	265	259
query57	1128	1061	1049	1049
query58	261	231	244	231
query59	3621	3224	3280	3224
query60	312	295	288	288
query61	96	99	93	93
query62	601	466	439	439
query63	328	297	294	294
query64	9091	2162	1659	1659
query65	3158	3089	3137	3089
query66	753	325	322	322
query67	15662	15246	14888	14888
query68	6272	543	547	543
query69	707	422	315	315
query70	1211	1143	1029	1029
query71	508	282	278	278
query72	8559	5526	5688	5526
query73	816	320	322	320
query74	5937	5515	5572	5515
query75	4715	2664	2657	2657
query76	4525	1068	969	969
query77	783	301	298	298
query78	9587	9149	8861	8861
query79	2300	524	535	524
query80	2504	464	471	464
query81	579	220	217	217
query82	753	107	103	103
query83	307	196	163	163
query84	259	88	86	86
query85	1218	283	330	283
query86	413	315	304	304
query87	3258	3099	3127	3099
query88	3932	2363	2354	2354
query89	495	393	377	377
query90	1886	188	185	185
query91	133	103	100	100
query92	57	49	51	49
query93	3118	514	517	514
query94	1104	215	210	210
query95	409	308	308	308
query96	596	271	273	271
query97	3269	3032	3009	3009
query98	221	196	194	194
query99	1142	846	845	845
Total cold run time: 275113 ms
Total hot run time: 173802 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf70e83501b2dea39ddcea6efcd71ef3a6147f53, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.05	0.04
query3	0.22	0.06	0.06
query4	1.65	0.09	0.09
query5	0.50	0.48	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.48	0.48
query10	0.54	0.51	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.77	0.78
query15	0.85	0.82	0.81
query16	0.36	0.36	0.37
query17	1.02	1.03	1.05
query18	0.24	0.22	0.23
query19	1.76	1.68	1.79
query20	0.02	0.01	0.01
query21	15.41	0.75	0.65
query22	4.24	7.62	1.50
query23	18.20	1.45	1.35
query24	2.12	0.22	0.22
query25	0.16	0.10	0.08
query26	0.27	0.18	0.17
query27	0.09	0.07	0.09
query28	13.28	1.01	0.99
query29	12.59	3.28	3.23
query30	0.26	0.06	0.06
query31	2.95	0.39	0.39
query32	3.23	0.47	0.46
query33	2.86	2.96	2.93
query34	17.22	4.40	4.42
query35	4.50	4.49	4.48
query36	0.65	0.47	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.13
query41	0.09	0.06	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.45 s
Total hot run time: 30.21 s

@github-actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by anyone and no changes requested.

@englefly englefly merged commit 68522bc into apache:master Jul 5, 2024
@englefly englefly deleted the big-num-null branch July 5, 2024 07:46
englefly added a commit that referenced this pull request Jul 5, 2024
#37236)

## Proposed changes
pick #37235
Issue Number: close #xxx

<!--Describe your changes.-->
englefly added a commit to englefly/incubator-doris that referenced this pull request Jul 8, 2024
…he#37235)

the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats
after step1, leftStats should be normalized to avoid abnormal cases,
such as ndv > rowCount or numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->
englefly added a commit that referenced this pull request Jul 9, 2024
…) (#37498)

pick from #37235 
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats after step1, leftStats
should be normalized to avoid abnormal cases, such as ndv > rowCount or
numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
## Proposed changes
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats
after step1, leftStats should be normalized to avoid abnormal cases,
such as ndv > rowCount or numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…e#37235 (apache#37236)

## Proposed changes
pick apache#37235
Issue Number: close #xxx

<!--Describe your changes.-->
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.1-merged p0_b reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants