Skip to content

[feat](nereids) adjust stats derive by delta row #39222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 1 commit into from
Sep 2, 2024
Merged

[feat](nereids) adjust stats derive by delta row #39222

englefly merged 1 commit into from
Sep 2, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Aug 12, 2024

Proposed changes

After analyzing, user may insert new rows.
analyzed rows: the rows have been analyzed
delta row: rows inserted after analyze job

if analyzed rows are filtered out, then we try to estimate filter result by delta row with unknown column stats.
Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly englefly mentioned this pull request Aug 12, 2024
Merged
@github-actions github-actions bot added the doing label Aug 12, 2024
@englefly
Copy link
Contributor Author

englefly commented Aug 12, 2024

run buildall

2 similar comments
@englefly
Copy link
Contributor Author

englefly commented Aug 12, 2024

run buildall

@englefly
Copy link
Contributor Author

englefly commented Aug 12, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

TPC-H: Total hot run time: 39472 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55247fed0d94147c287102075fc5b11bb8ca7b62, data reload: false

------ Round 1 ----------------------------------
q1	17636	4360	4272	4272
q2	2029	207	203	203
q3	11843	1089	1150	1089
q4	10517	675	701	675
q5	7518	2537	2495	2495
q6	229	146	142	142
q7	972	595	592	592
q8	9403	1930	1932	1930
q9	9516	6546	6583	6546
q10	7019	2205	2184	2184
q11	447	252	247	247
q12	401	218	217	217
q13	18855	2941	2969	2941
q14	287	235	234	234
q15	528	476	491	476
q16	502	388	398	388
q17	968	672	662	662
q18	7946	7372	7386	7372
q19	6263	938	954	938
q20	680	345	345	345
q21	6025	4544	4739	4544
q22	1080	987	980	980
Total cold run time: 120664 ms
Total hot run time: 39472 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4598	4291	4289	4289
q2	383	280	272	272
q3	2906	2807	2808	2807
q4	1962	1791	1809	1791
q5	5466	5430	5436	5430
q6	230	140	140	140
q7	2186	1780	1810	1780
q8	3297	3447	3413	3413
q9	8721	8879	8735	8735
q10	3457	3188	3278	3188
q11	629	539	514	514
q12	840	677	630	630
q13	17235	3158	3145	3145
q14	331	298	298	298
q15	535	482	494	482
q16	511	443	440	440
q17	1839	1519	1503	1503
q18	8168	8133	7779	7779
q19	3618	1610	1724	1610
q20	2197	1860	1863	1860
q21	5700	5179	5425	5179
q22	1118	1038	1009	1009
Total cold run time: 75927 ms
Total hot run time: 56294 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

TPC-DS: Total hot run time: 203205 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55247fed0d94147c287102075fc5b11bb8ca7b62, data reload: false

query1	1643	1366	1436	1366
query2	6325	2019	1914	1914
query3	13834	8291	318	318
query4	33256	23204	23358	23204
query5	3828	488	485	485
query6	282	187	172	172
query7	3961	293	285	285
query8	245	202	194	194
query9	8356	2465	2456	2456
query10	531	467	443	443
query11	17476	15154	15164	15154
query12	147	96	96	96
query13	1541	369	368	368
query14	9898	7505	7910	7505
query15	265	234	249	234
query16	7565	498	550	498
query17	1266	593	558	558
query18	1951	293	285	285
query19	227	147	149	147
query20	128	105	109	105
query21	204	106	101	101
query22	4616	4387	4260	4260
query23	34516	34433	33684	33684
query24	10741	2578	2570	2570
query25	527	366	367	366
query26	678	145	150	145
query27	2195	276	284	276
query28	5980	2047	2027	2027
query29	641	399	399	399
query30	247	151	152	151
query31	976	761	755	755
query32	93	50	56	50
query33	619	282	280	280
query34	856	457	468	457
query35	931	817	811	811
query36	1096	930	908	908
query37	132	75	79	75
query38	4258	4125	4152	4125
query39	1410	1387	1364	1364
query40	207	124	115	115
query41	47	46	45	45
query42	113	95	94	94
query43	502	459	460	459
query44	1064	734	726	726
query45	243	207	201	201
query46	1083	759	760	759
query47	1867	1761	1751	1751
query48	369	295	293	293
query49	847	427	428	427
query50	805	412	401	401
query51	6792	6730	6723	6723
query52	99	95	89	89
query53	262	185	182	182
query54	895	452	456	452
query55	76	81	78	78
query56	268	254	262	254
query57	1144	1071	1062	1062
query58	239	235	235	235
query59	2994	2871	2871	2871
query60	288	256	250	250
query61	107	96	100	96
query62	801	638	645	638
query63	213	181	180	180
query64	9144	2247	1778	1778
query65	3174	3148	3114	3114
query66	736	335	321	321
query67	15214	14792	14793	14792
query68	5148	552	544	544
query69	527	363	406	363
query70	1208	1057	1136	1057
query71	473	270	276	270
query72	17339	16812	16437	16437
query73	783	324	325	324
query74	9122	8770	8737	8737
query75	3470	2690	2655	2655
query76	3518	1022	1032	1022
query77	660	307	315	307
query78	9668	9226	8925	8925
query79	2429	519	560	519
query80	2338	478	487	478
query81	628	225	221	221
query82	1396	134	140	134
query83	295	147	149	147
query84	259	77	81	77
query85	1368	282	274	274
query86	437	302	307	302
query87	4656	4586	4501	4501
query88	3676	2476	2468	2468
query89	431	287	293	287
query90	1760	196	190	190
query91	120	97	96	96
query92	56	50	49	49
query93	3391	542	528	528
query94	794	282	341	282
query95	352	266	260	260
query96	605	277	277	277
query97	3211	3035	3027	3027
query98	219	202	205	202
query99	1475	1268	1274	1268
Total cold run time: 314846 ms
Total hot run time: 203205 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

ClickBench: Total hot run time: 29.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55247fed0d94147c287102075fc5b11bb8ca7b62, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.51	0.48	0.50
query6	1.12	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.50	0.50
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.58
query14	0.77	0.77	0.79
query15	0.85	0.81	0.82
query16	0.36	0.37	0.35
query17	1.02	0.97	1.06
query18	0.23	0.22	0.22
query19	1.83	1.79	1.77
query20	0.01	0.01	0.02
query21	15.41	0.72	0.65
query22	5.09	7.99	1.10
query23	18.20	1.36	1.30
query24	2.16	0.21	0.22
query25	0.15	0.09	0.08
query26	0.29	0.22	0.20
query27	0.46	0.22	0.22
query28	13.32	1.03	1.00
query29	12.63	3.30	3.27
query30	0.23	0.06	0.05
query31	2.87	0.39	0.38
query32	3.27	0.50	0.47
query33	2.86	2.92	2.90
query34	16.92	4.32	4.38
query35	4.47	4.43	4.40
query36	0.65	0.50	0.51
query37	0.19	0.15	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.10	0.05	0.04
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.53 s
Total hot run time: 29.82 s

@englefly
Copy link
Contributor Author

englefly commented Aug 12, 2024

run p0

@englefly
Copy link
Contributor Author

englefly commented Aug 12, 2024

run cloud_p0

@englefly
Copy link
Contributor Author

englefly commented Aug 13, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

TPC-H: Total hot run time: 39479 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55247fed0d94147c287102075fc5b11bb8ca7b62, data reload: false

------ Round 1 ----------------------------------
q1	17609	4333	4311	4311
q2	2019	181	172	172
q3	11813	957	1099	957
q4	10492	674	715	674
q5	7759	2792	2781	2781
q6	220	134	133	133
q7	953	584	595	584
q8	9533	2018	2043	2018
q9	10140	6512	6562	6512
q10	7040	2189	2252	2189
q11	458	246	247	246
q12	391	217	221	217
q13	18098	2985	2993	2985
q14	280	231	227	227
q15	546	487	480	480
q16	505	387	391	387
q17	981	751	758	751
q18	8022	7460	7371	7371
q19	4594	1037	1024	1024
q20	641	321	322	321
q21	5516	4544	4137	4137
q22	1116	1002	1013	1002
Total cold run time: 118726 ms
Total hot run time: 39479 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4403	4234	4267	4234
q2	378	276	266	266
q3	2899	2714	2746	2714
q4	1981	1647	1739	1647
q5	5706	5591	5496	5496
q6	221	131	139	131
q7	2169	1777	1757	1757
q8	3324	3429	3397	3397
q9	8768	8882	8722	8722
q10	3519	3224	3284	3224
q11	614	495	505	495
q12	829	643	663	643
q13	16969	3150	3219	3150
q14	319	308	292	292
q15	538	467	485	467
q16	497	451	427	427
q17	1817	1508	1501	1501
q18	8153	8024	7862	7862
q19	1740	1566	1613	1566
q20	2278	1902	1880	1880
q21	7533	5195	5277	5195
q22	1157	1065	1046	1046
Total cold run time: 75812 ms
Total hot run time: 56112 ms

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

TPC-DS: Total hot run time: 189939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55247fed0d94147c287102075fc5b11bb8ca7b62, data reload: false

query1	1253	900	855	855
query2	6321	1932	1795	1795
query3	10589	3832	3846	3832
query4	60267	26821	23222	23222
query5	5879	494	492	492
query6	518	169	163	163
query7	6488	286	305	286
query8	305	206	217	206
query9	9012	2431	2400	2400
query10	503	247	255	247
query11	18545	15015	15318	15015
query12	152	95	105	95
query13	1569	385	367	367
query14	12275	7704	7620	7620
query15	244	174	180	174
query16	7654	519	464	464
query17	1101	569	566	566
query18	2111	298	294	294
query19	287	148	137	137
query20	125	109	109	109
query21	205	99	100	99
query22	4452	4268	4196	4196
query23	34819	33588	33192	33192
query24	5620	2884	2944	2884
query25	540	404	410	404
query26	690	158	162	158
query27	1781	270	269	269
query28	3670	2029	2027	2027
query29	687	421	459	421
query30	226	148	144	144
query31	921	748	725	725
query32	85	51	51	51
query33	445	281	265	265
query34	846	449	475	449
query35	804	728	694	694
query36	1083	934	936	934
query37	131	77	76	76
query38	4022	3784	3913	3784
query39	1423	1380	1376	1376
query40	199	114	112	112
query41	46	40	47	40
query42	112	95	98	95
query43	494	465	468	465
query44	1065	727	724	724
query45	191	160	161	160
query46	1084	748	716	716
query47	1839	1736	1789	1736
query48	365	286	303	286
query49	739	403	427	403
query50	790	412	411	411
query51	6808	6766	6700	6700
query52	102	89	88	88
query53	250	180	179	179
query54	545	435	432	432
query55	75	73	75	73
query56	262	240	241	240
query57	1109	1069	1048	1048
query58	224	246	219	219
query59	2945	2617	2714	2617
query60	285	265	260	260
query61	102	96	91	91
query62	730	652	637	637
query63	206	176	180	176
query64	9040	2241	1750	1750
query65	3242	3137	3143	3137
query66	663	332	327	327
query67	15169	14884	14673	14673
query68	4809	544	559	544
query69	457	263	263	263
query70	1146	1099	1147	1099
query71	438	279	268	268
query72	7070	2269	2044	2044
query73	743	322	329	322
query74	9236	8698	8910	8698
query75	3386	2757	2642	2642
query76	3114	1019	972	972
query77	548	324	303	303
query78	9449	10721	9339	9339
query79	2555	531	538	531
query80	1161	504	487	487
query81	553	233	228	228
query82	595	135	131	131
query83	269	149	149	149
query84	271	83	79	79
query85	748	277	273	273
query86	478	290	296	290
query87	4341	4118	4273	4118
query88	4468	2461	2404	2404
query89	398	275	280	275
query90	2021	198	195	195
query91	121	96	94	94
query92	68	49	48	48
query93	3231	538	533	533
query94	915	285	301	285
query95	357	265	258	258
query96	614	275	273	273
query97	3215	3087	3112	3087
query98	218	206	192	192
query99	1538	1265	1233	1233
Total cold run time: 324860 ms
Total hot run time: 189939 ms

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

ClickBench: Total hot run time: 30.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55247fed0d94147c287102075fc5b11bb8ca7b62, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.07	0.08
query5	0.49	0.48	0.47
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.53	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.80	0.77	0.79
query15	0.89	0.82	0.82
query16	0.38	0.37	0.36
query17	1.03	0.95	0.99
query18	0.23	0.23	0.22
query19	1.86	1.86	1.72
query20	0.02	0.01	0.01
query21	15.40	0.75	0.65
query22	3.93	7.60	2.09
query23	18.31	1.32	1.31
query24	2.07	0.24	0.23
query25	0.14	0.09	0.08
query26	0.31	0.22	0.21
query27	0.45	0.24	0.23
query28	13.19	1.01	1.01
query29	12.70	3.35	3.31
query30	0.23	0.06	0.04
query31	2.87	0.40	0.40
query32	3.28	0.48	0.47
query33	2.96	2.97	3.01
query34	17.11	4.36	4.36
query35	4.44	4.39	4.42
query36	0.66	0.50	0.48
query37	0.19	0.15	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.06
query42	0.06	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 109.68 s
Total hot run time: 30.94 s

englefly added a commit that referenced this pull request Aug 14, 2024
@englefly
[feat](nereids) adjust stats derive by delta row (#39185)
c9813c8 
#39222
After analyzing, user may insert new rows. 
analyzed rows: the rows have been analyzed
delta row: rows inserted after analyze job

if analyzed rows are filtered out, then we try to estimate filter result
by delta row with unknown column stats.
  

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly
Copy link
Contributor Author

englefly commented Aug 16, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 16, 2024

TPC-H: Total hot run time: 38098 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

------ Round 1 ----------------------------------
q1	17911	4373	4304	4304
q2	2045	215	216	215
q3	11651	1077	1089	1077
q4	10549	834	704	704
q5	7813	2850	2821	2821
q6	267	158	159	158
q7	1009	651	638	638
q8	9629	2071	2091	2071
q9	8739	6559	6581	6559
q10	7084	2251	2167	2167
q11	467	269	270	269
q12	426	255	258	255
q13	17787	2985	3000	2985
q14	303	262	260	260
q15	564	536	529	529
q16	534	414	404	404
q17	973	749	771	749
q18	7382	6837	6651	6651
q19	6388	1167	1042	1042
q20	813	380	367	367
q21	3786	2867	2846	2846
q22	1115	1027	1027	1027
Total cold run time: 117235 ms
Total hot run time: 38098 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5254	4304	4291	4291
q2	425	290	302	290
q3	2826	2612	2712	2612
q4	2034	1710	1688	1688
q5	5673	5720	5619	5619
q6	250	153	155	153
q7	2175	1824	1755	1755
q8	3277	3510	3438	3438
q9	8799	8735	8730	8730
q10	3641	3373	3280	3280
q11	602	537	533	533
q12	820	675	680	675
q13	17236	3185	3018	3018
q14	329	289	297	289
q15	565	524	534	524
q16	520	462	458	458
q17	1858	1540	1536	1536
q18	8115	7884	7587	7587
q19	11278	1660	1576	1576
q20	2305	1892	1841	1841
q21	5593	5500	5486	5486
q22	1443	1058	1101	1058
Total cold run time: 85018 ms
Total hot run time: 56437 ms

@englefly
Copy link
Contributor Author

englefly commented Aug 21, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 21, 2024

TPC-H: Total hot run time: 38108 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

------ Round 1 ----------------------------------
q1	17838	4323	4345	4323
q2	2038	211	228	211
q3	11775	952	1053	952
q4	10537	774	756	756
q5	7782	2801	2782	2782
q6	265	158	158	158
q7	1035	674	669	669
q8	9583	2097	2085	2085
q9	8481	6531	6559	6531
q10	7073	2271	2239	2239
q11	491	274	270	270
q12	427	265	259	259
q13	17796	3052	2990	2990
q14	294	250	259	250
q15	562	518	543	518
q16	519	414	413	413
q17	998	699	756	699
q18	7334	6741	6671	6671
q19	6621	1112	1054	1054
q20	842	367	367	367
q21	3878	2974	2867	2867
q22	1137	1050	1044	1044
Total cold run time: 117306 ms
Total hot run time: 38108 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4569	4336	4272	4272
q2	431	306	306	306
q3	2795	2629	2636	2629
q4	2005	1720	1712	1712
q5	5629	5624	5717	5624
q6	239	149	148	148
q7	2167	1753	1748	1748
q8	3288	3524	3517	3517
q9	8910	8859	8883	8859
q10	3657	3339	3333	3333
q11	640	521	520	520
q12	820	656	634	634
q13	16304	3178	3147	3147
q14	317	286	285	285
q15	541	522	516	516
q16	510	462	462	462
q17	1824	1575	1550	1550
q18	8225	7858	7746	7746
q19	9430	1673	1559	1559
q20	2140	1900	1897	1897
q21	13889	5267	5292	5267
q22	1167	1084	1057	1057
Total cold run time: 89497 ms
Total hot run time: 56788 ms

@doris-robot
Copy link

doris-robot commented Aug 21, 2024

TPC-DS: Total hot run time: 196301 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

query1	1314	923	895	895
query2	6586	1977	1929	1929
query3	10657	3929	3762	3762
query4	58301	25869	23449	23449
query5	6267	718	702	702
query6	542	212	224	212
query7	6439	323	341	323
query8	554	453	454	453
query9	9321	2540	2484	2484
query10	585	347	342	342
query11	18432	15079	15575	15079
query12	183	130	127	127
query13	1651	433	457	433
query14	12161	7388	6906	6906
query15	254	194	190	190
query16	7445	486	517	486
query17	1211	601	609	601
query18	1807	339	339	339
query19	304	167	166	166
query20	160	144	140	140
query21	252	150	144	144
query22	4470	4329	4313	4313
query23	34442	33772	34428	33772
query24	5786	3030	2970	2970
query25	550	432	430	430
query26	704	186	182	182
query27	1736	311	316	311
query28	4081	2184	2155	2155
query29	691	471	451	451
query30	234	192	190	190
query31	1051	877	837	837
query32	96	75	78	75
query33	542	343	335	335
query34	903	493	524	493
query35	857	781	747	747
query36	1082	936	966	936
query37	161	104	98	98
query38	4072	3871	3904	3871
query39	1545	1491	1471	1471
query40	236	158	155	155
query41	138	138	136	136
query42	139	114	116	114
query43	551	511	511	511
query44	1105	781	778	778
query45	228	193	192	192
query46	1129	757	758	757
query47	1955	1871	1852	1852
query48	405	336	339	336
query49	908	580	576	576
query50	855	472	457	457
query51	7085	7156	7067	7067
query52	129	110	109	109
query53	298	230	222	222
query54	616	501	495	495
query55	93	90	89	89
query56	334	302	317	302
query57	1232	1151	1156	1151
query58	288	299	311	299
query59	3055	2857	2817	2817
query60	348	319	323	319
query61	149	145	149	145
query62	796	704	684	684
query63	259	228	225	225
query64	3362	1846	1834	1834
query65	3227	3164	3193	3164
query66	1024	678	721	678
query67	15309	15209	15106	15106
query68	8037	593	591	591
query69	671	419	331	331
query70	1274	1127	1106	1106
query71	527	309	328	309
query72	6441	2341	2065	2065
query73	820	356	352	352
query74	9204	8765	8894	8765
query75	3645	2743	2718	2718
query76	4092	1105	1088	1088
query77	850	435	487	435
query78	9869	9153	9283	9153
query79	5587	555	551	551
query80	2591	606	617	606
query81	611	267	257	257
query82	814	161	157	157
query83	356	211	209	209
query84	289	97	96	96
query85	981	361	351	351
query86	394	301	318	301
query87	4363	4235	4204	4204
query88	4513	2523	2527	2523
query89	437	326	321	321
query90	1977	229	231	229
query91	150	127	124	124
query92	89	75	110	75
query93	4344	551	549	549
query94	764	324	325	324
query95	380	299	290	290
query96	629	291	282	282
query97	3258	3096	3047	3047
query98	244	236	227	227
query99	1771	1314	1296	1296
Total cold run time: 332654 ms
Total hot run time: 196301 ms

@doris-robot
Copy link

doris-robot commented Aug 21, 2024

ClickBench: Total hot run time: 31.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.66	0.07	0.07
query5	0.53	0.51	0.50
query6	1.12	0.72	0.73
query7	0.02	0.02	0.02
query8	0.06	0.05	0.06
query9	0.54	0.49	0.49
query10	0.54	0.55	0.54
query11	0.15	0.12	0.12
query12	0.16	0.13	0.13
query13	0.61	0.60	0.59
query14	0.77	0.80	0.78
query15	0.82	0.81	0.83
query16	0.37	0.37	0.37
query17	1.04	1.05	1.05
query18	0.20	0.20	0.20
query19	1.81	1.74	1.66
query20	0.01	0.01	0.02
query21	15.40	0.66	0.65
query22	4.67	6.69	2.05
query23	18.23	1.54	1.22
query24	2.08	0.23	0.22
query25	0.15	0.09	0.08
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.30	1.03	1.01
query29	12.71	3.34	3.35
query30	0.44	0.26	0.24
query31	2.80	0.40	0.39
query32	3.28	0.49	0.49
query33	2.98	2.93	2.99
query34	17.11	4.35	4.37
query35	4.45	4.38	4.35
query36	0.67	0.52	0.52
query37	0.21	0.17	0.17
query38	0.18	0.17	0.17
query39	0.07	0.06	0.06
query40	0.18	0.15	0.16
query41	0.11	0.07	0.07
query42	0.09	0.07	0.07
query43	0.06	0.06	0.07
Total cold run time: 110.29 s
Total hot run time: 31.08 s

@englefly
Copy link
Contributor Author

englefly commented Aug 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 22, 2024

TPC-H: Total hot run time: 39056 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

------ Round 1 ----------------------------------
q1	18139	4706	4388	4388
q2	2061	243	209	209
q3	11190	1020	1240	1020
q4	10217	668	751	668
q5	7778	2957	2882	2882
q6	269	157	161	157
q7	1015	677	668	668
q8	9399	2191	2187	2187
q9	7125	6608	6594	6594
q10	7097	2247	2246	2246
q11	511	293	277	277
q12	435	263	258	258
q13	18753	3071	3029	3029
q14	324	275	264	264
q15	576	542	529	529
q16	548	421	417	417
q17	1004	700	713	700
q18	7628	7090	7036	7036
q19	1482	1079	1098	1079
q20	727	383	366	366
q21	4654	3047	3195	3047
q22	1166	1052	1035	1035
Total cold run time: 112098 ms
Total hot run time: 39056 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4428	4348	4318	4318
q2	419	322	320	320
q3	2952	2695	2697	2695
q4	1972	1751	1762	1751
q5	5720	5673	5798	5673
q6	245	150	151	150
q7	2309	1934	1853	1853
q8	3358	3507	3545	3507
q9	8930	8865	8897	8865
q10	3620	3439	3403	3403
q11	623	539	534	534
q12	864	648	721	648
q13	15367	3258	3270	3258
q14	338	304	304	304
q15	563	513	514	513
q16	509	467	459	459
q17	1860	1533	1559	1533
q18	8472	7848	7937	7848
q19	1774	1641	1641	1641
q20	2179	1985	1980	1980
q21	5656	5676	5494	5494
q22	1198	1109	1112	1109
Total cold run time: 73356 ms
Total hot run time: 57856 ms

@doris-robot
Copy link

doris-robot commented Aug 22, 2024

TPC-DS: Total hot run time: 197872 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

query1	1340	922	883	883
query2	6629	1973	1954	1954
query3	10709	4045	4099	4045
query4	59658	24336	23298	23298
query5	5703	686	637	637
query6	446	198	193	193
query7	5891	323	322	322
query8	458	370	364	364
query9	8986	2505	2526	2505
query10	554	348	336	336
query11	18254	15164	15613	15164
query12	187	129	131	129
query13	1630	471	458	458
query14	11523	7422	7807	7422
query15	278	209	196	196
query16	7539	512	510	510
query17	1185	643	611	611
query18	1829	360	347	347
query19	316	172	171	171
query20	166	138	137	137
query21	252	148	143	143
query22	4751	4418	4531	4418
query23	34553	33905	34108	33905
query24	6065	2989	2968	2968
query25	584	430	425	425
query26	712	189	183	183
query27	1830	324	318	318
query28	3989	2167	2156	2156
query29	729	456	454	454
query30	228	188	196	188
query31	1042	820	854	820
query32	104	77	99	77
query33	471	355	350	350
query34	888	508	518	508
query35	885	787	772	772
query36	1082	938	972	938
query37	165	109	106	106
query38	4007	3922	3905	3905
query39	1494	1482	1471	1471
query40	242	156	155	155
query41	136	133	137	133
query42	159	118	115	115
query43	540	511	487	487
query44	1111	790	791	790
query45	223	195	198	195
query46	1144	810	808	808
query47	1971	1851	1883	1851
query48	415	352	336	336
query49	922	587	590	587
query50	877	467	481	467
query51	7403	7187	7102	7102
query52	120	106	104	104
query53	296	221	220	220
query54	612	527	505	505
query55	90	89	86	86
query56	327	337	313	313
query57	1213	1130	1153	1130
query58	292	303	299	299
query59	2936	2943	2765	2765
query60	363	321	352	321
query61	150	143	148	143
query62	761	708	705	705
query63	258	232	230	230
query64	3462	1838	1901	1838
query65	3263	3201	3221	3201
query66	987	674	664	664
query67	15515	15266	15298	15266
query68	4483	615	608	608
query69	465	327	323	323
query70	1133	1158	1084	1084
query71	412	312	318	312
query72	6509	2395	2133	2133
query73	801	358	362	358
query74	9148	9054	8840	8840
query75	3450	2777	2787	2777
query76	1910	1094	1033	1033
query77	675	449	430	430
query78	9869	9136	9734	9136
query79	1247	566	554	554
query80	912	604	620	604
query81	599	267	260	260
query82	313	159	157	157
query83	314	213	215	213
query84	283	100	101	100
query85	939	366	360	360
query86	338	303	329	303
query87	4376	4293	4378	4293
query88	3521	2518	2522	2518
query89	430	322	327	322
query90	1751	231	231	231
query91	154	130	128	128
query92	84	77	77	77
query93	1777	552	558	552
query94	639	334	328	328
query95	390	292	301	292
query96	611	287	282	282
query97	3237	3108	3160	3108
query98	252	233	220	220
query99	1552	1333	1350	1333
Total cold run time: 315474 ms
Total hot run time: 197872 ms

@doris-robot
Copy link

doris-robot commented Aug 22, 2024

ClickBench: Total hot run time: 31.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59db45fae488867945d596b9d5634e7b5aff7a53, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.05	0.05
query3	0.23	0.06	0.06
query4	1.66	0.09	0.10
query5	0.50	0.52	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.50	0.50
query10	0.54	0.56	0.56
query11	0.16	0.12	0.13
query12	0.16	0.13	0.13
query13	0.62	0.60	0.60
query14	0.76	0.81	0.77
query15	0.85	0.81	0.83
query16	0.36	0.36	0.38
query17	0.96	1.06	0.99
query18	0.23	0.20	0.21
query19	1.83	1.70	1.76
query20	0.01	0.01	0.01
query21	15.41	0.68	0.67
query22	4.64	7.09	1.73
query23	18.27	1.34	1.23
query24	2.18	0.23	0.22
query25	0.15	0.08	0.10
query26	0.27	0.19	0.18
query27	0.10	0.09	0.08
query28	13.25	1.04	1.01
query29	12.65	3.45	3.44
query30	0.43	0.24	0.25
query31	2.83	0.40	0.40
query32	3.25	0.49	0.49
query33	3.00	3.02	2.94
query34	16.80	4.39	4.48
query35	4.53	4.48	4.50
query36	0.67	0.50	0.51
query37	0.21	0.18	0.18
query38	0.18	0.16	0.18
query39	0.07	0.06	0.06
query40	0.19	0.16	0.15
query41	0.11	0.07	0.07
query42	0.09	0.08	0.07
query43	0.07	0.06	0.06
Total cold run time: 110.11 s
Total hot run time: 31.09 s

Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request Aug 26, 2024
@englefly @Jibing-Li
[feat](nereids) adjust stats derive by delta row (apache#39185)
7062650 
apache#39222
After analyzing, user may insert new rows. 
analyzed rows: the rows have been analyzed
delta row: rows inserted after analyze job

if analyzed rows are filtered out, then we try to estimate filter result
by delta row with unknown column stats.
  

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly
Copy link
Contributor Author

englefly commented Aug 30, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 30, 2024

TPC-H: Total hot run time: 38670 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43b97a9536caee130bc02b7a228221efd3b03551, data reload: false

------ Round 1 ----------------------------------
q1	17970	4526	4439	4439
q2	2910	187	184	184
q3	11936	1121	1077	1077
q4	10480	721	738	721
q5	7784	2905	2872	2872
q6	232	145	147	145
q7	999	621	601	601
q8	9317	2058	2046	2046
q9	7037	6568	6566	6566
q10	7005	2441	2239	2239
q11	462	250	251	250
q12	396	233	228	228
q13	17768	3033	3029	3029
q14	279	243	244	243
q15	530	485	493	485
q16	592	509	505	505
q17	980	752	696	696
q18	7379	6932	6887	6887
q19	1396	1018	1054	1018
q20	671	342	334	334
q21	4202	3171	3081	3081
q22	1139	1024	1050	1024
Total cold run time: 111464 ms
Total hot run time: 38670 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4333	4453	4333
q2	372	267	276	267
q3	2889	2678	2629	2629
q4	1950	1628	1715	1628
q5	5409	5422	5439	5422
q6	220	133	135	133
q7	2103	1752	1767	1752
q8	3208	3357	3389	3357
q9	8437	8427	8450	8427
q10	3444	3241	3221	3221
q11	600	499	498	498
q12	816	597	621	597
q13	11144	3046	3026	3026
q14	306	294	286	286
q15	528	469	486	469
q16	583	545	547	545
q17	1823	1494	1493	1493
q18	7740	7509	7382	7382
q19	1685	1565	1502	1502
q20	2079	1824	1819	1819
q21	5417	5380	5330	5330
q22	1138	1049	1043	1043
Total cold run time: 66278 ms
Total hot run time: 55159 ms

@doris-robot
Copy link

doris-robot commented Aug 30, 2024

TPC-DS: Total hot run time: 189710 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43b97a9536caee130bc02b7a228221efd3b03551, data reload: false

query1	935	387	375	375
query2	6460	1982	1870	1870
query3	6648	210	237	210
query4	28043	23228	23186	23186
query5	4174	523	512	512
query6	248	169	172	169
query7	4580	307	298	298
query8	261	211	221	211
query9	8582	2476	2470	2470
query10	441	283	278	278
query11	16363	15092	14973	14973
query12	152	101	98	98
query13	1632	382	378	378
query14	10287	7564	7387	7387
query15	304	173	179	173
query16	7987	460	481	460
query17	1603	595	573	573
query18	1967	300	294	294
query19	227	158	155	155
query20	120	114	111	111
query21	219	109	108	108
query22	4548	4106	4129	4106
query23	34220	34581	33619	33619
query24	11254	3037	2927	2927
query25	570	410	417	410
query26	1090	170	166	166
query27	2782	288	287	287
query28	7467	2116	2095	2095
query29	651	442	431	431
query30	304	160	158	158
query31	1001	782	820	782
query32	99	60	63	60
query33	774	320	304	304
query34	939	483	512	483
query35	907	764	756	756
query36	1069	926	958	926
query37	154	98	93	93
query38	3996	3862	3883	3862
query39	1446	1400	1394	1394
query40	283	127	126	126
query41	50	50	49	49
query42	122	100	105	100
query43	525	489	487	487
query44	1203	761	756	756
query45	206	178	177	177
query46	1121	752	767	752
query47	1905	1796	1810	1796
query48	386	292	303	292
query49	1178	460	455	455
query50	830	419	430	419
query51	7208	7072	7102	7072
query52	100	91	91	91
query53	275	198	190	190
query54	884	471	484	471
query55	84	80	83	80
query56	299	278	289	278
query57	1220	1095	1065	1065
query58	264	254	248	248
query59	3184	2839	2828	2828
query60	315	289	287	287
query61	130	129	131	129
query62	839	670	650	650
query63	229	195	197	195
query64	5271	770	741	741
query65	3245	3175	3193	3175
query66	1315	356	361	356
query67	15730	15365	15251	15251
query68	4836	549	567	549
query69	435	315	289	289
query70	1158	1151	1143	1143
query71	404	288	284	284
query72	6279	4016	4031	4016
query73	793	332	336	332
query74	9091	8830	9008	8830
query75	3482	2728	2698	2698
query76	2864	1099	978	978
query77	529	341	358	341
query78	10498	9585	9057	9057
query79	3166	566	571	566
query80	1810	534	524	524
query81	593	244	233	233
query82	709	151	156	151
query83	271	153	157	153
query84	274	84	77	77
query85	1032	290	284	284
query86	329	301	295	295
query87	4731	4343	4345	4343
query88	3776	2346	2318	2318
query89	411	298	299	298
query90	1524	203	209	203
query91	133	110	104	104
query92	66	58	52	52
query93	2390	561	539	539
query94	574	298	317	298
query95	352	274	278	274
query96	632	268	273	268
query97	3228	3117	3075	3075
query98	235	217	209	209
query99	1761	1317	1265	1265
Total cold run time: 291271 ms
Total hot run time: 189710 ms

@doris-robot
Copy link

doris-robot commented Aug 30, 2024

ClickBench: Total hot run time: 31.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43b97a9536caee130bc02b7a228221efd3b03551, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.08	0.08
query5	0.49	0.49	0.51
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.47
query10	0.54	0.55	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.05	2.03	2.03
query15	0.90	0.83	0.84
query16	0.38	0.36	0.37
query17	1.04	0.99	1.01
query18	0.21	0.20	0.20
query19	1.82	1.74	1.74
query20	0.01	0.01	0.01
query21	15.41	0.68	0.67
query22	3.81	7.99	1.70
query23	18.24	1.41	1.39
query24	2.09	0.23	0.21
query25	0.16	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.26	1.04	1.01
query29	12.66	3.30	3.28
query30	0.24	0.06	0.05
query31	2.87	0.39	0.39
query32	3.27	0.48	0.48
query33	2.97	3.00	3.01
query34	17.27	4.40	4.46
query35	4.49	4.52	4.45
query36	0.64	0.50	0.49
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.63 s
Total hot run time: 31.92 s

@englefly
Copy link
Contributor Author

englefly commented Aug 30, 2024

run external

@englefly englefly requested a review from Jibing-Li September 1, 2024 01:25
@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 2, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 2, 2024

PR approved by anyone and no changes requested.

Jibing-Li
Jibing-Li approved these changes Sep 2, 2024
View reviewed changes
Copy link
Contributor

@Jibing-Li Jibing-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@englefly englefly merged commit 1bc6d78 into apache:master Sep 2, 2024
@englefly englefly deleted the delta_row_est branch September 2, 2024 02:39
@englefly englefly mentioned this pull request Sep 22, 2024
Closed
@morrySnow morrySnow added dev/3.0.x and removed doing labels Oct 10, 2024
@englefly englefly mentioned this pull request Oct 17, 2024
Merged
englefly added a commit that referenced this pull request Oct 17, 2024
@englefly
[feat](nereids) adjust stats derive by delta row #39222 (2.1) (#42025)
1245df6 
## Proposed changes
pick #39222
wait JiBin merge updateRows

Issue Number: close #xxx

<!--Describe your changes.-->
englefly added a commit to englefly/incubator-doris that referenced this pull request Oct 22, 2024
@englefly
[feat](nereids) adjust stats derive by delta row (apache#39222)
fbdd9b2 
## Proposed changes
After analyzing, user may insert new rows.
analyzed rows: the rows have been analyzed
delta row: rows inserted after analyze job

if analyzed rows are filtered out, then we try to estimate filter result
by delta row with unknown column stats.
Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly mentioned this pull request Oct 22, 2024
Merged
englefly added a commit that referenced this pull request Oct 22, 2024
@englefly
[Pick](nereids)table row related Prs 3.0 (#42237)
e3efb14 
## Proposed changes
pick #41464 #40529 #40349 #39222
Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@englefly @doris-robot @starocean999 @Jibing-Li @morrySnow