Skip to content

[Pick](nereids)table row related Prs 3.0 #42237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 5 commits into from
Oct 22, 2024
Merged

[Pick](nereids)table row related Prs 3.0 #42237

englefly merged 5 commits into from
Oct 22, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Oct 22, 2024

Proposed changes

pick #41464 #40529 #40349 #39222
Issue Number: close #xxx

@englefly
[feat](nereids) adjust stats derive by delta row (apache#39222)
fbdd9b2 
## Proposed changes
After analyzing, user may insert new rows.
analyzed rows: the rows have been analyzed
delta row: rows inserted after analyze job

if analyzed rows are filtered out, then we try to estimate filter result
by delta row with unknown column stats.
Issue Number: close #xxx

<!--Describe your changes.-->
@englefly
[feat](nereids) disable join reorder if any table row count is not av…
97acac6 
…ailable (apache#40349)

 join reorder
if a table is not analyzed, and its row count is not fully reported,
optimizer will disable join reorder.
fully report means after a table is newly created or truncated, all
tablets row count are reported.

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly
pick 41464
5f73aec
@doris-robot
Copy link

doris-robot commented Oct 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

englefly commented Oct 22, 2024

run buildall

@englefly englefly changed the title [Pick] table row related prs [Pick](nereids)table row related prs Oct 22, 2024
@englefly
Copy link
Contributor Author

englefly commented Oct 22, 2024

run buildall

@englefly englefly changed the title [Pick](nereids)table row related prs [Pick](nereids)table row related Prs 3.0 Oct 22, 2024
@doris-robot
Copy link

doris-robot commented Oct 22, 2024

TPC-H: Total hot run time: 40295 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7d0a5f1035e3df616976bb37d35b702e26ce93c, data reload: false

------ Round 1 ----------------------------------
q1	17572	7326	7253	7253
q2	2024	269	274	269
q3	12085	1235	1110	1110
q4	10570	745	776	745
q5	7747	2786	2733	2733
q6	234	153	155	153
q7	1002	614	618	614
q8	9567	1841	1958	1841
q9	8148	6391	6367	6367
q10	6980	2299	2289	2289
q11	436	244	245	244
q12	411	212	212	212
q13	17762	3000	2956	2956
q14	243	204	206	204
q15	555	503	511	503
q16	672	603	591	591
q17	965	603	470	470
q18	7066	6589	6594	6589
q19	1789	859	869	859
q20	571	269	265	265
q21	3862	3099	3050	3050
q22	1067	978	983	978
Total cold run time: 111328 ms
Total hot run time: 40295 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7345	7231	7243	7231
q2	317	224	224	224
q3	2964	2861	2804	2804
q4	2078	1809	1760	1760
q5	5681	5708	5712	5708
q6	229	141	141	141
q7	2221	1778	1758	1758
q8	3323	3487	3377	3377
q9	8830	8846	8798	8798
q10	3522	3512	3475	3475
q11	574	473	473	473
q12	801	581	617	581
q13	16572	3168	3164	3164
q14	321	273	267	267
q15	562	523	526	523
q16	724	651	674	651
q17	1842	1575	1599	1575
q18	8241	7886	7409	7409
q19	4894	1513	1469	1469
q20	2129	1884	1826	1826
q21	5351	5276	5222	5222
q22	1126	1034	1014	1014
Total cold run time: 79647 ms
Total hot run time: 59450 ms

@doris-robot
Copy link

doris-robot commented Oct 22, 2024

TPC-DS: Total hot run time: 188965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7d0a5f1035e3df616976bb37d35b702e26ce93c, data reload: false

query1	889	379	387	379
query2	6230	2047	1956	1956
query3	8696	202	202	202
query4	33709	23388	23494	23388
query5	3640	462	435	435
query6	272	170	158	158
query7	4189	319	295	295
query8	249	197	206	197
query9	9443	2637	2654	2637
query10	470	275	275	275
query11	17701	15238	15178	15178
query12	142	106	96	96
query13	1565	439	400	400
query14	8768	6949	6692	6692
query15	222	173	168	168
query16	7696	467	485	467
query17	1183	551	550	550
query18	1967	322	314	314
query19	205	163	167	163
query20	125	124	114	114
query21	206	107	110	107
query22	4770	4467	4517	4467
query23	35853	33794	33536	33536
query24	11447	2791	2818	2791
query25	524	403	390	390
query26	708	158	167	158
query27	2350	298	288	288
query28	6524	2484	2448	2448
query29	646	425	431	425
query30	273	158	148	148
query31	1039	754	776	754
query32	104	52	53	52
query33	699	292	291	291
query34	913	504	501	501
query35	863	719	717	717
query36	1092	920	948	920
query37	141	89	82	82
query38	4031	3868	3892	3868
query39	1478	1448	1411	1411
query40	206	101	95	95
query41	46	44	42	42
query42	120	100	100	100
query43	530	482	487	482
query44	1159	767	767	767
query45	196	169	166	166
query46	1128	702	721	702
query47	1901	1817	1851	1817
query48	437	348	334	334
query49	881	384	389	384
query50	812	406	426	406
query51	7062	6872	6868	6868
query52	99	91	93	91
query53	252	180	182	180
query54	1135	472	472	472
query55	77	76	76	76
query56	290	257	275	257
query57	1248	1183	1125	1125
query58	238	249	231	231
query59	3104	2876	3020	2876
query60	298	272	276	272
query61	128	117	117	117
query62	876	670	665	665
query63	220	181	188	181
query64	3773	614	607	607
query65	3246	3170	3142	3142
query66	853	297	304	297
query67	15821	15160	15235	15160
query68	4634	540	547	540
query69	600	286	292	286
query70	1147	1111	1124	1111
query71	441	271	267	267
query72	7856	3850	3822	3822
query73	769	341	342	341
query74	10285	8927	8893	8893
query75	4718	2564	2662	2564
query76	3834	919	868	868
query77	747	279	277	277
query78	9916	9308	10210	9308
query79	2637	587	563	563
query80	1256	434	438	434
query81	564	248	241	241
query82	534	142	136	136
query83	387	133	135	133
query84	298	80	74	74
query85	1748	285	300	285
query86	421	306	278	278
query87	4617	4329	4238	4238
query88	3921	2407	2409	2407
query89	399	288	285	285
query90	2060	181	188	181
query91	135	106	104	104
query92	62	45	45	45
query93	3648	530	520	520
query94	999	280	295	280
query95	346	251	249	249
query96	609	278	283	278
query97	3315	3151	3137	3137
query98	217	200	206	200
query99	1550	1310	1290	1290
Total cold run time: 301828 ms
Total hot run time: 188965 ms

@englefly
Copy link
Contributor Author

englefly commented Oct 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 22, 2024

TPC-H: Total hot run time: 40359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd19e794e1de151ccc022946f18cd82d63d4aee2, data reload: false

------ Round 1 ----------------------------------
q1	17576	7329	7256	7256
q2	2015	283	274	274
q3	12105	1017	1165	1017
q4	10550	742	736	736
q5	7755	2799	2748	2748
q6	231	147	155	147
q7	1031	630	612	612
q8	9346	1870	1906	1870
q9	6559	6438	6371	6371
q10	6959	2271	2303	2271
q11	431	251	249	249
q12	404	221	220	220
q13	17791	2984	2962	2962
q14	241	222	203	203
q15	553	519	515	515
q16	652	597	597	597
q17	947	530	529	529
q18	7169	6564	6607	6564
q19	1684	973	910	910
q20	592	274	278	274
q21	3868	3249	3053	3053
q22	1056	981	986	981
Total cold run time: 109515 ms
Total hot run time: 40359 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7316	7219	7283	7219
q2	323	231	230	230
q3	2894	2798	2838	2798
q4	2108	1823	1859	1823
q5	5682	5654	5675	5654
q6	225	147	150	147
q7	2203	1790	1751	1751
q8	3260	3492	3475	3475
q9	8807	8853	8788	8788
q10	3508	3508	3501	3501
q11	596	487	490	487
q12	777	577	575	575
q13	16418	3171	3109	3109
q14	300	265	264	264
q15	568	509	519	509
q16	694	664	672	664
q17	1818	1628	1607	1607
q18	8174	7640	7556	7556
q19	3303	1536	1486	1486
q20	2097	1860	1866	1860
q21	5328	5239	5277	5239
q22	1135	991	1004	991
Total cold run time: 77534 ms
Total hot run time: 59733 ms

@doris-robot
Copy link

doris-robot commented Oct 22, 2024

TPC-DS: Total hot run time: 191446 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd19e794e1de151ccc022946f18cd82d63d4aee2, data reload: false

query1	935	389	377	377
query2	6220	2143	2048	2048
query3	8721	200	210	200
query4	33593	23455	23533	23455
query5	3792	468	471	468
query6	273	169	170	169
query7	4198	307	301	301
query8	258	216	231	216
query9	9322	2651	2646	2646
query10	468	285	285	285
query11	17914	15134	15270	15134
query12	150	106	110	106
query13	1592	466	435	435
query14	9821	7030	7461	7030
query15	213	170	183	170
query16	7634	463	459	459
query17	1253	616	608	608
query18	1468	317	325	317
query19	219	164	157	157
query20	121	116	116	116
query21	209	110	107	107
query22	4693	4519	4643	4519
query23	34828	34292	34303	34292
query24	11573	3007	2888	2888
query25	605	400	407	400
query26	731	168	169	168
query27	2092	304	310	304
query28	6471	2496	2456	2456
query29	746	439	436	436
query30	282	160	156	156
query31	1022	777	748	748
query32	92	53	54	53
query33	642	298	297	297
query34	956	519	502	502
query35	863	724	743	724
query36	1116	949	939	939
query37	148	88	84	84
query38	4041	3898	3920	3898
query39	1469	1437	1419	1419
query40	202	100	99	99
query41	48	46	42	42
query42	125	100	100	100
query43	525	489	479	479
query44	1161	788	775	775
query45	195	165	165	165
query46	1171	727	716	716
query47	1904	1796	1819	1796
query48	439	349	341	341
query49	887	412	375	375
query50	837	425	412	412
query51	7080	6804	6935	6804
query52	102	96	89	89
query53	275	199	184	184
query54	1167	488	485	485
query55	76	81	76	76
query56	290	288	244	244
query57	1211	1122	1139	1122
query58	228	228	224	224
query59	3072	2915	2861	2861
query60	311	279	274	274
query61	106	103	96	96
query62	815	655	650	650
query63	221	187	181	181
query64	3697	643	634	634
query65	3295	3203	3207	3203
query66	787	314	308	308
query67	15972	15369	15381	15369
query68	4597	534	559	534
query69	611	288	286	286
query70	1149	1103	1091	1091
query71	445	304	272	272
query72	7900	3897	3924	3897
query73	785	345	350	345
query74	10424	8881	8937	8881
query75	4675	2617	2663	2617
query76	3569	932	985	932
query77	727	296	296	296
query78	9980	9477	10458	9477
query79	6192	615	590	590
query80	2405	457	447	447
query81	579	240	241	240
query82	767	140	144	140
query83	314	136	139	136
query84	293	81	76	76
query85	2148	304	285	285
query86	465	299	289	289
query87	4398	4374	4391	4374
query88	4380	2420	2456	2420
query89	475	290	296	290
query90	2139	189	187	187
query91	139	108	110	108
query92	63	47	46	46
query93	5234	534	533	533
query94	1169	289	290	289
query95	354	250	261	250
query96	644	288	281	281
query97	3305	3148	3145	3145
query98	230	198	204	198
query99	1601	1310	1303	1303
Total cold run time: 309099 ms
Total hot run time: 191446 ms

@englefly
Copy link
Contributor Author

englefly commented Oct 22, 2024

run p0

@englefly englefly merged commit e3efb14 into apache:branch-3.0 Oct 22, 2024
@englefly englefly deleted the pick-table-row branch November 30, 2024 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@englefly @doris-robot