Skip to content

branch-3.1: [enhance](paimon) opt count pushdown for paimon and refactor be logic (#46911) #52060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 3 commits into from
Jun 24, 2025
Merged

branch-3.1: [enhance](paimon) opt count pushdown for paimon and refactor be logic (#46911) #52060

morrySnow merged 3 commits into from
Jun 24, 2025

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jun 20, 2025

bp #46911

@suxiaogang223 suxiaogang223 requested a review from morrySnow as a code owner June 20, 2025 07:56
@Thearas
Copy link
Contributor

Thearas commented Jun 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 20, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 20, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.19% (1099/1321)
Line Coverage 66.46% (18510/27852)
Region Coverage 66.19% (9156/13833)
Branch Coverage 55.99% (4946/8834)

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 20, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 20, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.19% (1099/1321)
Line Coverage 66.44% (18584/27969)
Region Coverage 66.21% (9189/13878)
Branch Coverage 56.10% (4981/8878)

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 21, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 21, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.15% (1105/1329)
Line Coverage 66.25% (18796/28370)
Region Coverage 66.02% (9361/14178)
Branch Coverage 55.89% (5055/9044)

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.15% (1105/1329)
Line Coverage 66.25% (18794/28370)
Region Coverage 65.96% (9352/14178)
Branch Coverage 55.85% (5051/9044)

@suxiaogang223
[enhance](paimon) opt count pushdown for paimon and refactor be logic (…
2b7a1d2 
…apache#46911)

Related PR: apache#44038

1. Obtain directly from statistics for select count(*) from paimon_table
2. Refactor TableFormatReader, move same logic of IcebergTableReader,
PaimonTableReader and TransactionalHiveReader to TableFormatReader
@morrySnow
Copy link
Contributor

morrySnow commented Jun 23, 2025

run buildall

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.06% (1108/1334)
Line Coverage 66.17% (18848/28484)
Region Coverage 66.02% (9399/14237)
Branch Coverage 55.90% (5081/9090)

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

TPC-H: Total hot run time: 40040 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0199a6d96c9301a26b24126a03b78d961ea629ef, data reload: false

------ Round 1 ----------------------------------
q1	17653	7048	6618	6618
q2	2074	169	166	166
q3	10850	1114	1170	1114
q4	10257	807	779	779
q5	7747	2888	2879	2879
q6	216	135	134	134
q7	1005	631	603	603
q8	9359	2014	2035	2014
q9	6681	6413	6420	6413
q10	6993	2279	2302	2279
q11	477	265	253	253
q12	398	214	202	202
q13	17792	3020	2987	2987
q14	232	221	210	210
q15	512	462	468	462
q16	466	374	376	374
q17	987	537	519	519
q18	7394	6666	6647	6647
q19	1311	1040	1012	1012
q20	469	206	198	198
q21	3915	3201	3190	3190
q22	1079	995	987	987
Total cold run time: 107867 ms
Total hot run time: 40040 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6586	6572	6566	6566
q2	338	233	240	233
q3	2944	2735	2873	2735
q4	2005	1782	1833	1782
q5	5776	5739	5760	5739
q6	207	125	130	125
q7	2172	1758	1764	1758
q8	3393	3523	3572	3523
q9	8928	8787	8852	8787
q10	3577	3536	3459	3459
q11	602	493	492	492
q12	827	602	618	602
q13	8563	3181	3199	3181
q14	295	271	273	271
q15	505	467	457	457
q16	479	429	441	429
q17	1850	1632	1615	1615
q18	8287	7742	7925	7742
q19	1712	1478	1659	1478
q20	2158	1820	1796	1796
q21	5115	5021	4989	4989
q22	1123	1054	1048	1048
Total cold run time: 67442 ms
Total hot run time: 58807 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

TPC-DS: Total hot run time: 196818 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0199a6d96c9301a26b24126a03b78d961ea629ef, data reload: false

query1	1296	914	923	914
query2	6327	1846	1878	1846
query3	10833	4305	4386	4305
query4	61762	29303	23473	23473
query5	5180	465	458	458
query6	407	185	182	182
query7	5482	318	314	314
query8	329	235	228	228
query9	8673	2602	2616	2602
query10	447	277	251	251
query11	17823	15161	15750	15161
query12	173	109	104	104
query13	1498	450	440	440
query14	9898	7551	7528	7528
query15	209	180	172	172
query16	7194	484	489	484
query17	1146	578	606	578
query18	1892	318	308	308
query19	212	156	162	156
query20	120	118	110	110
query21	203	104	119	104
query22	4651	4481	4671	4481
query23	34841	33575	34152	33575
query24	6149	2906	2934	2906
query25	575	452	420	420
query26	695	170	172	170
query27	2067	360	357	357
query28	4203	2188	2169	2169
query29	723	431	440	431
query30	234	159	153	153
query31	1009	802	839	802
query32	69	54	54	54
query33	402	294	279	279
query34	941	521	555	521
query35	856	762	715	715
query36	1072	954	960	954
query37	113	69	71	69
query38	4056	3919	4004	3919
query39	1543	1479	1456	1456
query40	199	105	97	97
query41	52	46	47	46
query42	113	112	103	103
query43	534	474	475	474
query44	1217	850	817	817
query45	190	170	173	170
query46	1161	746	729	729
query47	2026	1932	1906	1906
query48	439	352	349	349
query49	747	402	398	398
query50	815	427	419	419
query51	7467	7350	7311	7311
query52	99	91	91	91
query53	259	183	185	183
query54	569	484	458	458
query55	80	77	75	75
query56	278	254	245	245
query57	1323	1188	1195	1188
query58	219	224	211	211
query59	3249	3096	3058	3058
query60	283	258	259	258
query61	109	104	111	104
query62	801	705	699	699
query63	222	187	185	185
query64	2031	691	630	630
query65	3380	3290	3288	3288
query66	736	292	287	287
query67	15881	15650	15411	15411
query68	4016	584	582	582
query69	434	261	272	261
query70	1214	1105	1112	1105
query71	348	271	258	258
query72	6425	4068	4017	4017
query73	739	348	356	348
query74	10223	9246	8980	8980
query75	3309	2674	2675	2674
query76	1964	1118	1035	1035
query77	503	268	266	266
query78	10623	9601	9571	9571
query79	1527	593	600	593
query80	880	429	432	429
query81	495	212	219	212
query82	1273	95	85	85
query83	243	140	138	138
query84	282	82	75	75
query85	890	307	289	289
query86	331	303	306	303
query87	4448	4254	4277	4254
query88	3761	2390	2348	2348
query89	417	288	286	286
query90	2022	183	186	183
query91	139	108	115	108
query92	63	49	58	49
query93	1987	559	558	558
query94	732	303	300	300
query95	358	260	255	255
query96	606	282	282	282
query97	3328	3134	3125	3125
query98	223	197	199	197
query99	1583	1313	1321	1313
Total cold run time: 315168 ms
Total hot run time: 196818 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

ClickBench: Total hot run time: 31.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0199a6d96c9301a26b24126a03b78d961ea629ef, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.64	0.09	0.08
query5	0.51	0.52	0.52
query6	1.14	0.74	0.75
query7	0.03	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.49	0.50
query10	0.58	0.56	0.58
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.61	0.60	0.60
query14	0.82	0.79	0.82
query15	0.86	0.83	0.84
query16	0.38	0.38	0.37
query17	1.10	1.06	1.05
query18	0.19	0.19	0.19
query19	1.88	1.72	1.80
query20	0.02	0.01	0.01
query21	15.43	0.68	0.67
query22	3.98	5.97	2.30
query23	18.29	1.33	1.23
query24	2.21	0.23	0.23
query25	0.14	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.28	0.59	0.56
query29	12.64	3.35	3.35
query30	0.24	0.06	0.05
query31	2.88	0.40	0.40
query32	3.23	0.48	0.48
query33	2.95	3.01	3.03
query34	17.04	4.53	4.50
query35	4.62	4.54	4.53
query36	0.67	0.49	0.48
query37	0.19	0.16	0.16
query38	0.16	0.16	0.16
query39	0.05	0.04	0.04
query40	0.18	0.13	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.77 s
Total hot run time: 31.1 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/83) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.13% (10901/26501)
Line Coverage 31.95% (93307/292072)
Region Coverage 31.02% (48105/155099)
Branch Coverage 27.49% (24644/89656)

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 23, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.06% (1108/1334)
Line Coverage 66.16% (18846/28484)
Region Coverage 66.02% (9399/14237)
Branch Coverage 55.84% (5076/9090)

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/83) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.14% (10902/26501)
Line Coverage 31.95% (93327/292072)
Region Coverage 31.01% (48101/155099)
Branch Coverage 27.49% (24646/89656)

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 23, 2025

run external

1 similar comment
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 23, 2025

run external

@morrySnow morrySnow merged commit 52d7650 into apache:branch-3.1 Jun 24, 2025
18 of 21 checks passed
@suxiaogang223 suxiaogang223 deleted the table_count_opt_3.1 branch June 27, 2025 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@suxiaogang223 @Thearas @hello-stephen @doris-robot @morrySnow