Skip to content

[fix](paimon) fix select count(*) from paimon table #51834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 6 commits into from
Jun 24, 2025
Merged

[fix](paimon) fix select count(*) from paimon table #51834

morningman merged 6 commits into from
Jun 24, 2025

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jun 17, 2025 •
edited
Loading

What problem does this PR solve?

Related PR: #46911

  1. Fixed the bug of querying count(*) from paimon table which is merge on read
  2. For each split of the Paimon table, determine whether count(*) optimization can be performed.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 17, 2025

run external

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

TPC-H: Total hot run time: 34102 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cddb856020306fb65b78e3ab0b78344a965e7df2, data reload: false

------ Round 1 ----------------------------------
q1	17598	5136	5110	5110
q2	1932	282	189	189
q3	10307	1300	734	734
q4	10224	1018	526	526
q5	7547	2408	2305	2305
q6	180	164	131	131
q7	899	733	611	611
q8	9313	1237	1082	1082
q9	6736	5066	5132	5066
q10	6939	2357	1961	1961
q11	490	282	283	282
q12	354	356	230	230
q13	17769	3646	3067	3067
q14	230	227	219	219
q15	574	484	480	480
q16	435	428	368	368
q17	600	851	370	370
q18	7514	7263	7237	7237
q19	2771	962	539	539
q20	326	333	221	221
q21	3697	3222	2400	2400
q22	1060	1016	974	974
Total cold run time: 107495 ms
Total hot run time: 34102 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5422	5214	5145	5145
q2	245	318	230	230
q3	2133	2623	2242	2242
q4	1373	1800	1346	1346
q5	4227	4202	4367	4202
q6	214	167	129	129
q7	2030	1891	1779	1779
q8	2605	2562	2510	2510
q9	7208	7064	7106	7064
q10	3053	3232	2826	2826
q11	577	509	497	497
q12	690	773	619	619
q13	3579	3891	3228	3228
q14	288	313	295	295
q15	527	487	485	485
q16	448	478	462	462
q17	1155	1502	1387	1387
q18	7349	7129	7185	7129
q19	811	875	1055	875
q20	1923	2028	1865	1865
q21	4799	4326	4233	4233
q22	1079	1039	1019	1019
Total cold run time: 51735 ms
Total hot run time: 49567 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

TPC-DS: Total hot run time: 185327 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cddb856020306fb65b78e3ab0b78344a965e7df2, data reload: false

query1	984	375	415	375
query2	6564	1837	1849	1837
query3	6737	230	225	225
query4	26685	23654	23277	23277
query5	4373	624	451	451
query6	323	209	199	199
query7	4618	491	284	284
query8	261	233	222	222
query9	8634	2632	2647	2632
query10	458	338	272	272
query11	15659	14973	14816	14816
query12	176	109	107	107
query13	1658	534	444	444
query14	9702	6139	6217	6139
query15	214	213	175	175
query16	7476	653	471	471
query17	1204	703	576	576
query18	2045	406	305	305
query19	206	208	164	164
query20	125	112	121	112
query21	221	134	109	109
query22	4098	4180	3979	3979
query23	33807	32953	33062	32953
query24	8389	2373	2398	2373
query25	559	485	384	384
query26	1238	269	155	155
query27	2726	511	342	342
query28	4308	2107	2093	2093
query29	714	536	424	424
query30	281	234	189	189
query31	923	851	744	744
query32	71	67	61	61
query33	552	387	314	314
query34	806	850	512	512
query35	786	819	724	724
query36	936	965	885	885
query37	108	103	76	76
query38	4143	4125	4118	4118
query39	1461	1468	1399	1399
query40	209	123	111	111
query41	63	57	60	57
query42	121	110	108	108
query43	494	491	456	456
query44	1304	836	832	832
query45	178	172	176	172
query46	842	1015	618	618
query47	1769	1761	1709	1709
query48	393	423	309	309
query49	754	478	422	422
query50	639	686	407	407
query51	4239	4136	4088	4088
query52	114	105	104	104
query53	221	250	189	189
query54	574	571	491	491
query55	91	85	90	85
query56	309	308	285	285
query57	1177	1191	1123	1123
query58	261	263	257	257
query59	2622	2607	2528	2528
query60	333	310	308	308
query61	129	123	141	123
query62	796	690	658	658
query63	233	192	192	192
query64	4249	1012	666	666
query65	4234	4202	4167	4167
query66	1076	412	322	322
query67	15782	15787	15486	15486
query68	7935	865	585	585
query69	476	295	267	267
query70	1165	1141	1130	1130
query71	474	312	294	294
query72	5527	4625	4609	4609
query73	687	569	353	353
query74	8915	8883	8641	8641
query75	3777	3193	2684	2684
query76	3607	1184	750	750
query77	777	382	287	287
query78	9920	10191	9440	9440
query79	2218	825	574	574
query80	604	505	440	440
query81	495	255	217	217
query82	475	130	100	100
query83	252	254	239	239
query84	251	108	90	90
query85	861	346	312	312
query86	390	307	289	289
query87	4373	4350	4343	4343
query88	3718	2301	2273	2273
query89	387	321	279	279
query90	1860	204	211	204
query91	146	157	122	122
query92	75	64	56	56
query93	1644	926	597	597
query94	660	408	294	294
query95	385	292	285	285
query96	504	562	275	275
query97	2733	2775	2641	2641
query98	244	209	206	206
query99	1648	1389	1261	1261
Total cold run time: 274618 ms
Total hot run time: 185327 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

ClickBench: Total hot run time: 29.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cddb856020306fb65b78e3ab0b78344a965e7df2, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.10
query5	0.43	0.42	0.40
query6	1.17	0.65	0.67
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.58	0.52	0.52
query10	0.58	0.58	0.56
query11	0.15	0.10	0.11
query12	0.16	0.11	0.11
query13	0.62	0.60	0.60
query14	0.80	0.80	0.83
query15	0.89	0.84	0.86
query16	0.38	0.38	0.39
query17	1.05	1.06	1.04
query18	0.23	0.21	0.22
query19	1.87	1.85	1.80
query20	0.01	0.01	0.01
query21	15.39	0.93	0.56
query22	0.76	1.15	0.71
query23	14.90	1.38	0.67
query24	6.88	2.21	1.00
query25	0.52	0.16	0.13
query26	0.67	0.16	0.14
query27	0.05	0.05	0.05
query28	10.41	0.86	0.44
query29	12.58	3.95	3.31
query30	0.25	0.10	0.06
query31	2.88	0.60	0.39
query32	3.23	0.56	0.46
query33	3.07	3.04	3.16
query34	16.20	5.32	4.76
query35	4.78	4.83	4.78
query36	0.69	0.50	0.49
query37	0.09	0.07	0.07
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.15	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.69 s
Total hot run time: 29.61 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

FE UT Coverage Report

Increment line coverage 50.91% (28/55) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-H: Total hot run time: 34018 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac684ed928c34eaabf6157ce07596b670a1307b3, data reload: false

------ Round 1 ----------------------------------
q1	17606	5160	5040	5040
q2	1950	297	198	198
q3	10257	1323	733	733
q4	10249	1044	538	538
q5	8068	2311	2386	2311
q6	210	165	132	132
q7	879	748	620	620
q8	9307	1328	1069	1069
q9	7079	5093	5134	5093
q10	6937	2399	1951	1951
q11	498	296	280	280
q12	352	348	218	218
q13	17801	3665	3147	3147
q14	229	240	215	215
q15	571	484	487	484
q16	433	423	375	375
q17	620	855	379	379
q18	7573	7142	7241	7142
q19	1563	950	557	557
q20	333	342	220	220
q21	3741	3193	2359	2359
q22	1048	1008	957	957
Total cold run time: 107304 ms
Total hot run time: 34018 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5255	5113	5085	5085
q2	245	328	221	221
q3	2201	2616	2293	2293
q4	1371	1802	1359	1359
q5	4200	4329	4411	4329
q6	221	174	125	125
q7	1995	1912	1817	1817
q8	2610	2655	2535	2535
q9	7107	7055	7139	7055
q10	3053	3224	2846	2846
q11	590	525	514	514
q12	665	782	618	618
q13	3528	3857	3263	3263
q14	283	294	285	285
q15	504	477	492	477
q16	453	513	453	453
q17	1151	1486	1353	1353
q18	7463	7230	7009	7009
q19	767	732	789	732
q20	1908	1986	1884	1884
q21	4705	4390	4180	4180
q22	1093	1033	994	994
Total cold run time: 51368 ms
Total hot run time: 49427 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-DS: Total hot run time: 185892 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac684ed928c34eaabf6157ce07596b670a1307b3, data reload: false

query1	986	403	415	403
query2	6541	1826	1840	1826
query3	6742	220	216	216
query4	26126	23416	22925	22925
query5	4353	626	484	484
query6	330	218	205	205
query7	4630	511	295	295
query8	291	229	215	215
query9	8621	2636	2629	2629
query10	471	331	293	293
query11	15646	15035	14838	14838
query12	170	116	110	110
query13	1667	546	429	429
query14	10107	6258	6309	6258
query15	220	200	173	173
query16	7660	649	461	461
query17	1207	751	610	610
query18	2032	411	315	315
query19	208	197	180	180
query20	133	115	127	115
query21	225	132	108	108
query22	4072	4090	3988	3988
query23	34095	33022	33171	33022
query24	8339	2439	2396	2396
query25	521	477	387	387
query26	1254	269	152	152
query27	2728	533	350	350
query28	4284	2146	2124	2124
query29	743	556	443	443
query30	287	213	194	194
query31	943	829	753	753
query32	74	66	69	66
query33	559	377	303	303
query34	808	876	540	540
query35	810	870	737	737
query36	935	994	895	895
query37	113	103	77	77
query38	4147	4104	4094	4094
query39	1523	1421	1403	1403
query40	211	122	108	108
query41	64	59	80	59
query42	135	113	109	109
query43	499	511	451	451
query44	1382	837	838	837
query45	180	177	173	173
query46	909	1057	653	653
query47	1731	1788	1699	1699
query48	387	452	317	317
query49	748	482	418	418
query50	655	700	408	408
query51	4186	4162	4141	4141
query52	112	114	105	105
query53	226	263	185	185
query54	595	566	514	514
query55	93	86	86	86
query56	304	318	290	290
query57	1194	1176	1111	1111
query58	273	274	269	269
query59	2524	2646	2633	2633
query60	329	323	319	319
query61	128	125	153	125
query62	814	718	671	671
query63	226	202	191	191
query64	4253	1026	678	678
query65	4276	4157	4153	4153
query66	1065	422	321	321
query67	15696	15628	15315	15315
query68	8197	901	537	537
query69	462	312	287	287
query70	1168	1137	1107	1107
query71	484	320	303	303
query72	5821	4719	4694	4694
query73	717	619	354	354
query74	8916	9014	8978	8978
query75	3886	3180	2687	2687
query76	3717	1213	788	788
query77	797	415	321	321
query78	10098	10178	9300	9300
query79	2297	841	603	603
query80	712	503	447	447
query81	482	273	225	225
query82	442	128	100	100
query83	279	254	233	233
query84	299	109	85	85
query85	785	361	325	325
query86	342	319	309	309
query87	4332	4471	4301	4301
query88	3460	2324	2297	2297
query89	387	334	284	284
query90	1972	216	212	212
query91	152	146	113	113
query92	79	62	61	61
query93	1566	969	590	590
query94	687	422	312	312
query95	387	296	289	289
query96	508	560	281	281
query97	2747	2797	2635	2635
query98	228	215	207	207
query99	1456	1383	1276	1276
Total cold run time: 275658 ms
Total hot run time: 185892 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

ClickBench: Total hot run time: 29.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ac684ed928c34eaabf6157ce07596b670a1307b3, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.42	0.40
query6	1.17	0.65	0.67
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.51	0.53
query10	0.57	0.58	0.57
query11	0.15	0.12	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.62
query14	0.79	0.81	0.80
query15	0.91	0.86	0.90
query16	0.39	0.37	0.40
query17	1.05	1.07	1.10
query18	0.22	0.21	0.21
query19	2.03	1.80	1.87
query20	0.01	0.01	0.02
query21	15.36	0.91	0.56
query22	0.75	1.09	0.60
query23	15.11	1.36	0.66
query24	6.65	1.61	0.83
query25	0.45	0.12	0.06
query26	0.62	0.16	0.14
query27	0.07	0.06	0.05
query28	9.27	0.91	0.45
query29	12.54	3.98	3.30
query30	0.26	0.10	0.07
query31	2.83	0.58	0.38
query32	3.24	0.56	0.46
query33	3.03	3.13	3.04
query34	15.92	5.39	4.77
query35	4.81	4.82	4.82
query36	0.71	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.15	0.15
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.22 s
Total hot run time: 29.38 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit f3d44a5 into apache:master Jun 24, 2025
27 of 28 checks passed
@suxiaogang223 suxiaogang223 mentioned this pull request Jun 25, 2025
Merged
16 tasks
hello-stephen pushed a commit that referenced this pull request Jun 25, 2025
@suxiaogang223
[fix](paimon) fix test_paimon_count case (#52263)
834fdb3 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #51834
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Jun 25, 2025
@suxiaogang223
[fix](paimon) fix select count(*) from paimon table (apache#51834)
597f8cc 
Related PR: apache#46911

1. Fixed the bug of querying count(*) from paimon table which is `merge
on read`
2. For each split of the Paimon table, determine whether count(*)
optimization can be performed.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Jun 25, 2025
@suxiaogang223
[fix](paimon) fix test_paimon_count case (apache#52263)
661dd6f 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#51834
@suxiaogang223 suxiaogang223 mentioned this pull request Jun 25, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 26, 2025
@suxiaogang223
branch-3.1: [fix](paimon) fix select count(*) from paimon table #51834 …
dbf0a55 
…#52263 (#52301)

main pick: #51834 
fix case: #52263
@suxiaogang223 suxiaogang223 deleted the fix_paimon_count branch June 27, 2025 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+1 more reviewer

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@suxiaogang223 @Thearas @doris-robot @hello-stephen @morningman @hubgeter @morrySnow