Skip to content

[follow-up](nereids) speedup huge InPredicate for tablet pruning #49386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 1 commit into from
Mar 26, 2025
Merged

[follow-up](nereids) speedup huge InPredicate for tablet pruning #49386

924060929 merged 1 commit into from
Mar 26, 2025

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Mar 24, 2025 •
edited
Loading

What problem does this PR solve?

skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up #47608

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929 924060929 added dev/2.1.x dev/3.0.x dev/3.1.x usercase Important user case type label labels Mar 24, 2025
@924060929
Copy link
Contributor Author

924060929 commented Mar 24, 2025

run buildall

This was referenced Mar 24, 2025
Merged
Merged
@doris-robot
Copy link

doris-robot commented Mar 24, 2025

TPC-H: Total hot run time: 34331 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7df7ae6c335af264d6cd28ec2e8ff1be662bc920, data reload: false

------ Round 1 ----------------------------------
q1	24437	5109	5051	5051
q2	2042	321	190	190
q3	10370	1271	680	680
q4	10225	989	532	532
q5	7530	2359	2329	2329
q6	187	161	134	134
q7	892	742	623	623
q8	9331	1294	1118	1118
q9	6960	5161	5246	5161
q10	6800	2337	1883	1883
q11	486	272	261	261
q12	343	353	222	222
q13	17761	3656	3067	3067
q14	225	228	206	206
q15	525	489	496	489
q16	623	613	587	587
q17	592	859	344	344
q18	7721	7230	7253	7230
q19	1225	950	573	573
q20	328	331	195	195
q21	4126	3468	2514	2514
q22	1071	1029	942	942
Total cold run time: 113800 ms
Total hot run time: 34331 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5170	5116	5127	5116
q2	232	334	225	225
q3	2192	2623	2278	2278
q4	1441	1818	1403	1403
q5	4532	4502	4393	4393
q6	212	177	128	128
q7	1942	1874	1721	1721
q8	2575	2551	2538	2538
q9	7234	7138	7121	7121
q10	3001	3159	2762	2762
q11	570	505	486	486
q12	727	785	634	634
q13	3498	3877	3225	3225
q14	296	302	267	267
q15	530	478	485	478
q16	669	666	632	632
q17	1110	1565	1332	1332
q18	7863	7619	7409	7409
q19	820	825	947	825
q20	1925	2000	1855	1855
q21	5371	4729	4570	4570
q22	1096	1022	1005	1005
Total cold run time: 53006 ms
Total hot run time: 50403 ms

@doris-robot
Copy link

doris-robot commented Mar 24, 2025

TPC-DS: Total hot run time: 186900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7df7ae6c335af264d6cd28ec2e8ff1be662bc920, data reload: false

query1	1018	473	479	473
query2	6541	1958	1949	1949
query3	6801	219	217	217
query4	26442	23771	23285	23285
query5	4306	662	484	484
query6	298	211	176	176
query7	4602	489	273	273
query8	296	245	237	237
query9	8575	2593	2586	2586
query10	467	303	256	256
query11	15480	15093	14845	14845
query12	158	110	106	106
query13	1655	518	406	406
query14	9463	6944	6328	6328
query15	228	191	167	167
query16	7779	638	489	489
query17	1228	707	558	558
query18	2020	405	302	302
query19	190	180	159	159
query20	124	119	115	115
query21	213	130	110	110
query22	4121	4341	4113	4113
query23	34027	33113	33123	33113
query24	7691	2373	2341	2341
query25	515	453	387	387
query26	1218	261	141	141
query27	2297	480	315	315
query28	4080	2416	2388	2388
query29	717	536	420	420
query30	284	216	188	188
query31	918	850	772	772
query32	74	59	67	59
query33	555	354	328	328
query34	775	818	485	485
query35	782	804	750	750
query36	941	991	895	895
query37	121	98	78	78
query38	4134	4125	4124	4124
query39	1448	1385	1401	1385
query40	208	113	106	106
query41	56	53	49	49
query42	123	100	103	100
query43	477	504	457	457
query44	1275	792	787	787
query45	175	172	165	165
query46	848	1011	602	602
query47	1770	1821	1722	1722
query48	374	412	309	309
query49	821	529	430	430
query50	675	730	417	417
query51	4228	4146	4166	4146
query52	102	103	92	92
query53	213	250	180	180
query54	481	477	405	405
query55	78	77	79	77
query56	261	279	274	274
query57	1146	1155	1068	1068
query58	247	234	253	234
query59	2630	2946	2688	2688
query60	278	288	253	253
query61	127	125	121	121
query62	810	742	648	648
query63	228	183	177	177
query64	4258	997	651	651
query65	4432	4295	4324	4295
query66	1116	402	295	295
query67	15779	15323	15478	15323
query68	8161	863	499	499
query69	483	296	263	263
query70	1165	1091	1015	1015
query71	496	296	269	269
query72	5619	5174	5137	5137
query73	705	662	340	340
query74	9219	8841	8854	8841
query75	3917	3197	2712	2712
query76	3736	1171	737	737
query77	784	437	273	273
query78	10036	10036	9334	9334
query79	2642	810	562	562
query80	650	508	430	430
query81	506	265	225	225
query82	705	125	96	96
query83	180	169	159	159
query84	243	93	75	75
query85	784	348	304	304
query86	389	299	276	276
query87	4453	4458	4562	4458
query88	3792	2215	2257	2215
query89	375	318	271	271
query90	1851	204	208	204
query91	148	147	109	109
query92	72	60	58	58
query93	1874	1044	573	573
query94	661	420	300	300
query95	357	270	263	263
query96	474	559	281	281
query97	3319	3413	3269	3269
query98	230	201	199	199
query99	1446	1389	1279	1279
Total cold run time: 275270 ms
Total hot run time: 186900 ms

@doris-robot
Copy link

doris-robot commented Mar 24, 2025

ClickBench: Total hot run time: 31.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7df7ae6c335af264d6cd28ec2e8ff1be662bc920, data reload: false

query1	0.03	0.04	0.03
query2	0.11	0.11	0.10
query3	0.24	0.18	0.20
query4	1.59	0.17	0.19
query5	0.56	0.54	0.56
query6	1.20	0.71	0.72
query7	0.03	0.01	0.02
query8	0.05	0.03	0.03
query9	0.58	0.54	0.53
query10	0.59	0.59	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.59
query14	2.67	2.81	2.82
query15	0.91	0.85	0.87
query16	0.39	0.38	0.38
query17	1.00	1.05	1.00
query18	0.21	0.19	0.19
query19	2.05	1.85	1.76
query20	0.02	0.01	0.01
query21	15.37	0.92	0.53
query22	0.76	1.16	0.70
query23	14.87	1.39	0.60
query24	6.94	1.23	1.13
query25	0.53	0.22	0.15
query26	0.55	0.16	0.13
query27	0.04	0.05	0.05
query28	9.77	0.92	0.43
query29	12.56	4.05	3.36
query30	0.25	0.09	0.06
query31	2.84	0.59	0.38
query32	3.23	0.54	0.48
query33	2.97	3.00	2.99
query34	15.90	5.16	4.51
query35	4.61	4.54	4.53
query36	0.67	0.50	0.47
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.49 s
Total hot run time: 31.48 s

@924060929
Copy link
Contributor Author

924060929 commented Mar 25, 2025

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 25, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 25, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 25, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

TPC-H: Total hot run time: 33862 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7df7ae6c335af264d6cd28ec2e8ff1be662bc920, data reload: false

------ Round 1 ----------------------------------
q1	25811	5082	5011	5011
q2	2075	289	164	164
q3	10395	1237	658	658
q4	10225	990	534	534
q5	7552	2323	2321	2321
q6	180	162	133	133
q7	909	746	612	612
q8	9329	1278	1137	1137
q9	6857	5116	5134	5116
q10	6828	2324	1874	1874
q11	487	271	258	258
q12	347	352	217	217
q13	17758	3642	3077	3077
q14	218	224	205	205
q15	538	469	484	469
q16	630	625	580	580
q17	568	846	334	334
q18	7599	7165	7039	7039
q19	1230	924	554	554
q20	322	335	201	201
q21	3837	3366	2407	2407
q22	1021	1008	961	961
Total cold run time: 114716 ms
Total hot run time: 33862 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5136	5061	5121	5061
q2	233	326	224	224
q3	2090	2663	2236	2236
q4	1362	1758	1357	1357
q5	4430	4407	4379	4379
q6	210	170	126	126
q7	1974	1924	1853	1853
q8	2617	2627	2563	2563
q9	7434	7373	7270	7270
q10	3034	3157	2729	2729
q11	583	508	474	474
q12	720	754	609	609
q13	3476	3889	3344	3344
q14	310	298	287	287
q15	526	465	471	465
q16	641	687	652	652
q17	1137	1572	1410	1410
q18	7735	7646	7558	7558
q19	791	823	942	823
q20	1896	1983	1839	1839
q21	5440	4988	4954	4954
q22	1079	1083	1021	1021
Total cold run time: 52854 ms
Total hot run time: 51234 ms

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

TPC-DS: Total hot run time: 194093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7df7ae6c335af264d6cd28ec2e8ff1be662bc920, data reload: false

query1	1454	1065	1061	1061
query2	6203	1953	1939	1939
query3	11052	4543	4428	4428
query4	52154	25457	23616	23616
query5	5116	659	524	524
query6	358	196	192	192
query7	4889	494	282	282
query8	316	253	238	238
query9	5566	2613	2607	2607
query10	452	309	261	261
query11	15256	15009	15187	15009
query12	162	115	112	112
query13	1098	508	394	394
query14	10276	6289	6278	6278
query15	195	192	180	180
query16	7149	655	476	476
query17	1069	714	579	579
query18	1599	403	321	321
query19	194	195	166	166
query20	131	121	121	121
query21	208	123	105	105
query22	4376	4294	4573	4294
query23	34003	33600	33448	33448
query24	7015	2443	2453	2443
query25	458	467	402	402
query26	676	283	153	153
query27	2316	498	348	348
query28	3134	2452	2452	2452
query29	603	576	440	440
query30	274	227	190	190
query31	877	870	824	824
query32	73	63	62	62
query33	472	363	307	307
query34	782	851	496	496
query35	811	857	767	767
query36	978	977	930	930
query37	120	102	78	78
query38	4127	4244	4119	4119
query39	1530	1455	1432	1432
query40	217	113	105	105
query41	54	51	51	51
query42	119	114	105	105
query43	516	498	480	480
query44	1299	832	816	816
query45	180	180	174	174
query46	851	1045	667	667
query47	1859	1926	1844	1844
query48	378	413	307	307
query49	707	521	443	443
query50	689	743	428	428
query51	4344	4244	4287	4244
query52	106	107	105	105
query53	232	260	219	219
query54	499	493	419	419
query55	89	82	90	82
query56	294	280	263	263
query57	1191	1217	1125	1125
query58	263	237	248	237
query59	2735	2873	2743	2743
query60	303	298	274	274
query61	137	128	128	128
query62	726	759	689	689
query63	227	190	190	190
query64	1504	1046	737	737
query65	4613	4505	4478	4478
query66	726	400	313	313
query67	15996	15554	15324	15324
query68	6562	872	505	505
query69	548	307	254	254
query70	1200	1122	1099	1099
query71	457	312	270	270
query72	6053	5068	4996	4996
query73	1296	568	337	337
query74	9039	8928	8737	8737
query75	3258	3207	2779	2779
query76	3869	1185	739	739
query77	504	368	290	290
query78	10152	9919	9276	9276
query79	2737	805	567	567
query80	683	513	441	441
query81	594	265	229	229
query82	696	125	91	91
query83	204	177	162	162
query84	287	99	81	81
query85	764	365	313	313
query86	353	304	298	298
query87	4534	4660	4508	4508
query88	3673	2269	2229	2229
query89	401	308	279	279
query90	1637	210	211	210
query91	139	139	112	112
query92	82	62	59	59
query93	2548	1038	576	576
query94	736	411	309	309
query95	358	277	269	269
query96	482	555	277	277
query97	3387	3371	3246	3246
query98	244	204	201	201
query99	1456	1394	1290	1290
Total cold run time: 297309 ms
Total hot run time: 194093 ms

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

ClickBench: Total hot run time: 31.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7df7ae6c335af264d6cd28ec2e8ff1be662bc920, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.10	0.10
query3	0.25	0.19	0.20
query4	1.59	0.20	0.19
query5	0.60	0.62	0.61
query6	1.18	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.54
query10	0.58	0.59	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.61	0.60
query14	2.84	2.72	2.79
query15	0.95	0.87	0.86
query16	0.38	0.38	0.39
query17	1.03	1.04	1.00
query18	0.22	0.19	0.20
query19	1.95	1.97	1.80
query20	0.02	0.01	0.01
query21	15.35	0.92	0.53
query22	0.76	1.37	0.77
query23	14.72	1.38	0.63
query24	7.09	2.31	0.74
query25	0.51	0.29	0.11
query26	0.71	0.16	0.15
query27	0.05	0.04	0.05
query28	9.14	0.89	0.44
query29	12.54	4.02	3.34
query30	0.25	0.09	0.06
query31	2.82	0.61	0.39
query32	3.24	0.56	0.49
query33	3.09	3.13	3.05
query34	15.85	5.13	4.50
query35	4.52	4.54	4.50
query36	0.66	0.51	0.49
query37	0.08	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.11 s
Total hot run time: 31.33 s

@morrySnow morrySnow changed the title [follow-up](nereids) speedup huge InPredicate for partition pruning #47608 [follow-up](nereids) speedup huge InPredicate for tablet pruning #47608 Mar 26, 2025
@morrySnow morrySnow changed the title [follow-up](nereids) speedup huge InPredicate for tablet pruning #47608 [follow-up](nereids) speedup huge InPredicate for tablet pruning Mar 26, 2025
@924060929 924060929 merged commit d9afcec into apache:master Mar 26, 2025
32 of 34 checks passed
@924060929 924060929 deleted the master-opt-in-predicate branch March 26, 2025 06:07
github-actions bot pushed a commit that referenced this pull request Mar 26, 2025
@924060929
[follow-up](nereids) speedup huge InPredicate for tablet pruning (#49386
8bd14dd 
)

skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up
#47608
@github-actions github-actions bot mentioned this pull request Mar 26, 2025
Closed
github-actions bot pushed a commit that referenced this pull request Mar 26, 2025
@924060929
[follow-up](nereids) speedup huge InPredicate for tablet pruning (#49386
bd77e3f 
)

skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up
#47608
@github-actions github-actions bot mentioned this pull request Mar 26, 2025
Closed
yiguolei pushed a commit that referenced this pull request Mar 28, 2025
@924060929
branch-2.1: [opt](nereids) skip run PruneOlapScanTablet when exists l…
cf1938d 
…ots of InPredicate (#49387)

cherry pick some parts from #47608 and #49386
dataroaring pushed a commit that referenced this pull request Apr 22, 2025
@924060929
branch-3.0: [opt](nereids) skip run PruneOlapScanTablet when exists l…
5d002d3 
…ots of InPredicate (#49388)

cherry pick some parts from #47608 and #49386
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@924060929 @koarz
[follow-up](nereids) speedup huge InPredicate for tablet pruning (apa…
01b08e3 
…che#49386)

skip run PruneOlapScanTablet when exists lots of InPredicate, follow-up
apache#47608
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@englefly englefly englefly approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@924060929 @hello-stephen @doris-robot @englefly @starocean999 @yiguolei @dataroaring @morrySnow