Skip to content

branch-3.0:[fix](parquet/orc) Disable string dictionary filtering when predicate express is not binary pred and in pred (#50749) #51267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 11, 2025
Merged

branch-3.0:[fix](parquet/orc) Disable string dictionary filtering when predicate express is not binary pred and in pred (#50749) #51267

dataroaring merged 1 commit into from
Jun 11, 2025

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented May 26, 2025

cherry-pick: #50749

@suxiaogang223
[fix](parquet/orc) Disable string dictionary filtering when predicate…
44d4978 
… express is not binary pred and in pred (apache#50749)

Related PR: apache#42113

Problem Summary:
In the previous fix, the conditions for when dictionary filtering can be
performed were wrong.
@hello-stephen
Copy link
Contributor

hello-stephen commented May 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented May 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-H: Total hot run time: 39699 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44d497806586a76cccd76ea78f8da59409af3664, data reload: false

------ Round 1 ----------------------------------
q1	17572	6757	6538	6538
q2	2061	172	166	166
q3	10594	1099	1198	1099
q4	10527	721	668	668
q5	7753	2821	2831	2821
q6	215	134	128	128
q7	951	623	598	598
q8	9379	1930	2043	1930
q9	6660	6427	6333	6333
q10	7019	2242	2305	2242
q11	466	258	256	256
q12	394	215	209	209
q13	17794	2974	2997	2974
q14	230	202	203	202
q15	506	471	461	461
q16	507	391	383	383
q17	972	589	585	585
q18	7241	6740	6872	6740
q19	1396	1036	1075	1036
q20	468	208	203	203
q21	4362	3179	3150	3150
q22	1119	983	977	977
Total cold run time: 108186 ms
Total hot run time: 39699 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6595	6552	6479	6479
q2	322	232	237	232
q3	2889	2758	2867	2758
q4	2057	1800	1810	1800
q5	5740	5797	5701	5701
q6	205	129	128	128
q7	2226	1783	1798	1783
q8	3319	3539	3511	3511
q9	8935	8760	8866	8760
q10	3590	3514	3510	3510
q11	598	520	487	487
q12	793	591	593	591
q13	7683	3170	3118	3118
q14	319	269	272	269
q15	511	483	462	462
q16	484	426	421	421
q17	1821	1628	1581	1581
q18	8262	7793	7807	7793
q19	1675	1573	1567	1567
q20	2104	1894	1830	1830
q21	5330	5260	5345	5260
q22	1111	1010	1020	1010
Total cold run time: 66569 ms
Total hot run time: 59051 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-DS: Total hot run time: 196487 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44d497806586a76cccd76ea78f8da59409af3664, data reload: false

query1	1279	884	898	884
query2	6376	1891	1885	1885
query3	10773	4214	4309	4214
query4	61349	28998	23823	23823
query5	5207	467	461	461
query6	406	169	170	169
query7	5476	313	308	308
query8	324	229	232	229
query9	8771	2608	2583	2583
query10	463	279	259	259
query11	17559	15068	15743	15068
query12	169	108	102	102
query13	1474	459	417	417
query14	10555	6731	6978	6731
query15	199	173	185	173
query16	7123	418	502	418
query17	1118	627	586	586
query18	1990	337	324	324
query19	204	160	157	157
query20	115	111	111	111
query21	207	104	105	104
query22	4542	4400	4702	4400
query23	34769	34022	34014	34014
query24	6193	2882	2959	2882
query25	549	434	433	433
query26	654	168	166	166
query27	1827	359	360	359
query28	4505	2146	2174	2146
query29	699	511	436	436
query30	243	167	155	155
query31	994	871	832	832
query32	74	56	58	56
query33	449	284	298	284
query34	925	515	519	515
query35	842	731	727	727
query36	1070	970	979	970
query37	111	73	67	67
query38	4073	4036	3963	3963
query39	1512	1473	1494	1473
query40	205	95	106	95
query41	48	47	48	47
query42	112	103	101	101
query43	531	504	491	491
query44	1172	827	817	817
query45	187	170	177	170
query46	1136	722	717	717
query47	2008	1908	1940	1908
query48	491	386	403	386
query49	726	390	388	388
query50	855	439	429	429
query51	7426	7107	7307	7107
query52	102	98	88	88
query53	263	179	184	179
query54	576	479	470	470
query55	84	85	77	77
query56	258	256	259	256
query57	1303	1210	1170	1170
query58	222	212	221	212
query59	3335	3117	3098	3098
query60	285	256	251	251
query61	118	107	109	107
query62	782	687	674	674
query63	211	189	185	185
query64	1869	650	644	644
query65	3242	3201	3191	3191
query66	706	289	297	289
query67	15894	15589	15534	15534
query68	4076	572	573	572
query69	435	267	279	267
query70	1113	1092	1120	1092
query71	339	253	250	250
query72	6415	4016	3998	3998
query73	742	356	360	356
query74	10137	9152	9349	9152
query75	3385	2656	2626	2626
query76	2096	1136	1030	1030
query77	488	279	272	272
query78	10549	9709	9481	9481
query79	1280	580	591	580
query80	836	418	441	418
query81	498	217	223	217
query82	1283	92	87	87
query83	242	149	141	141
query84	273	78	76	76
query85	859	309	290	290
query86	328	295	298	295
query87	4426	4231	4275	4231
query88	3684	2419	2382	2382
query89	424	300	299	299
query90	1986	190	189	189
query91	188	149	147	147
query92	61	50	50	50
query93	1655	542	541	541
query94	738	273	288	273
query95	356	260	255	255
query96	617	295	289	289
query97	3432	3182	3164	3164
query98	217	204	197	197
query99	1605	1337	1305	1305
Total cold run time: 314078 ms
Total hot run time: 196487 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

ClickBench: Total hot run time: 29.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44d497806586a76cccd76ea78f8da59409af3664, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.52	0.52	0.49
query6	1.16	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.55	0.55	0.54
query11	0.14	0.10	0.12
query12	0.14	0.10	0.11
query13	0.60	0.58	0.58
query14	0.76	0.80	0.79
query15	0.84	0.82	0.83
query16	0.40	0.39	0.39
query17	1.07	1.04	1.04
query18	0.24	0.23	0.23
query19	1.98	1.84	1.78
query20	0.01	0.01	0.01
query21	15.75	0.60	0.59
query22	3.00	2.39	1.54
query23	16.83	0.98	0.87
query24	2.62	0.88	1.96
query25	0.22	0.27	0.08
query26	0.36	0.13	0.14
query27	0.04	0.04	0.04
query28	9.99	0.49	0.47
query29	12.60	3.23	3.23
query30	0.24	0.06	0.05
query31	2.89	0.38	0.39
query32	3.25	0.46	0.45
query33	2.94	3.01	3.01
query34	17.17	4.56	4.49
query35	4.50	4.47	4.51
query36	0.68	0.48	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.53 s
Total hot run time: 29.8 s

@doris-robot
Copy link

doris-robot commented May 26, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.06% (10858/26443)
Line Coverage 31.87% (92683/290852)
Region Coverage 30.95% (47809/154467)
Branch Coverage 27.42% (24476/89256)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 80eae07 into apache:branch-3.0 Jun 11, 2025
21 of 24 checks passed
@suxiaogang223 suxiaogang223 deleted the fix_dict_30 branch June 27, 2025 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@suxiaogang223 @hello-stephen @doris-robot @morningman @dataroaring