Skip to content

branch-3.0: [Bug](runtime-filter) avoid wrong partitial ignore minmax filter #43195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Nov 6, 2024
Merged

branch-3.0: [Bug](runtime-filter) avoid wrong partitial ignore minmax filter #43195

dataroaring merged 2 commits into from
Nov 6, 2024

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Nov 4, 2024

Cherry-picked from #43078

@BiteTheDDDDt
[Bug](runtime-filter) avoid wrong partitial ignore minmax filter (#43078
2081c15 
)

In ##41667 we support ignore filter
partitial, however, when sync_filter_size is turned off, the final
filter may only contain part of the data because some filters are judged
to be 'in'.
For example:
there are rf001(2 instances, in_or_bloom),rf000(2 instances, min_max),
instance_1 has 1e8 row and rf001 change to bloom, the rf000 will not
ignored
instance_2 has 1 row and rf001 change to in, the rf000 will ignored
finally, rf000 applied and make wrong result
@doris-robot
Copy link

doris-robot commented Nov 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

doris-robot commented Nov 4, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 4, 2024

TPC-H: Total hot run time: 40610 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2081c15e96056e5f88b57a62fe097b2d68e1d7c3, data reload: false

------ Round 1 ----------------------------------
q1	17890	7456	7298	7298
q2	2060	159	145	145
q3	10668	1115	1105	1105
q4	10252	766	767	766
q5	7720	2847	2864	2847
q6	236	147	141	141
q7	982	608	590	590
q8	9339	1913	1977	1913
q9	6661	6424	6427	6424
q10	7053	2270	2298	2270
q11	463	254	266	254
q12	410	211	210	210
q13	17812	2958	2944	2944
q14	235	214	209	209
q15	567	509	525	509
q16	678	608	613	608
q17	982	535	556	535
q18	7605	6592	6731	6592
q19	1380	1047	1111	1047
q20	448	192	197	192
q21	3978	3025	3078	3025
q22	1108	990	986	986
Total cold run time: 108527 ms
Total hot run time: 40610 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7263	7259	7244	7244
q2	319	224	227	224
q3	2887	2708	2755	2708
q4	1914	1676	1663	1663
q5	5408	5403	5434	5403
q6	212	136	138	136
q7	2102	1634	1670	1634
q8	3181	3331	3408	3331
q9	8561	8539	8493	8493
q10	3434	3387	3392	3387
q11	577	484	485	484
q12	757	557	576	557
q13	16874	2949	2959	2949
q14	290	259	271	259
q15	553	494	511	494
q16	706	679	661	661
q17	1820	1561	1572	1561
q18	7528	7434	7322	7322
q19	1639	1606	1532	1532
q20	1994	1768	1779	1768
q21	5198	5032	5181	5032
q22	1077	982	977	977
Total cold run time: 74294 ms
Total hot run time: 57819 ms

@doris-robot
Copy link

doris-robot commented Nov 4, 2024

TPC-DS: Total hot run time: 189245 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2081c15e96056e5f88b57a62fe097b2d68e1d7c3, data reload: false

query1	979	389	383	383
query2	6520	2088	2092	2088
query3	6701	219	211	211
query4	33949	23340	23535	23340
query5	4354	459	444	444
query6	261	163	166	163
query7	4625	313	305	305
query8	312	230	218	218
query9	9620	2641	2637	2637
query10	452	271	248	248
query11	18152	15284	15081	15081
query12	149	104	102	102
query13	1624	430	403	403
query14	9756	6403	6271	6271
query15	225	172	172	172
query16	7375	464	497	464
query17	1191	555	552	552
query18	1966	308	290	290
query19	203	157	146	146
query20	117	110	109	109
query21	211	105	102	102
query22	4597	4441	4303	4303
query23	34681	33856	33978	33856
query24	12533	2823	2795	2795
query25	731	396	409	396
query26	1858	164	164	164
query27	3104	297	297	297
query28	8302	2487	2453	2453
query29	1269	431	423	423
query30	322	166	152	152
query31	1043	800	822	800
query32	97	58	59	58
query33	776	285	272	272
query34	1053	498	497	497
query35	941	744	702	702
query36	1082	930	954	930
query37	276	73	70	70
query38	3992	3890	3870	3870
query39	1483	1402	1403	1402
query40	297	99	99	99
query41	54	51	50	50
query42	108	101	96	96
query43	535	483	482	482
query44	1228	767	764	764
query45	188	170	161	161
query46	1126	701	707	701
query47	1910	1822	1816	1816
query48	450	370	364	364
query49	1297	388	411	388
query50	816	407	426	407
query51	7372	7043	7083	7043
query52	100	93	89	89
query53	254	182	182	182
query54	1139	449	445	445
query55	76	78	76	76
query56	250	245	238	238
query57	1232	1111	1093	1093
query58	249	212	208	208
query59	3269	3165	2966	2966
query60	283	255	253	253
query61	118	122	122	122
query62	882	662	646	646
query63	238	186	180	180
query64	5306	625	587	587
query65	3288	3192	3164	3164
query66	1499	325	296	296
query67	16036	15414	15180	15180
query68	4884	541	528	528
query69	452	249	274	249
query70	1131	1134	1149	1134
query71	420	256	254	254
query72	6348	3938	4075	3938
query73	766	333	347	333
query74	10145	8879	8884	8879
query75	3362	2605	2638	2605
query76	2874	1006	954	954
query77	380	262	264	262
query78	10562	9740	9503	9503
query79	8500	568	581	568
query80	2270	425	409	409
query81	577	240	241	240
query82	1366	115	115	115
query83	252	141	135	135
query84	291	77	78	77
query85	2325	304	283	283
query86	497	304	301	301
query87	4421	4376	4221	4221
query88	5851	2412	2398	2398
query89	561	292	286	286
query90	2136	176	181	176
query91	178	140	159	140
query92	68	47	50	47
query93	6153	556	536	536
query94	1032	270	279	270
query95	345	247	251	247
query96	625	279	293	279
query97	3435	3152	3143	3143
query98	214	193	192	192
query99	1865	1295	1278	1278
Total cold run time: 320768 ms
Total hot run time: 189245 ms

@BiteTheDDDDt
Copy link
Contributor

BiteTheDDDDt commented Nov 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 5, 2024

TPC-H: Total hot run time: 40384 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7134e22f7557ea316fe0c45fde83b47cf6e22100, data reload: false

------ Round 1 ----------------------------------
q1	17556	7376	7302	7302
q2	2077	164	158	158
q3	10663	1091	1155	1091
q4	10550	734	752	734
q5	7744	2818	2791	2791
q6	230	146	140	140
q7	985	613	590	590
q8	9337	1905	1985	1905
q9	6626	6461	6448	6448
q10	6991	2278	2295	2278
q11	447	256	260	256
q12	406	208	203	203
q13	17783	2957	2939	2939
q14	236	208	219	208
q15	563	517	499	499
q16	649	595	613	595
q17	986	545	538	538
q18	7166	6623	6645	6623
q19	1772	1021	1010	1010
q20	492	196	196	196
q21	3884	2990	2913	2913
q22	1086	976	967	967
Total cold run time: 108229 ms
Total hot run time: 40384 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7293	7214	7192	7192
q2	319	228	217	217
q3	2873	2809	2881	2809
q4	2026	1854	1778	1778
q5	5652	5716	5779	5716
q6	228	136	140	136
q7	2240	1743	1778	1743
q8	3345	3468	3526	3468
q9	8775	8898	8832	8832
q10	3561	3533	3489	3489
q11	596	509	495	495
q12	802	588	585	585
q13	16519	3115	3089	3089
q14	302	276	270	270
q15	554	521	494	494
q16	731	661	663	661
q17	1852	1630	1638	1630
q18	8291	7802	7451	7451
q19	2229	1637	1549	1549
q20	2054	1860	1858	1858
q21	5307	5229	5294	5229
q22	1117	1054	1004	1004
Total cold run time: 76666 ms
Total hot run time: 59695 ms

@doris-robot
Copy link

doris-robot commented Nov 5, 2024

TPC-DS: Total hot run time: 194264 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7134e22f7557ea316fe0c45fde83b47cf6e22100, data reload: false

query1	1239	898	913	898
query2	6228	2066	1982	1982
query3	10802	3799	3836	3799
query4	68868	28282	23384	23384
query5	5569	433	436	433
query6	471	171	161	161
query7	6294	303	311	303
query8	341	239	232	232
query9	9248	2671	2656	2656
query10	501	273	251	251
query11	18171	15403	15725	15403
query12	170	103	102	102
query13	1572	425	397	397
query14	11337	6847	6918	6847
query15	219	176	170	170
query16	7159	502	468	468
query17	1066	558	548	548
query18	1846	292	307	292
query19	196	147	147	147
query20	118	109	108	108
query21	211	99	104	99
query22	4301	4279	4283	4279
query23	34277	33651	34007	33651
query24	5617	2870	2887	2870
query25	502	412	412	412
query26	682	158	163	158
query27	1700	296	298	296
query28	4303	2539	2501	2501
query29	688	427	424	424
query30	240	155	151	151
query31	988	796	819	796
query32	66	54	56	54
query33	444	273	278	273
query34	902	498	505	498
query35	867	719	703	703
query36	1080	928	920	920
query37	115	71	73	71
query38	3871	3884	3874	3874
query39	1473	1458	1441	1441
query40	194	97	100	97
query41	50	50	49	49
query42	112	97	94	94
query43	522	502	488	488
query44	1144	795	788	788
query45	182	165	162	162
query46	1118	725	703	703
query47	1932	1795	1832	1795
query48	447	368	363	363
query49	743	388	400	388
query50	821	423	406	406
query51	7135	7097	7077	7077
query52	107	91	90	90
query53	254	181	181	181
query54	597	443	454	443
query55	78	77	75	75
query56	266	237	265	237
query57	1175	1087	1118	1087
query58	220	202	211	202
query59	3084	3202	3089	3089
query60	271	252	241	241
query61	102	102	113	102
query62	790	650	694	650
query63	216	185	178	178
query64	1646	614	619	614
query65	3242	3125	3175	3125
query66	683	308	310	308
query67	15841	15397	15360	15360
query68	4566	546	554	546
query69	403	253	248	248
query70	1153	1105	1101	1101
query71	361	250	274	250
query72	6372	3881	3812	3812
query73	755	331	334	331
query74	10047	8863	8955	8863
query75	3381	2653	2642	2642
query76	1911	964	982	964
query77	516	271	275	271
query78	10742	9622	9533	9533
query79	10564	593	589	589
query80	2112	418	414	414
query81	550	240	236	236
query82	1217	115	111	111
query83	327	137	135	135
query84	281	72	81	72
query85	1827	293	270	270
query86	463	278	296	278
query87	4472	4200	4160	4160
query88	5908	2363	2395	2363
query89	551	294	291	291
query90	2217	184	180	180
query91	168	136	138	136
query92	64	46	47	46
query93	7105	561	552	552
query94	982	285	294	285
query95	336	245	250	245
query96	621	278	281	278
query97	3436	3124	3128	3124
query98	218	195	193	193
query99	1608	1317	1301	1301
Total cold run time: 341948 ms
Total hot run time: 194264 ms

@dataroaring dataroaring merged commit 1688e65 into branch-3.0 Nov 6, 2024
@BiteTheDDDDt BiteTheDDDDt deleted the auto-pick-43078-branch-3.0 branch November 27, 2024 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@doris-robot @BiteTheDDDDt @dataroaring