[feature](inverted index) Add profile statistics for each condition in inverted index filters #47504

zzzxl1993 · 2025-02-05T07:10:52Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

select count() from httplogs where clientip match '232.71.0.0' and request match 'images';

IndexFilter:
- HitRows: 0ns
- fr_clientip: 10.392K (10392)
- fr_request: 28.601172M (28601172)
- ExecTime: 0ns
- ft_clientip: 2.65ms
- ft_request: 201.778ms

FilteredRows: Represents the count of rows that met the filtering conditions post-index filtering.
FilteredTime: Represents the time taken to complete the filtering operation.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-02-05T07:10:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zzzxl1993 · 2025-02-05T07:11:42Z

run buildall

doris-robot · 2025-02-05T08:58:17Z

TPC-H: Total hot run time: 32210 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

------ Round 1 ----------------------------------
q1	17584	5578	5403	5403
q2	2052	308	170	170
q3	10482	1209	755	755
q4	10203	987	527	527
q5	7532	2428	2144	2144
q6	195	170	136	136
q7	939	743	599	599
q8	9242	1401	1152	1152
q9	5193	5217	4855	4855
q10	6840	2341	1936	1936
q11	472	273	253	253
q12	341	362	218	218
q13	17752	3671	3110	3110
q14	233	222	202	202
q15	520	473	466	466
q16	613	618	585	585
q17	572	876	332	332
q18	7242	6363	6371	6363
q19	2757	969	540	540
q20	305	313	188	188
q21	2892	2171	1971	1971
q22	370	327	305	305
Total cold run time: 104331 ms
Total hot run time: 32210 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5653	5512	5501	5501
q2	242	330	237	237
q3	2288	2648	2322	2322
q4	1452	1816	1369	1369
q5	4301	4747	4666	4666
q6	172	161	131	131
q7	2081	2010	1859	1859
q8	2593	2897	2711	2711
q9	7268	7255	7245	7245
q10	3051	3305	2767	2767
q11	570	503	478	478
q12	646	756	571	571
q13	3520	3981	3271	3271
q14	279	308	273	273
q15	522	483	462	462
q16	650	674	664	664
q17	1233	1753	1270	1270
q18	7777	7554	7277	7277
q19	816	1163	1117	1117
q20	1973	2020	1910	1910
q21	5812	5431	5087	5087
q22	598	623	540	540
Total cold run time: 53497 ms
Total hot run time: 51728 ms

doris-robot · 2025-02-05T09:10:48Z

TPC-DS: Total hot run time: 193009 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	1342	959	910	910
query2	6317	2089	1998	1998
query3	11111	4591	4550	4550
query4	61359	29198	23508	23508
query5	5415	576	453	453
query6	447	210	185	185
query7	5537	512	301	301
query8	339	245	241	241
query9	8195	2620	2598	2598
query10	466	312	255	255
query11	17343	15407	15377	15377
query12	175	115	114	114
query13	1425	559	427	427
query14	10678	7828	7855	7828
query15	215	205	183	183
query16	7230	664	484	484
query17	1162	740	582	582
query18	1832	409	319	319
query19	214	191	162	162
query20	115	114	110	110
query21	220	128	109	109
query22	4504	4476	4769	4476
query23	34038	33501	34145	33501
query24	5878	2294	2395	2294
query25	462	447	412	412
query26	735	237	159	159
query27	2034	508	328	328
query28	3938	2474	2446	2446
query29	543	545	422	422
query30	207	188	152	152
query31	954	887	819	819
query32	71	62	54	54
query33	477	389	299	299
query34	770	889	562	562
query35	815	844	745	745
query36	1003	1070	968	968
query37	127	101	78	78
query38	4323	4312	4218	4218
query39	1520	1435	1419	1419
query40	201	114	99	99
query41	51	49	53	49
query42	121	107	106	106
query43	515	537	498	498
query44	1375	852	852	852
query45	182	186	171	171
query46	887	1056	664	664
query47	1945	1927	1862	1862
query48	384	396	320	320
query49	738	493	414	414
query50	680	688	396	396
query51	4275	4318	4229	4229
query52	105	104	94	94
query53	230	260	185	185
query54	502	516	420	420
query55	82	83	79	79
query56	257	269	251	251
query57	1185	1261	1155	1155
query58	258	239	249	239
query59	3115	3203	3128	3128
query60	293	290	266	266
query61	126	114	120	114
query62	757	701	663	663
query63	222	191	180	180
query64	1822	1028	655	655
query65	3241	3121	3144	3121
query66	749	401	301	301
query67	16185	15717	15499	15499
query68	4999	874	522	522
query69	482	301	254	254
query70	1194	1149	1137	1137
query71	406	287	259	259
query72	6044	3927	3809	3809
query73	788	757	370	370
query74	9805	8904	8607	8607
query75	3225	3163	2651	2651
query76	3799	1150	772	772
query77	482	367	266	266
query78	10039	9892	9421	9421
query79	2789	853	596	596
query80	1697	518	434	434
query81	564	277	236	236
query82	350	147	120	120
query83	270	174	155	155
query84	286	97	69	69
query85	755	355	306	306
query86	454	321	306	306
query87	4403	4397	4301	4301
query88	3910	2205	2206	2205
query89	389	322	298	298
query90	1594	184	183	183
query91	175	130	103	103
query92	66	58	53	53
query93	2926	866	518	518
query94	734	414	305	305
query95	323	265	252	252
query96	493	597	288	288
query97	2838	2882	2709	2709
query98	235	205	196	196
query99	1316	1364	1252	1252
Total cold run time: 311849 ms
Total hot run time: 193009 ms

doris-robot · 2025-02-05T09:16:04Z

ClickBench: Total hot run time: 30.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.42	0.40
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.55	0.56	0.56
query11	0.15	0.11	0.10
query12	0.15	0.12	0.12
query13	0.62	0.59	0.60
query14	2.75	2.73	2.79
query15	0.89	0.83	0.83
query16	0.39	0.39	0.38
query17	1.05	1.06	1.02
query18	0.22	0.20	0.20
query19	1.93	1.89	2.02
query20	0.01	0.01	0.01
query21	15.41	0.94	0.59
query22	0.75	0.81	0.70
query23	15.26	1.36	0.51
query24	2.69	1.33	0.49
query25	0.16	0.22	0.04
query26	0.40	0.16	0.13
query27	0.07	0.06	0.04
query28	12.97	1.03	0.43
query29	12.57	4.01	3.31
query30	0.25	0.08	0.06
query31	2.81	0.60	0.38
query32	3.23	0.55	0.46
query33	2.94	3.04	3.04
query34	16.58	5.26	4.51
query35	4.56	4.51	4.51
query36	0.65	0.49	0.48
query37	0.10	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.69 s
Total hot run time: 30.25 s

doris-robot · 2025-02-05T09:36:18Z

TeamCity be ut coverage result:
Function Coverage: 42.10% (11006/26145)
Line Coverage: 32.37% (92920/287072)
Region Coverage: 31.51% (47634/151158)
Branch Coverage: 27.54% (24106/87542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b35d7498119286144b585d6dc786458fc42d6a52_b35d7498119286144b585d6dc786458fc42d6a52/report/index.html

zzzxl1993 · 2025-02-05T10:26:49Z

run buildall

doris-robot · 2025-02-05T11:56:53Z

TPC-H: Total hot run time: 32686 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

------ Round 1 ----------------------------------
q1	17635	5532	5430	5430
q2	2064	335	191	191
q3	10454	1327	746	746
q4	10229	977	535	535
q5	7681	2479	2239	2239
q6	199	168	135	135
q7	932	798	605	605
q8	9247	1433	1266	1266
q9	5279	4949	4912	4912
q10	6818	2349	1875	1875
q11	463	280	257	257
q12	352	358	223	223
q13	17776	3729	3083	3083
q14	232	230	223	223
q15	517	465	479	465
q16	630	604	589	589
q17	596	891	335	335
q18	7089	6534	6512	6512
q19	1861	966	562	562
q20	328	342	183	183
q21	2988	2228	2019	2019
q22	361	330	301	301
Total cold run time: 103731 ms
Total hot run time: 32686 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5631	5608	5548	5548
q2	244	323	231	231
q3	2307	2639	2310	2310
q4	1448	1875	1359	1359
q5	4324	4786	4773	4773
q6	181	168	133	133
q7	2177	2023	1834	1834
q8	2641	2841	2722	2722
q9	7388	7271	7307	7271
q10	2976	3289	2817	2817
q11	580	524	488	488
q12	674	739	616	616
q13	3517	3913	3328	3328
q14	277	297	278	278
q15	505	486	469	469
q16	660	670	643	643
q17	1249	1740	1245	1245
q18	7801	7507	7346	7346
q19	854	1150	1094	1094
q20	2049	2045	1957	1957
q21	5690	5226	4963	4963
q22	567	614	546	546
Total cold run time: 53740 ms
Total hot run time: 51971 ms

doris-robot · 2025-02-05T12:08:27Z

TPC-DS: Total hot run time: 186506 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	978	382	361	361
query2	6525	2081	2070	2070
query3	6795	226	216	216
query4	33532	23479	23639	23479
query5	4399	637	465	465
query6	294	209	198	198
query7	4607	491	310	310
query8	300	246	229	229
query9	9660	2663	2639	2639
query10	467	329	266	266
query11	17905	15289	15366	15289
query12	175	108	104	104
query13	1679	526	385	385
query14	11434	8044	7263	7263
query15	212	199	197	197
query16	7113	615	460	460
query17	1584	729	546	546
query18	1362	370	299	299
query19	238	181	161	161
query20	117	109	110	109
query21	215	121	99	99
query22	4177	4310	4193	4193
query23	33805	32859	33407	32859
query24	6266	2325	2291	2291
query25	480	450	385	385
query26	789	274	157	157
query27	1972	468	330	330
query28	5295	2442	2423	2423
query29	530	565	468	468
query30	233	186	163	163
query31	996	874	785	785
query32	72	67	63	63
query33	514	354	307	307
query34	755	875	521	521
query35	861	829	737	737
query36	992	1044	929	929
query37	118	96	78	78
query38	4136	4090	4027	4027
query39	1459	1414	1435	1414
query40	211	115	102	102
query41	52	55	52	52
query42	128	104	103	103
query43	521	559	499	499
query44	1431	807	806	806
query45	180	169	165	165
query46	867	1044	646	646
query47	1821	1814	1766	1766
query48	386	406	316	316
query49	743	480	401	401
query50	669	658	395	395
query51	4176	4199	4119	4119
query52	107	102	91	91
query53	238	265	188	188
query54	487	497	419	419
query55	85	79	78	78
query56	280	282	247	247
query57	1149	1159	1080	1080
query58	256	234	240	234
query59	3047	3229	3167	3167
query60	280	261	254	254
query61	122	117	121	117
query62	805	716	660	660
query63	224	199	194	194
query64	3266	1016	657	657
query65	3263	3154	3168	3154
query66	878	418	307	307
query67	15648	15589	15557	15557
query68	2461	835	551	551
query69	423	298	265	265
query70	1198	1172	1126	1126
query71	322	295	259	259
query72	5068	4037	3881	3881
query73	630	756	361	361
query74	9834	9033	8909	8909
query75	3160	3207	2661	2661
query76	2145	1198	797	797
query77	353	375	305	305
query78	10053	10360	9273	9273
query79	998	915	596	596
query80	710	551	455	455
query81	477	278	236	236
query82	1141	155	126	126
query83	288	183	157	157
query84	239	88	78	78
query85	717	383	302	302
query86	340	307	303	303
query87	4400	4432	4382	4382
query88	3468	2194	2149	2149
query89	400	335	296	296
query90	1631	194	197	194
query91	137	145	112	112
query92	60	59	55	55
query93	991	909	536	536
query94	479	409	304	304
query95	337	276	268	268
query96	511	605	280	280
query97	2826	2870	2782	2782
query98	211	209	199	199
query99	1268	1462	1259	1259
Total cold run time: 271637 ms
Total hot run time: 186506 ms

doris-robot · 2025-02-05T12:13:45Z

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.10	0.11
query5	0.41	0.40	0.41
query6	1.13	0.65	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.56	0.56	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.77	2.86	2.93
query15	0.89	0.82	0.84
query16	0.40	0.38	0.36
query17	1.05	0.98	1.00
query18	0.23	0.21	0.22
query19	1.96	1.89	2.01
query20	0.02	0.01	0.02
query21	15.36	0.90	0.58
query22	0.76	0.74	0.76
query23	15.24	1.48	0.51
query24	3.00	0.91	1.48
query25	0.14	0.21	0.06
query26	0.22	0.16	0.14
query27	0.06	0.05	0.04
query28	13.98	0.98	0.44
query29	12.59	4.01	3.30
query30	0.25	0.09	0.06
query31	2.84	0.57	0.39
query32	3.25	0.56	0.46
query33	3.02	3.02	3.02
query34	16.57	5.11	4.43
query35	4.53	4.47	4.51
query36	0.63	0.49	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.87 s
Total hot run time: 30.65 s

doris-robot · 2025-02-05T12:46:05Z

TeamCity be ut coverage result:
Function Coverage: 42.11% (11017/26164)
Line Coverage: 32.38% (92993/287187)
Region Coverage: 31.54% (47703/151223)
Branch Coverage: 27.55% (24129/87578)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b35d7498119286144b585d6dc786458fc42d6a52_b35d7498119286144b585d6dc786458fc42d6a52/report/index.html

zzzxl1993 · 2025-02-06T09:39:27Z

run buildall

doris-robot · 2025-02-06T11:10:27Z

TPC-H: Total hot run time: 31459 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 221058fdf17f446ce59b89d6b42e2a1289459d9c, data reload: false

------ Round 1 ----------------------------------
q1	17609	5345	5070	5070
q2	2057	304	170	170
q3	10413	1345	712	712
q4	10228	1026	549	549
q5	8196	2533	2320	2320
q6	190	169	137	137
q7	927	765	616	616
q8	9289	1320	1095	1095
q9	5028	4614	4841	4614
q10	6834	2315	1877	1877
q11	478	269	252	252
q12	349	361	216	216
q13	17781	4435	3109	3109
q14	224	236	199	199
q15	517	469	447	447
q16	628	611	593	593
q17	591	880	354	354
q18	6804	6275	6170	6170
q19	1383	967	565	565
q20	332	340	194	194
q21	2940	2262	1900	1900
q22	366	331	300	300
Total cold run time: 103164 ms
Total hot run time: 31459 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5178	5124	5161	5124
q2	241	334	231	231
q3	2205	2724	2255	2255
q4	1435	1866	1391	1391
q5	4245	4125	4160	4125
q6	205	172	128	128
q7	1891	1883	1809	1809
q8	2603	2759	2611	2611
q9	7203	7090	7107	7090
q10	3016	3196	2741	2741
q11	574	508	488	488
q12	710	801	616	616
q13	3474	3874	3245	3245
q14	301	305	265	265
q15	493	459	457	457
q16	643	663	646	646
q17	1148	1595	1325	1325
q18	7640	7385	7264	7264
q19	819	796	881	796
q20	1955	1997	1856	1856
q21	5589	4949	4853	4853
q22	626	587	538	538
Total cold run time: 52194 ms
Total hot run time: 49854 ms

doris-robot · 2025-02-06T11:21:54Z

TPC-DS: Total hot run time: 184440 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 221058fdf17f446ce59b89d6b42e2a1289459d9c, data reload: false

query1	964	395	377	377
query2	6519	1863	1832	1832
query3	6793	211	210	210
query4	26514	23362	23686	23362
query5	4363	684	477	477
query6	313	210	193	193
query7	4612	505	310	310
query8	301	266	236	236
query9	8611	2597	2599	2597
query10	501	329	277	277
query11	15950	15166	14952	14952
query12	162	106	107	106
query13	1655	530	387	387
query14	9124	6297	6232	6232
query15	226	191	167	167
query16	7185	669	481	481
query17	1204	715	560	560
query18	1949	408	325	325
query19	198	198	167	167
query20	129	119	119	119
query21	211	125	103	103
query22	4370	4335	4326	4326
query23	34028	32913	33064	32913
query24	8393	2444	2473	2444
query25	552	447	399	399
query26	1233	263	155	155
query27	2739	473	334	334
query28	4345	2443	2395	2395
query29	818	547	421	421
query30	252	193	160	160
query31	952	890	789	789
query32	93	67	63	63
query33	566	388	304	304
query34	796	843	495	495
query35	817	815	735	735
query36	950	981	931	931
query37	122	104	76	76
query38	4133	4223	4224	4223
query39	1485	1375	1443	1375
query40	205	116	102	102
query41	52	56	53	53
query42	127	104	103	103
query43	516	528	501	501
query44	1279	786	779	779
query45	179	168	163	163
query46	870	1054	661	661
query47	1746	1792	1708	1708
query48	390	417	295	295
query49	769	519	400	400
query50	705	714	424	424
query51	4276	4162	4124	4124
query52	119	103	93	93
query53	237	273	192	192
query54	474	502	407	407
query55	80	77	86	77
query56	270	273	263	263
query57	1121	1141	1079	1079
query58	257	241	246	241
query59	2598	2726	2680	2680
query60	286	273	263	263
query61	120	115	116	115
query62	821	717	660	660
query63	226	192	191	191
query64	4562	1026	665	665
query65	3211	3109	3122	3109
query66	1161	408	310	310
query67	15862	15490	15588	15490
query68	8057	771	504	504
query69	531	297	281	281
query70	1211	1071	1146	1071
query71	395	282	277	277
query72	5737	3598	3737	3598
query73	745	721	354	354
query74	9071	9163	9107	9107
query75	3157	3183	2692	2692
query76	3260	1194	763	763
query77	474	364	296	296
query78	9984	10120	9291	9291
query79	1438	841	592	592
query80	616	551	473	473
query81	559	271	231	231
query82	219	151	123	123
query83	182	186	157	157
query84	254	97	73	73
query85	760	366	311	311
query86	382	297	286	286
query87	4408	4681	4339	4339
query88	2867	2225	2205	2205
query89	384	315	285	285
query90	1942	202	259	202
query91	142	144	110	110
query92	79	67	60	60
query93	1650	1043	578	578
query94	667	416	273	273
query95	351	285	263	263
query96	490	561	271	271
query97	2785	2800	2739	2739
query98	223	209	203	203
query99	1301	1432	1276	1276
Total cold run time: 270968 ms
Total hot run time: 184440 ms

doris-robot · 2025-02-06T11:27:10Z

ClickBench: Total hot run time: 30.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 221058fdf17f446ce59b89d6b42e2a1289459d9c, data reload: false

query1	0.04	0.03	0.03
query2	0.10	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.43	0.42	0.42
query6	1.15	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.52	0.53
query10	0.58	0.57	0.57
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.59	0.62
query14	2.67	2.72	2.71
query15	0.93	0.84	0.84
query16	0.37	0.38	0.37
query17	1.01	1.03	1.05
query18	0.22	0.20	0.20
query19	1.94	1.79	1.98
query20	0.01	0.01	0.01
query21	15.36	0.90	0.58
query22	0.75	1.13	0.76
query23	14.85	1.35	0.61
query24	6.61	2.05	0.48
query25	0.52	0.19	0.08
query26	0.65	0.16	0.13
query27	0.06	0.05	0.05
query28	9.45	0.82	0.44
query29	12.57	3.92	3.29
query30	0.25	0.09	0.06
query31	2.84	0.60	0.39
query32	3.23	0.55	0.47
query33	3.06	2.99	3.01
query34	15.77	5.15	4.55
query35	4.60	4.61	4.58
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.59 s
Total hot run time: 30.46 s

zzzxl1993 · 2025-02-06T15:51:53Z

run buildall

doris-robot · 2025-02-06T17:39:04Z

TPC-H: Total hot run time: 31645 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1cd6dcc3d24e75a244255e30fdcdae4fac017bc, data reload: false

------ Round 1 ----------------------------------
q1	17599	5266	5086	5086
q2	2050	293	171	171
q3	10484	1256	751	751
q4	10233	1061	538	538
q5	7716	2381	2362	2362
q6	191	164	131	131
q7	917	757	598	598
q8	9308	1328	1145	1145
q9	4877	4751	4837	4751
q10	6828	2323	1869	1869
q11	474	275	258	258
q12	347	351	230	230
q13	17781	3668	3137	3137
q14	233	217	206	206
q15	513	474	471	471
q16	620	625	600	600
q17	578	889	329	329
q18	6769	6191	6060	6060
q19	1486	938	561	561
q20	314	316	186	186
q21	2804	2140	1917	1917
q22	358	344	288	288
Total cold run time: 102480 ms
Total hot run time: 31645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5141	5115	5106	5106
q2	231	345	226	226
q3	2206	2731	2343	2343
q4	1393	1838	1379	1379
q5	4224	4126	4191	4126
q6	212	165	126	126
q7	1896	1823	1723	1723
q8	2620	2543	2620	2543
q9	7238	7120	7173	7120
q10	3057	3205	2788	2788
q11	586	510	495	495
q12	707	806	653	653
q13	3405	3919	3391	3391
q14	284	297	272	272
q15	516	456	450	450
q16	634	689	648	648
q17	1140	1628	1336	1336
q18	7522	7339	7382	7339
q19	806	869	907	869
q20	1975	2035	1850	1850
q21	5480	5030	4902	4902
q22	659	588	539	539
Total cold run time: 51932 ms
Total hot run time: 50224 ms

doris-robot · 2025-02-06T17:51:11Z

TPC-DS: Total hot run time: 190603 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1cd6dcc3d24e75a244255e30fdcdae4fac017bc, data reload: false

query1	1313	951	968	951
query2	6257	1906	1847	1847
query3	10961	4594	4407	4407
query4	56575	25996	23358	23358
query5	5261	513	484	484
query6	364	195	190	190
query7	5112	516	299	299
query8	326	247	229	229
query9	6450	2581	2591	2581
query10	430	313	252	252
query11	15160	15119	14912	14912
query12	162	108	104	104
query13	1153	500	374	374
query14	10961	6281	6319	6281
query15	211	197	176	176
query16	7082	664	472	472
query17	1050	722	558	558
query18	1575	418	312	312
query19	196	185	158	158
query20	134	129	119	119
query21	206	122	103	103
query22	4793	4710	4685	4685
query23	33783	33470	33297	33297
query24	5713	2431	2428	2428
query25	480	456	416	416
query26	706	286	160	160
query27	1718	501	331	331
query28	2952	2509	2464	2464
query29	589	543	410	410
query30	220	188	154	154
query31	932	891	829	829
query32	75	66	64	64
query33	453	373	308	308
query34	768	844	500	500
query35	815	852	763	763
query36	972	1009	922	922
query37	134	108	79	79
query38	4330	4355	4252	4252
query39	1513	1481	1440	1440
query40	218	172	113	113
query41	58	53	56	53
query42	126	111	111	111
query43	511	537	500	500
query44	1319	815	848	815
query45	189	173	176	173
query46	904	1070	659	659
query47	1859	1931	1806	1806
query48	397	430	314	314
query49	712	487	417	417
query50	706	769	416	416
query51	4232	4285	4301	4285
query52	115	107	98	98
query53	232	265	218	218
query54	503	495	415	415
query55	81	81	77	77
query56	271	281	274	274
query57	1200	1200	1158	1158
query58	243	234	238	234
query59	2681	3102	2775	2775
query60	291	276	283	276
query61	117	133	139	133
query62	742	735	675	675
query63	229	191	187	187
query64	1523	1050	677	677
query65	3241	3120	3159	3120
query66	777	406	293	293
query67	16149	15553	15630	15553
query68	5417	786	520	520
query69	504	344	266	266
query70	1208	1119	1101	1101
query71	417	289	258	258
query72	5948	3678	3767	3678
query73	1198	751	348	348
query74	8952	8990	8757	8757
query75	3341	3177	2794	2794
query76	3864	1165	746	746
query77	562	366	300	300
query78	10080	10115	9321	9321
query79	2119	830	590	590
query80	591	581	441	441
query81	491	272	243	243
query82	288	153	123	123
query83	172	168	158	158
query84	290	96	81	81
query85	803	360	306	306
query86	411	312	282	282
query87	4373	4604	4320	4320
query88	3518	2203	2195	2195
query89	409	320	281	281
query90	1857	195	194	194
query91	135	141	113	113
query92	74	62	58	58
query93	1894	986	575	575
query94	645	414	293	293
query95	343	265	254	254
query96	486	549	277	277
query97	2843	2845	2749	2749
query98	234	205	200	200
query99	1302	1382	1287	1287
Total cold run time: 296280 ms
Total hot run time: 190603 ms

doris-robot · 2025-02-06T17:56:30Z

ClickBench: Total hot run time: 30.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1cd6dcc3d24e75a244255e30fdcdae4fac017bc, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.08	0.07
query4	1.62	0.11	0.10
query5	0.42	0.41	0.40
query6	1.17	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.58	0.58	0.58
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	2.78	2.83	2.71
query15	0.94	0.86	0.84
query16	0.39	0.38	0.37
query17	0.99	1.01	1.02
query18	0.22	0.20	0.19
query19	1.91	1.75	1.97
query20	0.01	0.01	0.01
query21	15.38	0.92	0.55
query22	0.75	1.25	0.65
query23	14.92	1.37	0.60
query24	6.72	1.78	1.38
query25	0.53	0.26	0.10
query26	0.54	0.16	0.13
query27	0.06	0.05	0.05
query28	10.72	0.86	0.42
query29	12.58	3.88	3.22
query30	0.25	0.09	0.06
query31	2.82	0.58	0.40
query32	3.22	0.54	0.47
query33	2.97	2.94	3.01
query34	15.85	5.13	4.50
query35	4.49	4.52	4.54
query36	0.68	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 105.89 s
Total hot run time: 30.89 s

doris-robot · 2025-02-07T04:08:40Z

TeamCity be ut coverage result:
Function Coverage: 42.07% (11022/26197)
Line Coverage: 32.34% (93026/287661)
Region Coverage: 31.52% (47718/151394)
Branch Coverage: 27.53% (24138/87676)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a1cd6dcc3d24e75a244255e30fdcdae4fac017bc_a1cd6dcc3d24e75a244255e30fdcdae4fac017bc/report/index.html

be/src/olap/inverted_index_profile.h

airborne12 · 2025-02-13T01:38:05Z

be/src/olap/inverted_index_profile.h

+            COUNTER_UPDATE(filter_rows, stats.filter_rows);
+        }
+
+        for (auto stats : statistics->stats) {


should we add a limit for max columns?

zzzxl1993 · 2025-02-18T06:12:27Z

run buildall

qidaye

LGTM

github-actions · 2025-02-18T06:32:51Z

PR approved by at least one committer and no changes requested.

…n inverted index filters

zzzxl1993 · 2025-02-20T09:22:00Z

run buildall

doris-robot · 2025-02-20T09:40:34Z

TPC-H: Total hot run time: 31416 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4cca179cd4de26b05c936d907aac1a2117409abf, data reload: false

------ Round 1 ----------------------------------
q1	17637	5354	5073	5073
q2	2050	305	178	178
q3	10408	1297	715	715
q4	10218	1025	536	536
q5	7533	2477	2292	2292
q6	195	168	136	136
q7	910	760	601	601
q8	9312	1351	1139	1139
q9	5115	4720	4559	4559
q10	6839	2314	1894	1894
q11	485	269	253	253
q12	355	361	217	217
q13	17760	3623	3066	3066
q14	231	219	203	203
q15	507	476	454	454
q16	610	621	587	587
q17	582	864	332	332
q18	6682	6267	6228	6228
q19	1213	957	542	542
q20	314	327	189	189
q21	2757	2156	1915	1915
q22	361	329	307	307
Total cold run time: 102074 ms
Total hot run time: 31416 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5099	5039	5095	5039
q2	247	340	234	234
q3	2191	2715	2314	2314
q4	1461	1842	1381	1381
q5	4233	4132	4171	4132
q6	206	162	126	126
q7	1870	1838	1693	1693
q8	2610	2586	2506	2506
q9	7309	7137	7066	7066
q10	3033	3167	2773	2773
q11	587	511	509	509
q12	678	757	614	614
q13	3485	3871	3226	3226
q14	270	290	264	264
q15	517	496	457	457
q16	635	675	655	655
q17	1153	1613	1301	1301
q18	7488	7281	7134	7134
q19	795	831	916	831
q20	2004	1999	1849	1849
q21	5401	4926	4824	4824
q22	645	576	554	554
Total cold run time: 51917 ms
Total hot run time: 49482 ms

doris-robot · 2025-02-20T09:51:54Z

TPC-DS: Total hot run time: 183774 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4cca179cd4de26b05c936d907aac1a2117409abf, data reload: false

query1	974	369	376	369
query2	6520	1833	1814	1814
query3	6794	214	211	211
query4	26869	23672	23667	23667
query5	4317	648	508	508
query6	293	182	179	179
query7	4619	484	293	293
query8	293	233	214	214
query9	8597	2563	2547	2547
query10	442	323	244	244
query11	15640	15072	15300	15072
query12	179	113	105	105
query13	1655	532	406	406
query14	9606	6371	6018	6018
query15	225	191	182	182
query16	7722	626	476	476
query17	1197	691	558	558
query18	1968	391	289	289
query19	193	180	152	152
query20	118	113	113	113
query21	204	157	105	105
query22	4236	4309	4552	4309
query23	34423	33568	32953	32953
query24	7868	2374	2371	2371
query25	510	448	377	377
query26	1231	283	161	161
query27	2255	510	331	331
query28	4042	2407	2401	2401
query29	734	569	446	446
query30	230	182	157	157
query31	905	852	765	765
query32	71	66	69	66
query33	577	355	308	308
query34	767	838	505	505
query35	814	816	740	740
query36	980	958	873	873
query37	119	94	75	75
query38	4178	4164	4044	4044
query39	1521	1386	1383	1383
query40	205	110	100	100
query41	56	49	51	49
query42	123	100	103	100
query43	478	507	473	473
query44	1266	768	758	758
query45	174	173	157	157
query46	867	1032	655	655
query47	1753	1798	1718	1718
query48	382	405	307	307
query49	782	491	416	416
query50	696	739	409	409
query51	4203	4201	4100	4100
query52	107	102	97	97
query53	225	260	216	216
query54	484	479	397	397
query55	80	75	78	75
query56	253	269	239	239
query57	1129	1148	1039	1039
query58	248	239	239	239
query59	2652	2818	2631	2631
query60	288	272	241	241
query61	120	119	116	116
query62	795	735	652	652
query63	239	179	192	179
query64	4252	998	677	677
query65	3220	3156	3109	3109
query66	1064	396	296	296
query67	15978	15606	15403	15403
query68	3005	795	551	551
query69	460	317	284	284
query70	1202	1146	1126	1126
query71	371	311	264	264
query72	5259	3566	3784	3566
query73	721	736	352	352
query74	9093	9371	8992	8992
query75	3207	3190	2674	2674
query76	3073	1171	744	744
query77	470	366	283	283
query78	10019	10120	9279	9279
query79	2060	822	601	601
query80	1562	517	446	446
query81	580	276	246	246
query82	407	128	96	96
query83	279	173	152	152
query84	241	89	74	74
query85	797	349	307	307
query86	447	291	273	273
query87	4446	4473	4361	4361
query88	2995	2210	2222	2210
query89	390	351	286	286
query90	1773	193	190	190
query91	134	143	110	110
query92	62	59	53	53
query93	1187	991	593	593
query94	670	399	291	291
query95	346	267	261	261
query96	490	534	274	274
query97	2800	2922	2758	2758
query98	237	206	195	195
query99	1284	1384	1247	1247
Total cold run time: 266072 ms
Total hot run time: 183774 ms

doris-robot · 2025-02-20T09:57:12Z

ClickBench: Total hot run time: 30.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4cca179cd4de26b05c936d907aac1a2117409abf, data reload: false

query1	0.06	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.08	0.06
query4	1.60	0.10	0.10
query5	0.41	0.41	0.40
query6	1.18	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.53	0.52
query10	0.56	0.57	0.57
query11	0.16	0.10	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	2.71	2.72	2.73
query15	0.90	0.85	0.85
query16	0.38	0.38	0.37
query17	1.03	1.03	1.04
query18	0.21	0.20	0.20
query19	1.96	1.76	1.96
query20	0.02	0.01	0.01
query21	15.36	0.90	0.54
query22	0.75	1.19	0.63
query23	14.99	1.36	0.58
query24	7.80	1.18	1.11
query25	0.47	0.20	0.14
query26	0.64	0.16	0.13
query27	0.06	0.05	0.05
query28	9.65	0.84	0.43
query29	12.60	3.97	3.28
query30	0.25	0.08	0.06
query31	2.85	0.58	0.37
query32	3.23	0.55	0.47
query33	2.97	3.01	2.98
query34	15.71	5.18	4.50
query35	4.52	4.52	4.53
query36	0.69	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.93 s
Total hot run time: 30.74 s

doris-robot · 2025-02-20T10:17:27Z

TeamCity be ut coverage result:
Function Coverage: 43.83% (11641/26558)
Line Coverage: 33.76% (97600/289129)
Region Coverage: 32.85% (49961/152082)
Branch Coverage: 28.56% (25106/87906)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4cca179cd4de26b05c936d907aac1a2117409abf_4cca179cd4de26b05c936d907aac1a2117409abf/report/index.html

airborne12

LGTM

github-actions · 2025-02-21T02:17:58Z

PR approved by at least one committer and no changes requested.

qidaye

LGTM

…n inverted index filters (apache#47504) Problem Summary: select count() from httplogs where clientip match '232.71.0.0' and request match 'images'; IndexFilter: - HitRows: 0ns - fr_clientip: 10.392K (10392) - fr_request: 28.601172M (28601172) - ExecTime: 0ns - ft_clientip: 2.65ms - ft_request: 201.778ms FilteredRows: Represents the count of rows that met the filtering conditions post-index filtering. FilteredTime: Represents the time taken to complete the filtering operation.

…n inverted index filters (#48459) #47504

…n inverted index filters (apache#47504) Problem Summary: select count() from httplogs where clientip match '232.71.0.0' and request match 'images'; IndexFilter: - HitRows: 0ns - fr_clientip: 10.392K (10392) - fr_request: 28.601172M (28601172) - ExecTime: 0ns - ft_clientip: 2.65ms - ft_request: 201.778ms FilteredRows: Represents the count of rows that met the filtering conditions post-index filtering. FilteredTime: Represents the time taken to complete the filtering operation.

#48826 #48950 (#52377) Cherry-picked #47504 #48826 #48950

…n inverted index filters (apache#47504) Problem Summary: select count() from httplogs where clientip match '232.71.0.0' and request match 'images'; IndexFilter: - HitRows: 0ns - fr_clientip: 10.392K (10392) - fr_request: 28.601172M (28601172) - ExecTime: 0ns - ft_clientip: 2.65ms - ft_request: 201.778ms FilteredRows: Represents the count of rows that met the filtering conditions post-index filtering. FilteredTime: Represents the time taken to complete the filtering operation.

pick #47504 pick #48826 pick #48950

pick apache#47504 pick apache#48826 pick apache#48950

zzzxl1993 force-pushed the 202501231529 branch from c75c6d0 to b35d749 Compare February 5, 2025 07:11

zzzxl1993 force-pushed the 202501231529 branch from b35d749 to 221058f Compare February 6, 2025 09:39

zzzxl1993 force-pushed the 202501231529 branch from 221058f to a1cd6dc Compare February 6, 2025 15:51

qidaye reviewed Feb 7, 2025

View reviewed changes

be/src/olap/inverted_index_profile.h Outdated Show resolved Hide resolved

airborne12 reviewed Feb 13, 2025

View reviewed changes

zzzxl1993 force-pushed the 202501231529 branch from a1cd6dc to 1e3877a Compare February 18, 2025 06:12

qidaye previously approved these changes Feb 18, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2025

[feature](inverted index) Add profile statistics for each condition i…

4cca179

…n inverted index filters

zzzxl1993 force-pushed the 202501231529 branch from 4e621d3 to 4cca179 Compare February 20, 2025 09:21

airborne12 approved these changes Feb 21, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 21, 2025

qidaye approved these changes Feb 21, 2025

View reviewed changes

qidaye merged commit 198f532 into apache:master Feb 21, 2025
24 of 26 checks passed

zzzxl1993 mentioned this pull request Feb 28, 2025

[feature](inverted index) Add profile statistics for each condition in inverted index filters #48459

Merged

yiguolei pushed a commit that referenced this pull request Mar 1, 2025

[feature](inverted index) Add profile statistics for each condition i…

cd3e1dc

…n inverted index filters (#48459) #47504

yiguolei added the dev/2.1.9-merged label Mar 1, 2025

yiguolei mentioned this pull request Mar 25, 2025

2.1.9 Release Note #49443

Closed

zzzxl1993 mentioned this pull request May 14, 2025

[opt](inverted index) enhance inverted index profile #50876

Merged

zzzxl1993 mentioned this pull request Jun 26, 2025

[fix](inverted index) enhance inverted index profile #52377

Merged

morrySnow pushed a commit that referenced this pull request Jun 27, 2025

branch-3.1: [fix](inverted index) enhance inverted index profile #47504

c306240

#48826 #48950 (#52377) Cherry-picked #47504 #48826 #48950

morrySnow added the dev/3.1.0-merged label Jun 27, 2025

dataroaring pushed a commit that referenced this pull request Jun 27, 2025

[opt](inverted index) enhance inverted index profile (#50876)

3d43145

pick #47504 pick #48826 pick #48950

dataroaring added the dev/3.0.7-merged label Jun 27, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025

[opt](inverted index) enhance inverted index profile (apache#50876)

8b9be16

pick apache#47504 pick apache#48826 pick apache#48950

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

[feature](inverted index) Add profile statistics for each condition in inverted index filters #47504

[feature](inverted index) Add profile statistics for each condition in inverted index filters #47504

Uh oh!

Conversation

zzzxl1993 commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 5, 2025

Uh oh!

zzzxl1993 commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

zzzxl1993 commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

zzzxl1993 commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

zzzxl1993 commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 7, 2025

Uh oh!

Uh oh!

airborne12 Feb 13, 2025

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Feb 18, 2025

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 18, 2025

Uh oh!

zzzxl1993 commented Feb 20, 2025

Uh oh!

doris-robot commented Feb 20, 2025

Uh oh!

doris-robot commented Feb 20, 2025

Uh oh!

doris-robot commented Feb 20, 2025

Uh oh!

doris-robot commented Feb 20, 2025

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 21, 2025

Uh oh!

zzzxl1993 commented Feb 5, 2025 •

edited

Loading