Skip to content

[opt](inverted index) enhance inverted index profile #50876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 27, 2025
Merged

[opt](inverted index) enhance inverted index profile #50876

dataroaring merged 1 commit into from
Jun 27, 2025

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented May 14, 2025 •
edited
Loading

pick #47504
pick #48826
pick #48950
pick #51495

@zzzxl1993 zzzxl1993 requested a review from dataroaring as a code owner May 14, 2025 02:52
@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented May 14, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented May 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-H: Total hot run time: 40433 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5daa1c39fa3b3e8dc53fe74325ac3b5cec061810, data reload: false

------ Round 1 ----------------------------------
q1	17663	7137	6690	6690
q2	2072	166	163	163
q3	10650	1147	1262	1147
q4	10320	755	733	733
q5	7803	2873	2742	2742
q6	234	136	137	136
q7	975	624	596	596
q8	9347	2041	2079	2041
q9	6740	6471	6419	6419
q10	6986	2303	2372	2303
q11	451	255	260	255
q12	400	213	215	213
q13	17801	3007	2994	2994
q14	240	204	203	203
q15	504	489	473	473
q16	643	593	586	586
q17	992	581	534	534
q18	7438	6573	6721	6573
q19	1401	1109	1084	1084
q20	492	197	204	197
q21	4156	3433	3359	3359
q22	1127	1011	992	992
Total cold run time: 108435 ms
Total hot run time: 40433 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6621	6602	6534	6534
q2	337	235	234	234
q3	2918	2825	3062	2825
q4	2095	1855	1842	1842
q5	5653	5789	5730	5730
q6	202	125	127	125
q7	2193	1817	1791	1791
q8	3382	3580	3554	3554
q9	8832	8887	8903	8887
q10	3559	3548	3569	3548
q11	600	492	500	492
q12	805	622	609	609
q13	9243	3187	3085	3085
q14	310	294	272	272
q15	516	467	457	457
q16	678	671	642	642
q17	1852	1594	1583	1583
q18	8276	7796	7702	7702
q19	1708	1661	1618	1618
q20	2115	1859	1851	1851
q21	5687	5250	5252	5250
q22	1140	1032	1003	1003
Total cold run time: 68722 ms
Total hot run time: 59634 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-DS: Total hot run time: 189616 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5daa1c39fa3b3e8dc53fe74325ac3b5cec061810, data reload: false

query1	991	380	378	378
query2	6524	2084	2070	2070
query3	6712	214	221	214
query4	33872	23687	24092	23687
query5	4364	475	452	452
query6	281	190	198	190
query7	4637	314	315	314
query8	305	221	208	208
query9	9570	2596	2590	2590
query10	463	262	253	253
query11	18137	15246	15121	15121
query12	154	100	103	100
query13	1640	430	411	411
query14	10243	7563	7132	7132
query15	254	173	186	173
query16	8140	467	446	446
query17	1682	578	565	565
query18	2164	326	321	321
query19	368	154	151	151
query20	118	106	109	106
query21	211	106	107	106
query22	4411	4247	4197	4197
query23	34346	33826	33176	33176
query24	11271	2857	2850	2850
query25	685	405	417	405
query26	1420	174	172	172
query27	2826	342	349	342
query28	7932	2400	2415	2400
query29	903	458	438	438
query30	330	168	169	168
query31	1040	824	816	816
query32	100	59	60	59
query33	806	304	288	288
query34	961	495	534	495
query35	861	747	721	721
query36	1092	938	959	938
query37	140	70	69	69
query38	3988	3855	3798	3798
query39	1486	1439	1456	1439
query40	285	107	107	107
query41	55	52	51	51
query42	120	101	102	101
query43	527	499	491	491
query44	1279	811	794	794
query45	192	173	176	173
query46	1152	703	713	703
query47	1969	1830	1832	1830
query48	473	384	383	383
query49	1185	413	409	409
query50	819	410	409	409
query51	7427	7298	7185	7185
query52	109	93	92	92
query53	262	188	193	188
query54	1178	467	463	463
query55	79	80	78	78
query56	285	261	255	255
query57	1264	1136	1132	1132
query58	247	215	242	215
query59	3299	3060	2965	2965
query60	280	265	247	247
query61	143	113	110	110
query62	901	676	682	676
query63	225	191	194	191
query64	5371	679	654	654
query65	3290	3240	3246	3240
query66	1448	318	309	309
query67	15968	15580	15430	15430
query68	4844	578	563	563
query69	443	270	270	270
query70	1202	1140	1152	1140
query71	341	254	251	251
query72	6456	2488	3876	2488
query73	749	344	343	343
query74	10572	9362	8938	8938
query75	3357	2666	2669	2666
query76	2917	1120	1028	1028
query77	376	275	271	271
query78	10514	9703	9611	9611
query79	1101	603	593	593
query80	765	429	419	419
query81	516	240	237	237
query82	1257	94	92	92
query83	235	165	151	151
query84	232	84	77	77
query85	1104	312	281	281
query86	325	295	303	295
query87	4364	4267	4228	4228
query88	3635	2399	2351	2351
query89	411	300	297	297
query90	2099	192	187	187
query91	188	153	155	153
query92	58	51	52	51
query93	1077	553	547	547
query94	792	289	307	289
query95	355	264	259	259
query96	611	279	289	279
query97	3298	3145	3129	3129
query98	220	199	194	194
query99	1517	1304	1298	1298
Total cold run time: 301191 ms
Total hot run time: 189616 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

ClickBench: Total hot run time: 31.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5daa1c39fa3b3e8dc53fe74325ac3b5cec061810, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.60	0.10	0.11
query5	0.52	0.51	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.50
query10	0.55	0.54	0.56
query11	0.13	0.10	0.10
query12	0.14	0.10	0.11
query13	0.61	0.61	0.60
query14	2.87	2.88	2.85
query15	0.90	0.84	0.84
query16	0.37	0.38	0.37
query17	1.06	1.01	1.00
query18	0.23	0.21	0.22
query19	1.96	1.88	2.03
query20	0.02	0.01	0.01
query21	15.37	0.57	0.58
query22	2.61	2.08	2.68
query23	16.91	1.11	0.79
query24	2.67	1.08	0.34
query25	0.32	0.16	0.04
query26	0.39	0.13	0.14
query27	0.05	0.05	0.05
query28	11.43	0.51	0.52
query29	12.56	3.24	3.18
query30	0.24	0.06	0.06
query31	2.85	0.39	0.40
query32	3.23	0.47	0.46
query33	3.00	3.02	3.03
query34	16.77	4.52	4.50
query35	4.60	4.53	4.60
query36	0.68	0.48	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.10	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 107.26 s
Total hot run time: 31.93 s

@doris-robot
Copy link

doris-robot commented May 14, 2025

BE UT Coverage Report

Increment line coverage 40.00% (14/35) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.98% (10812/26383)
Line Coverage 31.82% (92320/290126)
Region Coverage 30.91% (47663/154212)
Branch Coverage 27.41% (24423/89104)

@zzzxl1993 zzzxl1993 changed the title [feature](inverted index) Add profile statistics for each condition in inverted index filters [opt](inverted index) Add profile statistics for each condition in inverted index filters Jun 3, 2025
@zzzxl1993 zzzxl1993 changed the title [opt](inverted index) Add profile statistics for each condition in inverted index filters [opt](inverted index) enhance inverted index profile Jun 26, 2025
@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jun 26, 2025

run buildall

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jun 26, 2025

run buildall

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jun 26, 2025

run buildall

@zzzxl1993
[feature](inverted index) Add profile statistics for each condition i…
c3eb0b3 
…n inverted index filters (apache#47504)

Problem Summary:

select count() from httplogs where clientip match '232.71.0.0' and
request match 'images';

IndexFilter:
      -  HitRows:  0ns
          -  fr_clientip:  10.392K  (10392)
          -  fr_request:  28.601172M  (28601172)
      -  ExecTime:  0ns
          -  ft_clientip:  2.65ms
          -  ft_request:  201.778ms

FilteredRows: Represents the count of rows that met the filtering
conditions post-index filtering.
FilteredTime: Represents the time taken to complete the filtering
operation.
@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jun 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

TPC-H: Total hot run time: 39779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3eb0b3d3111c874a0450b65bba57dee68a2d516, data reload: false

------ Round 1 ----------------------------------
q1	17593	6993	6667	6667
q2	2046	174	178	174
q3	10680	1071	1165	1071
q4	10424	781	724	724
q5	7745	2843	2867	2843
q6	217	135	133	133
q7	979	620	595	595
q8	9356	1968	1979	1968
q9	6554	6395	6438	6395
q10	7046	2270	2294	2270
q11	467	263	258	258
q12	395	210	206	206
q13	17805	2983	2996	2983
q14	239	205	222	205
q15	513	462	455	455
q16	478	369	370	369
q17	986	581	511	511
q18	7317	6652	6712	6652
q19	1421	1047	1002	1002
q20	467	201	199	199
q21	3948	3137	3107	3107
q22	1073	992	1000	992
Total cold run time: 107749 ms
Total hot run time: 39779 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6643	6580	6585	6580
q2	331	227	234	227
q3	2889	2780	2886	2780
q4	2056	1874	1866	1866
q5	5794	5736	5761	5736
q6	221	131	127	127
q7	2174	1816	1806	1806
q8	3369	3562	3532	3532
q9	8987	8811	8902	8811
q10	3559	3583	3527	3527
q11	601	509	483	483
q12	832	604	622	604
q13	9275	3087	3181	3087
q14	313	271	266	266
q15	510	456	458	456
q16	493	447	458	447
q17	1856	1642	1597	1597
q18	8264	7753	7830	7753
q19	1694	1684	1639	1639
q20	2162	1782	1794	1782
q21	5158	5188	5000	5000
q22	1156	1045	1039	1039
Total cold run time: 68337 ms
Total hot run time: 59145 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

TPC-DS: Total hot run time: 196803 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3eb0b3d3111c874a0450b65bba57dee68a2d516, data reload: false

query1	1317	907	907	907
query2	6270	1907	1949	1907
query3	10924	4522	4454	4454
query4	61504	29345	23707	23707
query5	5273	467	457	457
query6	393	169	176	169
query7	5434	307	302	302
query8	301	215	221	215
query9	8321	2588	2577	2577
query10	473	273	267	267
query11	17820	15172	15719	15172
query12	163	107	109	107
query13	1391	435	431	431
query14	10263	7355	7138	7138
query15	193	185	185	185
query16	7157	526	485	485
query17	1150	604	574	574
query18	1832	311	303	303
query19	216	161	160	160
query20	117	111	110	110
query21	205	104	101	101
query22	4672	4496	4591	4496
query23	34495	34246	33970	33970
query24	6142	2916	2926	2916
query25	547	403	403	403
query26	664	162	168	162
query27	1727	362	359	359
query28	3863	2151	2108	2108
query29	708	482	461	461
query30	229	159	156	156
query31	986	816	848	816
query32	71	60	59	59
query33	419	304	326	304
query34	913	503	543	503
query35	862	756	749	749
query36	1087	976	1017	976
query37	121	73	67	67
query38	4051	4015	3932	3932
query39	1555	1619	1496	1496
query40	208	108	113	108
query41	55	51	49	49
query42	112	103	104	103
query43	551	493	489	489
query44	1235	836	834	834
query45	188	172	174	172
query46	1152	733	747	733
query47	2015	1915	1915	1915
query48	476	388	411	388
query49	749	423	407	407
query50	832	430	436	430
query51	7489	7363	7221	7221
query52	116	94	96	94
query53	265	184	186	184
query54	570	488	487	487
query55	82	82	78	78
query56	298	240	251	240
query57	1328	1230	1182	1182
query58	218	205	205	205
query59	3213	3060	2975	2975
query60	287	255	283	255
query61	128	106	107	106
query62	780	690	686	686
query63	213	186	188	186
query64	1360	648	633	633
query65	3338	3167	3192	3167
query66	706	297	296	296
query67	15878	15422	15588	15422
query68	4073	583	577	577
query69	428	264	270	264
query70	1191	1072	1138	1072
query71	348	267	255	255
query72	6327	3999	4090	3999
query73	743	346	367	346
query74	10277	9007	8990	8990
query75	3367	2606	2611	2606
query76	1896	1097	1034	1034
query77	490	272	269	269
query78	10626	9606	9461	9461
query79	1424	594	582	582
query80	871	420	437	420
query81	497	221	216	216
query82	1279	88	86	86
query83	245	142	151	142
query84	282	79	72	72
query85	873	293	294	293
query86	346	303	301	301
query87	4315	4198	4230	4198
query88	3416	2380	2335	2335
query89	421	289	288	288
query90	1971	185	185	185
query91	197	141	145	141
query92	66	48	50	48
query93	1729	550	537	537
query94	743	285	299	285
query95	357	263	256	256
query96	606	276	282	276
query97	3321	3183	3082	3082
query98	215	201	195	195
query99	1636	1294	1286	1286
Total cold run time: 312247 ms
Total hot run time: 196803 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

ClickBench: Total hot run time: 29.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3eb0b3d3111c874a0450b65bba57dee68a2d516, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.53	0.48	0.51
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.55	0.51	0.50
query10	0.56	0.53	0.54
query11	0.14	0.10	0.10
query12	0.16	0.10	0.11
query13	0.62	0.59	0.59
query14	0.77	0.79	0.80
query15	0.83	0.84	0.83
query16	0.38	0.38	0.38
query17	1.06	1.05	1.07
query18	0.23	0.21	0.20
query19	1.92	1.77	1.84
query20	0.01	0.00	0.01
query21	15.43	0.59	0.58
query22	2.25	1.74	1.39
query23	16.69	1.15	0.85
query24	3.02	1.23	2.00
query25	0.28	0.12	0.06
query26	0.61	0.14	0.13
query27	0.05	0.04	0.04
query28	9.45	0.49	0.51
query29	12.64	3.22	3.18
query30	0.25	0.06	0.06
query31	2.85	0.39	0.37
query32	3.25	0.46	0.46
query33	2.96	2.99	3.03
query34	16.91	4.49	4.51
query35	4.56	4.53	4.52
query36	0.68	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.27 s
Total hot run time: 29.93 s

dataroaring
dataroaring approved these changes Jun 27, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 3d43145 into apache:branch-3.0 Jun 27, 2025
21 of 23 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@zzzxl1993 @koarz
[opt](inverted index) enhance inverted index profile (apache#50876)
8b9be16 
pick apache#47504
pick apache#48826
pick apache#48950
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zzzxl1993 @Thearas @doris-robot @dataroaring