Skip to content

[opt](index) add more inverted index query profile #36545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 5 commits into from
Jun 20, 2024
Merged

[opt](index) add more inverted index query profile #36545

xiaokang merged 5 commits into from
Jun 20, 2024

Conversation

@xiaokang
Copy link
Contributor

@xiaokang xiaokang commented Jun 19, 2024

add more inverted index query profile:

  • InvertedIndexQueryNullBitmapTime
  • InvertedIndexQueryFileExistsTime
  • InvertedIndexSearcherCacheHit
  • InvertedIndexSearcherCacheMiss

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xiaokang
Copy link
Contributor Author

xiaokang commented Jun 19, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

zzzxl1993
zzzxl1993 approved these changes Jun 19, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
}

bool InvertedIndexReader::indexExists(io::Path& index_file_path) {
// SCOPED_RAW_TIMER(&stats->inverted_index_query_file_exists_timer);
Copy link
Contributor

@zzzxl1993 zzzxl1993 Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can delete it

@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2024

PR approved by anyone and no changes requested.

@xiaokang
Copy link
Contributor Author

xiaokang commented Jun 19, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

@xiaokang
Copy link
Contributor Author

xiaokang commented Jun 19, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TPC-H: Total hot run time: 49877 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8d108993705fb4a8f49b40f5b7e0c9ce9f6b7a8, data reload: false

------ Round 1 ----------------------------------
q1	18001	4418	4392	4392
q2	2072	158	153	153
q3	10428	1898	1951	1898
q4	10241	1255	1342	1255
q5	8687	4134	3968	3968
q6	261	125	124	124
q7	2085	1584	1579	1579
q8	9257	2823	2740	2740
q9	10668	10281	10261	10261
q10	8648	3493	3491	3491
q11	422	241	243	241
q12	467	298	304	298
q13	18343	3945	4035	3945
q14	357	320	327	320
q15	514	454	462	454
q16	679	570	570	570
q17	1153	985	971	971
q18	7381	6978	6787	6787
q19	1785	1655	1647	1647
q20	523	302	317	302
q21	4489	4039	4073	4039
q22	506	450	442	442
Total cold run time: 116967 ms
Total hot run time: 49877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4369	4372	4311	4311
q2	327	218	241	218
q3	4181	4165	4215	4165
q4	2760	2738	2744	2738
q5	7244	7142	7161	7142
q6	237	117	116	116
q7	3285	2849	2795	2795
q8	4386	4522	4482	4482
q9	16993	16842	16854	16842
q10	4281	4291	4268	4268
q11	772	706	670	670
q12	1041	844	854	844
q13	6673	3737	3734	3734
q14	444	425	430	425
q15	525	465	458	458
q16	738	685	673	673
q17	3908	3841	3892	3841
q18	8828	8833	8706	8706
q19	1742	1741	1624	1624
q20	2357	2151	2109	2109
q21	9094	8564	8515	8515
q22	1083	1011	940	940
Total cold run time: 85268 ms
Total hot run time: 79616 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8096/21416)
Line Coverage: 29.45% (66149/224638)
Region Coverage: 28.93% (34102/117889)
Branch Coverage: 24.80% (17505/70576)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8d108993705fb4a8f49b40f5b7e0c9ce9f6b7a8_a8d108993705fb4a8f49b40f5b7e0c9ce9f6b7a8/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TPC-DS: Total hot run time: 204087 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8d108993705fb4a8f49b40f5b7e0c9ce9f6b7a8, data reload: false

query1	943	421	376	376
query2	6544	2875	2763	2763
query3	6926	206	204	204
query4	20807	18023	17822	17822
query5	19728	6519	6521	6519
query6	289	220	229	220
query7	4159	294	302	294
query8	411	466	382	382
query9	3115	2696	2641	2641
query10	418	296	302	296
query11	11273	10706	10732	10706
query12	119	79	72	72
query13	5596	689	694	689
query14	17683	13873	13335	13335
query15	366	252	244	244
query16	6445	287	271	271
query17	1724	1424	881	881
query18	2308	426	413	413
query19	214	152	148	148
query20	76	81	82	81
query21	190	98	97	97
query22	5118	5103	5027	5027
query23	32534	31932	31837	31837
query24	7009	6570	6512	6512
query25	533	450	433	433
query26	494	159	162	159
query27	1767	296	297	296
query28	6182	2383	2349	2349
query29	2830	2680	2672	2672
query30	244	164	167	164
query31	900	731	767	731
query32	72	64	58	58
query33	402	254	252	252
query34	863	478	479	478
query35	1146	915	937	915
query36	1191	1007	1151	1007
query37	87	60	59	59
query38	3122	2921	2940	2921
query39	1355	1340	1314	1314
query40	213	98	93	93
query41	45	42	45	42
query42	86	79	88	79
query43	823	681	667	667
query44	1137	720	728	720
query45	251	237	236	236
query46	1240	942	974	942
query47	2004	1893	1849	1849
query48	1001	704	697	697
query49	633	365	384	365
query50	867	588	592	588
query51	4782	4640	4628	4628
query52	98	83	95	83
query53	443	313	333	313
query54	2634	2488	2455	2455
query55	97	77	85	77
query56	235	197	212	197
query57	1166	1196	1129	1129
query58	208	198	207	198
query59	4013	3847	3863	3847
query60	209	207	208	207
query61	99	96	98	96
query62	888	468	471	468
query63	483	338	342	338
query64	2496	1562	1400	1400
query65	3652	3526	3553	3526
query66	752	379	381	379
query67	16022	15009	16331	15009
query68	8427	663	640	640
query69	566	350	352	350
query70	1650	1479	1393	1393
query71	377	309	317	309
query72	6573	3540	3524	3524
query73	735	330	321	321
query74	6373	5847	5833	5833
query75	4658	3666	3659	3659
query76	4875	1158	1103	1103
query77	677	275	258	258
query78	12606	12749	13075	12749
query79	12690	638	643	638
query80	805	401	432	401
query81	499	234	234	234
query82	873	96	96	96
query83	181	132	133	132
query84	260	71	74	71
query85	745	321	328	321
query86	335	309	296	296
query87	3270	3033	3062	3033
query88	5887	2344	2315	2315
query89	397	306	304	304
query90	2125	199	211	199
query91	185	147	138	138
query92	58	53	51	51
query93	5362	563	566	563
query94	994	212	207	207
query95	1112	1045	1057	1045
query96	636	327	327	327
query97	6432	6268	6313	6268
query98	187	182	176	176
query99	2645	922	867	867
Total cold run time: 314955 ms
Total hot run time: 204087 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

ClickBench: Total hot run time: 31.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8d108993705fb4a8f49b40f5b7e0c9ce9f6b7a8, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.26	0.05	0.04
query4	1.79	0.07	0.07
query5	0.53	0.53	0.53
query6	1.27	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.48
query10	0.54	0.53	0.54
query11	0.11	0.09	0.08
query12	0.11	0.10	0.09
query13	0.64	0.61	0.61
query14	0.79	0.79	0.79
query15	0.77	0.76	0.76
query16	0.40	0.37	0.37
query17	1.03	1.00	1.01
query18	0.23	0.27	0.24
query19	1.83	1.87	1.87
query20	0.02	0.01	0.01
query21	15.46	0.57	0.54
query22	2.27	2.05	1.69
query23	17.07	1.00	0.93
query24	4.24	3.96	1.50
query25	0.41	0.14	0.05
query26	0.78	0.15	0.14
query27	0.04	0.04	0.04
query28	5.50	0.78	0.77
query29	12.62	2.30	2.32
query30	0.57	0.53	0.49
query31	2.81	0.39	0.38
query32	3.37	0.50	0.49
query33	3.06	3.11	3.09
query34	15.25	4.81	4.80
query35	4.84	4.83	4.85
query36	1.06	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 100.71 s
Total hot run time: 31.46 s

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a8d108993705fb4a8f49b40f5b7e0c9ce9f6b7a8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@xiaokang
Copy link
Contributor Author

xiaokang commented Jun 19, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8096/21416)
Line Coverage: 29.44% (66152/224664)
Region Coverage: 28.92% (34100/117901)
Branch Coverage: 24.80% (17506/70590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d14a90c6b181b2fe65016f920d55bb86a253c645_d14a90c6b181b2fe65016f920d55bb86a253c645/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TPC-H: Total hot run time: 50185 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d14a90c6b181b2fe65016f920d55bb86a253c645, data reload: false

------ Round 1 ----------------------------------
q1	17825	4429	4402	4402
q2	2072	157	151	151
q3	10254	1879	1863	1863
q4	10171	1259	1371	1259
q5	8475	3969	3952	3952
q6	232	148	130	130
q7	2073	1627	1600	1600
q8	9322	2754	2726	2726
q9	10711	10452	10180	10180
q10	8662	3544	3532	3532
q11	418	238	254	238
q12	475	309	305	305
q13	18342	4015	4056	4015
q14	356	326	332	326
q15	501	462	456	456
q16	692	575	572	572
q17	1149	986	972	972
q18	7416	6960	6948	6948
q19	1800	1730	1700	1700
q20	528	298	317	298
q21	4462	4157	4126	4126
q22	532	443	434	434
Total cold run time: 116468 ms
Total hot run time: 50185 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4314	4311	4311
q2	324	227	225	225
q3	4184	4153	4139	4139
q4	2741	2740	2756	2740
q5	7122	7074	7131	7074
q6	237	121	119	119
q7	3256	2851	2871	2851
q8	4399	4469	4515	4469
q9	16992	16781	16782	16781
q10	4242	4285	4274	4274
q11	755	692	690	690
q12	1039	865	880	865
q13	6991	3729	3750	3729
q14	451	427	416	416
q15	504	467	453	453
q16	751	672	678	672
q17	3877	3868	3799	3799
q18	8831	8781	8842	8781
q19	1734	1704	1708	1704
q20	2398	2131	2094	2094
q21	8530	8478	8439	8439
q22	1059	989	968	968
Total cold run time: 84742 ms
Total hot run time: 79593 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TPC-DS: Total hot run time: 204878 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d14a90c6b181b2fe65016f920d55bb86a253c645, data reload: false

query1	947	437	391	391
query2	6554	3004	2873	2873
query3	6924	208	203	203
query4	20718	18010	17982	17982
query5	19739	6575	6554	6554
query6	293	225	229	225
query7	4170	307	317	307
query8	448	398	490	398
query9	3190	2741	2682	2682
query10	407	300	304	300
query11	11453	10846	10730	10730
query12	127	76	75	75
query13	5640	701	684	684
query14	18136	13873	13401	13401
query15	364	237	249	237
query16	6468	293	265	265
query17	1711	1443	900	900
query18	2301	424	422	422
query19	220	152	152	152
query20	76	79	81	79
query21	190	95	98	95
query22	5281	5052	5042	5042
query23	32774	31947	31933	31933
query24	6952	6594	6482	6482
query25	512	452	434	434
query26	533	163	164	163
query27	1882	297	300	297
query28	6201	2407	2355	2355
query29	2853	2733	2792	2733
query30	244	175	180	175
query31	947	760	753	753
query32	73	65	66	65
query33	401	261	268	261
query34	855	485	492	485
query35	1153	902	887	887
query36	1270	1166	1279	1166
query37	95	62	61	61
query38	3125	2900	2924	2900
query39	1379	1349	1339	1339
query40	206	100	95	95
query41	56	49	45	45
query42	81	80	86	80
query43	707	728	659	659
query44	1118	714	735	714
query45	250	237	236	236
query46	1233	977	992	977
query47	1825	1824	1825	1824
query48	1037	700	723	700
query49	631	381	377	377
query50	864	644	621	621
query51	4841	4674	4664	4664
query52	99	85	92	85
query53	454	324	318	318
query54	2681	2466	2462	2462
query55	97	83	84	83
query56	246	242	211	211
query57	1286	1147	1065	1065
query58	229	204	205	204
query59	4152	3805	4214	3805
query60	219	216	217	216
query61	99	97	103	97
query62	835	485	461	461
query63	499	347	349	347
query64	2494	1565	1517	1517
query65	3660	3617	3569	3569
query66	772	383	394	383
query67	16671	16580	15401	15401
query68	8718	681	646	646
query69	582	347	334	334
query70	1590	1502	1369	1369
query71	405	317	311	311
query72	6655	3546	3513	3513
query73	752	330	327	327
query74	6442	5838	5794	5794
query75	5181	3835	3736	3736
query76	5099	1106	1166	1106
query77	805	261	258	258
query78	12749	12047	11848	11848
query79	7650	670	683	670
query80	1149	417	422	417
query81	496	244	241	241
query82	1673	101	98	98
query83	176	146	136	136
query84	260	75	73	73
query85	883	326	321	321
query86	325	303	312	303
query87	3337	3041	3041	3041
query88	4801	2343	2345	2343
query89	392	305	318	305
query90	1890	210	210	210
query91	181	155	144	144
query92	60	55	57	55
query93	5545	589	605	589
query94	682	224	218	218
query95	1132	1096	1061	1061
query96	679	328	332	328
query97	6476	6358	6399	6358
query98	201	173	182	173
query99	2970	850	873	850
Total cold run time: 313884 ms
Total hot run time: 204878 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d14a90c6b181b2fe65016f920d55bb86a253c645, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.02	0.03
query3	0.24	0.06	0.05
query4	1.78	0.06	0.07
query5	0.55	0.53	0.52
query6	1.24	0.63	0.62
query7	0.01	0.01	0.02
query8	0.03	0.03	0.02
query9	0.53	0.49	0.49
query10	0.55	0.54	0.53
query11	0.11	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.62
query14	0.78	0.79	0.78
query15	0.77	0.76	0.75
query16	0.36	0.40	0.37
query17	1.02	1.01	0.99
query18	0.23	0.25	0.23
query19	1.92	1.86	1.87
query20	0.01	0.02	0.01
query21	15.45	0.55	0.55
query22	2.16	1.66	1.68
query23	17.45	0.95	0.98
query24	4.65	1.21	3.47
query25	0.39	0.11	0.06
query26	0.75	0.16	0.16
query27	0.05	0.04	0.04
query28	5.82	0.76	0.70
query29	12.76	2.22	2.27
query30	0.62	0.54	0.53
query31	2.80	0.39	0.40
query32	3.36	0.50	0.50
query33	3.11	3.10	3.07
query34	15.25	4.84	4.77
query35	4.86	4.85	4.86
query36	1.06	1.01	1.03
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.01	0.01
Total cold run time: 101.9 s
Total hot run time: 31.02 s

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d14a90c6b181b2fe65016f920d55bb86a253c645 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@xiaokang xiaokang merged commit e2e947e into apache:branch-2.0 Jun 20, 2024
@xiaokang xiaokang mentioned this pull request Jun 20, 2024
Merged
xiaokang added a commit that referenced this pull request Jun 21, 2024
@xiaokang
[fix](index profile) fix missing update inverted_index_searcher_cache…
a59c15a 
…_hit/miss in #36545  (#36636)
@xiaokang xiaokang mentioned this pull request Jun 23, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@xiaokang
[opt](index) add more inverted index query profile (apache#36545)
944a8fe 
add more inverted index query profile:
- InvertedIndexQueryNullBitmapTime
- InvertedIndexQueryFileExistsTime
- InvertedIndexSearcherCacheHit
- InvertedIndexSearcherCacheMiss
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@xiaokang
[fix](index profile) fix missing update inverted_index_searcher_cache…
7715b44 
…_hit/miss in apache#36545  (apache#36636)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

dev/2.0.12-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xiaokang @doris-robot @zzzxl1993