Skip to content

Conversation

@zzzxl1993
Copy link
Contributor

… profile (apache#44863)

Add NumInvertedIndexRemoteIOTotal to count the number of remote IO
operations in the inverted index
Related PR: apache#43542
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993 zzzxl1993 changed the title [opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in… [opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile #44863 Dec 19, 2024
@zzzxl1993
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}

Status InvertedIndexFileReader::_init_from_v2(int32_t read_buffer_size) {
Status InvertedIndexFileReader::_init_from(int32_t read_buffer_size, const io::IOContext* io_ctx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_init_from' exceeds recommended size/complexity thresholds [readability-function-size]

Status InvertedIndexFileReader::_init_from(int32_t read_buffer_size, const io::IOContext* io_ctx) {
                                ^
Additional context

be/src/olap/rowset/segment_v2/inverted_index_file_reader.cpp:49: 94 lines including whitespace and comments (threshold 80)

Status InvertedIndexFileReader::_init_from(int32_t read_buffer_size, const io::IOContext* io_ctx) {
                                ^

@doris-robot
Copy link

TPC-H: Total hot run time: 40795 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f2f8e0ef2143c4c501739cb2dba8bdf82ac0344, data reload: false

------ Round 1 ----------------------------------
q1	17574	7538	7355	7355
q2	2065	158	162	158
q3	10748	1112	1196	1112
q4	10570	729	775	729
q5	7726	2816	2774	2774
q6	233	147	141	141
q7	960	620	608	608
q8	9573	1929	1973	1929
q9	7867	6412	6408	6408
q10	7002	2307	2296	2296
q11	454	259	255	255
q12	410	214	211	211
q13	17797	2976	2966	2966
q14	232	208	213	208
q15	550	520	525	520
q16	685	607	596	596
q17	991	604	513	513
q18	7344	6659	6561	6561
q19	2212	1060	1085	1060
q20	461	210	201	201
q21	3986	3225	3211	3211
q22	1102	983	986	983
Total cold run time: 110542 ms
Total hot run time: 40795 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7315	7300	7277	7277
q2	319	234	233	233
q3	3074	2883	2916	2883
q4	2045	1775	1764	1764
q5	5688	5712	5809	5712
q6	226	147	144	144
q7	2245	1768	1752	1752
q8	3382	3471	3384	3384
q9	8891	8905	8853	8853
q10	3575	3497	3523	3497
q11	602	497	494	494
q12	829	627	575	575
q13	16511	3144	3124	3124
q14	300	275	265	265
q15	562	528	521	521
q16	718	678	676	676
q17	1863	1630	1588	1588
q18	8183	7819	7628	7628
q19	8453	1552	1469	1469
q20	2100	1857	1828	1828
q21	5407	5369	5263	5263
q22	1155	1012	1034	1012
Total cold run time: 83443 ms
Total hot run time: 59942 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196333 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f2f8e0ef2143c4c501739cb2dba8bdf82ac0344, data reload: false

query1	1280	923	922	922
query2	6231	2130	2052	2052
query3	10950	4279	3944	3944
query4	66253	29337	23593	23593
query5	5260	446	428	428
query6	468	163	164	163
query7	5679	305	308	305
query8	315	235	235	235
query9	9267	2680	2671	2671
query10	504	262	252	252
query11	17740	15316	15797	15316
query12	156	102	107	102
query13	1559	448	416	416
query14	10370	7267	7315	7267
query15	214	173	172	172
query16	7175	545	514	514
query17	1085	564	593	564
query18	1876	322	319	319
query19	216	157	162	157
query20	122	113	118	113
query21	70	46	51	46
query22	4855	4700	4653	4653
query23	34953	34121	34041	34041
query24	6266	2931	2966	2931
query25	553	425	413	413
query26	664	174	173	173
query27	1812	299	307	299
query28	4543	2573	2527	2527
query29	702	467	456	456
query30	260	166	166	166
query31	1004	834	847	834
query32	65	52	55	52
query33	403	280	286	280
query34	902	504	510	504
query35	869	752	727	727
query36	1081	923	946	923
query37	115	71	70	70
query38	4175	3981	4043	3981
query39	1506	1471	1448	1448
query40	141	77	79	77
query41	45	48	52	48
query42	109	98	101	98
query43	525	497	504	497
query44	1200	814	784	784
query45	185	168	168	168
query46	1157	723	756	723
query47	1997	1862	1914	1862
query48	446	370	366	366
query49	726	374	355	355
query50	814	417	411	411
query51	7304	7167	7116	7116
query52	99	90	86	86
query53	250	194	177	177
query54	548	438	437	437
query55	70	73	74	73
query56	243	234	218	218
query57	1194	1081	1095	1081
query58	201	197	199	197
query59	3197	3072	2988	2988
query60	272	253	234	234
query61	104	103	106	103
query62	772	650	652	650
query63	203	185	189	185
query64	1499	639	619	619
query65	3235	3162	3178	3162
query66	716	310	297	297
query67	15877	15295	15760	15295
query68	4600	551	540	540
query69	452	258	248	248
query70	1091	1042	1064	1042
query71	401	242	247	242
query72	6644	3856	3916	3856
query73	766	335	338	335
query74	10343	8848	8970	8848
query75	3340	2548	2607	2548
query76	2124	1163	1037	1037
query77	568	256	259	256
query78	10728	9810	9703	9703
query79	2989	594	577	577
query80	2094	402	411	402
query81	551	232	240	232
query82	889	115	109	109
query83	271	139	137	137
query84	291	84	84	84
query85	1276	296	284	284
query86	481	298	298	298
query87	4407	4276	4284	4276
query88	5067	2349	2385	2349
query89	410	288	290	288
query90	2086	190	184	184
query91	183	147	143	143
query92	67	47	49	47
query93	5413	529	515	515
query94	861	274	289	274
query95	349	248	241	241
query96	603	277	281	277
query97	3461	3173	3161	3161
query98	226	204	193	193
query99	1969	1297	1271	1271
Total cold run time: 329648 ms
Total hot run time: 196333 ms

@qidaye qidaye merged commit e807b37 into apache:branch-3.0 Dec 19, 2024
22 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants