Skip to content

branch-3.0: [opt](Inverted index)Avoid repeated calculations of suffix paths #48137 #48153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 21, 2025
Merged

branch-3.0: [opt](Inverted index)Avoid repeated calculations of suffix paths #48137 #48153

dataroaring merged 1 commit into from
Feb 21, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Feb 20, 2025

Cherry-picked from #48137

[opt](Inverted index)Avoid repeated calculations of suffix paths (#48137
f38bc45 
)

Introduced a new local variable `escaped_suffix` to store the result of
`escape_for_path_name(suffix_path)`.
This change improves performance by avoiding repeated function calls.
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 20, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-H: Total hot run time: 40577 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f38bc45b6ddf50d17aac9286e59de7215959c5ee, data reload: false

------ Round 1 ----------------------------------
q1	17578	7387	7204	7204
q2	2065	175	158	158
q3	10689	1068	1205	1068
q4	10560	727	813	727
q5	7754	2854	2785	2785
q6	235	149	147	147
q7	963	610	605	605
q8	9358	1969	2002	1969
q9	6613	6376	6405	6376
q10	7023	2290	2310	2290
q11	469	280	259	259
q12	400	211	210	210
q13	17947	2985	3053	2985
q14	270	215	208	208
q15	564	536	526	526
q16	684	583	593	583
q17	967	522	543	522
q18	7223	6630	6682	6630
q19	1394	1115	1014	1014
q20	476	212	199	199
q21	3949	3208	3125	3125
q22	1074	987	1001	987
Total cold run time: 108255 ms
Total hot run time: 40577 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7269	7204	7184	7184
q2	334	232	224	224
q3	2825	2928	2893	2893
q4	1992	1809	1839	1809
q5	5683	5678	5731	5678
q6	228	142	142	142
q7	2288	1785	1822	1785
q8	3346	3505	3516	3505
q9	8773	8878	8817	8817
q10	3600	3576	3567	3567
q11	599	511	490	490
q12	790	577	595	577
q13	9054	3215	3148	3148
q14	316	265	286	265
q15	584	528	522	522
q16	686	669	657	657
q17	1835	1668	1608	1608
q18	8225	7675	7733	7675
q19	1642	1596	1503	1503
q20	2045	1845	1862	1845
q21	5461	5415	5417	5415
q22	1107	1063	997	997
Total cold run time: 68682 ms
Total hot run time: 60306 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-DS: Total hot run time: 198398 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f38bc45b6ddf50d17aac9286e59de7215959c5ee, data reload: false

query1	1276	920	906	906
query2	6250	2095	2055	2055
query3	10812	4528	4114	4114
query4	61885	29904	24047	24047
query5	5152	460	444	444
query6	416	175	173	173
query7	5432	311	318	311
query8	314	222	218	218
query9	8117	2714	2699	2699
query10	448	266	263	263
query11	17668	15285	15849	15285
query12	156	113	101	101
query13	1401	447	412	412
query14	10445	7496	6852	6852
query15	208	181	184	181
query16	7200	498	495	495
query17	1141	572	585	572
query18	1746	307	313	307
query19	198	152	150	150
query20	112	113	110	110
query21	193	104	102	102
query22	4886	4752	4686	4686
query23	34212	34212	33956	33956
query24	6222	2873	2956	2873
query25	538	430	427	427
query26	674	170	169	169
query27	1797	363	361	361
query28	4257	2487	2470	2470
query29	730	480	462	462
query30	239	167	169	167
query31	997	836	860	836
query32	77	59	60	59
query33	454	286	292	286
query34	930	520	539	520
query35	899	743	713	713
query36	1112	970	956	956
query37	130	83	75	75
query38	4104	4071	4013	4013
query39	1496	1454	1468	1454
query40	204	103	104	103
query41	51	52	49	49
query42	116	98	103	98
query43	544	499	495	495
query44	1183	868	842	842
query45	190	164	169	164
query46	1163	732	732	732
query47	2034	1890	1911	1890
query48	476	381	383	381
query49	704	399	386	386
query50	852	431	429	429
query51	7410	7221	7257	7221
query52	101	94	90	90
query53	266	188	187	187
query54	556	466	458	458
query55	79	77	74	74
query56	257	258	250	250
query57	1313	1149	1167	1149
query58	220	215	224	215
query59	3282	3151	3087	3087
query60	276	238	259	238
query61	106	119	117	117
query62	849	744	733	733
query63	215	186	182	182
query64	1356	679	641	641
query65	3245	3211	3211	3211
query66	682	294	305	294
query67	15831	15584	15566	15566
query68	4113	606	576	576
query69	421	263	264	263
query70	1175	1138	1161	1138
query71	374	255	263	255
query72	6359	4046	3957	3957
query73	769	356	364	356
query74	9314	9197	9072	9072
query75	3302	2610	2666	2610
query76	1969	1014	1122	1014
query77	496	268	273	268
query78	10579	9701	9615	9615
query79	1402	595	596	595
query80	875	433	418	418
query81	511	245	234	234
query82	1177	118	120	118
query83	250	145	141	141
query84	280	80	75	75
query85	910	308	320	308
query86	325	297	310	297
query87	4345	4345	4317	4317
query88	3620	2415	2383	2383
query89	416	295	291	291
query90	2005	188	185	185
query91	175	144	144	144
query92	65	48	51	48
query93	1857	556	556	556
query94	758	290	290	290
query95	358	259	253	253
query96	607	292	285	285
query97	3286	3147	3192	3147
query98	211	199	207	199
query99	1711	1428	1433	1428
Total cold run time: 311900 ms
Total hot run time: 198398 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

ClickBench: Total hot run time: 32.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f38bc45b6ddf50d17aac9286e59de7215959c5ee, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.52	0.50	0.49
query6	1.14	0.73	0.76
query7	0.02	0.02	0.02
query8	0.06	0.03	0.03
query9	0.56	0.48	0.49
query10	0.55	0.55	0.54
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.60	0.60	0.59
query14	2.86	2.85	2.78
query15	0.88	0.82	0.83
query16	0.38	0.38	0.38
query17	1.05	1.06	1.06
query18	0.22	0.22	0.21
query19	1.97	1.87	1.98
query20	0.01	0.01	0.01
query21	15.35	0.58	0.57
query22	3.05	2.08	2.74
query23	17.04	1.07	0.70
query24	2.98	1.93	0.46
query25	0.22	0.22	0.03
query26	0.61	0.14	0.13
query27	0.04	0.04	0.04
query28	10.70	1.10	1.08
query29	12.57	3.22	3.23
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.24	0.46	0.46
query33	2.99	3.05	2.99
query34	17.02	4.46	4.43
query35	4.61	4.52	4.50
query36	0.68	0.49	0.52
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.75 s
Total hot run time: 32.38 s

dataroaring
dataroaring approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit a11296a into branch-3.0 Feb 21, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48137-branch-3.0 branch February 21, 2025 00:32
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring