Skip to content

branch-3.1: [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 #57055 #57137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Oct 30, 2025
Merged

branch-3.1: [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 #57055 #57137

morrySnow merged 2 commits into from
Oct 30, 2025

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Oct 20, 2025 •
edited
Loading

@zzzxl1993 zzzxl1993 requested a review from morrySnow as a code owner October 20, 2025 02:59
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.29% (1264/1536)
Line Coverage 66.12% (22678/34299)
Region Coverage 67.46% (11298/16747)
Branch Coverage 57.12% (5969/10450)

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Oct 20, 2025

run buildall

2 similar comments
@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Oct 20, 2025

run buildall

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Oct 20, 2025

run buildall

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.29% (1264/1536)
Line Coverage 66.14% (22686/34299)
Region Coverage 67.44% (11295/16747)
Branch Coverage 57.19% (5976/10450)

@zzzxl1993
[opt](inverted index) Add built_in_analyzer parser alias and modify c…
b9b9598 
…ustom analyzer logic (apache#57055)

Add built-in parser alias and support same-name config for custom
analyzer components
@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.29% (1264/1536)
Line Coverage 66.12% (22678/34299)
Region Coverage 67.39% (11286/16747)
Branch Coverage 57.13% (5970/10450)

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

TPC-H: Total hot run time: 32771 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9b9598f6c96a8b912e0bb65c35b68e2ee86bfed, data reload: false

------ Round 1 ----------------------------------
q1	17938	5581	5458	5458
q2	2025	402	281	281
q3	12461	1258	772	772
q4	10529	878	458	458
q5	9333	2396	2126	2126
q6	182	168	134	134
q7	911	751	613	613
q8	9337	1444	1131	1131
q9	5249	4949	4946	4946
q10	6763	2266	1819	1819
q11	459	276	274	274
q12	344	352	211	211
q13	17785	3636	3045	3045
q14	218	230	215	215
q15	523	472	457	457
q16	416	434	378	378
q17	577	870	358	358
q18	7058	6373	6391	6373
q19	1343	966	543	543
q20	331	341	210	210
q21	2953	2172	1992	1992
q22	1031	1084	977	977
Total cold run time: 107766 ms
Total hot run time: 32771 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5564	5500	5564	5500
q2	232	325	241	241
q3	2291	2626	2361	2361
q4	1373	1832	1383	1383
q5	4421	5077	5029	5029
q6	167	164	129	129
q7	2055	2021	1795	1795
q8	2669	2841	2753	2753
q9	7286	7221	7211	7211
q10	3089	3245	2773	2773
q11	579	524	496	496
q12	665	758	637	637
q13	3456	3784	3202	3202
q14	280	296	258	258
q15	518	478	472	472
q16	447	490	446	446
q17	1206	1711	1284	1284
q18	7633	7419	7248	7248
q19	821	1092	1136	1092
q20	1991	2056	1892	1892
q21	5320	4944	4678	4678
q22	1089	1035	1064	1035
Total cold run time: 53152 ms
Total hot run time: 51915 ms

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

TPC-DS: Total hot run time: 192819 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9b9598f6c96a8b912e0bb65c35b68e2ee86bfed, data reload: false

query1	963	406	410	406
query2	6207	1939	1864	1864
query3	8684	209	204	204
query4	33695	23977	23408	23408
query5	3640	587	469	469
query6	275	186	177	177
query7	4197	482	328	328
query8	292	252	237	237
query9	9309	2649	2638	2638
query10	460	323	264	264
query11	17816	15445	15210	15210
query12	159	104	108	104
query13	1549	533	430	430
query14	10396	6765	6778	6765
query15	265	192	171	171
query16	8061	640	528	528
query17	1555	745	580	580
query18	2140	418	313	313
query19	212	189	167	167
query20	128	125	118	118
query21	213	126	105	105
query22	4559	4612	4675	4612
query23	35280	34256	33982	33982
query24	7511	2723	2799	2723
query25	548	503	445	445
query26	1212	294	182	182
query27	2116	510	387	387
query28	5442	2217	2202	2202
query29	790	630	488	488
query30	254	195	173	173
query31	1017	909	867	867
query32	91	57	58	57
query33	517	394	319	319
query34	764	862	534	534
query35	811	811	746	746
query36	999	1056	974	974
query37	108	101	76	76
query38	3999	4055	3974	3974
query39	1535	1497	1520	1497
query40	217	126	112	112
query41	53	53	51	51
query42	133	109	106	106
query43	518	549	513	513
query44	1383	854	867	854
query45	181	177	173	173
query46	884	1065	681	681
query47	1992	1983	1943	1943
query48	413	450	361	361
query49	778	501	406	406
query50	684	682	440	440
query51	7331	7351	7333	7333
query52	103	101	96	96
query53	244	266	194	194
query54	563	582	483	483
query55	87	82	80	80
query56	279	281	254	254
query57	1286	1276	1237	1237
query58	238	220	220	220
query59	3028	3217	3046	3046
query60	299	293	268	268
query61	122	121	116	116
query62	790	765	705	705
query63	239	214	197	197
query64	4448	1020	649	649
query65	3408	3302	3342	3302
query66	1093	410	324	324
query67	16232	15860	15752	15752
query68	7747	833	537	537
query69	497	316	273	273
query70	1197	1162	1107	1107
query71	369	299	267	267
query72	5747	3920	3826	3826
query73	645	757	358	358
query74	10478	9115	9092	9092
query75	3208	3168	2698	2698
query76	3130	1174	790	790
query77	505	376	285	285
query78	10370	10565	9558	9558
query79	3240	901	596	596
query80	703	530	433	433
query81	501	262	233	233
query82	578	121	88	88
query83	163	156	144	144
query84	255	102	76	76
query85	784	352	302	302
query86	381	316	305	305
query87	4315	4328	4227	4227
query88	5164	2450	2395	2395
query89	388	334	294	294
query90	1827	187	182	182
query91	144	142	114	114
query92	65	56	52	52
query93	2127	903	533	533
query94	691	385	293	293
query95	336	277	271	271
query96	484	599	291	291
query97	3204	3241	3104	3104
query98	229	207	195	195
query99	1557	1394	1280	1280
Total cold run time: 295824 ms
Total hot run time: 192819 ms

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

ClickBench: Total hot run time: 29.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9b9598f6c96a8b912e0bb65c35b68e2ee86bfed, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.06
query4	1.65	0.08	0.08
query5	0.52	0.49	0.52
query6	1.12	0.75	0.75
query7	0.03	0.02	0.01
query8	0.06	0.05	0.05
query9	0.56	0.51	0.50
query10	0.55	0.56	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.59
query14	0.80	0.78	0.80
query15	0.85	0.83	0.85
query16	0.39	0.37	0.38
query17	1.07	1.06	1.02
query18	0.18	0.19	0.19
query19	1.99	1.88	1.91
query20	0.01	0.02	0.01
query21	15.37	0.94	0.66
query22	0.76	0.74	0.69
query23	14.83	1.45	0.67
query24	2.20	0.36	0.22
query25	0.15	0.08	0.09
query26	0.28	0.19	0.18
query27	0.08	0.08	0.08
query28	13.42	1.25	0.56
query29	12.68	4.05	3.38
query30	0.25	0.09	0.06
query31	2.83	0.61	0.39
query32	3.22	0.56	0.48
query33	3.03	3.07	3.06
query34	16.58	5.22	4.63
query35	4.63	4.66	4.60
query36	0.62	0.49	0.48
query37	0.18	0.16	0.17
query38	0.16	0.16	0.15
query39	0.06	0.05	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.05
Total cold run time: 102.76 s
Total hot run time: 29.3 s

airborne12
airborne12 approved these changes Oct 21, 2025
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow morrySnow changed the title [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer branch-3.1: [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 #57055 Oct 30, 2025
@morrySnow morrySnow merged commit 347775b into apache:branch-3.1 Oct 30, 2025
20 of 21 checks passed
@morrySnow morrySnow mentioned this pull request Nov 13, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zzzxl1993 @hello-stephen @doris-robot @airborne12 @morrySnow