[fix](bf index) add ngram bf index validation in nereids index definition check #45780

airborne12 · 2024-12-23T03:30:15Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #44973

Problem Summary:

Need to check ngram bf index properties in index definition for nereids.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-23T03:30:20Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2024-12-23T04:39:33Z

run buildall

github-actions · 2024-12-23T05:34:18Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-23T05:34:20Z

PR approved by anyone and no changes requested.

doris-robot · 2024-12-23T05:58:25Z

TPC-H: Total hot run time: 40071 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit beb78e5f31ae751d3a9ce01d4bd84de362925b29, data reload: false

------ Round 1 ----------------------------------
q1	17586	7557	7382	7382
q2	2067	192	175	175
q3	10648	1084	1216	1084
q4	10592	709	748	709
q5	7616	2744	2675	2675
q6	244	148	150	148
q7	996	636	605	605
q8	9269	1865	1966	1865
q9	6762	6413	6401	6401
q10	7045	2304	2314	2304
q11	476	270	261	261
q12	438	228	223	223
q13	17775	2990	2967	2967
q14	263	219	212	212
q15	570	497	514	497
q16	668	604	607	604
q17	996	567	539	539
q18	7280	6768	6702	6702
q19	1332	999	1004	999
q20	460	189	176	176
q21	4100	3236	3236	3236
q22	376	338	307	307
Total cold run time: 107559 ms
Total hot run time: 40071 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7376	7273	7336	7273
q2	330	236	227	227
q3	2924	2834	2962	2834
q4	2083	1896	1961	1896
q5	5684	5666	5678	5666
q6	233	141	141	141
q7	2264	1881	1831	1831
q8	3444	3567	3585	3567
q9	8800	8954	8986	8954
q10	3613	3580	3612	3580
q11	603	512	518	512
q12	824	611	613	611
q13	12386	3168	3131	3131
q14	323	286	275	275
q15	567	512	504	504
q16	685	637	644	637
q17	1868	1660	1619	1619
q18	8231	7780	7659	7659
q19	1708	1499	1538	1499
q20	2100	1848	1875	1848
q21	5624	5677	5420	5420
q22	675	587	598	587
Total cold run time: 72345 ms
Total hot run time: 60271 ms

doris-robot · 2024-12-23T06:10:48Z

TPC-DS: Total hot run time: 196920 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit beb78e5f31ae751d3a9ce01d4bd84de362925b29, data reload: false

query1	1313	977	905	905
query2	6226	2454	2374	2374
query3	11063	4858	4881	4858
query4	33525	23463	23348	23348
query5	4794	465	468	465
query6	300	189	190	189
query7	4005	310	307	307
query8	311	249	237	237
query9	9649	2689	2695	2689
query10	501	260	274	260
query11	17919	15202	15102	15102
query12	163	102	102	102
query13	1600	453	430	430
query14	9718	6990	7496	6990
query15	277	194	194	194
query16	8089	486	468	468
query17	1724	608	606	606
query18	2193	313	342	313
query19	365	177	199	177
query20	122	113	111	111
query21	211	110	107	107
query22	4907	4340	4441	4340
query23	35166	34281	33774	33774
query24	10297	2476	2455	2455
query25	650	415	395	395
query26	1256	158	152	152
query27	2617	340	340	340
query28	7873	2480	2457	2457
query29	879	425	424	424
query30	228	150	151	150
query31	1053	826	879	826
query32	100	54	56	54
query33	765	293	301	293
query34	1148	535	517	517
query35	895	783	757	757
query36	1126	944	967	944
query37	155	79	74	74
query38	4259	4251	4176	4176
query39	1496	1458	1462	1458
query40	209	104	102	102
query41	46	45	46	45
query42	114	109	107	107
query43	553	509	519	509
query44	1311	850	847	847
query45	192	170	174	170
query46	1174	739	730	730
query47	2080	1938	1947	1938
query48	420	321	321	321
query49	929	403	388	388
query50	824	414	393	393
query51	7351	7201	7193	7193
query52	109	93	87	87
query53	264	185	179	179
query54	1198	420	429	420
query55	85	85	81	81
query56	267	242	236	236
query57	1302	1208	1150	1150
query58	228	244	226	226
query59	3504	3278	3223	3223
query60	288	247	248	247
query61	114	113	110	110
query62	844	706	676	676
query63	232	196	187	187
query64	4079	707	668	668
query65	3328	3291	3286	3286
query66	760	313	316	313
query67	16552	15500	15484	15484
query68	5613	563	544	544
query69	514	257	254	254
query70	1211	1125	1135	1125
query71	473	249	250	249
query72	7044	4030	4135	4030
query73	787	374	365	365
query74	10189	9108	8942	8942
query75	3663	2615	2620	2615
query76	3759	1099	1066	1066
query77	598	280	269	269
query78	10206	9605	9423	9423
query79	2063	610	616	610
query80	1349	431	423	423
query81	518	247	234	234
query82	470	127	116	116
query83	190	144	142	142
query84	280	69	74	69
query85	1113	348	306	306
query86	349	305	306	305
query87	4657	4458	4339	4339
query88	4028	2210	2200	2200
query89	429	288	299	288
query90	2178	188	185	185
query91	138	102	105	102
query92	69	49	50	49
query93	3824	539	538	538
query94	889	292	309	292
query95	349	250	245	245
query96	620	279	289	279
query97	2869	2693	2673	2673
query98	223	195	192	192
query99	1596	1323	1317	1317
Total cold run time: 309553 ms
Total hot run time: 196920 ms

doris-robot · 2024-12-23T06:16:20Z

ClickBench: Total hot run time: 32.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit beb78e5f31ae751d3a9ce01d4bd84de362925b29, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.11
query5	0.44	0.40	0.43
query6	1.15	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.51	0.52
query10	0.55	0.59	0.56
query11	0.14	0.10	0.11
query12	0.14	0.10	0.11
query13	0.61	0.61	0.59
query14	2.79	2.71	2.76
query15	0.90	0.83	0.82
query16	0.38	0.39	0.39
query17	0.96	1.05	1.06
query18	0.22	0.21	0.21
query19	1.92	1.84	1.92
query20	0.01	0.01	0.01
query21	15.37	0.58	0.58
query22	2.30	3.31	1.51
query23	16.86	1.15	0.81
query24	3.32	1.03	1.19
query25	0.32	0.18	0.11
query26	0.39	0.14	0.13
query27	0.05	0.04	0.04
query28	10.28	1.12	1.07
query29	12.58	3.19	3.21
query30	0.25	0.07	0.06
query31	2.88	0.40	0.39
query32	3.23	0.46	0.46
query33	3.09	3.06	3.19
query34	17.26	4.50	4.54
query35	4.52	4.54	4.51
query36	0.67	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.66 s
Total hot run time: 32.45 s

…tion check

airborne12 · 2024-12-30T09:04:37Z

run buildall

doris-robot · 2024-12-30T09:47:11Z

TPC-H: Total hot run time: 33127 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 326486b30da1ab5c782edd1683423487ec6aef36, data reload: false

------ Round 1 ----------------------------------
q1	17564	6129	6169	6129
q2	2049	300	185	185
q3	10544	1245	768	768
q4	10205	885	446	446
q5	7553	2182	2011	2011
q6	211	184	150	150
q7	917	775	623	623
q8	9223	1372	1234	1234
q9	5320	4893	4921	4893
q10	6780	2317	1895	1895
q11	480	281	251	251
q12	349	377	231	231
q13	17819	3598	3028	3028
q14	238	228	224	224
q15	575	519	493	493
q16	634	630	603	603
q17	590	857	317	317
q18	6861	6661	6511	6511
q19	1887	957	571	571
q20	306	326	196	196
q21	3096	2342	2049	2049
q22	363	340	319	319
Total cold run time: 103564 ms
Total hot run time: 33127 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6293	6227	6255	6227
q2	255	326	234	234
q3	2207	2644	2306	2306
q4	1437	1884	1375	1375
q5	4299	4840	4816	4816
q6	196	180	149	149
q7	2080	1960	1810	1810
q8	2703	2826	2688	2688
q9	7359	7290	7241	7241
q10	3087	3380	2898	2898
q11	587	512	495	495
q12	699	794	704	704
q13	3352	3738	3141	3141
q14	284	308	273	273
q15	584	524	505	505
q16	671	698	672	672
q17	1274	1728	1280	1280
q18	7686	7462	7405	7405
q19	878	1212	1112	1112
q20	1979	2033	1828	1828
q21	5787	5469	5019	5019
q22	649	649	592	592
Total cold run time: 54346 ms
Total hot run time: 52770 ms

doris-robot · 2024-12-30T09:59:40Z

TPC-DS: Total hot run time: 197764 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 326486b30da1ab5c782edd1683423487ec6aef36, data reload: false

query1	1296	944	911	911
query2	6499	2406	2356	2356
query3	10966	4692	4476	4476
query4	39143	23616	23531	23531
query5	5079	644	482	482
query6	261	208	207	207
query7	3656	509	323	323
query8	324	258	255	255
query9	6323	2754	2755	2754
query10	430	310	272	272
query11	15915	15388	15564	15388
query12	166	109	106	106
query13	993	561	429	429
query14	11011	7769	7299	7299
query15	265	221	191	191
query16	7908	633	475	475
query17	1545	789	579	579
query18	2111	403	334	334
query19	198	190	161	161
query20	127	110	109	109
query21	207	127	108	108
query22	4674	4531	4423	4423
query23	35415	33581	33842	33581
query24	6182	2373	2385	2373
query25	501	476	396	396
query26	794	289	163	163
query27	2321	488	342	342
query28	5247	2524	2481	2481
query29	692	619	424	424
query30	210	192	156	156
query31	970	938	880	880
query32	94	62	58	58
query33	500	366	322	322
query34	790	886	528	528
query35	826	823	817	817
query36	1038	1047	978	978
query37	130	100	77	77
query38	4224	4160	4359	4160
query39	1539	1502	1486	1486
query40	206	124	109	109
query41	45	44	45	44
query42	131	111	110	110
query43	548	541	519	519
query44	1346	842	836	836
query45	184	173	177	173
query46	932	1075	699	699
query47	1974	1963	1947	1947
query48	385	414	314	314
query49	714	490	412	412
query50	701	705	410	410
query51	7322	7268	7396	7268
query52	108	107	95	95
query53	237	268	188	188
query54	500	498	431	431
query55	90	84	77	77
query56	270	286	252	252
query57	1255	1235	1160	1160
query58	245	233	222	222
query59	3174	3193	3148	3148
query60	283	298	255	255
query61	108	108	108	108
query62	876	833	760	760
query63	246	208	198	198
query64	3371	1049	719	719
query65	3425	3276	3248	3248
query66	697	421	314	314
query67	16580	15645	15488	15488
query68	10050	773	531	531
query69	442	296	262	262
query70	1207	1171	1179	1171
query71	434	297	267	267
query72	6311	3856	3905	3856
query73	768	790	362	362
query74	10259	9068	8926	8926
query75	4477	3161	2662	2662
query76	5322	1212	799	799
query77	962	376	278	278
query78	10078	10192	9411	9411
query79	5335	895	581	581
query80	754	537	443	443
query81	478	271	236	236
query82	283	162	183	162
query83	197	163	150	150
query84	292	85	75	75
query85	770	364	317	317
query86	357	296	306	296
query87	4790	4680	4499	4499
query88	4257	2230	2219	2219
query89	454	346	291	291
query90	2035	191	190	190
query91	133	136	105	105
query92	68	56	55	55
query93	3242	892	549	549
query94	695	402	298	298
query95	332	271	259	259
query96	494	616	287	287
query97	2731	2786	2693	2693
query98	238	205	194	194
query99	1652	1551	1445	1445
Total cold run time: 306420 ms
Total hot run time: 197764 ms

doris-robot · 2024-12-30T10:05:01Z

ClickBench: Total hot run time: 31.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 326486b30da1ab5c782edd1683423487ec6aef36, data reload: false

query1	0.03	0.05	0.02
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.60	0.10	0.11
query5	0.42	0.43	0.42
query6	1.15	0.67	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.49
query10	0.56	0.59	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.62	0.60
query14	2.72	2.72	2.75
query15	0.90	0.83	0.82
query16	0.37	0.40	0.39
query17	1.04	1.06	1.09
query18	0.22	0.22	0.21
query19	1.92	1.91	1.98
query20	0.01	0.02	0.01
query21	15.35	0.90	0.57
query22	0.75	0.80	0.73
query23	15.23	1.40	0.58
query24	2.78	1.47	1.50
query25	0.13	0.15	0.08
query26	0.36	0.14	0.14
query27	0.06	0.06	0.05
query28	14.17	1.53	1.05
query29	12.58	3.97	3.29
query30	0.25	0.09	0.07
query31	2.80	0.61	0.38
query32	3.23	0.55	0.46
query33	3.13	3.14	3.06
query34	16.77	5.09	4.44
query35	4.47	4.45	4.49
query36	0.67	0.50	0.50
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.14	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.98 s
Total hot run time: 31.91 s

qidaye

LGTM

github-actions · 2024-12-31T09:26:50Z

PR approved by at least one committer and no changes requested.

eldenmoon

LGTM

…tion check (apache#45780) Related PR: apache#44973 Problem Summary: Need to check ngram bf index properties in index definition for nereids.

…tion check (#45780) (#46215) pick #45780

…tion check (apache#45780) Related PR: apache#44973 Problem Summary: Need to check ngram bf index properties in index definition for nereids.

…index definition check #45780 (#46216) cherry pick from #45780

morrySnow added dev/2.1.x dev/3.0.x labels Dec 23, 2024

starocean999 previously approved these changes Dec 23, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024

github-actions bot added the reviewed label Dec 23, 2024

airborne12 added 3 commits December 30, 2024 11:38

[fix](bf index) add ngram bf index validation in nereids index defini…

bc8b474

…tion check

fix format

bc453b1

fix

326486b

airborne12 dismissed starocean999’s stale review via 326486b December 30, 2024 09:04

airborne12 force-pushed the fix-nereids-bf-index branch from beb78e5 to 326486b Compare December 30, 2024 09:04

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 30, 2024

qidaye approved these changes Dec 31, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2024

eldenmoon approved these changes Dec 31, 2024

View reviewed changes

starocean999 approved these changes Dec 31, 2024

View reviewed changes

airborne12 merged commit 8781666 into apache:master Dec 31, 2024
27 of 28 checks passed

airborne12 deleted the fix-nereids-bf-index branch December 31, 2024 09:41

github-actions bot added the dev/3.0.x-conflict label Dec 31, 2024

github-actions bot added the dev/2.1.x-conflict label Dec 31, 2024

This was referenced Dec 31, 2024

[fix](bf index) add ngram bf index validation in nereids index definition check (#45780) #46215

Merged

branch-2.1: [fix](bf index) add ngram bf index validation in nereids index definition check #45780 #46216

Merged

airborne12 added a commit that referenced this pull request Jan 1, 2025

[fix](bf index) add ngram bf index validation in nereids index defini…

5ced8ba

…tion check (#45780) (#46215) pick #45780

wm1581066 added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jan 2, 2025

airborne12 added a commit that referenced this pull request Jan 8, 2025

branch-2.1: [fix](bf index) add ngram bf index validation in nereids …

4b3422f

…index definition check #45780 (#46216) cherry pick from #45780

airborne12 added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Jan 8, 2025

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

[fix](bf index) add ngram bf index validation in nereids index definition check #45780

[fix](bf index) add ngram bf index validation in nereids index definition check #45780

Uh oh!

Conversation

airborne12 commented Dec 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 23, 2024

Uh oh!

airborne12 commented Dec 23, 2024

Uh oh!

github-actions bot commented Dec 23, 2024

Uh oh!

github-actions bot commented Dec 23, 2024

Uh oh!

doris-robot commented Dec 23, 2024

Uh oh!

doris-robot commented Dec 23, 2024

Uh oh!

doris-robot commented Dec 23, 2024

Uh oh!

airborne12 commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 31, 2024

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

airborne12 commented Dec 23, 2024 •

edited

Loading