[FIX](array_index)fix array tokenized index parser and array_contains bugs #34683

amorynan · 2024-05-11T01:42:42Z

Proposed changes

fix array tokenized inverted index write with char_string_reader used by _analyzer
fix array_contains semantics as match_equals and fix nested null value match
Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-11T01:42:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

amorynan · 2024-05-11T01:42:50Z

run buildall

airborne12 · 2024-05-11T01:47:45Z

be/src/vec/functions/array/function_array_index.h

 struct ParamValue {
    PrimitiveType type;
    Field value;
+    [[maybe_unused]] bool is_evaled_inverted_idx = false;


where should it be used ?

it will be in array_match_any

doris-robot · 2024-05-11T02:12:06Z

TeamCity be ut coverage result:
Function Coverage: 35.68% (8981/25174)
Line Coverage: 27.34% (74261/271578)
Region Coverage: 26.58% (38389/144430)
Branch Coverage: 23.40% (19580/83664)
Coverage Report: http://coverage.selectdb-in.cc/coverage/34aee14505ced566731872b03b7f6e89ce66c6f5_34aee14505ced566731872b03b7f6e89ce66c6f5/report/index.html

doris-robot · 2024-05-11T02:20:49Z

TPC-H: Total hot run time: 41787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34aee14505ced566731872b03b7f6e89ce66c6f5, data reload: false

------ Round 1 ----------------------------------
q1	17597	4264	4243	4243
q2	2015	177	188	177
q3	10498	1189	1185	1185
q4	10200	831	845	831
q5	7473	2725	2741	2725
q6	227	129	131	129
q7	1006	624	610	610
q8	9233	2145	2126	2126
q9	9337	6730	6727	6727
q10	9185	3851	3830	3830
q11	435	247	246	246
q12	461	226	230	226
q13	17489	3230	3133	3133
q14	250	209	209	209
q15	510	471	466	466
q16	512	399	398	398
q17	971	682	748	682
q18	8276	7710	7820	7710
q19	2231	1562	1537	1537
q20	663	324	319	319
q21	5338	4368	3990	3990
q22	346	296	288	288
Total cold run time: 114253 ms
Total hot run time: 41787 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4593	4377	4406	4377
q2	374	285	268	268
q3	3100	2924	2782	2782
q4	1850	1597	1659	1597
q5	5464	5480	5481	5480
q6	209	121	122	121
q7	2333	1962	1922	1922
q8	3237	3356	3413	3356
q9	8610	8722	8651	8651
q10	3910	3815	3810	3810
q11	581	505	491	491
q12	794	621	599	599
q13	16041	3097	3134	3097
q14	296	279	274	274
q15	528	475	507	475
q16	480	428	409	409
q17	1751	1477	1452	1452
q18	7823	7563	7480	7480
q19	1666	1610	1495	1495
q20	1955	1784	1768	1768
q21	9247	4858	4712	4712
q22	591	511	506	506
Total cold run time: 75433 ms
Total hot run time: 55122 ms

xiaokang

LGTM

doris-robot · 2024-05-11T02:32:28Z

TPC-DS: Total hot run time: 186464 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 34aee14505ced566731872b03b7f6e89ce66c6f5, data reload: false

query1	929	354	353	353
query2	6639	2261	2280	2261
query3	6651	219	219	219
query4	24199	21499	21265	21265
query5	4103	409	417	409
query6	266	190	174	174
query7	4590	293	287	287
query8	251	206	181	181
query9	8677	2388	2365	2365
query10	436	256	239	239
query11	14813	14205	14384	14205
query12	134	93	89	89
query13	1645	367	365	365
query14	10019	6911	8185	6911
query15	217	173	171	171
query16	7144	287	279	279
query17	981	568	569	568
query18	1679	282	283	282
query19	205	157	160	157
query20	90	84	85	84
query21	198	132	132	132
query22	5002	4809	4793	4793
query23	33791	33642	33412	33412
query24	5118	2915	3076	2915
query25	482	384	378	378
query26	684	159	154	154
query27	1834	320	325	320
query28	3785	2064	2051	2051
query29	861	633	623	623
query30	211	157	155	155
query31	929	768	736	736
query32	60	53	57	53
query33	399	256	257	256
query34	861	490	481	481
query35	763	673	686	673
query36	1051	890	926	890
query37	106	71	75	71
query38	2852	2763	2728	2728
query39	1610	1549	1554	1549
query40	200	129	125	125
query41	43	41	40	40
query42	106	95	102	95
query43	574	503	546	503
query44	1055	725	726	725
query45	267	248	259	248
query46	1065	732	719	719
query47	1953	1900	1888	1888
query48	380	297	303	297
query49	790	475	383	383
query50	769	397	387	387
query51	6911	6780	6800	6780
query52	106	87	93	87
query53	351	288	279	279
query54	529	420	431	420
query55	80	78	76	76
query56	237	221	224	221
query57	1239	1133	1141	1133
query58	208	202	197	197
query59	3447	2938	3383	2938
query60	253	241	242	241
query61	94	87	92	87
query62	602	465	482	465
query63	302	280	281	280
query64	7633	7380	7340	7340
query65	3129	3071	3067	3067
query66	657	338	343	338
query67	15715	15021	14919	14919
query68	9029	539	535	535
query69	575	305	314	305
query70	1248	1110	1134	1110
query71	562	275	265	265
query72	8222	2503	2362	2362
query73	1426	322	323	322
query74	6730	6531	6528	6528
query75	4815	2597	2613	2597
query76	4834	879	988	879
query77	748	264	261	261
query78	10719	10238	10172	10172
query79	7069	515	512	512
query80	1100	434	432	432
query81	454	222	224	222
query82	643	96	93	93
query83	216	166	168	166
query84	270	82	85	82
query85	817	313	272	272
query86	373	306	308	306
query87	3344	3097	3106	3097
query88	4297	2401	2393	2393
query89	523	385	391	385
query90	2166	189	181	181
query91	123	97	95	95
query92	61	52	49	49
query93	5915	504	492	492
query94	929	180	181	180
query95	386	311	295	295
query96	597	266	269	266
query97	3107	2980	2969	2969
query98	230	229	230	229
query99	1199	903	900	900
Total cold run time: 285687 ms
Total hot run time: 186464 ms

github-actions · 2024-05-11T02:32:40Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-11T02:32:42Z

PR approved by anyone and no changes requested.

amorynan · 2024-05-12T07:38:12Z

run buildall

doris-robot · 2024-05-12T07:58:07Z

TPC-H: Total hot run time: 41945 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de4dade5a24c6a593f48b7e2640ba3720cc114bb, data reload: false

------ Round 1 ----------------------------------
q1	17650	4287	4235	4235
q2	2022	185	185	185
q3	10516	1282	1251	1251
q4	10206	849	827	827
q5	7484	2771	2715	2715
q6	222	132	138	132
q7	1062	617	585	585
q8	9237	2185	2103	2103
q9	9807	6739	6767	6739
q10	9457	4123	3851	3851
q11	435	240	250	240
q12	440	227	242	227
q13	17229	3221	3255	3221
q14	288	223	211	211
q15	523	476	482	476
q16	499	393	388	388
q17	980	628	688	628
q18	8419	7806	7765	7765
q19	5892	1579	1506	1506
q20	642	331	320	320
q21	5295	4062	4148	4062
q22	351	284	278	278
Total cold run time: 118656 ms
Total hot run time: 41945 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4535	4369	4857	4369
q2	378	261	283	261
q3	3143	2925	2873	2873
q4	1962	1673	1655	1655
q5	5483	5513	5508	5508
q6	217	129	122	122
q7	2366	1993	1954	1954
q8	3224	3389	3415	3389
q9	8672	8720	8724	8720
q10	3963	3748	3825	3748
q11	611	487	497	487
q12	803	623	655	623
q13	16997	3196	3155	3155
q14	303	262	277	262
q15	524	492	469	469
q16	506	443	441	441
q17	1792	1516	1477	1477
q18	7632	7470	7384	7384
q19	1664	1555	1564	1555
q20	1937	1783	1750	1750
q21	8314	4810	4919	4810
q22	563	491	478	478
Total cold run time: 75589 ms
Total hot run time: 55490 ms

doris-robot · 2024-05-12T08:09:36Z

TPC-DS: Total hot run time: 186452 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de4dade5a24c6a593f48b7e2640ba3720cc114bb, data reload: false

query1	904	377	343	343
query2	6462	2508	2424	2424
query3	6644	204	208	204
query4	24066	21325	21212	21212
query5	4109	427	428	427
query6	266	181	195	181
query7	4597	290	291	290
query8	245	202	197	197
query9	8765	2432	2438	2432
query10	427	255	263	255
query11	15002	14237	14203	14203
query12	133	90	90	90
query13	1646	385	379	379
query14	9679	8036	6793	6793
query15	265	176	171	171
query16	8052	259	255	255
query17	1801	544	542	542
query18	2093	278	271	271
query19	209	142	154	142
query20	92	86	83	83
query21	193	127	131	127
query22	5085	4818	4814	4814
query23	34112	33691	33515	33515
query24	9789	2813	2869	2813
query25	562	361	369	361
query26	695	165	157	157
query27	2149	326	329	326
query28	5757	2080	2063	2063
query29	864	598	590	590
query30	224	150	151	150
query31	983	759	743	743
query32	89	53	56	53
query33	640	261	250	250
query34	910	476	482	476
query35	786	703	674	674
query36	1133	893	908	893
query37	102	68	73	68
query38	2894	2762	2754	2754
query39	1633	1561	1547	1547
query40	197	131	125	125
query41	41	39	39	39
query42	103	96	99	96
query43	570	547	579	547
query44	1116	724	746	724
query45	271	259	254	254
query46	1088	720	706	706
query47	1939	1894	1877	1877
query48	373	306	302	302
query49	858	398	406	398
query50	829	399	402	399
query51	6686	6715	6656	6656
query52	100	91	95	91
query53	352	280	282	280
query54	841	436	431	431
query55	75	71	73	71
query56	249	223	227	223
query57	1226	1124	1148	1124
query58	233	222	204	204
query59	3336	3426	3265	3265
query60	255	235	272	235
query61	89	90	84	84
query62	642	492	463	463
query63	305	281	276	276
query64	8417	7405	7339	7339
query65	3176	3091	3100	3091
query66	801	339	336	336
query67	15483	14970	15218	14970
query68	4691	545	540	540
query69	530	304	309	304
query70	1215	1173	1166	1166
query71	399	274	278	274
query72	7988	2565	2365	2365
query73	720	326	329	326
query74	6541	6129	6133	6129
query75	3552	2621	2580	2580
query76	3115	997	988	988
query77	606	264	268	264
query78	10509	10184	9999	9999
query79	1758	514	516	514
query80	830	435	433	433
query81	521	218	219	218
query82	734	93	96	93
query83	236	163	167	163
query84	249	87	83	83
query85	1416	271	309	271
query86	456	302	325	302
query87	3324	3120	3068	3068
query88	4125	2452	2466	2452
query89	473	380	383	380
query90	1965	192	187	187
query91	126	96	100	96
query92	62	53	51	51
query93	2013	503	495	495
query94	1221	179	190	179
query95	395	304	304	304
query96	582	282	271	271
query97	3128	2974	2992	2974
query98	248	221	213	213
query99	1219	898	896	896
Total cold run time: 278655 ms
Total hot run time: 186452 ms

doris-robot · 2024-05-12T09:15:57Z

TeamCity be ut coverage result:
Function Coverage: 35.65% (8983/25201)
Line Coverage: 27.31% (74258/271867)
Region Coverage: 26.55% (38381/144577)
Branch Coverage: 23.36% (19569/83768)
Coverage Report: http://coverage.selectdb-in.cc/coverage/de4dade5a24c6a593f48b7e2640ba3720cc114bb_de4dade5a24c6a593f48b7e2640ba3720cc114bb/report/index.html

xiaokang

LGTM

github-actions · 2024-05-13T00:53:20Z

PR approved by at least one committer and no changes requested.

airborne12

LGTM

… bugs (apache#34683) * fix array tokenized index parser and array_contains bugs * delete useless param

here with some array with inverted index bugfix: see also: #34766 #35086 #34683 #34076

…#35926) ## Proposed changes see also for branch-2.1: #35837 and master: #34766 #35086 #34683 #34076

…pache#34683 apache#34076 (apache#35926) ## Proposed changes see also for branch-2.1: apache#35837 and master: apache#34766 apache#35086 apache#34683 apache#34076

fix array tokenized index parser and array_contains bugs

34aee14

airborne12 reviewed May 11, 2024

View reviewed changes

xiaokang previously approved these changes May 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 11, 2024

github-actions bot added the reviewed label May 11, 2024

delete useless param

de4dade

amorynan dismissed xiaokang’s stale review via de4dade May 12, 2024 07:36

github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 12, 2024

xiaokang approved these changes May 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 13, 2024

airborne12 approved these changes May 13, 2024

View reviewed changes

eldenmoon merged commit e8288f1 into apache:master May 13, 2024

ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024

[FIX](array_index)fix array tokenized index parser and array_contains…

ede90e7

… bugs (apache#34683) * fix array tokenized index parser and array_contains bugs * delete useless param

yiguolei added the 2.1.0-conflict label May 18, 2024

This was referenced Jun 4, 2024

[FIX] Pick array inverted index bugfix #35837

Merged

[FIX](array)fix array with inverted index #35926

Merged

xiaokang pushed a commit that referenced this pull request Jun 6, 2024

[FIX] Pick array inverted index bugfix (#35837)

b5a35b9

here with some array with inverted index bugfix: see also: #34766 #35086 #34683 #34076

xiaokang removed the 2.1.0-conflict label Jun 12, 2024

xiaokang added the dev/2.1.4-merged label Jun 12, 2024

xiaokang pushed a commit that referenced this pull request Jun 12, 2024

[FIX](array)fix array with inverted index #35837 #34766 #34683 #34076 (…

0bd451d

…#35926) ## Proposed changes see also for branch-2.1: #35837 and master: #34766 #35086 #34683 #34076

xiaokang added the dev/2.0.12-merged label Jun 12, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

[FIX](array_index)fix array tokenized index parser and array_contains bugs #34683

[FIX](array_index)fix array tokenized index parser and array_contains bugs #34683

Uh oh!

Conversation

amorynan commented May 11, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented May 11, 2024

Uh oh!

amorynan commented May 11, 2024

Uh oh!

airborne12 May 11, 2024

Choose a reason for hiding this comment

Uh oh!

amorynan May 11, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented May 11, 2024

Uh oh!

doris-robot commented May 11, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented May 11, 2024

Uh oh!

github-actions bot commented May 11, 2024

Uh oh!

github-actions bot commented May 11, 2024

Uh oh!

amorynan commented May 12, 2024

Uh oh!

doris-robot commented May 12, 2024

Uh oh!

doris-robot commented May 12, 2024

Uh oh!

doris-robot commented May 12, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 13, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants