Skip to content

[feature](inverted index) add ordered functionality to match_phrase query #37752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Jul 15, 2024
Merged

[feature](inverted index) add ordered functionality to match_phrase query #37752

airborne12 merged 1 commit into from
Jul 15, 2024

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Jul 14, 2024 •
edited by airborne12
Loading

Proposed changes

  1. select count() from tbl where b match_phrase 'the brown ~2+';

pick from #36356

@doris-robot
Copy link

doris-robot commented Jul 14, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jul 14, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 14, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index/query/phrase_query.h

#include "common/status.h"
// clang-format off
#include <CLucene.h> // IWYU pragma: keep
Copy link
Contributor

@github-actions github-actions bot Jul 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'CLucene.h' file not found [clang-diagnostic-error]

#include <CLucene.h> // IWYU pragma: keep
         ^

@doris-robot
Copy link

doris-robot commented Jul 14, 2024

TeamCity be ut coverage result:
Function Coverage: 37.81% (8115/21464)
Line Coverage: 29.49% (66511/225573)
Region Coverage: 28.95% (34275/118404)
Branch Coverage: 24.82% (17604/70920)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e31439dad728832635fdd3c01bf48b09defe557_4e31439dad728832635fdd3c01bf48b09defe557/report/index.html

@doris-robot
Copy link

doris-robot commented Jul 14, 2024

TPC-H: Total hot run time: 49466 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e31439dad728832635fdd3c01bf48b09defe557, data reload: false

------ Round 1 ----------------------------------
q1	18025	4376	4299	4299
q2	2067	156	149	149
q3	10435	1876	1928	1876
q4	10352	1267	1281	1267
q5	8741	3815	3952	3815
q6	231	148	126	126
q7	1992	1595	1603	1595
q8	9277	2720	2687	2687
q9	10587	10132	10129	10129
q10	8638	3500	3482	3482
q11	431	248	251	248
q12	474	311	307	307
q13	18376	3952	4048	3952
q14	343	328	333	328
q15	505	454	469	454
q16	675	566	569	566
q17	1124	927	952	927
q18	7251	6880	6997	6880
q19	1848	1608	1571	1571
q20	558	303	300	300
q21	4409	4128	4070	4070
q22	523	438	444	438
Total cold run time: 116862 ms
Total hot run time: 49466 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4303	4304	4303
q2	319	229	226	226
q3	4206	4155	4156	4155
q4	2749	2710	2736	2710
q5	7137	6985	7015	6985
q6	243	123	121	121
q7	3217	2837	2857	2837
q8	4313	4498	4443	4443
q9	16785	16700	16683	16683
q10	4217	4241	4269	4241
q11	782	689	709	689
q12	1026	869	868	868
q13	6949	3726	3720	3720
q14	457	448	427	427
q15	502	452	456	452
q16	744	682	683	682
q17	3834	3879	3841	3841
q18	8899	8752	8919	8752
q19	1709	1692	1630	1630
q20	2400	2093	2084	2084
q21	8489	8470	8471	8470
q22	1038	957	951	951
Total cold run time: 84335 ms
Total hot run time: 79270 ms

@doris-robot
Copy link

doris-robot commented Jul 14, 2024

TPC-DS: Total hot run time: 203759 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e31439dad728832635fdd3c01bf48b09defe557, data reload: false

query1	920	424	374	374
query2	6546	3027	2665	2665
query3	6918	210	211	210
query4	21542	18090	18069	18069
query5	19956	6581	6666	6581
query6	290	226	249	226
query7	4165	304	311	304
query8	450	447	445	445
query9	3161	2694	2630	2630
query10	409	300	320	300
query11	11332	10712	10759	10712
query12	129	76	74	74
query13	5601	697	720	697
query14	17902	13105	13726	13105
query15	366	252	260	252
query16	6473	297	263	263
query17	1689	1446	900	900
query18	2303	420	412	412
query19	206	152	149	149
query20	81	81	78	78
query21	196	95	96	95
query22	5179	4985	5041	4985
query23	32697	31893	32040	31893
query24	6914	6567	6522	6522
query25	539	439	429	429
query26	533	173	163	163
query27	1850	304	297	297
query28	6176	2340	2321	2321
query29	2846	2786	2642	2642
query30	243	168	165	165
query31	900	735	748	735
query32	69	62	63	62
query33	409	256	262	256
query34	852	483	480	480
query35	1122	912	931	912
query36	1229	1151	1194	1151
query37	94	61	64	61
query38	3068	2910	2944	2910
query39	1384	1348	1332	1332
query40	205	95	97	95
query41	46	43	43	43
query42	96	84	82	82
query43	734	665	686	665
query44	1130	722	721	721
query45	247	239	236	236
query46	1231	964	983	964
query47	1961	2132	1726	1726
query48	1051	743	730	730
query49	609	387	372	372
query50	863	588	624	588
query51	4741	4654	4682	4654
query52	94	75	82	75
query53	446	327	328	327
query54	2682	2458	2471	2458
query55	93	80	80	80
query56	249	212	218	212
query57	1283	1059	1194	1059
query58	224	205	196	196
query59	4395	4209	4116	4116
query60	212	208	206	206
query61	96	92	93	92
query62	823	462	453	453
query63	495	348	344	344
query64	2506	1555	1524	1524
query65	3635	3527	3528	3527
query66	784	383	393	383
query67	15891	15420	14678	14678
query68	11165	673	646	646
query69	604	350	369	350
query70	2148	1503	1423	1423
query71	421	300	318	300
query72	6618	3454	3490	3454
query73	750	322	322	322
query74	6396	5902	5917	5902
query75	5590	3738	3673	3673
query76	6958	1140	1207	1140
query77	1200	260	266	260
query78	12618	12748	11945	11945
query79	10085	705	660	660
query80	650	408	403	403
query81	459	234	231	231
query82	583	111	99	99
query83	185	136	135	135
query84	261	72	76	72
query85	752	334	342	334
query86	351	323	311	311
query87	3280	2985	3008	2985
query88	4560	2344	2367	2344
query89	445	296	314	296
query90	2120	216	210	210
query91	174	138	155	138
query92	58	53	52	52
query93	5997	596	552	552
query94	808	214	212	212
query95	1128	1071	1042	1042
query96	661	348	338	338
query97	6425	6453	6391	6391
query98	185	173	177	173
query99	2833	903	964	903
Total cold run time: 320029 ms
Total hot run time: 203759 ms

@doris-robot
Copy link

doris-robot commented Jul 14, 2024

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e31439dad728832635fdd3c01bf48b09defe557, data reload: false

query1	0.02	0.02	0.01
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.79	0.07	0.07
query5	0.53	0.52	0.52
query6	1.23	0.63	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.51	0.50	0.48
query10	0.55	0.55	0.54
query11	0.13	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.60
query14	0.78	0.81	0.78
query15	0.77	0.76	0.75
query16	0.37	0.38	0.36
query17	1.01	0.96	0.98
query18	0.24	0.24	0.24
query19	1.87	1.80	1.84
query20	0.01	0.01	0.01
query21	15.50	0.56	0.56
query22	1.98	2.11	1.60
query23	17.27	1.02	0.85
query24	4.65	3.12	1.47
query25	0.39	0.10	0.06
query26	0.71	0.17	0.16
query27	0.04	0.04	0.04
query28	5.90	0.77	0.77
query29	12.62	2.32	2.09
query30	0.55	0.50	0.53
query31	2.80	0.39	0.39
query32	3.36	0.50	0.49
query33	3.08	3.09	3.11
query34	15.27	4.77	4.77
query35	4.83	4.85	4.86
query36	1.05	1.01	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 101.28 s
Total hot run time: 30.96 s

@doris-robot
Copy link

doris-robot commented Jul 14, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4e31439dad728832635fdd3c01bf48b09defe557 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@airborne12 airborne12 changed the title [feture](inverted index) add ordered functionality to match_phrase query [feature](inverted index) add ordered functionality to match_phrase query Jul 15, 2024
airborne12
airborne12 approved these changes Jul 15, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 15, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 15, 2024

PR approved by anyone and no changes requested.

@airborne12 airborne12 merged commit 8d32c8f into apache:branch-2.0 Jul 15, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zzzxl1993
[feature](inverted index) add ordered functionality to match_phrase q…
08e6577 
…uery (apache#37752)

## Proposed changes

1. select count() from tbl where b match_phrase 'the brown ~2+';

pick from apache#36356
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@airborne12 airborne12 airborne12 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zzzxl1993 @doris-robot @airborne12 @xiaokang