Skip to content

[Fix](inverted index) fix wrong no need read data when same column in inverted index and like function #36687 #38651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Aug 1, 2024
Merged

[Fix](inverted index) fix wrong no need read data when same column in inverted index and like function #36687 #38651

airborne12 merged 1 commit into from
Aug 1, 2024

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Aug 1, 2024

cherry pick from #36687

@airborne12
[Fix](inverted index) fix wrong no need read data when same column in…
4aa5743 
… inverted index and like function (apache#36687)

## Proposed changes

SQL like SELECT COUNT() FROM table WHERE a >= '2024-06-15 00:00:00' AND
b LIKE '%xxx%' AND b MATCH 'xxx' for inverted index tables may cause
incorrect optimization in the segment iterator, leading to the
no-read-data optimization being applied incorrectly.
@airborne12
Copy link
Member Author

airborne12 commented Aug 1, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 1, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 1, 2024

TPC-H: Total hot run time: 49726 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4aa57436b1d7f872bdf1d0f057225ac504c8a048, data reload: false

------ Round 1 ----------------------------------
q1	17849	4419	4355	4355
q2	2040	159	147	147
q3	10418	1912	1956	1912
q4	10304	1236	1293	1236
q5	8435	3908	3903	3903
q6	239	124	125	124
q7	2021	1612	1581	1581
q8	9321	2696	2687	2687
q9	10699	10261	10203	10203
q10	8594	3509	3509	3509
q11	429	249	239	239
q12	472	301	312	301
q13	18294	3980	4019	3980
q14	360	328	323	323
q15	508	462	465	462
q16	673	568	571	568
q17	1120	931	952	931
q18	7402	6907	6856	6856
q19	1735	1693	1610	1610
q20	557	308	308	308
q21	4404	4054	4046	4046
q22	548	449	445	445
Total cold run time: 116422 ms
Total hot run time: 49726 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4355	4289	4319	4289
q2	315	224	214	214
q3	4181	4141	4154	4141
q4	2753	2723	2742	2723
q5	7137	7109	7076	7076
q6	240	118	118	118
q7	3223	2824	2859	2824
q8	4394	4470	4514	4470
q9	16917	16800	16709	16709
q10	4241	4236	4245	4236
q11	752	688	718	688
q12	1022	857	868	857
q13	6768	3724	3735	3724
q14	452	421	427	421
q15	501	466	460	460
q16	745	680	675	675
q17	3865	3921	3852	3852
q18	8726	8651	8799	8651
q19	1744	1689	1689	1689
q20	2383	2089	2149	2089
q21	8528	8464	8398	8398
q22	1089	984	959	959
Total cold run time: 84331 ms
Total hot run time: 79263 ms

@doris-robot
Copy link

doris-robot commented Aug 1, 2024

TeamCity be ut coverage result:
Function Coverage: 37.74% (8102/21466)
Line Coverage: 29.38% (66346/225803)
Region Coverage: 28.89% (34233/118477)
Branch Coverage: 24.79% (17596/70982)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4aa57436b1d7f872bdf1d0f057225ac504c8a048_4aa57436b1d7f872bdf1d0f057225ac504c8a048/report/index.html

@doris-robot
Copy link

doris-robot commented Aug 1, 2024

TPC-DS: Total hot run time: 203701 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4aa57436b1d7f872bdf1d0f057225ac504c8a048, data reload: false

query1	926	424	382	382
query2	6530	2813	2633	2633
query3	6939	209	204	204
query4	20504	17924	17893	17893
query5	19746	6525	6522	6522
query6	292	225	236	225
query7	4156	307	322	307
query8	441	405	415	405
query9	3158	2694	2628	2628
query10	401	302	297	297
query11	11197	10666	10718	10666
query12	119	82	73	73
query13	5598	696	668	668
query14	17773	13568	13405	13405
query15	365	251	254	251
query16	6484	281	267	267
query17	1741	1465	891	891
query18	2305	420	417	417
query19	206	149	148	148
query20	80	80	80	80
query21	192	95	95	95
query22	5190	5011	5011	5011
query23	32551	31881	31699	31699
query24	7006	6551	6484	6484
query25	550	421	430	421
query26	531	169	164	164
query27	1854	307	302	302
query28	6115	2362	2321	2321
query29	2983	2777	2703	2703
query30	243	169	167	167
query31	904	733	737	733
query32	72	61	59	59
query33	396	251	262	251
query34	849	490	479	479
query35	1112	929	925	925
query36	1205	1321	1108	1108
query37	88	58	60	58
query38	3074	2893	2900	2893
query39	1376	1333	1308	1308
query40	204	97	95	95
query41	48	43	44	43
query42	83	89	88	88
query43	701	657	661	657
query44	1133	725	722	722
query45	252	237	236	236
query46	1223	976	973	973
query47	1888	1900	1641	1641
query48	1005	705	716	705
query49	604	375	370	370
query50	866	612	639	612
query51	4796	4687	4686	4686
query52	94	83	82	82
query53	453	336	320	320
query54	2652	2477	2495	2477
query55	97	85	91	85
query56	253	209	214	209
query57	1110	1055	995	995
query58	208	207	214	207
query59	4346	3809	4196	3809
query60	225	236	228	228
query61	97	95	98	95
query62	747	544	499	499
query63	481	352	349	349
query64	2618	1548	1413	1413
query65	3625	3553	3541	3541
query66	812	397	388	388
query67	15845	17190	15927	15927
query68	6357	623	651	623
query69	561	356	368	356
query70	1556	1378	1386	1378
query71	403	317	344	317
query72	6537	3473	3481	3473
query73	750	319	309	309
query74	6186	5937	5872	5872
query75	4617	3778	3798	3778
query76	4268	1105	1135	1105
query77	545	267	260	260
query78	12684	11498	11561	11498
query79	8320	661	680	661
query80	3116	414	408	408
query81	529	239	231	231
query82	1635	98	95	95
query83	200	140	133	133
query84	262	72	71	71
query85	1418	334	344	334
query86	373	295	291	291
query87	3248	3023	3005	3005
query88	5317	2330	2335	2330
query89	395	286	285	285
query90	1742	209	218	209
query91	183	151	145	145
query92	61	54	54	54
query93	5186	573	579	573
query94	891	200	208	200
query95	1102	1072	1056	1056
query96	640	321	327	321
query97	6425	6269	6404	6269
query98	177	182	173	173
query99	2924	954	893	893
Total cold run time: 310326 ms
Total hot run time: 203701 ms

@doris-robot
Copy link

doris-robot commented Aug 1, 2024

ClickBench: Total hot run time: 30.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4aa57436b1d7f872bdf1d0f057225ac504c8a048, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.06
query5	0.53	0.52	0.52
query6	1.23	0.61	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.47
query10	0.52	0.53	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.60	0.62
query14	0.77	0.79	0.80
query15	0.78	0.76	0.76
query16	0.35	0.37	0.38
query17	1.02	1.02	1.01
query18	0.23	0.25	0.23
query19	1.93	1.86	1.86
query20	0.02	0.01	0.01
query21	15.55	0.54	0.55
query22	2.17	3.29	1.55
query23	17.29	0.93	0.94
query24	8.36	1.04	0.74
query25	0.40	0.12	0.05
query26	0.82	0.15	0.15
query27	0.05	0.03	0.05
query28	5.06	0.75	0.77
query29	12.63	2.26	2.22
query30	0.60	0.49	0.52
query31	2.81	0.40	0.37
query32	3.36	0.49	0.50
query33	3.08	3.09	3.08
query34	15.27	4.80	4.81
query35	4.84	4.86	4.82
query36	1.05	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.03
query39	0.03	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.69 s
Total hot run time: 30.37 s

@doris-robot
Copy link

doris-robot commented Aug 1, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4aa57436b1d7f872bdf1d0f057225ac504c8a048 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@airborne12 airborne12 merged commit 22665e2 into apache:branch-2.0 Aug 1, 2024
@airborne12 airborne12 deleted the pick_36687_to_origin_branch-2.0 branch August 1, 2024 04:18
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@airborne12
[Fix](inverted index) fix wrong no need read data when same column in…
19d34b5 
… inverted index and like function apache#36687 (apache#38651)

cherry pick from apache#36687
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@airborne12 @doris-robot