Skip to content

branch-3.0: [fix](ES Catalog)Only like on keyword can be applied to wildcard query #43399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 2 commits into from
Nov 8, 2024
Merged

branch-3.0: [fix](ES Catalog)Only like on keyword can be applied to wildcard query #43399

airborne12 merged 2 commits into from
Nov 8, 2024

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Nov 7, 2024

Cherry-picked from #41176

@qidaye
[fix](ES Catalog)Only like on keyword can be applied to wildcard query (
dd635d0 
#41176)

## Proposed changes

We map `text` and `keyword` both to `string` type in Doris. When enable
`like_push_down`, we translate like to wildcard query in ES, which will
lead unexpected result in `text` field. We should stick to `keyword`
with wildcard query.
1. Add `column2typeMap` in `EsTable` to save the mapping of column_name
to ES field data type.
2. Add new class `EsSchemaCacheValue` to get schema and column to type
map
3. Init `column2typeMap` when cache init and build query process of ES
external table
4. Support LIKE functionCallExpr for Nereids planner.
5. Add end to end like predicate test cases and UTs
@doris-robot
Copy link

doris-robot commented Nov 7, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

doris-robot commented Nov 7, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 7, 2024

TPC-H: Total hot run time: 40709 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd635d0578537e6088b461bdfcac16166541757b, data reload: false

------ Round 1 ----------------------------------
q1	17592	7414	7335	7335
q2	2060	165	178	165
q3	10656	1084	1113	1084
q4	10552	769	780	769
q5	7764	2782	2760	2760
q6	234	146	146	146
q7	966	624	614	614
q8	9386	1944	1980	1944
q9	6551	6357	6392	6357
q10	6943	2275	2264	2264
q11	458	258	264	258
q12	412	212	204	204
q13	17801	2971	2977	2971
q14	237	211	224	211
q15	562	526	512	512
q16	675	607	601	601
q17	983	580	581	580
q18	7294	6803	6648	6648
q19	1374	1074	1050	1050
q20	486	195	193	193
q21	3875	3095	3072	3072
q22	1076	1011	971	971
Total cold run time: 107937 ms
Total hot run time: 40709 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7273	7200	7192	7192
q2	426	297	303	297
q3	2853	2820	2842	2820
q4	2067	1846	1834	1834
q5	5661	5679	5700	5679
q6	224	141	139	139
q7	2263	1745	1760	1745
q8	3338	3588	3497	3497
q9	8722	8830	8816	8816
q10	3517	3514	3488	3488
q11	603	490	501	490
q12	813	596	599	596
q13	16593	3139	3142	3139
q14	308	280	265	265
q15	603	559	559	559
q16	728	687	664	664
q17	1841	1595	1568	1568
q18	8275	7673	7447	7447
q19	2438	1675	1649	1649
q20	2052	1836	1824	1824
q21	5407	5285	5268	5268
q22	1109	1025	1009	1009
Total cold run time: 77114 ms
Total hot run time: 59985 ms

@doris-robot
Copy link

doris-robot commented Nov 7, 2024

TPC-DS: Total hot run time: 192849 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd635d0578537e6088b461bdfcac16166541757b, data reload: false

query1	2441	2244	2419	2244
query2	6258	2139	2122	2122
query3	14028	10027	235	235
query4	33242	23322	23514	23322
query5	3754	422	420	420
query6	267	191	183	183
query7	3984	305	308	305
query8	316	228	243	228
query9	9622	2712	2696	2696
query10	467	263	277	263
query11	17859	15104	15154	15104
query12	149	103	98	98
query13	1544	413	397	397
query14	8882	6912	6862	6862
query15	236	182	183	182
query16	7588	510	489	489
query17	1567	596	598	596
query18	2180	620	660	620
query19	256	195	192	192
query20	131	117	113	113
query21	210	105	105	105
query22	4716	4249	4362	4249
query23	34446	34721	33630	33630
query24	12749	3356	3341	3341
query25	528	410	401	401
query26	1162	193	180	180
query27	2412	290	307	290
query28	7605	2487	2475	2475
query29	679	442	436	436
query30	399	305	312	305
query31	1032	802	787	787
query32	90	54	54	54
query33	776	293	284	284
query34	915	509	498	498
query35	857	727	735	727
query36	1073	919	924	919
query37	195	73	75	73
query38	3900	3863	3914	3863
query39	1498	1416	1412	1412
query40	255	100	100	100
query41	53	50	49	49
query42	111	97	96	96
query43	532	495	489	489
query44	1260	792	783	783
query45	180	164	166	164
query46	1133	721	736	721
query47	1903	1864	1842	1842
query48	463	371	389	371
query49	1059	391	393	391
query50	812	399	398	398
query51	7028	7126	7060	7060
query52	99	89	90	89
query53	255	184	188	184
query54	1250	439	440	439
query55	73	73	77	73
query56	249	232	226	226
query57	1181	1127	1070	1070
query58	225	199	201	199
query59	3214	2880	2978	2880
query60	272	249	251	249
query61	109	131	124	124
query62	866	642	663	642
query63	217	187	192	187
query64	5312	638	607	607
query65	3247	3162	3138	3138
query66	1277	306	298	298
query67	15866	15325	15370	15325
query68	4629	572	594	572
query69	429	252	248	248
query70	1158	1094	1151	1094
query71	385	262	246	246
query72	6262	3931	3908	3908
query73	768	346	339	339
query74	10138	8843	8896	8843
query75	3359	2607	2607	2607
query76	2668	910	962	910
query77	369	259	258	258
query78	10549	9642	9542	9542
query79	7127	596	587	587
query80	2060	443	424	424
query81	570	240	258	240
query82	1723	116	113	113
query83	287	154	154	154
query84	295	81	83	81
query85	1732	306	297	297
query86	473	294	290	290
query87	4377	4225	4280	4225
query88	5472	2359	2383	2359
query89	552	291	281	281
query90	2053	188	179	179
query91	184	147	142	142
query92	65	49	47	47
query93	6588	545	540	540
query94	940	287	281	281
query95	348	247	252	247
query96	630	282	286	282
query97	3408	3140	3152	3140
query98	219	194	196	194
query99	1849	1290	1285	1285
Total cold run time: 320180 ms
Total hot run time: 192849 ms

@qidaye qidaye closed this Nov 7, 2024
@qidaye qidaye reopened this Nov 7, 2024
airborne12
airborne12 approved these changes Nov 7, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@airborne12
Copy link
Member

airborne12 commented Nov 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 8, 2024

TPC-H: Total hot run time: 41473 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f133feb14c0f61a7b6ceb5ae0927df9ae28e00a5, data reload: false

------ Round 1 ----------------------------------
q1	17757	7482	7362	7362
q2	2084	179	170	170
q3	11007	1131	1138	1131
q4	10555	773	807	773
q5	7774	2815	2835	2815
q6	236	148	147	147
q7	989	632	615	615
q8	10236	1995	2038	1995
q9	6656	6422	6482	6422
q10	7003	2381	2335	2335
q11	458	266	265	265
q12	405	211	215	211
q13	17787	3024	3137	3024
q14	255	232	248	232
q15	575	521	501	501
q16	674	619	613	613
q17	1017	559	611	559
q18	8896	7034	7067	7034
q19	7115	1017	1077	1017
q20	466	197	195	195
q21	4292	3080	3141	3080
q22	1085	1011	977	977
Total cold run time: 117322 ms
Total hot run time: 41473 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7556	7274	7444	7274
q2	450	328	327	327
q3	3063	2928	2893	2893
q4	1958	1786	1732	1732
q5	5685	5748	5699	5699
q6	221	140	142	140
q7	2153	1823	1734	1734
q8	3281	3552	3538	3538
q9	8828	8807	8823	8807
q10	3560	3521	3479	3479
q11	593	500	527	500
q12	816	573	594	573
q13	16722	3154	3221	3154
q14	303	295	286	286
q15	594	544	534	534
q16	694	674	648	648
q17	1891	1632	1625	1625
q18	8136	7804	7794	7794
q19	1703	1521	1682	1521
q20	2089	1872	1899	1872
q21	8091	5326	5360	5326
q22	1135	1024	975	975
Total cold run time: 79522 ms
Total hot run time: 60431 ms

@doris-robot
Copy link

doris-robot commented Nov 8, 2024

TPC-DS: Total hot run time: 192554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f133feb14c0f61a7b6ceb5ae0927df9ae28e00a5, data reload: false

query1	1176	775	735	735
query2	6238	2095	2101	2095
query3	14177	10063	235	235
query4	33341	23737	23436	23436
query5	4747	449	430	430
query6	280	187	185	185
query7	4000	316	324	316
query8	300	233	235	233
query9	9460	2738	2733	2733
query10	499	275	264	264
query11	17875	15181	15148	15148
query12	154	100	101	100
query13	1542	422	425	422
query14	9126	7204	7591	7204
query15	237	189	172	172
query16	7738	495	492	492
query17	1567	562	572	562
query18	1714	606	620	606
query19	248	188	173	173
query20	123	111	120	111
query21	206	103	101	101
query22	4521	4047	4301	4047
query23	34642	34243	34110	34110
query24	9289	3342	3378	3342
query25	572	373	390	373
query26	733	179	177	177
query27	2000	293	312	293
query28	6309	2522	2496	2496
query29	673	419	414	414
query30	395	298	307	298
query31	1010	795	796	795
query32	91	51	53	51
query33	583	271	263	263
query34	888	492	507	492
query35	859	710	715	710
query36	1053	955	932	932
query37	109	73	68	68
query38	3991	3865	3900	3865
query39	1482	1416	1442	1416
query40	202	103	100	100
query41	53	50	51	50
query42	106	97	99	97
query43	547	503	499	499
query44	1154	794	789	789
query45	186	165	169	165
query46	1157	727	716	716
query47	1941	1810	1853	1810
query48	463	363	377	363
query49	906	397	408	397
query50	819	415	414	414
query51	7177	7152	7067	7067
query52	106	89	95	89
query53	254	187	184	184
query54	658	460	458	458
query55	79	83	75	75
query56	261	254	247	247
query57	1203	1099	1120	1099
query58	229	205	220	205
query59	3180	2928	3015	2928
query60	284	269	266	266
query61	131	126	128	126
query62	831	671	657	657
query63	216	193	184	184
query64	3765	729	693	693
query65	3262	3187	3202	3187
query66	816	329	319	319
query67	15819	15569	15377	15377
query68	4543	582	579	579
query69	428	268	276	268
query70	1150	1138	1125	1125
query71	365	253	261	253
query72	6434	3951	3914	3914
query73	757	338	339	338
query74	10184	8848	8908	8848
query75	3313	2649	2608	2608
query76	2366	1060	1079	1060
query77	385	271	262	262
query78	10733	9947	9449	9449
query79	8785	588	601	588
query80	2671	424	416	416
query81	578	244	240	240
query82	1701	113	114	113
query83	282	165	158	158
query84	287	86	80	80
query85	2199	294	282	282
query86	490	296	274	274
query87	4518	4221	4314	4221
query88	5792	2418	2416	2416
query89	579	288	296	288
query90	2144	180	179	179
query91	176	141	142	141
query92	65	49	51	49
query93	7630	540	535	535
query94	990	274	281	274
query95	347	247	251	247
query96	619	298	287	287
query97	3325	3138	3118	3118
query98	215	199	193	193
query99	1764	1336	1295	1295
Total cold run time: 315568 ms
Total hot run time: 192554 ms

@airborne12 airborne12 merged commit be7d43e into branch-3.0 Nov 8, 2024
@airborne12 airborne12 deleted the auto-pick-41176-branch-3.0 branch November 8, 2024 04:18
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@doris-robot @airborne12 @qidaye