Skip to content

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Apr 26, 2024

Proposed changes

In SQL syntax, col != '' equals col.length() > 0.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

In SQL syntax, `col != ''` equals `col.length() > 0`.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Apr 26, 2024

run buildall

morningman
morningman previously approved these changes Apr 26, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

morrySnow
morrySnow previously approved these changes Apr 26, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 26, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40774 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85f24324b0cb0594e72ea0668e705d170093c702, data reload: false

------ Round 1 ----------------------------------
q1	17600	4365	4255	4255
q2	2007	192	194	192
q3	10653	1242	1215	1215
q4	10661	848	733	733
q5	7800	2763	2835	2763
q6	220	135	137	135
q7	1057	612	597	597
q8	9826	2101	2082	2082
q9	10028	7360	7257	7257
q10	8718	3693	3730	3693
q11	446	233	234	233
q12	403	230	214	214
q13	17796	2921	2966	2921
q14	268	224	223	223
q15	520	501	473	473
q16	522	380	374	374
q17	959	682	651	651
q18	8118	7387	7272	7272
q19	1622	1540	1530	1530
q20	645	314	303	303
q21	5026	3392	4061	3392
q22	339	266	268	266
Total cold run time: 115234 ms
Total hot run time: 40774 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4251	4211	4294	4211
q2	366	269	261	261
q3	2930	2749	2725	2725
q4	1862	1606	1553	1553
q5	5294	5274	5315	5274
q6	208	121	125	121
q7	2226	1871	1847	1847
q8	3203	3350	3354	3350
q9	9239	9289	9244	9244
q10	3883	3648	3668	3648
q11	584	486	483	483
q12	786	620	579	579
q13	16098	2973	2936	2936
q14	305	267	276	267
q15	511	463	476	463
q16	459	412	425	412
q17	1757	1458	1475	1458
q18	7620	7574	7444	7444
q19	1619	1500	1486	1486
q20	1972	1779	1752	1752
q21	4841	4768	4972	4768
q22	541	485	472	472
Total cold run time: 70555 ms
Total hot run time: 54754 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185932 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85f24324b0cb0594e72ea0668e705d170093c702, data reload: false

query1	918	344	352	344
query2	6462	2445	2392	2392
query3	6653	205	205	205
query4	22964	21463	21611	21463
query5	4235	416	407	407
query6	265	172	172	172
query7	4585	281	276	276
query8	243	182	186	182
query9	8605	2318	2312	2312
query10	594	262	254	254
query11	14775	14336	14314	14314
query12	139	89	86	86
query13	1648	374	358	358
query14	10600	7550	6790	6790
query15	256	183	190	183
query16	8156	263	262	262
query17	1913	578	546	546
query18	2106	272	322	272
query19	208	145	151	145
query20	90	84	84	84
query21	199	128	126	126
query22	4992	4801	4777	4777
query23	33817	33069	33048	33048
query24	11714	2914	2862	2862
query25	640	360	379	360
query26	1752	148	149	148
query27	2999	309	312	309
query28	7322	1980	1972	1972
query29	1029	601	592	592
query30	294	152	150	150
query31	968	725	725	725
query32	90	56	64	56
query33	760	242	235	235
query34	1092	465	489	465
query35	840	701	692	692
query36	1020	933	885	885
query37	275	64	65	64
query38	3368	3217	3227	3217
query39	1600	1511	1531	1511
query40	272	127	126	126
query41	40	40	43	40
query42	102	98	94	94
query43	573	534	548	534
query44	1172	714	728	714
query45	282	258	258	258
query46	1082	722	723	722
query47	1928	1838	1855	1838
query48	363	293	284	284
query49	1185	396	386	386
query50	750	376	384	376
query51	6804	6528	6702	6528
query52	104	94	89	89
query53	353	281	306	281
query54	319	226	231	226
query55	80	72	70	70
query56	244	221	222	221
query57	1223	1112	1123	1112
query58	222	195	193	193
query59	3506	3170	3085	3085
query60	264	237	231	231
query61	90	88	87	87
query62	646	450	456	450
query63	301	277	273	273
query64	9495	7205	7150	7150
query65	3092	3051	3051	3051
query66	1375	349	333	333
query67	15445	14993	14888	14888
query68	5205	526	529	526
query69	481	294	302	294
query70	1203	1083	1157	1083
query71	397	266	273	266
query72	7139	2640	2421	2421
query73	702	317	315	315
query74	6858	6435	6433	6433
query75	3498	2654	2686	2654
query76	3409	1062	991	991
query77	392	260	262	260
query78	10925	10312	10290	10290
query79	3429	513	522	513
query80	1738	416	420	416
query81	546	220	224	220
query82	868	91	89	89
query83	278	163	166	163
query84	270	89	83	83
query85	2174	265	262	262
query86	518	320	273	273
query87	3515	3264	3261	3261
query88	4626	2312	2321	2312
query89	499	362	386	362
query90	2012	181	178	178
query91	125	95	99	95
query92	62	48	48	48
query93	5004	506	491	491
query94	1247	180	179	179
query95	404	301	308	301
query96	594	263	262	262
query97	3159	2959	2965	2959
query98	239	280	212	212
query99	1233	839	873	839
Total cold run time: 294247 ms
Total hot run time: 185932 ms

@qidaye qidaye dismissed stale reviews from morrySnow and morningman via 975d24c April 26, 2024 07:12
@qidaye
Copy link
Contributor Author

qidaye commented Apr 26, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 26, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41318 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 975d24c102c0f6a3727013b95f7e9f28c82c36fe, data reload: false

------ Round 1 ----------------------------------
q1	17603	4349	4244	4244
q2	2007	193	195	193
q3	10438	1262	1149	1149
q4	10204	780	831	780
q5	7531	2756	2708	2708
q6	219	136	132	132
q7	1081	653	638	638
q8	9236	2141	2112	2112
q9	9610	6854	6840	6840
q10	9503	3933	3923	3923
q11	447	232	239	232
q12	496	220	220	220
q13	18293	3089	3257	3089
q14	270	223	223	223
q15	512	479	479	479
q16	521	423	399	399
q17	981	644	698	644
q18	8355	7835	7671	7671
q19	3664	1568	1536	1536
q20	636	325	329	325
q21	5262	3503	4239	3503
q22	341	278	281	278
Total cold run time: 117210 ms
Total hot run time: 41318 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4498	4410	4375	4375
q2	372	280	281	280
q3	3265	2932	2944	2932
q4	1989	1601	1623	1601
q5	5472	5535	5527	5527
q6	216	127	128	127
q7	2356	1975	1967	1967
q8	3289	3431	3426	3426
q9	8819	8914	8928	8914
q10	4113	3715	3823	3715
q11	605	504	505	504
q12	802	650	659	650
q13	15972	3147	3117	3117
q14	314	295	287	287
q15	524	502	497	497
q16	516	446	441	441
q17	1792	1538	1522	1522
q18	7887	7499	7467	7467
q19	2594	1563	1537	1537
q20	2021	1811	1788	1788
q21	13865	4923	4817	4817
q22	581	491	477	477
Total cold run time: 81862 ms
Total hot run time: 55968 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 975d24c102c0f6a3727013b95f7e9f28c82c36fe, data reload: false

query1	925	373	350	350
query2	6443	2570	2652	2570
query3	6913	211	223	211
query4	24208	21999	21588	21588
query5	4312	428	426	426
query6	753	194	187	187
query7	5752	293	288	288
query8	578	209	189	189
query9	11151	2327	2312	2312
query10	446	241	255	241
query11	14801	14302	14592	14302
query12	141	94	94	94
query13	1645	366	361	361
query14	8974	8730	8374	8374
query15	252	185	187	185
query16	8146	257	259	257
query17	1916	591	584	584
query18	2096	270	267	267
query19	310	151	147	147
query20	92	84	85	84
query21	194	131	128	128
query22	5047	4840	4830	4830
query23	34185	33280	33310	33280
query24	10558	2985	2928	2928
query25	582	376	374	374
query26	689	160	149	149
query27	2096	318	325	318
query28	5815	2010	1988	1988
query29	850	609	603	603
query30	239	155	157	155
query31	938	743	739	739
query32	97	51	53	51
query33	725	248	241	241
query34	956	478	482	478
query35	795	672	672	672
query36	1062	909	896	896
query37	104	63	67	63
query38	3185	3004	3017	3004
query39	1596	1537	1531	1531
query40	194	124	128	124
query41	39	36	37	36
query42	104	94	96	94
query43	583	546	551	546
query44	1150	721	727	721
query45	284	261	273	261
query46	1065	713	721	713
query47	1945	1843	1863	1843
query48	374	282	293	282
query49	805	386	421	386
query50	768	388	373	373
query51	6927	6825	6759	6759
query52	108	89	87	87
query53	344	284	279	279
query54	302	234	238	234
query55	82	71	71	71
query56	239	220	213	213
query57	1185	1108	1157	1108
query58	220	213	193	193
query59	3395	3443	3157	3157
query60	257	229	263	229
query61	92	90	96	90
query62	602	426	436	426
query63	304	277	286	277
query64	8191	7160	7099	7099
query65	3122	3029	3067	3029
query66	821	337	360	337
query67	15558	15031	14913	14913
query68	5214	521	544	521
query69	488	303	297	297
query70	1115	1117	1146	1117
query71	416	265	264	264
query72	7591	2637	2427	2427
query73	696	316	321	316
query74	6627	6213	6052	6052
query75	3371	2613	2715	2613
query76	3461	1011	914	914
query77	559	260	265	260
query78	11016	10463	10261	10261
query79	3529	521	516	516
query80	2053	432	430	430
query81	540	222	223	222
query82	865	98	93	93
query83	304	176	172	172
query84	263	90	93	90
query85	1884	283	267	267
query86	476	307	310	307
query87	3241	3112	3098	3098
query88	4544	2316	2319	2316
query89	494	377	374	374
query90	2064	178	176	176
query91	126	101	97	97
query92	56	46	48	46
query93	5175	513	495	495
query94	1271	190	224	190
query95	380	293	296	293
query96	608	265	263	263
query97	3159	2931	2949	2931
query98	239	221	217	217
query99	1206	873	860	860
Total cold run time: 291658 ms
Total hot run time: 187820 ms

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye
Copy link
Contributor Author

qidaye commented Apr 26, 2024

run p0

2 similar comments
@qidaye
Copy link
Contributor Author

qidaye commented Apr 26, 2024

run p0

@qidaye
Copy link
Contributor Author

qidaye commented Apr 26, 2024

run p0

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 26, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@qidaye qidaye merged commit 3e36eab into apache:master Apr 26, 2024
@qidaye qidaye deleted the fix_es_catalog_column_empty branch April 26, 2024 11:46
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Apr 26, 2024
…34151)

In SQL syntax, `col != ''` equals `col.length() > 0`.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

---------

Co-authored-by: Luennng <luennng@gmail.com>
morningman pushed a commit that referenced this pull request Apr 26, 2024
yiguolei pushed a commit that referenced this pull request Apr 26, 2024
In SQL syntax, `col != ''` equals `col.length() > 0`.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

---------

Co-authored-by: Luennng <luennng@gmail.com>
dataroaring pushed a commit that referenced this pull request Apr 27, 2024
In SQL syntax, `col != ''` equals `col.length() > 0`.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

---------

Co-authored-by: Luennng <luennng@gmail.com>
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.10-merged dev/2.1.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants