Skip to content

[fix](ES catalog)Make col != '' behavior consistent with SQL (#34151) #34190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Apr 26, 2024
Merged

[fix](ES catalog)Make col != '' behavior consistent with SQL (#34151) #34190

morningman merged 1 commit into from
Apr 26, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Apr 26, 2024 •
edited
Loading

Proposed changes

pick #34151
In SQL syntax, col != '' equals col.length() > 0.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@qidaye @luennng
[fix](ES catalog)Make col != '' behavior consistent with SQL (apache#…
71d60f5 
…34151)

In SQL syntax, `col != ''` equals `col.length() > 0`.
It means that this column must exist in ES doc fields and its content is not empty.
In this PR, we make a special translation for this binary predicate to keep the behavior of both consistent.

---------

Co-authored-by: Luennng <luennng@gmail.com>
@doris-robot
Copy link

doris-robot commented Apr 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Apr 26, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 26, 2024

TPC-H: Total hot run time: 49807 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 71d60f54892b6a522830b4895945a23ce6a5ee86, data reload: false

------ Round 1 ----------------------------------
q1	17821	4404	4317	4317
q2	2023	149	139	139
q3	10469	1859	1940	1859
q4	10332	1266	1327	1266
q5	8389	3926	3928	3926
q6	225	122	122	122
q7	2016	1577	1560	1560
q8	9267	2700	2705	2700
q9	10836	10405	10669	10405
q10	8582	3493	3491	3491
q11	414	231	240	231
q12	452	298	305	298
q13	18351	3987	4023	3987
q14	353	321	322	321
q15	498	452	460	452
q16	672	577	577	577
q17	1114	951	950	950
q18	7151	6981	6864	6864
q19	1706	1566	1539	1539
q20	530	331	309	309
q21	4385	4104	4099	4099
q22	478	395	402	395
Total cold run time: 116064 ms
Total hot run time: 49807 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4311	4289	4269	4269
q2	320	225	228	225
q3	4179	4175	4178	4175
q4	2737	2744	2728	2728
q5	7148	7068	7058	7058
q6	237	118	118	118
q7	3245	2853	2836	2836
q8	4318	4475	4495	4475
q9	17370	17251	17102	17102
q10	4237	4225	4245	4225
q11	746	680	731	680
q12	1018	857	845	845
q13	6243	3758	3770	3758
q14	451	411	415	411
q15	503	450	451	450
q16	734	676	688	676
q17	3803	3942	3981	3942
q18	8851	8823	8676	8676
q19	1700	1691	1658	1658
q20	2339	2119	2122	2119
q21	8532	8371	8413	8371
q22	1004	980	968	968
Total cold run time: 84026 ms
Total hot run time: 79765 ms

@doris-robot
Copy link

doris-robot commented Apr 26, 2024

TPC-DS: Total hot run time: 202878 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 71d60f54892b6a522830b4895945a23ce6a5ee86, data reload: false

query1	908	384	374	374
query2	6544	2769	2514	2514
query3	6914	197	196	196
query4	19917	17874	18054	17874
query5	19729	6411	6451	6411
query6	275	219	232	219
query7	4154	291	298	291
query8	244	273	237	237
query9	3148	2711	2647	2647
query10	398	290	300	290
query11	11276	10734	10727	10727
query12	116	76	68	68
query13	5578	697	665	665
query14	17288	13049	13205	13049
query15	366	217	234	217
query16	6478	279	254	254
query17	1728	1472	884	884
query18	2306	407	395	395
query19	205	153	145	145
query20	80	77	77	77
query21	186	89	93	89
query22	5138	4991	5046	4991
query23	32556	31669	32103	31669
query24	7107	6536	6469	6469
query25	508	434	425	425
query26	529	156	159	156
query27	1875	288	292	288
query28	6236	2409	2368	2368
query29	2802	2562	2897	2562
query30	240	161	160	160
query31	905	742	730	730
query32	68	60	56	56
query33	396	249	257	249
query34	848	447	464	447
query35	1119	903	942	903
query36	1288	1213	1104	1104
query37	87	63	63	63
query38	3036	2895	2906	2895
query39	1366	1318	1321	1318
query40	196	98	92	92
query41	37	39	34	34
query42	84	88	77	77
query43	791	852	732	732
query44	1110	729	714	714
query45	237	225	225	225
query46	1217	970	945	945
query47	1745	1712	1770	1712
query48	1007	693	700	693
query49	616	361	364	361
query50	868	596	646	596
query51	4718	4652	4620	4620
query52	84	75	74	74
query53	439	312	305	305
query54	2637	2461	2446	2446
query55	96	79	81	79
query56	204	205	198	198
query57	1429	1112	1142	1112
query58	200	197	209	197
query59	4041	3885	4216	3885
query60	213	183	201	183
query61	94	87	88	87
query62	849	440	484	440
query63	465	336	329	329
query64	2417	1568	1420	1420
query65	3631	3549	3535	3535
query66	785	368	380	368
query67	16198	15452	16511	15452
query68	10177	669	662	662
query69	572	327	339	327
query70	1869	1296	1514	1296
query71	415	302	309	302
query72	6456	3469	3531	3469
query73	730	323	323	323
query74	6346	5880	5858	5858
query75	5401	3688	3638	3638
query76	6599	1157	1227	1157
query77	1143	260	247	247
query78	12617	12864	12730	12730
query79	11389	640	644	640
query80	655	401	395	395
query81	494	229	229	229
query82	494	97	92	92
query83	172	133	134	133
query84	258	69	68	68
query85	702	311	300	300
query86	315	291	278	278
query87	3217	3012	2976	2976
query88	5167	2319	2316	2316
query89	397	293	279	279
query90	2023	185	196	185
query91	167	134	131	131
query92	55	48	50	48
query93	5229	567	578	567
query94	707	211	206	206
query95	1098	1038	1052	1038
query96	641	330	326	326
query97	6577	6261	6599	6261
query98	187	173	162	162
query99	2691	974	852	852
Total cold run time: 315310 ms
Total hot run time: 202878 ms

@doris-robot
Copy link

doris-robot commented Apr 26, 2024

ClickBench: Total hot run time: 30.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 71d60f54892b6a522830b4895945a23ce6a5ee86, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.07	0.06
query5	0.53	0.52	0.52
query6	1.26	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.49
query10	0.53	0.54	0.55
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.61
query14	0.78	0.80	0.79
query15	0.76	0.76	0.76
query16	0.35	0.37	0.37
query17	1.02	1.02	1.01
query18	0.23	0.25	0.26
query19	1.87	1.80	1.87
query20	0.01	0.01	0.01
query21	15.46	0.59	0.57
query22	2.22	2.75	1.64
query23	17.19	1.02	0.81
query24	5.67	1.06	3.70
query25	0.39	0.12	0.06
query26	0.84	0.15	0.15
query27	0.04	0.04	0.03
query28	4.98	0.75	0.72
query29	12.62	2.41	2.26
query30	0.54	0.59	0.52
query31	2.80	0.38	0.38
query32	3.37	0.48	0.50
query33	3.04	3.08	3.10
query34	15.27	4.82	4.81
query35	4.92	4.83	4.85
query36	1.05	1.01	1.00
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 101.68 s
Total hot run time: 30.74 s

@doris-robot
Copy link

doris-robot commented Apr 26, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 71d60f54892b6a522830b4895945a23ce6a5ee86 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s

@morningman morningman merged commit ddf292a into apache:branch-2.0 Apr 26, 2024
@qidaye qidaye deleted the pick_es_catalog_empty_column branch April 28, 2024 04:17
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@qidaye @luennng
[fix](ES catalog)Make col != '' behavior consistent with SQL (apache#…
2fe4a51 
…34151) (apache#34190)

pick apache#34151

Co-authored-by: Luennng <luennng@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qidaye @doris-robot @morningman