Skip to content

[fix](ES Catalog)Fix int parse error when querying by doc_values #40385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qidaye merged 2 commits into from
Sep 9, 2024
Merged

[fix](ES Catalog)Fix int parse error when querying by doc_values #40385

qidaye merged 2 commits into from
Sep 9, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Sep 4, 2024

Proposed changes

When querying by doc_values, the result may be embrace by quotes, which will lead the int parsing error.

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Sep 4, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

TPC-H: Total hot run time: 38431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3a633f356f6861f7f6c49f10063fd826197e73b, data reload: false

------ Round 1 ----------------------------------
q1	17624	4496	4320	4320
q2	2020	183	179	179
q3	11846	990	1298	990
q4	10516	715	698	698
q5	7765	2884	2800	2800
q6	231	143	140	140
q7	953	620	609	609
q8	9320	2127	2149	2127
q9	7219	6565	6522	6522
q10	7022	2269	2227	2227
q11	453	241	244	241
q12	393	221	227	221
q13	17763	3113	3105	3105
q14	284	240	240	240
q15	538	499	498	498
q16	613	545	514	514
q17	993	663	794	663
q18	7394	6865	6886	6865
q19	1385	1034	1155	1034
q20	659	339	342	339
q21	4017	3077	3094	3077
q22	1142	1028	1022	1022
Total cold run time: 110150 ms
Total hot run time: 38431 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4360	4319	4324	4319
q2	387	274	276	274
q3	2916	2703	2696	2696
q4	2006	1746	1634	1634
q5	5737	5708	5812	5708
q6	228	140	134	134
q7	2229	1872	1862	1862
q8	3277	3488	3532	3488
q9	8954	8901	8856	8856
q10	3683	3439	3383	3383
q11	595	512	533	512
q12	867	694	672	672
q13	15116	3294	3280	3280
q14	328	290	294	290
q15	531	491	501	491
q16	603	577	594	577
q17	1861	1601	1529	1529
q18	8177	7795	7890	7795
q19	1747	1635	1606	1606
q20	2163	1915	1954	1915
q21	5749	5735	5311	5311
q22	1153	1101	1100	1100
Total cold run time: 72667 ms
Total hot run time: 57432 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

TeamCity be ut coverage result:
Function Coverage: 36.84% (9393/25497)
Line Coverage: 28.28% (77457/273925)
Region Coverage: 27.66% (39959/144444)
Branch Coverage: 24.32% (20339/83644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b3a633f356f6861f7f6c49f10063fd826197e73b_b3a633f356f6861f7f6c49f10063fd826197e73b/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

TPC-DS: Total hot run time: 194085 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3a633f356f6861f7f6c49f10063fd826197e73b, data reload: false

query1	1264	897	882	882
query2	6451	2109	1970	1970
query3	10729	4246	4147	4147
query4	60342	26429	23195	23195
query5	5390	517	507	507
query6	417	164	154	154
query7	5783	310	301	301
query8	298	216	228	216
query9	8852	2509	2516	2509
query10	511	277	273	273
query11	18392	15061	15509	15061
query12	152	102	101	101
query13	1563	414	406	406
query14	10916	7424	7486	7424
query15	242	180	183	180
query16	7574	503	519	503
query17	1109	602	603	602
query18	2097	315	314	314
query19	304	170	164	164
query20	134	114	110	110
query21	213	111	111	111
query22	4516	4571	4501	4501
query23	34366	33491	33826	33491
query24	5994	2974	2850	2850
query25	553	411	410	410
query26	691	161	156	156
query27	1784	284	285	284
query28	3854	2074	2047	2047
query29	720	429	430	429
query30	246	161	165	161
query31	934	787	779	779
query32	92	56	56	56
query33	481	314	294	294
query34	863	503	491	491
query35	824	724	764	724
query36	1066	954	937	937
query37	149	91	90	90
query38	3946	4013	3862	3862
query39	1452	1392	1393	1392
query40	205	125	122	122
query41	51	49	48	48
query42	122	103	103	103
query43	524	486	494	486
query44	1128	751	760	751
query45	197	165	167	165
query46	1103	786	735	735
query47	1967	1795	1828	1795
query48	380	320	307	307
query49	786	451	470	451
query50	839	425	433	425
query51	6935	6943	6857	6857
query52	102	87	91	87
query53	255	184	184	184
query54	595	471	482	471
query55	77	77	81	77
query56	307	272	277	272
query57	1206	1089	1069	1069
query58	231	254	246	246
query59	3199	2923	2871	2871
query60	300	278	270	270
query61	101	98	101	98
query62	761	652	672	652
query63	213	189	188	188
query64	2884	683	668	668
query65	3192	3171	3156	3156
query66	714	338	335	335
query67	15447	15325	15172	15172
query68	4312	573	578	573
query69	439	293	288	288
query70	1180	1081	1033	1033
query71	374	285	297	285
query72	6582	4055	4035	4035
query73	760	329	347	329
query74	9457	8888	9013	8888
query75	3413	2681	2739	2681
query76	1765	982	976	976
query77	564	330	310	310
query78	9769	9134	9014	9014
query79	1306	547	540	540
query80	954	515	547	515
query81	567	232	243	232
query82	251	147	151	147
query83	176	151	158	151
query84	263	80	76	76
query85	913	289	341	289
query86	414	299	309	299
query87	4437	4351	4335	4335
query88	3148	2394	2388	2388
query89	392	294	285	285
query90	1851	195	195	195
query91	132	105	98	98
query92	65	51	57	51
query93	2073	544	552	544
query94	787	303	303	303
query95	351	258	259	258
query96	605	268	269	268
query97	3192	3080	3072	3072
query98	216	211	201	201
query99	1642	1281	1288	1281
Total cold run time: 311435 ms
Total hot run time: 194085 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

ClickBench: Total hot run time: 32.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3a633f356f6861f7f6c49f10063fd826197e73b, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.52	0.50	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.56	0.50	0.50
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.59	0.59
query14	2.02	2.13	2.11
query15	0.85	0.82	0.84
query16	0.37	0.38	0.37
query17	1.05	1.06	1.06
query18	0.21	0.21	0.21
query19	1.89	1.83	1.84
query20	0.01	0.00	0.01
query21	15.41	0.68	0.66
query22	4.15	6.77	1.79
query23	18.30	1.43	1.29
query24	2.08	0.23	0.22
query25	0.16	0.08	0.08
query26	0.28	0.18	0.17
query27	0.08	0.07	0.08
query28	13.27	1.01	1.00
query29	12.63	3.38	3.39
query30	0.24	0.06	0.05
query31	2.85	0.41	0.39
query32	3.25	0.48	0.49
query33	2.96	3.00	3.05
query34	17.04	4.41	4.45
query35	4.49	4.38	4.46
query36	0.66	0.47	0.47
query37	0.18	0.15	0.16
query38	0.17	0.14	0.15
query39	0.06	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.8 s
Total hot run time: 32.17 s

@qidaye qidaye force-pushed the fix_doc_value_int_parse_error branch from b3a633f to c5d2617 Compare September 5, 2024 13:21
@qidaye
Copy link
Contributor Author

qidaye commented Sep 5, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TeamCity be ut coverage result:
Function Coverage: 36.83% (9394/25507)
Line Coverage: 28.26% (77463/274110)
Region Coverage: 27.67% (39995/144562)
Branch Coverage: 24.30% (20346/83730)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c5d26176bf484aea74fb2991256aeb9bc96fce7c_c5d26176bf484aea74fb2991256aeb9bc96fce7c/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TPC-H: Total hot run time: 37812 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5d26176bf484aea74fb2991256aeb9bc96fce7c, data reload: false

------ Round 1 ----------------------------------
q1	17607	4434	4304	4304
q2	2019	187	178	178
q3	11712	970	1071	970
q4	10524	701	662	662
q5	7747	2821	2823	2821
q6	229	137	134	134
q7	963	597	595	595
q8	9313	2043	2035	2035
q9	7129	6514	6571	6514
q10	6999	2218	2181	2181
q11	451	235	246	235
q12	398	220	219	219
q13	18060	3049	3062	3049
q14	297	252	233	233
q15	526	483	483	483
q16	586	517	523	517
q17	972	663	684	663
q18	7288	6805	6888	6805
q19	1396	1017	1153	1017
q20	661	352	328	328
q21	4536	3020	2858	2858
q22	1098	1012	1011	1011
Total cold run time: 110511 ms
Total hot run time: 37812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4295	4297	4295
q2	379	293	280	280
q3	2888	2709	2652	2652
q4	1959	1611	1674	1611
q5	5609	5647	5660	5647
q6	243	138	135	135
q7	2207	1804	1819	1804
q8	3279	3407	3415	3407
q9	8763	8826	8703	8703
q10	3540	3363	3424	3363
q11	612	521	507	507
q12	844	658	648	648
q13	14083	3257	3195	3195
q14	305	284	288	284
q15	529	498	492	492
q16	608	567	584	567
q17	1833	1528	1544	1528
q18	8129	7869	7702	7702
q19	1743	1731	1541	1541
q20	2158	1917	1890	1890
q21	5798	5525	5641	5525
q22	1114	1042	1055	1042
Total cold run time: 70981 ms
Total hot run time: 56818 ms

@morningman morningman added dev/2.1.x dev/3.0.x and removed approved Indicates a PR has been approved by one committer. reviewed labels Sep 5, 2024
@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TPC-DS: Total hot run time: 193077 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5d26176bf484aea74fb2991256aeb9bc96fce7c, data reload: false

query1	1259	879	855	855
query2	6320	1910	1867	1867
query3	10612	3939	3936	3936
query4	59080	24742	23284	23284
query5	5510	490	501	490
query6	412	182	160	160
query7	5791	296	291	291
query8	321	230	230	230
query9	9221	2499	2499	2499
query10	491	270	266	266
query11	18632	15238	15318	15238
query12	160	107	110	107
query13	1581	415	425	415
query14	11069	7332	7265	7265
query15	237	175	175	175
query16	7645	441	452	441
query17	1206	575	581	575
query18	1958	289	302	289
query19	297	146	151	146
query20	117	113	106	106
query21	209	108	104	104
query22	4664	4429	4397	4397
query23	34284	33637	33529	33529
query24	5943	2860	2849	2849
query25	519	388	387	387
query26	690	150	148	148
query27	1802	276	269	269
query28	3922	2061	2047	2047
query29	688	396	404	396
query30	232	153	157	153
query31	916	767	785	767
query32	82	52	85	52
query33	474	282	285	282
query34	886	478	467	467
query35	838	703	742	703
query36	1018	920	936	920
query37	147	84	79	79
query38	4010	3911	3897	3897
query39	1448	1442	1380	1380
query40	192	109	115	109
query41	46	46	44	44
query42	115	94	96	94
query43	487	466	462	462
query44	1068	753	753	753
query45	194	166	163	163
query46	1093	735	731	731
query47	1914	1799	1839	1799
query48	375	296	294	294
query49	764	429	451	429
query50	827	429	405	405
query51	6946	6870	6890	6870
query52	98	88	120	88
query53	248	178	178	178
query54	549	451	447	447
query55	77	73	73	73
query56	288	262	258	258
query57	1220	1060	1064	1060
query58	213	218	231	218
query59	2968	2813	2770	2770
query60	301	274	268	268
query61	106	100	101	100
query62	748	659	655	655
query63	221	182	186	182
query64	2849	721	666	666
query65	3192	3122	3169	3122
query66	675	333	359	333
query67	15427	15489	15326	15326
query68	2994	598	587	587
query69	396	286	281	281
query70	1200	1088	1112	1088
query71	341	282	280	280
query72	5087	4043	3969	3969
query73	748	324	322	322
query74	9128	8891	8958	8891
query75	3393	2680	2705	2680
query76	1397	933	1023	933
query77	552	327	310	310
query78	10435	9295	9066	9066
query79	1059	542	532	532
query80	691	511	530	511
query81	469	234	237	234
query82	238	141	150	141
query83	181	151	152	151
query84	246	76	72	72
query85	680	298	279	279
query86	320	301	277	277
query87	4354	4269	4265	4265
query88	3196	2378	2367	2367
query89	388	281	284	281
query90	1922	186	184	184
query91	124	102	98	98
query92	55	48	52	48
query93	1058	533	530	530
query94	682	290	274	274
query95	312	245	246	245
query96	586	273	270	270
query97	3171	3081	3143	3081
query98	211	197	203	197
query99	1539	1275	1304	1275
Total cold run time: 305045 ms
Total hot run time: 193077 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

ClickBench: Total hot run time: 31.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c5d26176bf484aea74fb2991256aeb9bc96fce7c, data reload: false

query1	0.05	0.05	0.04
query2	0.09	0.03	0.04
query3	0.23	0.06	0.05
query4	1.68	0.07	0.07
query5	0.52	0.49	0.48
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.54	0.56	0.55
query11	0.16	0.11	0.11
query12	0.16	0.13	0.12
query13	0.60	0.59	0.59
query14	2.08	2.04	2.12
query15	0.87	0.84	0.82
query16	0.36	0.38	0.37
query17	1.01	1.03	0.97
query18	0.21	0.21	0.20
query19	1.86	1.75	1.77
query20	0.01	0.01	0.02
query21	15.39	0.67	0.65
query22	4.41	7.33	1.96
query23	18.29	1.34	1.28
query24	2.10	0.22	0.21
query25	0.16	0.08	0.07
query26	0.26	0.18	0.18
query27	0.08	0.07	0.07
query28	13.31	1.02	1.02
query29	12.66	3.38	3.33
query30	0.25	0.05	0.05
query31	2.88	0.40	0.40
query32	3.24	0.50	0.48
query33	2.99	2.98	2.98
query34	16.98	4.31	4.36
query35	4.47	4.47	4.44
query36	0.67	0.47	0.47
query37	0.19	0.16	0.16
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.08	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.05 s
Total hot run time: 31.95 s

eldenmoon
eldenmoon approved these changes Sep 9, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by anyone and no changes requested.

@qidaye qidaye merged commit 39c5fb4 into apache:master Sep 9, 2024
@qidaye qidaye deleted the fix_doc_value_int_parse_error branch September 9, 2024 02:20
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 9, 2024
@qidaye
[fix](ES Catalog)Fix int parse error when querying by doc_values (apa…
19ae695 
…che#40385)

When querying by doc_values, the result may be embrace by quotes, which
will lead the int parsing error.
@qidaye qidaye mentioned this pull request Sep 9, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Sep 9, 2024
@qidaye
[fix](ES Catalog)Fix int parse error when querying by doc_values (apa…
0867cd5 
…che#40385)

When querying by doc_values, the result may be embrace by quotes, which
will lead the int parsing error.
@qidaye qidaye mentioned this pull request Sep 9, 2024
Merged
xiaokang pushed a commit that referenced this pull request Sep 9, 2024
@qidaye
[fix](ES Catalog)Fix int parse error when querying by doc_values (#40385
9dea5ce 
) (#40522)
qidaye added a commit that referenced this pull request Sep 9, 2024
@qidaye
[fix](ES Catalog)Fix int parse error when querying by doc_values (#40385
314f6ae 
) (#40521)

bp #40385
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@qidaye @dataroaring
[fix](ES Catalog)Fix int parse error when querying by doc_values (#40385
ba1dd8c 
)

When querying by doc_values, the result may be embrace by quotes, which
will lead the int parsing error.
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@qidaye @doris-robot @morningman @eldenmoon @xiaokang @dataroaring