Skip to content

[Fix](JsonReader) Return correct status when parse failed (#39206) #39269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eldenmoon merged 1 commit into from
Aug 14, 2024
Merged

[Fix](JsonReader) Return correct status when parse failed (#39206) #39269

eldenmoon merged 1 commit into from
Aug 14, 2024

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Aug 13, 2024

@eldenmoon
[Fix](JsonReader) Return correct status when parse failed (apache#39206)
c500be5 
When using `JsonFunctions::extract_from_object`, but input obj is not
object type but other types like null, then Status should be
`simdjson::INCORRECT_TYPE` to fill the default value in the later
process. Example, json path is `$.city.name`, but input json is `{"city"
: null}` then `Status::NotFound` should be returned, in the following,
column will be filled with default values.
@doris-robot
Copy link

doris-robot commented Aug 13, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

eldenmoon commented Aug 13, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 13, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

TPC-H: Total hot run time: 49706 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c500be53f472d508948f62f60015ba4e21edf938, data reload: false

------ Round 1 ----------------------------------
q1	17733	4475	4352	4352
q2	2065	160	151	151
q3	10250	1902	1968	1902
q4	10150	1221	1314	1221
q5	8484	3933	3955	3933
q6	236	129	125	125
q7	2082	1621	1602	1602
q8	9513	2727	2715	2715
q9	11039	10225	10186	10186
q10	8663	3531	3523	3523
q11	408	237	235	235
q12	464	305	312	305
q13	18347	3942	4006	3942
q14	347	328	337	328
q15	499	465	453	453
q16	687	563	567	563
q17	1128	967	996	967
q18	7267	6735	6903	6735
q19	1824	1643	1640	1640
q20	551	333	310	310
q21	4426	4164	4068	4068
q22	530	450	456	450
Total cold run time: 116693 ms
Total hot run time: 49706 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4312	4253	4284	4253
q2	324	235	224	224
q3	4193	4129	4135	4129
q4	2741	2745	2731	2731
q5	7098	7058	7055	7055
q6	240	120	121	120
q7	3261	2861	2810	2810
q8	4373	4464	4526	4464
q9	16893	16797	16771	16771
q10	4208	4264	4371	4264
q11	748	717	666	666
q12	1030	839	855	839
q13	7664	3738	3754	3738
q14	461	418	423	418
q15	501	463	450	450
q16	743	682	684	682
q17	3817	3820	3909	3820
q18	8802	8731	8760	8731
q19	1735	1734	1690	1690
q20	2415	2132	2076	2076
q21	8417	8492	8362	8362
q22	1054	1001	977	977
Total cold run time: 85030 ms
Total hot run time: 79270 ms

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

TPC-DS: Total hot run time: 203644 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c500be53f472d508948f62f60015ba4e21edf938, data reload: false

query1	919	389	419	389
query2	6536	2929	2643	2643
query3	6924	210	199	199
query4	21341	18011	17973	17973
query5	19734	6531	6589	6531
query6	278	224	246	224
query7	4159	306	308	306
query8	450	433	443	433
query9	3159	2736	2676	2676
query10	406	306	305	305
query11	11318	10773	10791	10773
query12	127	77	71	71
query13	5624	708	697	697
query14	17863	13508	13431	13431
query15	365	259	259	259
query16	6469	283	270	270
query17	1694	1453	882	882
query18	2304	418	414	414
query19	203	148	147	147
query20	75	78	79	78
query21	190	96	92	92
query22	5251	5064	5039	5039
query23	32546	31847	32086	31847
query24	6994	6467	6623	6467
query25	529	457	438	438
query26	528	161	160	160
query27	1898	294	289	289
query28	6157	2395	2333	2333
query29	2984	2677	2742	2677
query30	235	169	167	167
query31	904	748	733	733
query32	69	61	61	61
query33	405	259	263	259
query34	828	472	496	472
query35	1126	943	893	893
query36	1281	1238	1060	1060
query37	92	60	60	60
query38	3014	2983	2946	2946
query39	1360	1311	1336	1311
query40	224	90	92	90
query41	46	44	44	44
query42	86	82	83	82
query43	728	770	767	767
query44	1143	725	733	725
query45	249	239	233	233
query46	1225	951	953	951
query47	1844	1695	1746	1695
query48	1038	705	730	705
query49	634	373	362	362
query50	854	621	589	589
query51	4737	4671	4639	4639
query52	99	87	95	87
query53	449	325	328	325
query54	2663	2462	2473	2462
query55	89	75	84	75
query56	242	206	219	206
query57	1091	1167	1086	1086
query58	212	205	204	204
query59	4358	4163	4186	4163
query60	214	219	201	201
query61	97	96	122	96
query62	887	505	457	457
query63	490	350	340	340
query64	2468	1560	1444	1444
query65	3682	3559	3559	3559
query66	846	378	372	372
query67	15311	15822	15007	15007
query68	10431	652	647	647
query69	581	362	333	333
query70	2169	1444	1415	1415
query71	420	301	318	301
query72	6531	3468	3485	3468
query73	734	317	320	317
query74	6321	5874	5885	5874
query75	5374	3675	3680	3675
query76	6949	1154	1185	1154
query77	1174	271	263	263
query78	12596	11544	11577	11544
query79	8665	652	643	643
query80	1003	409	400	400
query81	460	234	233	233
query82	831	100	99	99
query83	168	133	129	129
query84	256	72	71	71
query85	890	333	331	331
query86	328	287	287	287
query87	3298	3062	3053	3053
query88	4216	2268	2298	2268
query89	480	287	280	280
query90	1941	208	208	208
query91	182	142	142	142
query92	59	54	51	51
query93	7091	585	576	576
query94	700	205	201	201
query95	1112	1067	1045	1045
query96	637	331	321	321
query97	6489	6360	6323	6323
query98	187	184	177	177
query99	3121	844	892	844
Total cold run time: 317739 ms
Total hot run time: 203644 ms

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c500be53f472d508948f62f60015ba4e21edf938, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.02	0.02
query3	0.25	0.05	0.04
query4	1.79	0.07	0.05
query5	0.54	0.53	0.52
query6	1.26	0.62	0.61
query7	0.02	0.01	0.00
query8	0.04	0.02	0.02
query9	0.51	0.49	0.48
query10	0.55	0.53	0.52
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.63	0.61	0.61
query14	0.80	0.78	0.79
query15	0.77	0.76	0.76
query16	0.38	0.35	0.37
query17	1.00	1.01	1.01
query18	0.22	0.25	0.25
query19	1.90	1.86	1.89
query20	0.01	0.01	0.02
query21	15.48	0.55	0.55
query22	2.07	1.82	1.77
query23	17.09	0.98	1.07
query24	7.56	0.76	0.98
query25	0.35	0.14	0.04
query26	0.79	0.16	0.16
query27	0.04	0.05	0.03
query28	5.83	0.77	0.72
query29	12.64	2.30	2.35
query30	0.58	0.54	0.52
query31	2.81	0.38	0.38
query32	3.36	0.51	0.50
query33	3.06	3.10	3.03
query34	15.25	4.81	4.80
query35	4.84	4.83	4.83
query36	1.05	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.01	0.02	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 104.19 s
Total hot run time: 30.72 s

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c500be53f472d508948f62f60015ba4e21edf938 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@doris-robot
Copy link

doris-robot commented Aug 13, 2024

TeamCity be ut coverage result:
Function Coverage: 37.72% (8105/21485)
Line Coverage: 29.36% (66416/226212)
Region Coverage: 28.86% (34256/118712)
Branch Coverage: 24.74% (17607/71180)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c500be53f472d508948f62f60015ba4e21edf938_c500be53f472d508948f62f60015ba4e21edf938/report/index.html

@eldenmoon eldenmoon merged commit b4d25a8 into apache:branch-2.0 Aug 14, 2024
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@eldenmoon
[Fix](JsonReader) Return correct status when parse failed (apache#39206…
d437cbb 
…) (apache#39269)

apache#39206
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@eldenmoon @doris-robot