Skip to content

[fix](es) Fix array not supporting json subtype #55738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Sep 14, 2025
Merged

[fix](es) Fix array not supporting json subtype #55738

morningman merged 1 commit into from
Sep 14, 2025

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Sep 6, 2025 •
edited
Loading

An issue introduced by #40614, which omitted handling of json subtypes when parsing single value for array column.

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk zy-kkk force-pushed the support_es_array_json branch from 6c785a2 to 5e04207 Compare September 11, 2025 03:59
@zy-kkk zy-kkk force-pushed the support_es_array_json branch from a35a2e2 to ccebc43 Compare September 11, 2025 05:26
@zy-kkk zy-kkk marked this pull request as ready for review September 11, 2025 05:26
@zy-kkk
Copy link
Member Author

zy-kkk commented Sep 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 11, 2025

TPC-H: Total hot run time: 34230 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccebc430a3bd00bf208058fa015fc9d50e5d60d5, data reload: false

------ Round 1 ----------------------------------
q1	17623	5146	5007	5007
q2	2007	340	211	211
q3	10245	1268	705	705
q4	10234	1041	528	528
q5	7534	2465	2273	2273
q6	181	164	134	134
q7	934	758	648	648
q8	9363	1291	1059	1059
q9	6914	5078	5119	5078
q10	6949	2391	1962	1962
q11	497	297	283	283
q12	347	363	217	217
q13	17777	3624	3055	3055
q14	240	243	216	216
q15	579	480	475	475
q16	1023	996	965	965
q17	587	862	358	358
q18	7431	7044	6999	6999
q19	1580	944	565	565
q20	341	326	230	230
q21	3586	2564	2293	2293
q22	1074	1044	969	969
Total cold run time: 107046 ms
Total hot run time: 34230 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5168	5050	5028	5028
q2	249	334	228	228
q3	2168	2623	2331	2331
q4	1305	1752	1349	1349
q5	4333	4312	4558	4312
q6	235	180	138	138
q7	2066	2015	1815	1815
q8	2667	2612	2512	2512
q9	7458	7340	7402	7340
q10	3168	3297	2896	2896
q11	581	523	493	493
q12	705	793	629	629
q13	3537	3904	3300	3300
q14	443	327	276	276
q15	525	479	470	470
q16	1068	1128	1066	1066
q17	1124	1550	1434	1434
q18	8124	7657	7797	7657
q19	794	819	832	819
q20	2052	2049	1849	1849
q21	4928	4396	4280	4280
q22	1078	1031	1015	1015
Total cold run time: 53776 ms
Total hot run time: 51237 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2025

TPC-DS: Total hot run time: 188818 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccebc430a3bd00bf208058fa015fc9d50e5d60d5, data reload: false

query1	1051	486	410	410
query2	6558	1679	1675	1675
query3	6748	232	228	228
query4	26582	23721	23337	23337
query5	4381	642	515	515
query6	367	250	232	232
query7	4681	501	294	294
query8	297	261	275	261
query9	8656	2883	2888	2883
query10	491	368	297	297
query11	15738	15053	14782	14782
query12	180	130	120	120
query13	1679	566	441	441
query14	10261	9285	9153	9153
query15	217	187	172	172
query16	7296	674	501	501
query17	1228	739	613	613
query18	2004	428	334	334
query19	202	192	167	167
query20	133	128	122	122
query21	206	132	118	118
query22	4046	4246	4030	4030
query23	33764	32900	32927	32900
query24	8176	2352	2415	2352
query25	580	516	450	450
query26	1240	277	166	166
query27	2739	504	360	360
query28	4379	2261	2204	2204
query29	829	606	489	489
query30	292	221	193	193
query31	899	801	721	721
query32	86	83	81	81
query33	569	394	360	360
query34	781	860	519	519
query35	843	815	769	769
query36	977	1041	908	908
query37	127	114	95	95
query38	3492	3520	3428	3428
query39	1465	1434	1420	1420
query40	235	141	125	125
query41	63	60	61	60
query42	136	116	124	116
query43	518	481	463	463
query44	1379	868	875	868
query45	188	178	176	176
query46	857	1017	653	653
query47	1791	1779	1733	1733
query48	408	423	336	336
query49	799	529	455	455
query50	649	723	411	411
query51	3981	4104	3840	3840
query52	118	118	106	106
query53	248	271	202	202
query54	610	610	536	536
query55	94	94	91	91
query56	340	347	327	327
query57	1189	1197	1125	1125
query58	342	290	286	286
query59	2605	2658	2559	2559
query60	359	359	349	349
query61	169	155	150	150
query62	827	733	669	669
query63	229	194	194	194
query64	4500	1133	814	814
query65	4033	3948	3963	3948
query66	1172	450	370	370
query67	15477	15150	15201	15150
query68	6689	924	580	580
query69	521	334	293	293
query70	1346	1261	1290	1261
query71	520	367	329	329
query72	6256	5330	5117	5117
query73	719	680	353	353
query74	8922	9145	8803	8803
query75	3243	3260	2743	2743
query76	3151	1178	739	739
query77	500	404	335	335
query78	9638	9817	8866	8866
query79	2977	807	582	582
query80	685	588	532	532
query81	514	264	227	227
query82	709	164	235	164
query83	278	278	264	264
query84	264	107	99	99
query85	920	457	419	419
query86	389	318	321	318
query87	3698	3668	3624	3624
query88	3852	2266	2237	2237
query89	399	325	298	298
query90	1877	234	241	234
query91	174	163	135	135
query92	95	80	75	75
query93	2659	982	647	647
query94	717	424	319	319
query95	417	339	336	336
query96	489	591	278	278
query97	2901	2982	2872	2872
query98	263	223	225	223
query99	1337	1434	1304	1304
Total cold run time: 274675 ms
Total hot run time: 188818 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2025

ClickBench: Total hot run time: 30.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccebc430a3bd00bf208058fa015fc9d50e5d60d5, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.09	0.08
query4	1.61	0.11	0.12
query5	0.28	0.27	0.26
query6	1.18	0.65	0.62
query7	0.03	0.03	0.02
query8	0.05	0.04	0.05
query9	0.62	0.54	0.52
query10	0.59	0.58	0.58
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.64	0.62
query14	1.02	1.03	1.01
query15	0.87	0.85	0.88
query16	0.42	0.39	0.42
query17	1.04	1.09	1.06
query18	0.21	0.20	0.20
query19	1.91	1.84	1.82
query20	0.02	0.01	0.01
query21	15.40	0.96	0.61
query22	0.77	1.15	0.79
query23	14.84	1.41	0.63
query24	7.06	1.04	1.59
query25	0.46	0.14	0.20
query26	0.59	0.16	0.13
query27	0.06	0.06	0.06
query28	10.54	0.95	0.43
query29	12.56	3.96	3.28
query30	0.29	0.13	0.11
query31	2.82	0.59	0.38
query32	3.23	0.55	0.49
query33	3.04	3.15	3.12
query34	16.01	5.52	4.81
query35	4.99	4.88	4.93
query36	0.70	0.51	0.49
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.19	0.15	0.15
query41	0.08	0.04	0.03
query42	0.03	0.04	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.11 s
Total hot run time: 30.28 s

@doris-robot
Copy link

doris-robot commented Sep 11, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.98% (17300/33284)
Line Coverage 37.32% (157688/422540)
Region Coverage 31.95% (120332/376665)
Branch Coverage 33.33% (52827/158495)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 11, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.73% (23115/32682)
Line Coverage 57.15% (241330/422285)
Region Coverage 52.53% (200696/382093)
Branch Coverage 54.14% (86299/159398)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 12, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 12, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 3fccf20 into apache:master Sep 14, 2025
29 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 14, 2025
@zy-kkk
[fix](es) Fix array not supporting json subtype (#55738)
ce470bf 
An issue introduced by #40614, which
omitted handling of json subtypes when parsing single value for array
column.
@github-actions github-actions bot mentioned this pull request Sep 14, 2025
Closed
github-actions bot pushed a commit that referenced this pull request Sep 14, 2025
@zy-kkk
[fix](es) Fix array not supporting json subtype (#55738)
c539c98 
An issue introduced by #40614, which
omitted handling of json subtypes when parsing single value for array
column.
@github-actions github-actions bot mentioned this pull request Sep 14, 2025
Closed
github-actions bot pushed a commit that referenced this pull request Sep 14, 2025
@zy-kkk
[fix](es) Fix array not supporting json subtype (#55738)
1cf9786 
An issue introduced by #40614, which
omitted handling of json subtypes when parsing single value for array
column.
@github-actions github-actions bot mentioned this pull request Sep 14, 2025
Closed
@zy-kkk zy-kkk deleted the support_es_array_json branch September 15, 2025 02:54
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Sep 28, 2025
@zy-kkk
[fix](es) Fix array not supporting json subtype (apache#55738)
7c770e5 
An issue introduced by apache#40614, which
omitted handling of json subtypes when parsing single value for array
column.
@zy-kkk zy-kkk mentioned this pull request Sep 28, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 28, 2025
@zy-kkk
branch-3.1: [fix](es) Fix array not supporting json subtype #55738 (#…
977d707 
…56535)

pick  #55738
yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Sep 28, 2025
@selectdb-robot @zy-kkk
branch-selectdb-doris-3.0: [fix](es) Fix array not supporting json su…
a2072bf 
…btype apache#55738 (apache#4667)

cherry-picks from apache#55738
Related to apache#4473

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Sep 28, 2025
@dataroaring
Revert "branch-selectdb-doris-3.0: [fix](es) Fix array not supporting…
00cf24d 
… json subtype apache#55738" (apache#4810)

Reverts selectdb/selectdb-core#4667
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Nov 12, 2025
@zy-kkk
branch-3.0: [fix](es) Fix array not supporting json subtype apache#55738
3385c37 
 (apache#4821)

pick apache#55738
w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Dec 26, 2025
@zy-kkk
cloud-4.0: [fix](es) Fix array not supporting json subtype (apache#4986)
1315413 
pick apache#55738
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zy-kkk @hello-stephen @doris-robot @morningman @CalvinKirs @dataroaring @morrySnow