Skip to content

branch-3.0: [Fix](Serde) Support hive compatible output format #49036 #49830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
github-actions wants to merge 1 commit into from
Closed

branch-3.0: [Fix](Serde) Support hive compatible output format #49036 #49830

github-actions wants to merge 1 commit into from

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Apr 7, 2025

Cherry-picked from #49036

@github-actions github-actions bot requested a review from dataroaring as a code owner April 7, 2025 07:13
@Thearas
Copy link
Contributor

Thearas commented Apr 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Apr 7, 2025
@dataroaring dataroaring reopened this Apr 7, 2025
@Thearas
Copy link
Contributor

Thearas commented Apr 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

TeamCity cloud ut coverage result:
Function Coverage: 82.99% (1083/1305)
Line Coverage: 65.91% (17915/27182)
Region Coverage: 65.45% (8839/13504)
Branch Coverage: 55.35% (4769/8616)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b5d4c6bb53f514abf512bb51f3d5cbe3586a4af0_b5d4c6bb53f514abf512bb51f3d5cbe3586a4af0_cloud/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

TPC-H: Total hot run time: 40399 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5d4c6bb53f514abf512bb51f3d5cbe3586a4af0, data reload: false

------ Round 1 ----------------------------------
q1	18005	6943	6632	6632
q2	2075	170	158	158
q3	10566	1129	1148	1129
q4	10229	748	761	748
q5	7752	2900	2850	2850
q6	224	130	131	130
q7	978	605	623	605
q8	9358	1973	2017	1973
q9	6720	6458	6454	6454
q10	6992	2274	2288	2274
q11	458	263	256	256
q12	445	213	203	203
q13	17804	2986	3015	2986
q14	234	209	209	209
q15	514	471	466	466
q16	668	569	585	569
q17	982	581	514	514
q18	7297	6770	6746	6746
q19	1381	1119	1083	1083
q20	480	196	201	196
q21	4245	3291	3228	3228
q22	1102	990	1034	990
Total cold run time: 108509 ms
Total hot run time: 40399 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6652	6583	6585	6583
q2	329	223	234	223
q3	2897	2806	2805	2805
q4	1930	1741	1706	1706
q5	5411	5476	5482	5476
q6	203	126	124	124
q7	2123	1701	1711	1701
q8	3251	3423	3488	3423
q9	8680	8561	8691	8561
q10	3498	3458	3413	3413
q11	585	485	504	485
q12	776	592	627	592
q13	9570	2971	3009	2971
q14	301	256	253	253
q15	504	452	459	452
q16	676	620	654	620
q17	1806	1592	1575	1575
q18	7721	7358	7483	7358
q19	1654	1478	1542	1478
q20	2054	1815	1814	1814
q21	5355	5078	5173	5078
q22	1080	987	963	963
Total cold run time: 67056 ms
Total hot run time: 57654 ms

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

TPC-DS: Total hot run time: 191061 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b5d4c6bb53f514abf512bb51f3d5cbe3586a4af0, data reload: false

query1	973	387	372	372
query2	6541	2019	2027	2019
query3	6700	217	214	214
query4	33973	23047	23348	23047
query5	4505	443	443	443
query6	273	181	186	181
query7	4623	322	317	317
query8	290	226	225	225
query9	9655	2634	2613	2613
query10	483	259	266	259
query11	18149	15130	15238	15130
query12	162	107	102	102
query13	1617	410	400	400
query14	11071	7460	7012	7012
query15	307	174	181	174
query16	8117	454	467	454
query17	1789	591	561	561
query18	2143	305	304	304
query19	388	164	157	157
query20	120	110	108	108
query21	209	104	104	104
query22	4435	4409	4188	4188
query23	34414	33667	33828	33667
query24	11225	2889	2818	2818
query25	687	418	415	415
query26	1206	174	167	167
query27	2801	344	346	344
query28	7822	2424	2404	2404
query29	876	455	438	438
query30	328	163	165	163
query31	1034	814	796	796
query32	101	59	61	59
query33	794	310	339	310
query34	901	488	516	488
query35	865	706	719	706
query36	1104	962	969	962
query37	145	67	74	67
query38	4150	3819	3841	3819
query39	1514	1449	1459	1449
query40	211	108	101	101
query41	54	53	50	50
query42	115	104	99	99
query43	528	478	481	478
query44	1241	809	803	803
query45	189	173	168	168
query46	1169	726	734	726
query47	2004	1864	1899	1864
query48	476	383	383	383
query49	1179	414	414	414
query50	816	422	423	422
query51	7408	7159	7060	7060
query52	103	90	92	90
query53	260	186	186	186
query54	1323	476	482	476
query55	85	80	79	79
query56	284	255	261	255
query57	1233	1149	1100	1100
query58	246	213	223	213
query59	3173	2959	3209	2959
query60	307	272	261	261
query61	132	129	170	129
query62	874	679	677	677
query63	218	199	191	191
query64	5406	662	638	638
query65	3273	3221	3194	3194
query66	1438	311	311	311
query67	16145	15786	15743	15743
query68	4910	575	579	575
query69	457	292	267	267
query70	1169	1061	1022	1022
query71	338	260	272	260
query72	6352	4053	3984	3984
query73	757	346	348	346
query74	10541	9248	9065	9065
query75	3406	2638	2658	2638
query76	3211	1191	1053	1053
query77	377	284	279	279
query78	10566	9578	9548	9548
query79	2036	585	594	585
query80	1146	441	429	429
query81	559	238	239	238
query82	895	87	88	87
query83	224	147	143	143
query84	245	86	91	86
query85	1278	315	296	296
query86	436	310	302	302
query87	4412	4334	4247	4247
query88	4011	2397	2375	2375
query89	405	301	296	296
query90	2066	187	188	187
query91	182	151	149	149
query92	65	51	52	51
query93	1572	569	554	554
query94	918	294	309	294
query95	360	261	255	255
query96	612	283	293	283
query97	3327	3157	3202	3157
query98	228	200	203	200
query99	1517	1294	1284	1284
Total cold run time: 304958 ms
Total hot run time: 191061 ms

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

ClickBench: Total hot run time: 32.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b5d4c6bb53f514abf512bb51f3d5cbe3586a4af0, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.26	0.07	0.07
query4	1.59	0.11	0.10
query5	0.52	0.51	0.51
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.55	0.58	0.59
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.61
query14	2.85	2.84	2.73
query15	0.90	0.82	0.85
query16	0.41	0.38	0.39
query17	1.05	1.06	1.06
query18	0.23	0.22	0.22
query19	1.95	1.75	2.04
query20	0.01	0.01	0.02
query21	15.36	0.60	0.58
query22	2.60	1.75	1.70
query23	16.99	0.96	0.83
query24	3.79	1.25	0.75
query25	0.36	0.11	0.08
query26	0.44	0.14	0.14
query27	0.06	0.04	0.04
query28	9.86	0.51	0.48
query29	12.57	3.25	3.25
query30	0.24	0.06	0.06
query31	2.87	0.38	0.39
query32	3.23	0.47	0.46
query33	2.98	3.02	3.03
query34	17.16	4.56	4.56
query35	4.54	4.52	4.49
query36	0.66	0.50	0.47
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.29 s
Total hot run time: 32.01 s

@BePPPower @dataroaring
[Fix](Serde) Support hive compatible output format (#49036)
dab8723 
Problem Summary:

The output format of complex data types are different between Hive and
Doris, such as array, map and struct.
When user migrate from Hive to Doris, they expect the same format so
that they don't need to modify their business code.

This PR mainly changes:

Add a new option to  session variable `serde_dialect`: If set to hive,
the output format returned to MySQL client of some datatypes will be
changed:

Array
Doris: ["abc", "def", "", null, 1]
Hive: ["abc","def","",null,true]

Map
Doris: {"k1":null, "k2":"v3"}
Hive: {"k1":null,"k2":"v3"}

Struct
Doris: {"s_id":100, "s_name":"abc , "", "s_address":null}
Hive: {"s_id":100,"s_name":"abc ,"","s_address":null}

Related #37039
@dataroaring dataroaring force-pushed the auto-pick-49036-branch-3.0 branch from b5d4c6b to dab8723 Compare April 30, 2025 09:36
@dataroaring
Copy link
Contributor

dataroaring commented Apr 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 30, 2025

TeamCity cloud ut coverage result:
Function Coverage: 83.08% (1090/1312)
Line Coverage: 66.05% (18035/27304)
Region Coverage: 65.56% (8879/13544)
Branch Coverage: 55.45% (4791/8640)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dab8723e091be87ccadc35e5b55272cd81026888_dab8723e091be87ccadc35e5b55272cd81026888_cloud/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 30, 2025

TPC-H: Total hot run time: 40691 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dab8723e091be87ccadc35e5b55272cd81026888, data reload: false

------ Round 1 ----------------------------------
q1	17651	7243	6680	6680
q2	2089	176	184	176
q3	10614	1110	1171	1110
q4	10548	800	809	800
q5	7738	2908	2922	2908
q6	222	135	133	133
q7	990	638	624	624
q8	9371	1977	2102	1977
q9	6636	6427	6411	6411
q10	7005	2300	2339	2300
q11	457	263	269	263
q12	414	215	213	213
q13	17765	3010	2999	2999
q14	240	207	221	207
q15	519	478	464	464
q16	677	574	590	574
q17	1010	552	545	545
q18	7400	6698	6624	6624
q19	1412	1139	1122	1122
q20	489	208	217	208
q21	4100	3427	3385	3385
q22	1081	988	968	968
Total cold run time: 108428 ms
Total hot run time: 40691 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6684	6628	6605	6605
q2	337	232	233	232
q3	2936	2910	3015	2910
q4	2118	1813	1837	1813
q5	5856	5770	5753	5753
q6	211	127	130	127
q7	2273	1827	1838	1827
q8	3415	3596	3562	3562
q9	8784	8877	8934	8877
q10	3567	3564	3504	3504
q11	587	488	497	488
q12	790	628	615	615
q13	9358	3126	3240	3126
q14	314	264	279	264
q15	510	464	462	462
q16	706	671	654	654
q17	1849	1662	1622	1622
q18	8246	7892	7712	7712
q19	1716	1625	1586	1586
q20	2150	1829	1817	1817
q21	5615	5442	5467	5442
q22	1112	1059	993	993
Total cold run time: 69134 ms
Total hot run time: 59991 ms

@doris-robot
Copy link

doris-robot commented Apr 30, 2025

BE UT Coverage Report

Increment line coverage 22.86% (8/35) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.56% (10661/26287)
Line Coverage 31.30% (90422/288925)
Region Coverage 30.43% (46685/153410)
Branch Coverage 26.92% (23879/88712)

@doris-robot
Copy link

doris-robot commented Apr 30, 2025

TPC-DS: Total hot run time: 197477 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dab8723e091be87ccadc35e5b55272cd81026888, data reload: false

query1	1309	926	901	901
query2	6151	2029	1959	1959
query3	10851	4404	4491	4404
query4	61054	28965	23512	23512
query5	5226	481	461	461
query6	405	182	180	180
query7	5493	327	326	326
query8	304	228	215	215
query9	8627	2632	2621	2621
query10	473	279	269	269
query11	17758	15255	15668	15255
query12	163	103	105	103
query13	1451	465	442	442
query14	10444	7159	7346	7159
query15	206	177	181	177
query16	7037	500	512	500
query17	1169	600	611	600
query18	1864	327	324	324
query19	214	165	150	150
query20	117	108	110	108
query21	209	101	102	101
query22	4760	4366	4640	4366
query23	35006	34290	33820	33820
query24	6148	2980	2930	2930
query25	495	393	400	393
query26	697	172	169	169
query27	1959	364	352	352
query28	4281	2460	2477	2460
query29	695	466	440	440
query30	243	162	161	161
query31	990	825	837	825
query32	74	55	60	55
query33	472	301	313	301
query34	922	521	527	521
query35	862	733	716	716
query36	1066	976	955	955
query37	123	72	72	72
query38	4087	3930	4005	3930
query39	1614	1495	1475	1475
query40	211	119	106	106
query41	52	54	49	49
query42	118	108	104	104
query43	536	483	476	476
query44	1219	813	825	813
query45	197	170	180	170
query46	1176	718	749	718
query47	1982	1891	1892	1891
query48	487	392	385	385
query49	727	426	430	426
query50	850	452	459	452
query51	7323	7289	7191	7191
query52	117	95	94	94
query53	273	191	196	191
query54	592	496	490	490
query55	83	85	86	85
query56	289	270	267	267
query57	1235	1142	1139	1139
query58	232	227	252	227
query59	3153	2981	3081	2981
query60	317	274	265	265
query61	121	150	112	112
query62	768	677	701	677
query63	227	204	205	204
query64	1444	688	642	642
query65	3374	3238	3215	3215
query66	722	291	304	291
query67	15775	15496	15505	15496
query68	4226	571	578	571
query69	444	275	271	271
query70	1195	1117	1082	1082
query71	358	263	273	263
query72	6164	4052	4013	4013
query73	756	365	366	365
query74	9484	9062	9019	9019
query75	3319	2657	2668	2657
query76	1912	1057	1038	1038
query77	505	275	279	275
query78	10563	9638	9524	9524
query79	2288	656	635	635
query80	1376	440	430	430
query81	534	245	237	237
query82	1285	92	91	91
query83	281	144	147	144
query84	280	76	77	76
query85	1026	319	304	304
query86	380	305	294	294
query87	4394	4232	4237	4232
query88	3725	2431	2402	2402
query89	411	299	295	295
query90	1989	185	186	185
query91	185	151	147	147
query92	63	50	51	50
query93	2960	576	567	567
query94	762	299	291	291
query95	361	268	259	259
query96	620	284	282	282
query97	3335	3119	3147	3119
query98	211	209	202	202
query99	1576	1339	1292	1292
Total cold run time: 315587 ms
Total hot run time: 197477 ms

@doris-robot
Copy link

doris-robot commented Apr 30, 2025

ClickBench: Total hot run time: 32.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dab8723e091be87ccadc35e5b55272cd81026888, data reload: false

query1	0.03	0.03	0.04
query2	0.09	0.04	0.05
query3	0.23	0.06	0.06
query4	1.63	0.09	0.10
query5	0.52	0.50	0.50
query6	1.14	0.75	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.51
query10	0.55	0.58	0.54
query11	0.16	0.11	0.12
query12	0.16	0.13	0.12
query13	0.62	0.59	0.60
query14	2.75	2.73	2.76
query15	0.92	0.85	0.84
query16	0.38	0.38	0.37
query17	1.06	0.99	0.98
query18	0.21	0.20	0.20
query19	1.84	1.80	2.11
query20	0.02	0.02	0.01
query21	15.38	0.68	0.66
query22	4.42	7.31	2.01
query23	18.19	1.37	1.26
query24	2.24	0.22	0.22
query25	0.14	0.08	0.09
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.28	0.61	0.60
query29	12.68	3.42	3.37
query30	0.24	0.06	0.06
query31	2.85	0.41	0.42
query32	3.23	0.50	0.49
query33	3.00	3.00	3.04
query34	17.17	4.54	4.53
query35	4.53	4.57	4.56
query36	0.66	0.48	0.48
query37	0.19	0.17	0.17
query38	0.16	0.15	0.15
query39	0.06	0.04	0.04
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.07	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 112.08 s
Total hot run time: 32.92 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Thearas @doris-robot @dataroaring @BePPPower