[opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #33786

suxiaogang223 · 2024-04-17T09:53:23Z

Proposed changes

Followup: #25364
support mapping Paimon column type "Row" to Doris type "Struct"

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-17T09:53:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman

Please update document and test case

suxiaogang223 · 2024-04-19T09:16:40Z

update doc apache/doris-website#577

suxiaogang223 · 2024-04-27T18:17:08Z

run buildall

doris-robot · 2024-04-27T19:05:30Z

TPC-DS: Total hot run time: 188064 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac9bce385a7e1a93d7cd161e1fbb0fce66ebb6e5, data reload: false

query1	909	370	357	357
query2	6778	2285	2295	2285
query3	6823	206	209	206
query4	23560	21970	22080	21970
query5	4409	417	428	417
query6	275	179	172	172
query7	4573	280	280	280
query8	255	187	178	178
query9	8647	2346	2334	2334
query10	413	229	243	229
query11	15285	14807	15015	14807
query12	156	88	93	88
query13	1633	363	373	363
query14	10791	7697	8480	7697
query15	251	182	177	177
query16	7698	266	256	256
query17	2217	600	587	587
query18	2127	304	300	300
query19	319	169	171	169
query20	93	93	88	88
query21	216	151	145	145
query22	5406	5127	5179	5127
query23	34477	33235	33181	33181
query24	10627	2876	2829	2829
query25	590	360	367	360
query26	1156	146	139	139
query27	2339	313	323	313
query28	6859	1989	1995	1989
query29	887	619	614	614
query30	206	156	154	154
query31	950	720	717	717
query32	81	48	50	48
query33	740	239	229	229
query34	1044	467	468	467
query35	826	672	674	672
query36	1046	862	876	862
query37	145	62	64	62
query38	3166	3013	3017	3013
query39	1578	1521	1539	1521
query40	199	129	122	122
query41	40	38	37	37
query42	102	99	98	98
query43	562	551	544	544
query44	1240	745	739	739
query45	272	266	259	259
query46	1088	725	722	722
query47	1951	1874	1840	1840
query48	365	286	288	286
query49	810	416	380	380
query50	753	377	365	365
query51	7011	6766	6774	6766
query52	99	92	83	83
query53	355	284	276	276
query54	312	229	225	225
query55	75	69	70	69
query56	231	216	214	214
query57	1199	1116	1121	1116
query58	216	188	191	188
query59	3469	3005	3024	3005
query60	257	224	226	224
query61	87	88	87	87
query62	612	444	448	444
query63	301	273	273	273
query64	8585	7258	7162	7162
query65	3163	3034	3048	3034
query66	750	324	328	324
query67	15481	15305	14817	14817
query68	5251	534	544	534
query69	497	296	289	289
query70	1140	1107	1157	1107
query71	408	264	270	264
query72	7905	2605	2417	2417
query73	706	326	327	326
query74	6625	6467	6421	6421
query75	3744	2654	2693	2654
query76	3399	935	1013	935
query77	649	264	253	253
query78	11259	10345	10421	10345
query79	5240	511	526	511
query80	2124	490	424	424
query81	541	232	227	227
query82	819	98	92	92
query83	313	166	164	164
query84	269	84	83	83
query85	1932	277	255	255
query86	474	311	305	305
query87	3346	3118	3122	3118
query88	5049	2298	2304	2298
query89	519	384	373	373
query90	1983	177	182	177
query91	127	95	98	95
query92	70	46	45	45
query93	5198	518	511	511
query94	1202	179	177	177
query95	382	292	306	292
query96	614	263	253	253
query97	3156	2948	2948	2948
query98	245	216	221	216
query99	1243	866	854	854
Total cold run time: 295565 ms
Total hot run time: 188064 ms

github-actions · 2024-04-28T02:23:39Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-28T02:23:41Z

PR approved by anyone and no changes requested.

suxiaogang223 · 2024-04-28T02:58:47Z

run buildall

suxiaogang223 · 2024-04-28T03:13:01Z

run buildall

doris-robot · 2024-04-28T03:44:52Z

TPC-H: Total hot run time: 41291 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84a68808c42b67604b2ea3445f7b76eeba674fda, data reload: false

------ Round 1 ----------------------------------
q1	17629	4286	4211	4211
q2	2013	191	194	191
q3	10471	1245	1279	1245
q4	10197	827	778	778
q5	7517	2750	2791	2750
q6	223	130	129	129
q7	1056	623	622	622
q8	9236	2133	2094	2094
q9	9544	6799	6776	6776
q10	8956	3989	3905	3905
q11	442	243	241	241
q12	454	236	235	235
q13	18159	3144	3178	3144
q14	256	232	230	230
q15	500	465	485	465
q16	496	408	398	398
q17	956	714	759	714
q18	8390	7808	7722	7722
q19	4177	1538	1482	1482
q20	650	329	316	316
q21	5091	3373	3461	3373
q22	357	278	270	270
Total cold run time: 116770 ms
Total hot run time: 41291 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4500	4460	4329	4329
q2	380	274	272	272
q3	3169	2926	2859	2859
q4	1952	1692	1588	1588
q5	5318	5540	5523	5523
q6	209	122	119	119
q7	2338	1977	1981	1977
q8	3236	3424	3396	3396
q9	8816	8846	8949	8846
q10	4070	3819	3691	3691
q11	589	491	494	491
q12	853	627	627	627
q13	17045	3119	3116	3116
q14	328	297	281	281
q15	525	493	486	486
q16	511	428	441	428
q17	1786	1507	1496	1496
q18	7670	7476	7458	7458
q19	1633	1551	1494	1494
q20	1997	1763	1763	1763
q21	9617	4683	4877	4683
q22	579	476	472	472
Total cold run time: 77121 ms
Total hot run time: 55395 ms

doris-robot · 2024-04-28T03:56:16Z

TPC-DS: Total hot run time: 186184 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84a68808c42b67604b2ea3445f7b76eeba674fda, data reload: false

query1	916	357	347	347
query2	6437	2326	2218	2218
query3	6645	212	207	207
query4	23544	21410	21063	21063
query5	4184	421	426	421
query6	260	172	165	165
query7	4588	290	280	280
query8	242	194	186	186
query9	8390	2345	2333	2333
query10	439	242	250	242
query11	14769	14279	14244	14244
query12	135	91	90	90
query13	1668	367	367	367
query14	8585	7778	8100	7778
query15	212	174	175	174
query16	7127	268	263	263
query17	1020	566	552	552
query18	1930	287	276	276
query19	196	169	151	151
query20	99	95	84	84
query21	197	127	132	127
query22	5131	4909	4812	4812
query23	33761	33292	33255	33255
query24	5038	2936	2990	2936
query25	467	393	360	360
query26	686	165	148	148
query27	1834	322	323	322
query28	3747	2016	2012	2012
query29	826	603	595	595
query30	209	151	150	150
query31	915	731	713	713
query32	58	51	53	51
query33	374	267	240	240
query34	862	475	471	471
query35	751	688	659	659
query36	1066	918	911	911
query37	98	69	67	67
query38	3128	3044	3024	3024
query39	1613	1542	1539	1539
query40	193	123	128	123
query41	41	39	38	38
query42	105	98	93	93
query43	554	532	555	532
query44	1042	724	742	724
query45	292	273	254	254
query46	1069	701	755	701
query47	1929	1885	1856	1856
query48	375	292	288	288
query49	747	387	393	387
query50	782	382	372	372
query51	6792	6763	6799	6763
query52	102	88	90	88
query53	348	273	276	273
query54	254	232	230	230
query55	73	70	71	70
query56	240	219	220	219
query57	1175	1140	1153	1140
query58	209	192	190	190
query59	3333	3255	3032	3032
query60	255	255	235	235
query61	91	87	101	87
query62	556	431	428	428
query63	304	276	281	276
query64	7439	7188	7122	7122
query65	3123	3011	3024	3011
query66	666	348	332	332
query67	15411	15188	14965	14965
query68	5227	563	530	530
query69	498	302	302	302
query70	1183	1159	1165	1159
query71	440	269	263	263
query72	7734	2658	2401	2401
query73	693	320	321	320
query74	6454	6191	6072	6072
query75	3389	2674	2660	2660
query76	2747	981	879	879
query77	383	278	259	259
query78	10994	10376	10254	10254
query79	6671	523	518	518
query80	1005	459	439	439
query81	485	223	218	218
query82	1151	97	97	97
query83	232	176	168	168
query84	274	83	83	83
query85	897	276	265	265
query86	449	313	303	303
query87	3238	3080	3131	3080
query88	4925	2312	2320	2312
query89	507	377	366	366
query90	2100	183	182	182
query91	124	116	105	105
query92	69	47	46	46
query93	6232	527	512	512
query94	1177	180	177	177
query95	371	298	295	295
query96	614	259	258	258
query97	3096	2947	2934	2934
query98	231	219	214	214
query99	1195	851	874	851
Total cold run time: 274432 ms
Total hot run time: 186184 ms

morningman

LGTM

github-actions · 2024-04-28T04:25:03Z

PR approved by at least one committer and no changes requested.

…"Struct" (#34239) backport: #33786

…"Struct" #33786 (#34242) backport: #33786

…"Struct" apache#33786 (apache#34242) backport: apache#33786

xiaokang added the dev/2.0.x label Apr 18, 2024

xiaokang changed the title ~~[opt](paimon) support map row of paimon to struct of doris~~ [opt](paimon) support mapping row of paimon to struct of doris Apr 18, 2024

morningman changed the title ~~[opt](paimon) support mapping row of paimon to struct of doris~~ [opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" Apr 19, 2024

morningman reviewed Apr 19, 2024

View reviewed changes

morningman added the dev/2.1.x label Apr 26, 2024

support map row of paimon to struct of doris

3c4aeb2

suxiaogang223 force-pushed the paimon_struct_type branch from c8906e4 to 3c4aeb2 Compare April 26, 2024 08:51

suxiaogang223 added 2 commits April 28, 2024 01:51

change unpackStruct as Hidi impl

353d648

add regress-test

ac9bce3

AshinGau previously approved these changes Apr 28, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 28, 2024

set force_jni_scanner=true to make sure use PaimonJniReader

9d8a033

suxiaogang223 dismissed AshinGau’s stale review via 9d8a033 April 28, 2024 02:54

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 28, 2024

remove unuseful comment

77f6da2

fix

84a6880

morningman approved these changes Apr 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2024

AshinGau approved these changes Apr 28, 2024

View reviewed changes

This was referenced Apr 28, 2024

[opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #34239

Merged

[opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #34242

Merged

morningman merged commit fc7c49e into apache:master Apr 28, 2024

morningman pushed a commit that referenced this pull request Apr 28, 2024

[opt](paimon) support mapping Paimon column type "Row" to Doris type …

11039ad

…"Struct" (#34239) backport: #33786

morningman added dev/2.1.3-merged and removed dev/2.1.x labels Apr 28, 2024

morningman pushed a commit that referenced this pull request Apr 28, 2024

[opt](paimon) support mapping Paimon column type "Row" to Doris type …

7237743

…"Struct" #33786 (#34242) backport: #33786

morningman added dev/2.0.10-merged and removed dev/2.0.x labels Apr 28, 2024

suxiaogang223 deleted the paimon_struct_type branch August 15, 2024 08:51

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[opt](paimon) support mapping Paimon column type "Row" to Doris type …

4f6f709

…"Struct" apache#33786 (apache#34242) backport: apache#33786

[opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #33786

[opt](paimon) support mapping Paimon column type "Row" to Doris type "Struct" #33786

Uh oh!

Conversation

suxiaogang223 commented Apr 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

suxiaogang223 commented Apr 19, 2024

Uh oh!

suxiaogang223 commented Apr 27, 2024

Uh oh!

doris-robot commented Apr 27, 2024

Uh oh!

github-actions bot commented Apr 28, 2024

Uh oh!

github-actions bot commented Apr 28, 2024

Uh oh!

suxiaogang223 commented Apr 28, 2024

Uh oh!

suxiaogang223 commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

suxiaogang223 commented Apr 17, 2024 •

edited

Loading