[feature](iceberg)support iceberg insert grammar #34257

wuwenchi · 2024-04-28T12:14:26Z

Proposed changes

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-28T12:14:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-04-28T12:19:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-29T03:54:02Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-06T06:33:29Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-06T06:47:32Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-06T07:13:31Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-06T08:00:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-10T03:29:20Z

clang-tidy review says "All clean, LGTM! 👍"

refactor

wuwenchi · 2024-05-16T06:45:56Z

run buildall

github-actions · 2024-05-16T06:51:04Z

clang-tidy review says "All clean, LGTM! 👍"

wuwenchi · 2024-05-16T09:32:16Z

run buildall

github-actions · 2024-05-16T09:37:07Z

clang-tidy review says "All clean, LGTM! 👍"

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java

fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java

...va/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java

fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java

github-actions · 2024-05-16T12:38:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-05-16T13:11:14Z

clang-tidy review says "All clean, LGTM! 👍"

wuwenchi · 2024-05-16T13:12:40Z

run buildall

wuwenchi · 2024-05-17T03:26:13Z

run buildall

github-actions · 2024-05-17T03:31:40Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-17T04:54:47Z

TPC-DS: Total hot run time: 187209 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86a406858bf5f0101df4a8e0be10483dbaa10b44, data reload: false

query1	917	397	367	367
query2	6240	2397	2168	2168
query3	6653	209	209	209
query4	23234	21752	21988	21752
query5	3835	445	422	422
query6	280	189	173	173
query7	4546	292	292	292
query8	236	184	207	184
query9	8442	2418	2403	2403
query10	408	269	242	242
query11	15438	14859	14823	14823
query12	122	85	86	85
query13	1605	393	364	364
query14	9433	8324	6774	6774
query15	241	172	163	163
query16	8148	254	248	248
query17	1731	561	548	548
query18	2097	278	283	278
query19	232	143	145	143
query20	86	81	83	81
query21	197	126	123	123
query22	5131	4884	4810	4810
query23	34377	33666	33640	33640
query24	10865	2889	2919	2889
query25	610	364	368	364
query26	1306	158	150	150
query27	3056	321	314	314
query28	7275	2004	2011	2004
query29	857	642	600	600
query30	237	171	169	169
query31	987	741	736	736
query32	91	48	52	48
query33	734	250	241	241
query34	1057	472	471	471
query35	803	664	676	664
query36	1072	965	928	928
query37	144	67	70	67
query38	2905	2763	2807	2763
query39	1609	1563	1552	1552
query40	203	122	123	122
query41	45	43	48	43
query42	98	100	98	98
query43	576	563	519	519
query44	1187	716	733	716
query45	261	254	252	252
query46	1077	722	722	722
query47	1970	1860	1877	1860
query48	366	300	299	299
query49	881	417	380	380
query50	745	380	371	371
query51	6814	6648	6697	6648
query52	99	94	90	90
query53	353	276	277	276
query54	860	418	430	418
query55	78	70	72	70
query56	235	211	223	211
query57	1220	1151	1147	1147
query58	216	210	196	196
query59	3367	3308	3162	3162
query60	255	229	237	229
query61	90	91	86	86
query62	669	473	467	467
query63	310	278	283	278
query64	8738	7352	7365	7352
query65	3146	3100	3121	3100
query66	1370	326	343	326
query67	15707	15124	15060	15060
query68	4609	523	528	523
query69	525	300	310	300
query70	1190	1137	1144	1137
query71	428	275	264	264
query72	8099	2590	2413	2413
query73	719	315	321	315
query74	6514	6075	6183	6075
query75	3649	2660	2610	2610
query76	3324	989	1004	989
query77	573	261	265	261
query78	10573	10176	10103	10103
query79	2476	508	504	504
query80	1091	435	430	430
query81	540	246	247	246
query82	1335	94	93	93
query83	273	167	166	166
query84	242	96	83	83
query85	1532	263	258	258
query86	500	308	325	308
query87	3302	3111	3101	3101
query88	3946	2285	2286	2285
query89	457	377	375	375
query90	1941	186	184	184
query91	126	97	98	97
query92	60	51	48	48
query93	2144	487	482	482
query94	1193	185	184	184
query95	405	296	290	290
query96	594	269	258	258
query97	3199	2975	3016	2975
query98	239	216	222	216
query99	1219	886	914	886
Total cold run time: 285190 ms
Total hot run time: 187209 ms

doris-robot · 2024-05-17T05:00:18Z

ClickBench: Total hot run time: 30.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86a406858bf5f0101df4a8e0be10483dbaa10b44, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.48
query10	0.54	0.54	0.52
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.60	0.59	0.60
query14	0.76	0.78	0.78
query15	0.82	0.82	0.81
query16	0.36	0.36	0.36
query17	1.04	0.99	1.01
query18	0.22	0.22	0.27
query19	1.79	1.79	1.79
query20	0.02	0.01	0.01
query21	15.73	0.65	0.65
query22	4.47	6.97	2.19
query23	18.28	1.44	1.22
query24	1.96	0.23	0.20
query25	0.14	0.08	0.08
query26	0.27	0.17	0.18
query27	0.08	0.09	0.08
query28	13.33	1.01	0.99
query29	13.81	3.32	3.27
query30	0.24	0.06	0.05
query31	2.92	0.40	0.38
query32	3.24	0.48	0.47
query33	2.76	2.86	2.81
query34	17.19	4.41	4.53
query35	4.48	4.51	4.65
query36	0.64	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.15
query41	0.10	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.97 s
Total hot run time: 30.79 s

wuwenchi · 2024-05-17T07:42:58Z

run buildall

github-actions · 2024-05-17T07:46:59Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

github-actions · 2024-05-17T15:17:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-17T15:17:25Z

PR approved by anyone and no changes requested.

kaka11chen

LGTM

(cherry picked from commit 09cb8b9)

…3629 bp: #34257 #33629

wuwenchi marked this pull request as draft April 28, 2024 12:14

github-actions bot added the meta-change label Apr 28, 2024

wuwenchi force-pushed the support-iceberg-sql branch from 88fe782 to 3eb24f3 Compare May 6, 2024 06:42

wuwenchi force-pushed the support-iceberg-sql branch from c9cd995 to eb3299a Compare May 10, 2024 03:23

wuwenchi added 6 commits May 16, 2024 11:36

fix

00c0520

refactor

cb746f2

refactor

add data location

3dbb70e

add compression type

c619ef0

modify file and compression type

901b6f0

fix

5cf9d92

wuwenchi force-pushed the support-iceberg-sql branch from 1845d0b to 5cf9d92 Compare May 16, 2024 06:45

wuwenchi marked this pull request as ready for review May 16, 2024 06:45

fix

7de4e62

morningman reviewed May 16, 2024

View reviewed changes

fix

f2beced

fix

38b8e98

fix

86a4068

fix test

7b45515

morningman approved these changes May 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 17, 2024

github-actions bot added the reviewed label May 17, 2024

morningman added dev/2.1.x dev/3.0.x labels May 17, 2024

kaka11chen approved these changes May 19, 2024

View reviewed changes

morningman merged commit 09cb8b9 into apache:master May 20, 2024

wuwenchi added a commit to wuwenchi/doris that referenced this pull request May 21, 2024

[feature](iceberg)support iceberg insert grammar (apache#34257)

97ce056

(cherry picked from commit 09cb8b9)

wuwenchi mentioned this pull request May 21, 2024

[feature](iceberg)Support write to iceberg for 2.1 #35103

Merged

yiguolei added the 2.1.0-conflict label May 21, 2024

morningman pushed a commit that referenced this pull request May 21, 2024

[feature](iceberg)Support write to iceberg for 2.1 (#35103) #34257 #3…

009ab77

…3629 bp: #34257 #33629

morningman added dev/2.1.4-merged and removed 2.1.0-conflict dev/2.1.x labels May 21, 2024

dataroaring pushed a commit that referenced this pull request May 26, 2024

[feature](iceberg)support iceberg insert grammar (#34257)

d9879fb

gavinchou added dev/3.0.0-merged and removed dev/3.0.x labels Jun 11, 2024

[feature](iceberg)support iceberg insert grammar #34257

[feature](iceberg)support iceberg insert grammar #34257

Uh oh!

Conversation

wuwenchi commented Apr 28, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

github-actions bot commented Apr 28, 2024

Uh oh!

github-actions bot commented Apr 29, 2024

Uh oh!

github-actions bot commented May 6, 2024

Uh oh!

github-actions bot commented May 6, 2024

Uh oh!

github-actions bot commented May 6, 2024

Uh oh!

github-actions bot commented May 6, 2024

Uh oh!

github-actions bot commented May 10, 2024

Uh oh!

wuwenchi commented May 16, 2024

Uh oh!

github-actions bot commented May 16, 2024

Uh oh!

wuwenchi commented May 16, 2024

Uh oh!

github-actions bot commented May 16, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 16, 2024

Uh oh!

github-actions bot commented May 16, 2024

Uh oh!

wuwenchi commented May 16, 2024

Uh oh!

wuwenchi commented May 17, 2024

Uh oh!

github-actions bot commented May 17, 2024

Uh oh!

doris-robot commented May 17, 2024

Uh oh!

doris-robot commented May 17, 2024

Uh oh!

wuwenchi commented May 17, 2024

Uh oh!

github-actions bot commented May 17, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 17, 2024

Uh oh!

github-actions bot commented May 17, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants