[enchement](iceberg)support read iceberg partition evolution table. #45367

hubgeter · 2024-12-12T14:25:39Z

What problem does this PR solve?

Problem Summary:
Supports reading iceberg tables after ADD DROP REPLACE partition. Before this, reading such tables will cause be core.

https://iceberg.apache.org/docs/1.7.0/spark-ddl/#alter-table-add-partition-field

Release note

Supports reading Iceberg tables after partition evolution.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2024-12-12T14:25:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hubgeter · 2024-12-12T14:26:37Z

run buildall

hubgeter · 2024-12-12T14:35:51Z

run buildall

doris-robot · 2024-12-12T14:54:42Z

TPC-H: Total hot run time: 39810 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e153e577e8da3fb64795f062406464cc46f96953, data reload: false

------ Round 1 ----------------------------------
q1	17607	7419	7267	7267
q2	2046	177	173	173
q3	10536	1129	1122	1122
q4	10231	730	670	670
q5	7595	2693	2697	2693
q6	236	155	152	152
q7	977	619	621	619
q8	9240	1828	1862	1828
q9	6606	6429	6464	6429
q10	7037	2285	2302	2285
q11	472	254	263	254
q12	436	223	225	223
q13	17782	2981	2988	2981
q14	235	207	208	207
q15	573	545	529	529
q16	654	602	596	596
q17	991	582	528	528
q18	7220	6955	6675	6675
q19	1346	1010	936	936
q20	458	187	190	187
q21	4251	3142	3403	3142
q22	388	315	314	314
Total cold run time: 106917 ms
Total hot run time: 39810 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7240	7248	7231	7231
q2	330	236	232	232
q3	2910	2764	3086	2764
q4	2132	1870	1868	1868
q5	5555	5715	5599	5599
q6	230	148	142	142
q7	2237	1774	1834	1774
q8	3401	3534	3493	3493
q9	8954	8985	8978	8978
q10	3586	3541	3540	3540
q11	600	512	502	502
q12	843	611	635	611
q13	12208	3131	3167	3131
q14	305	277	263	263
q15	591	540	520	520
q16	648	644	654	644
q17	1781	1600	1572	1572
q18	7888	7317	7271	7271
q19	1703	1506	1521	1506
q20	2025	1821	1788	1788
q21	5432	5262	5346	5262
q22	627	561	546	546
Total cold run time: 71226 ms
Total hot run time: 59237 ms

doris-robot · 2024-12-12T15:06:42Z

TPC-DS: Total hot run time: 189440 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e153e577e8da3fb64795f062406464cc46f96953, data reload: false

query1	972	384	376	376
query2	6516	2196	2074	2074
query3	6713	218	225	218
query4	33935	23346	23395	23346
query5	4374	484	472	472
query6	291	184	194	184
query7	4618	301	316	301
query8	307	239	243	239
query9	9612	2730	2708	2708
query10	493	247	234	234
query11	18001	15472	15070	15070
query12	160	99	99	99
query13	1643	424	418	418
query14	10240	6653	7106	6653
query15	243	175	176	175
query16	8181	452	465	452
query17	1639	570	557	557
query18	2128	292	289	289
query19	363	152	151	151
query20	118	114	120	114
query21	222	104	103	103
query22	4528	4106	4318	4106
query23	34962	33857	33716	33716
query24	10482	2485	2410	2410
query25	563	362	378	362
query26	1051	148	151	148
query27	2802	278	302	278
query28	8001	2447	2421	2421
query29	658	400	411	400
query30	297	151	146	146
query31	1013	812	817	812
query32	89	58	56	56
query33	772	299	289	289
query34	943	505	512	505
query35	951	758	721	721
query36	1091	949	937	937
query37	117	75	74	74
query38	4515	4256	4268	4256
query39	1476	1377	1386	1377
query40	276	99	99	99
query41	46	43	70	43
query42	112	100	100	100
query43	540	477	504	477
query44	1273	840	823	823
query45	188	162	168	162
query46	1148	702	722	702
query47	1939	1864	1870	1864
query48	405	315	318	315
query49	1165	395	389	389
query50	805	395	388	388
query51	7275	7003	7243	7003
query52	98	94	89	89
query53	279	184	180	180
query54	1213	399	417	399
query55	84	76	76	76
query56	265	241	243	241
query57	1236	1157	1086	1086
query58	230	214	229	214
query59	3308	2979	3266	2979
query60	270	246	255	246
query61	110	107	109	107
query62	900	667	679	667
query63	215	184	187	184
query64	5261	779	733	733
query65	3242	3265	3231	3231
query66	1436	309	317	309
query67	15951	15755	15358	15358
query68	4488	570	562	562
query69	417	250	254	250
query70	1197	1124	1135	1124
query71	342	241	250	241
query72	6253	3819	4047	3819
query73	753	363	372	363
query74	10056	8930	8940	8930
query75	3393	2628	2667	2628
query76	2907	1107	1224	1107
query77	459	286	340	286
query78	10280	9391	9513	9391
query79	1529	593	609	593
query80	1064	435	441	435
query81	518	230	231	230
query82	928	120	121	120
query83	251	145	142	142
query84	237	73	74	73
query85	1278	309	383	309
query86	352	304	302	302
query87	4745	4471	4576	4471
query88	3328	2329	2203	2203
query89	399	288	303	288
query90	1975	184	194	184
query91	141	104	106	104
query92	59	52	47	47
query93	1080	545	553	545
query94	974	297	290	290
query95	353	251	256	251
query96	625	283	277	277
query97	2864	2675	2696	2675
query98	211	203	195	195
query99	1550	1308	1294	1294
Total cold run time: 299356 ms
Total hot run time: 189440 ms

doris-robot · 2024-12-12T15:12:10Z

ClickBench: Total hot run time: 32.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e153e577e8da3fb64795f062406464cc46f96953, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.45	0.42	0.40
query6	1.14	0.65	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.54	0.49
query10	0.55	0.57	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.61
query14	2.72	2.76	2.73
query15	0.90	0.84	0.82
query16	0.38	0.39	0.38
query17	1.07	1.06	1.07
query18	0.23	0.22	0.20
query19	1.97	1.86	2.03
query20	0.02	0.01	0.02
query21	15.36	0.59	0.58
query22	2.98	2.19	1.76
query23	17.17	0.83	0.79
query24	2.63	0.75	1.72
query25	0.29	0.29	0.04
query26	0.43	0.13	0.12
query27	0.03	0.04	0.04
query28	10.67	1.11	1.07
query29	12.54	3.19	3.21
query30	0.25	0.06	0.06
query31	2.86	0.37	0.38
query32	3.28	0.46	0.46
query33	3.06	3.03	3.02
query34	17.05	4.39	4.43
query35	4.52	4.42	4.43
query36	0.67	0.47	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 107.14 s
Total hot run time: 32.09 s

morningman

LGTM

github-actions · 2024-12-13T02:51:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-13T02:51:04Z

PR approved by anyone and no changes requested.

hubgeter · 2024-12-13T07:58:18Z

run buildall

hubgeter · 2024-12-13T10:38:25Z

run buildall

github-actions · 2024-12-13T10:42:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-13T11:16:30Z

TeamCity be ut coverage result:
Function Coverage: 38.75% (10107/26082)
Line Coverage: 29.68% (84819/285766)
Region Coverage: 28.75% (43547/151449)
Branch Coverage: 25.31% (22121/87416)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fd339bca843f70b3634f01afd2f702134b6e3097_fd339bca843f70b3634f01afd2f702134b6e3097/report/index.html

hubgeter · 2024-12-15T13:57:52Z

run buildall

doris-robot · 2024-12-15T14:37:21Z

TeamCity be ut coverage result:
Function Coverage: 38.76% (10109/26082)
Line Coverage: 29.70% (84885/285787)
Region Coverage: 28.77% (43570/151462)
Branch Coverage: 25.32% (22138/87426)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fd339bca843f70b3634f01afd2f702134b6e3097_fd339bca843f70b3634f01afd2f702134b6e3097/report/index.html

hubgeter · 2024-12-15T15:01:56Z

run buildall

github-actions · 2024-12-15T15:06:22Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-15T15:30:10Z

TeamCity be ut coverage result:
Function Coverage: 38.57% (10061/26082)
Line Coverage: 29.49% (84277/285787)
Region Coverage: 28.61% (43334/151462)
Branch Coverage: 25.21% (22039/87426)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b8fd474a55c3d57f729953728f2dfc9b1a314b3d_b8fd474a55c3d57f729953728f2dfc9b1a314b3d/report/index.html

github-actions · 2024-12-17T09:25:06Z

PR approved by at least one committer and no changes requested.

…45367) ### What problem does this PR solve? Problem Summary: Supports reading iceberg tables after `ADD DROP REPLACE` partition. Before this, reading such tables will cause be core. https://iceberg.apache.org/docs/1.7.0/spark-ddl/#alter-table-add-partition-field ### Release note [fix](iceberg) Supports reading Iceberg tables after partition evolution.

…pache#45367) ### What problem does this PR solve? Problem Summary: Supports reading iceberg tables after `ADD DROP REPLACE` partition. Before this, reading such tables will cause be core. https://iceberg.apache.org/docs/1.7.0/spark-ddl/#alter-table-add-partition-field ### Release note [fix](iceberg) Supports reading Iceberg tables after partition evolution.

…45367) (#45569) cherry-pick #45367 Co-authored-by: daidai <changyuwei@selectdb.com>

…ion table. #45367 (#45563) Cherry-picked from #45367 Co-authored-by: daidai <changyuwei@selectdb.com>

morningman previously approved these changes Dec 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 13, 2024

github-actions bot added the reviewed label Dec 13, 2024

hubgeter dismissed morningman’s stale review via 6992805 December 13, 2024 07:58

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 13, 2024

hubgeter added 6 commits December 15, 2024 23:01

[enchement](iceberg)support read iceberg partition evolution table

48d9fe1

fix case

ffc4b74

fix log

f9425a9

fix case/

c255b58

fix iceberg write

2eeb69e

fix case

b8fd474

hubgeter force-pushed the iceberg_partition_evolution branch from fd339bc to b8fd474 Compare December 15, 2024 15:01

morningman added the dev/2.1.x label Dec 17, 2024

morningman added the dev/3.0.x label Dec 17, 2024

morningman approved these changes Dec 17, 2024

View reviewed changes

wuwenchi approved these changes Dec 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 17, 2024

morningman merged commit a6da226 into apache:master Dec 17, 2024
24 of 30 checks passed

github-actions bot mentioned this pull request Dec 17, 2024

branch-3.0: [enchement](iceberg)support read iceberg partition evolution table. #45367 #45563

Merged

github-actions bot added the dev/2.1.x-conflict label Dec 17, 2024

morningman mentioned this pull request Dec 18, 2024

[enchement](iceberg)support read iceberg partition evolution table. (#45367) #45569

Merged

morningman added a commit that referenced this pull request Dec 20, 2024

[enchement](iceberg)support read iceberg partition evolution table. (#…

19c0e89

…45367) (#45569) cherry-pick #45367 Co-authored-by: daidai <changyuwei@selectdb.com>

morningman added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Dec 20, 2024

morningman pushed a commit that referenced this pull request Dec 20, 2024

branch-3.0: [enchement](iceberg)support read iceberg partition evolut…

0671640

…ion table. #45367 (#45563) Cherry-picked from #45367 Co-authored-by: daidai <changyuwei@selectdb.com>

morningman added dev/3.0.4-merged and removed dev/3.0.x labels Dec 20, 2024

yiguolei mentioned this pull request Jan 19, 2025

Release Note 2.1.8 #47198

Closed

wm1581066 added the usercase Important user case type label label Feb 6, 2025

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

suxiaogang223 mentioned this pull request Nov 7, 2025

branch-3.0: [fix](iceberg) support reading iceberg partition table without partition value stored in file #57795

Open

16 tasks

[enchement](iceberg)support read iceberg partition evolution table. #45367

[enchement](iceberg)support read iceberg partition evolution table. #45367

Uh oh!

Conversation

hubgeter commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 12, 2024

Uh oh!

hubgeter commented Dec 12, 2024

Uh oh!

hubgeter commented Dec 12, 2024

Uh oh!

doris-robot commented Dec 12, 2024

Uh oh!

doris-robot commented Dec 12, 2024

Uh oh!

doris-robot commented Dec 12, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 13, 2024

Uh oh!

github-actions bot commented Dec 13, 2024

Uh oh!

hubgeter commented Dec 13, 2024

Uh oh!

hubgeter commented Dec 13, 2024

Uh oh!

github-actions bot commented Dec 13, 2024

Uh oh!

doris-robot commented Dec 13, 2024

Uh oh!

hubgeter commented Dec 15, 2024

Uh oh!

doris-robot commented Dec 15, 2024

Uh oh!

hubgeter commented Dec 15, 2024

Uh oh!

github-actions bot commented Dec 15, 2024

Uh oh!

doris-robot commented Dec 15, 2024

Uh oh!

github-actions bot commented Dec 17, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hubgeter commented Dec 12, 2024 •

edited

Loading