Skip to content

[enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases #59716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
suxiaogang223 wants to merge 8 commits into
base: master
Choose a base branch
from
Open

[enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases #59716

suxiaogang223 wants to merge 8 commits into from
+624 −197

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jan 9, 2026 •
edited
Loading

What problem does this PR solve?

Description

Changes

This PR refactors the Iceberg metadata cache structure to improve code organization and adds comprehensive test cases for table cache behavior.

Main Changes

1. Refactored IcebergMetadataCache

  • Introduced IcebergTableCacheValue to encapsulate table-related metadata
  • Removed redundant snapshotListCache and snapshotCache
  • Merged snapshot information into IcebergTableCacheValue with lazy loading
  • Simplified cache structure from 3 separate caches to 2: tableCache and viewCache

Before:

private LoadingCache<IcebergMetadataCacheKey, List<Snapshot>> snapshotListCache;
private LoadingCache<IcebergMetadataCacheKey, Table> tableCache;
private LoadingCache<IcebergMetadataCacheKey, IcebergSnapshotCacheValue> snapshotCache;

After:

private LoadingCache<IcebergMetadataCacheKey, IcebergTableCacheValue> tableCache;
private LoadingCache<IcebergMetadataCacheKey, View> viewCache;

2. Lazy Loading for Snapshot Cache

  • Snapshot cache is now loaded on-demand through IcebergTableCacheValue.getSnapshotCacheValue()
  • Reduced unnecessary memory footprint for queries that don't require snapshot information
  • Snapshot information is mainly used for MTMV scenarios

3. Simplified Cache API

  • getIcebergTable(): Returns the Table object directly from IcebergTableCacheValue
  • getSnapshotCache(): Returns snapshot cache value with lazy loading
  • getSnapshotList(): Returns snapshot list from the Table object

4. Test Cases

  • Added comprehensive test case test_iceberg_table_cache to verify cache behavior
  • Tests cover both cache-enabled and cache-disabled scenarios
  • Validated external modifications (INSERT, DELETE, UPDATE, schema changes) are properly handled

Benefits

Aspect Improvement
Memory Usage Reduced by eliminating duplicate caching of snapshot information
Code Structure Cleaner with single IcebergTableCacheValue instead of multiple separate caches
Performance Better with lazy loading of snapshot cache only when needed
Maintainability Simpler cache management logic

Test Results

  • Added regression test: test_iceberg_table_cache.groovy
  • Tests validate cache behavior with TTL and external modifications
  • Verified cache invalidation works correctly with REFRESH TABLE
  • Test scenarios include:
    • DML operations (INSERT, DELETE, UPDATE, INSERT OVERWRITE)
    • Schema changes (ADD/DROP/RENAME COLUMN, ALTER COLUMN TYPE)
    • Partition evolution (ADD/DROP/REPLACE PARTITION FIELD)

Related Files

Core Changes:

  • IcebergMetadataCache.java - Refactored cache structure
  • IcebergTableCacheValue.java - New class to encapsulate table metadata
  • IcebergExternalCatalog.java - Updated cache-related configurations

Tests:

  • test_iceberg_table_cache.groovy - Comprehensive cache behavior tests
  • Suite.groovy - Updated getSparkIcebergContainerName() implementation

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 9, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223 suxiaogang223 marked this pull request as draft January 9, 2026 08:14
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jan 9, 2026

run external

@suxiaogang223 suxiaogang223 marked this pull request as ready for review January 12, 2026 08:48
@suxiaogang223 suxiaogang223 changed the title Refact meta cache [improvement](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases Jan 12, 2026
@suxiaogang223 suxiaogang223 changed the title [improvement](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases [enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases Jan 12, 2026
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jan 12, 2026

run buildall

@suxiaogang223 suxiaogang223 mentioned this pull request Jan 12, 2026
Open
8 tasks
@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 12, 2026

FE UT Coverage Report

Increment line coverage 0.00% (0/36) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Jan 12, 2026

TPC-H: Total hot run time: 32189 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1574f3a0840831ea973580b5275b27182d07b2bf, data reload: false

------ Round 1 ----------------------------------
q1	17636	4183	4086	4086
q2	2077	377	245	245
q3	10116	1275	724	724
q4	10208	855	309	309
q5	7538	2107	1841	1841
q6	187	168	135	135
q7	911	811	671	671
q8	9273	1414	1159	1159
q9	5100	4604	4559	4559
q10	6839	1852	1436	1436
q11	537	277	275	275
q12	735	756	590	590
q13	17802	3842	3078	3078
q14	297	298	274	274
q15	609	517	510	510
q16	667	684	647	647
q17	676	764	578	578
q18	6747	6583	6973	6583
q19	1355	996	669	669
q20	451	404	269	269
q21	3285	2654	2506	2506
q22	1170	1126	1045	1045
Total cold run time: 104216 ms
Total hot run time: 32189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4248	4257	4248
q2	324	411	333	333
q3	2299	2747	2449	2449
q4	1516	1815	1484	1484
q5	4601	4340	4422	4340
q6	224	177	129	129
q7	1958	1941	1816	1816
q8	2570	2413	2394	2394
q9	7135	7364	7264	7264
q10	2508	2691	2307	2307
q11	579	495	464	464
q12	701	723	561	561
q13	3349	3817	3040	3040
q14	268	289	266	266
q15	525	490	482	482
q16	616	674	616	616
q17	1099	1321	1362	1321
q18	7417	7489	7337	7337
q19	808	774	777	774
q20	1896	1974	1872	1872
q21	4485	4339	4196	4196
q22	1096	1020	995	995
Total cold run time: 50369 ms
Total hot run time: 48688 ms

@doris-robot
Copy link

doris-robot commented Jan 12, 2026

TPC-DS: Total hot run time: 173580 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1574f3a0840831ea973580b5275b27182d07b2bf, data reload: false

query5	4444	598	436	436
query6	334	231	222	222
query7	4218	466	269	269
query8	330	258	246	246
query9	8770	2715	2657	2657
query10	508	354	336	336
query11	15270	15196	14803	14803
query12	173	112	111	111
query13	1261	477	368	368
query14	6214	2967	2704	2704
query14_1	2632	2581	2635	2581
query15	206	194	175	175
query16	1012	462	468	462
query17	1112	690	588	588
query18	2507	450	353	353
query19	229	228	198	198
query20	126	117	115	115
query21	224	145	125	125
query22	3994	3948	3896	3896
query23	16088	15712	15492	15492
query23_1	15825	15645	15676	15645
query24	7694	1600	1175	1175
query24_1	1185	1186	1213	1186
query25	580	521	451	451
query26	1240	281	165	165
query27	2754	481	311	311
query28	4515	2151	2142	2142
query29	796	569	476	476
query30	315	247	211	211
query31	799	639	560	560
query32	82	77	76	76
query33	554	346	300	300
query34	909	886	521	521
query35	718	784	681	681
query36	921	925	824	824
query37	133	95	79	79
query38	2740	2667	2641	2641
query39	797	756	729	729
query39_1	721	726	747	726
query40	229	136	118	118
query41	74	68	69	68
query42	106	103	102	102
query43	446	433	412	412
query44	1324	729	717	717
query45	192	188	181	181
query46	854	950	598	598
query47	1370	1456	1308	1308
query48	319	341	248	248
query49	638	419	353	353
query50	647	280	216	216
query51	3752	3799	3749	3749
query52	106	116	111	111
query53	304	327	274	274
query54	304	268	266	266
query55	81	77	72	72
query56	343	299	278	278
query57	1036	983	954	954
query58	267	246	254	246
query59	2187	2247	2036	2036
query60	316	321	298	298
query61	165	157	153	153
query62	420	352	312	312
query63	294	261	266	261
query64	4860	1316	994	994
query65	3835	3776	3749	3749
query66	1428	420	311	311
query67	15460	16105	15120	15120
query68	6029	992	713	713
query69	517	353	312	312
query70	1076	958	927	927
query71	418	294	276	276
query72	6008	3420	3527	3420
query73	774	721	300	300
query74	8871	8891	8623	8623
query75	2817	2831	2450	2450
query76	3456	1050	633	633
query77	534	377	274	274
query78	9874	9836	9158	9158
query79	1584	944	598	598
query80	704	572	463	463
query81	513	257	226	226
query82	449	147	110	110
query83	272	258	239	239
query84	257	118	101	101
query85	902	512	465	465
query86	433	302	318	302
query87	2872	2850	2763	2763
query88	3224	2230	2231	2230
query89	399	351	345	345
query90	2148	156	158	156
query91	175	169	141	141
query92	80	64	68	64
query93	1495	904	530	530
query94	637	319	297	297
query95	582	381	296	296
query96	580	452	214	214
query97	2371	2391	2331	2331
query98	239	200	202	200
query99	578	594	490	490
Total cold run time: 254239 ms
Total hot run time: 173580 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 12, 2026

FE Regression Coverage Report

Increment line coverage 0.00% (0/36) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jan 26, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

TPC-H: Total hot run time: 32499 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35ab7872eaf6d9ed159f68593e694e5cd4ba731d, data reload: false

------ Round 1 ----------------------------------
q1	17650	5227	5099	5099
q2	2080	307	200	200
q3	10154	1297	735	735
q4	10219	885	315	315
q5	7555	2161	1894	1894
q6	196	178	147	147
q7	884	745	627	627
q8	9257	1375	1077	1077
q9	5452	4941	4818	4818
q10	6775	1952	1561	1561
q11	521	309	275	275
q12	337	387	230	230
q13	17782	4035	3236	3236
q14	230	244	217	217
q15	879	838	808	808
q16	685	680	613	613
q17	626	819	478	478
q18	6771	6449	6522	6449
q19	1240	983	594	594
q20	398	337	227	227
q21	2611	2008	1933	1933
q22	1046	992	966	966
Total cold run time: 103348 ms
Total hot run time: 32499 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5324	5299	5284	5284
q2	261	334	262	262
q3	2207	2650	2266	2266
q4	1344	1737	1321	1321
q5	4285	4164	4322	4164
q6	302	205	163	163
q7	2351	1994	1836	1836
q8	2581	2397	2380	2380
q9	7611	7677	7487	7487
q10	2874	3173	2599	2599
q11	556	471	466	466
q12	702	781	680	680
q13	3922	4415	3537	3537
q14	318	328	289	289
q15	854	800	825	800
q16	680	766	720	720
q17	1175	1348	1383	1348
q18	8413	7967	7980	7967
q19	913	882	891	882
q20	2060	2176	1991	1991
q21	4828	4597	4186	4186
q22	1037	1012	971	971
Total cold run time: 54598 ms
Total hot run time: 51599 ms

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

ClickBench: Total hot run time: 28.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35ab7872eaf6d9ed159f68593e694e5cd4ba731d, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.28	0.26	0.25
query6	1.17	0.67	0.67
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.56	0.51	0.50
query10	0.55	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.62	0.61
query14	1.07	1.07	1.07
query15	0.86	0.87	0.88
query16	0.40	0.39	0.39
query17	1.16	1.16	1.14
query18	0.23	0.21	0.22
query19	2.09	2.04	1.96
query20	0.02	0.01	0.02
query21	15.40	0.27	0.14
query22	5.50	0.05	0.05
query23	15.82	0.28	0.10
query24	2.22	0.25	0.56
query25	0.09	0.07	0.11
query26	0.14	0.13	0.12
query27	0.09	0.08	0.07
query28	4.74	1.11	0.96
query29	12.57	3.97	3.19
query30	0.27	0.14	0.11
query31	2.82	0.63	0.39
query32	3.28	0.60	0.49
query33	3.27	3.22	3.20
query34	16.47	5.43	4.74
query35	4.77	4.84	4.80
query36	0.66	0.49	0.50
query37	0.11	0.08	0.07
query38	0.08	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.04	0.02
query43	0.05	0.04	0.04
Total cold run time: 100.1 s
Total hot run time: 28.32 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 26, 2026

FE UT Coverage Report

Increment line coverage 0.00% (0/37) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 26, 2026

FE Regression Coverage Report

Increment line coverage 81.08% (30/37) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jan 27, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 27, 2026

TPC-H: Total hot run time: 32768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 991bc75b6530dc1f39b1b3d9dec7880b810253b7, data reload: false

------ Round 1 ----------------------------------
q1	17622	5238	5057	5057
q2	2076	303	197	197
q3	10209	1359	799	799
q4	10252	808	330	330
q5	8239	2209	1958	1958
q6	230	186	154	154
q7	905	765	613	613
q8	9272	1543	1137	1137
q9	5400	4839	4828	4828
q10	6915	1974	1559	1559
q11	545	305	278	278
q12	410	378	234	234
q13	17774	4415	3263	3263
q14	250	237	221	221
q15	912	804	838	804
q16	705	678	627	627
q17	643	719	559	559
q18	6913	6469	6490	6469
q19	1658	982	651	651
q20	406	345	228	228
q21	2819	2121	1855	1855
q22	1040	1006	947	947
Total cold run time: 105195 ms
Total hot run time: 32768 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5456	5288	5331	5288
q2	258	348	266	266
q3	2243	2732	2258	2258
q4	1403	1791	1335	1335
q5	4223	4199	5026	4199
q6	261	207	156	156
q7	2166	2039	1803	1803
q8	2560	2426	2455	2426
q9	7640	7742	7715	7715
q10	2866	3044	2569	2569
q11	557	480	449	449
q12	696	798	668	668
q13	3829	4501	3668	3668
q14	331	457	348	348
q15	862	851	830	830
q16	666	729	679	679
q17	1175	1354	1350	1350
q18	8076	7933	7786	7786
q19	983	951	914	914
q20	2137	2156	2046	2046
q21	5022	4443	4177	4177
q22	1107	1070	951	951
Total cold run time: 54517 ms
Total hot run time: 51881 ms

@doris-robot
Copy link

doris-robot commented Jan 27, 2026

ClickBench: Total hot run time: 28.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 991bc75b6530dc1f39b1b3d9dec7880b810253b7, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.08	0.08
query4	1.64	0.11	0.11
query5	0.27	0.24	0.25
query6	1.17	0.68	0.66
query7	0.03	0.03	0.02
query8	0.06	0.04	0.04
query9	0.55	0.53	0.50
query10	0.55	0.54	0.56
query11	0.14	0.09	0.10
query12	0.14	0.10	0.10
query13	0.63	0.61	0.61
query14	1.07	1.06	1.05
query15	0.88	0.85	0.88
query16	0.40	0.40	0.39
query17	1.16	1.15	1.12
query18	0.22	0.21	0.21
query19	2.05	1.92	1.90
query20	0.02	0.02	0.01
query21	15.46	0.26	0.13
query22	4.97	0.05	0.05
query23	15.84	0.28	0.10
query24	1.95	0.88	0.25
query25	0.11	0.07	0.06
query26	0.15	0.13	0.14
query27	0.08	0.06	0.06
query28	4.50	1.16	0.97
query29	12.58	3.86	3.17
query30	0.28	0.13	0.12
query31	2.81	0.61	0.40
query32	3.24	0.59	0.50
query33	3.19	3.25	3.23
query34	16.03	5.36	4.80
query35	4.87	4.78	4.77
query36	0.64	0.50	0.50
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.16	0.16
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.06	0.04	0.04
Total cold run time: 98.71 s
Total hot run time: 28.25 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 27, 2026

FE UT Coverage Report

Increment line coverage 0.00% (0/92) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jan 27, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 27, 2026

TPC-H: Total hot run time: 33103 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 991bc75b6530dc1f39b1b3d9dec7880b810253b7, data reload: false

------ Round 1 ----------------------------------
q1	17606	5320	5061	5061
q2	2074	344	200	200
q3	10138	1287	752	752
q4	10229	920	324	324
q5	7555	2187	1951	1951
q6	201	181	149	149
q7	883	725	605	605
q8	9276	1589	1141	1141
q9	5177	4759	4805	4759
q10	6873	1959	1534	1534
q11	547	295	275	275
q12	337	375	229	229
q13	17822	4088	3268	3268
q14	237	247	225	225
q15	881	829	809	809
q16	678	715	616	616
q17	654	786	478	478
q18	6827	6645	7692	6645
q19	1114	1042	667	667
q20	437	390	239	239
q21	2997	2359	2101	2101
q22	1144	1075	1185	1075
Total cold run time: 103687 ms
Total hot run time: 33103 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5518	5526	5682	5526
q2	250	337	256	256
q3	2397	2860	2555	2555
q4	1527	1854	1394	1394
q5	4521	4515	4672	4515
q6	232	207	163	163
q7	2270	2023	1774	1774
q8	2565	2332	2335	2332
q9	7579	7465	7544	7465
q10	2923	3055	2695	2695
q11	554	464	462	462
q12	679	754	618	618
q13	3626	4047	3238	3238
q14	277	280	265	265
q15	842	794	782	782
q16	646	683	633	633
q17	1093	1238	1249	1238
q18	7604	7240	7255	7240
q19	845	790	811	790
q20	1977	2046	1891	1891
q21	4581	4259	4067	4067
q22	1104	1005	985	985
Total cold run time: 53610 ms
Total hot run time: 50884 ms

@doris-robot
Copy link

doris-robot commented Jan 27, 2026

ClickBench: Total hot run time: 28.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 991bc75b6530dc1f39b1b3d9dec7880b810253b7, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.04	0.04
query3	0.25	0.08	0.09
query4	1.60	0.11	0.11
query5	0.29	0.24	0.25
query6	1.17	0.67	0.68
query7	0.04	0.03	0.02
query8	0.06	0.04	0.03
query9	0.56	0.50	0.48
query10	0.56	0.55	0.55
query11	0.14	0.09	0.09
query12	0.14	0.10	0.11
query13	0.63	0.61	0.62
query14	1.04	1.05	1.07
query15	0.87	0.88	0.88
query16	0.39	0.39	0.38
query17	1.13	1.08	1.16
query18	0.23	0.21	0.20
query19	2.14	2.03	2.06
query20	0.02	0.01	0.02
query21	15.43	0.24	0.14
query22	5.28	0.05	0.04
query23	16.10	0.28	0.10
query24	1.04	0.79	0.26
query25	0.09	0.05	0.07
query26	0.14	0.13	0.12
query27	0.09	0.06	0.05
query28	4.13	1.15	0.96
query29	12.57	3.88	3.15
query30	0.28	0.13	0.12
query31	2.82	0.67	0.40
query32	3.23	0.60	0.49
query33	3.20	3.21	3.32
query34	16.40	5.39	4.75
query35	4.83	4.81	4.79
query36	0.66	0.49	0.48
query37	0.11	0.08	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.17	0.15
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.3 s
Total hot run time: 28.19 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 27, 2026

FE UT Coverage Report

Increment line coverage 0.00% (0/92) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 27, 2026

FE Regression Coverage Report

Increment line coverage 46.74% (43/92) 🎉
Increment coverage report
Complete coverage report

@morningman morningman requested a review from Copilot January 27, 2026 14:42
Copilot AI reviewed Jan 27, 2026
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the Iceberg metadata cache structure to improve code organization, reduce memory overhead, and add comprehensive test coverage for table cache behavior. The main improvements include consolidating three separate caches into two, implementing lazy loading for snapshot cache, and fixing a spelling error in a method name.

Changes:

  • Introduced IcebergTableCacheValue to encapsulate table metadata with lazy-loaded snapshot cache
  • Removed redundant snapshotListCache and snapshotCache, consolidating them into IcebergTableCacheValue
  • Renamed getLastedIcebergSnapshot to getLatestIcebergSnapshot (spelling correction)
  • Added comprehensive test suite covering DML operations, schema changes, and partition evolution

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
IcebergTableCacheValue.java New class implementing lazy-loaded snapshot cache with thread-safe double-checked locking
IcebergMetadataCache.java Simplified cache structure from 3 caches to 2; removed snapshot-specific caches and cache statistics
IcebergUtils.java Refactored method signatures to support passing Table instances; renamed methods for consistency
IcebergExternalCatalog.java Removed deprecated ICEBERG_SNAPSHOT_META_CACHE_TTL_SECOND property
IcebergExternalTable.java Updated to use new simplified cache API methods
IcebergDlaTable.java Updated to use new cache API for Hive-based Iceberg tables
HMSExternalTable.java Updated to use renamed snapshot cache methods
test_iceberg_table_cache.groovy Comprehensive new test covering cache behavior with DML, DDL, and partition operations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman left review comments

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

dev/4.0.x kind/need-document

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@suxiaogang223 @hello-stephen @doris-robot @morningman