[fix](Index)Make column unique ids in index dynamically computed #48988

qidaye · 2025-03-12T13:37:16Z

What problem does this PR solve?

Issue Number: close #48989

Problem Summary:

We introduced Index.columnUniqueIds in 3.0.3, and make sure Index.columnUniqueIds is not empty in 3.0.4.
But when we upgrade Doris from 3.0.2 and before to 3.0.4, the Index.columnUniqueIds will be null.

Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null.
We need to initialize the column unique ids associated with the index correctly.

Key Changes

Removed stored columnUniqueIds from Index class:

Previously, column unique IDs were stored as a field in the Index class
Now they're dynamically computed when needed rather than stored redundantly
Added dynamic computation method:

New method getColumnUniqueIds(List<Column> schema) computes IDs at runtime
Column IDs are looked up by matching index column names with schema columns

Updated method signatures:

Changed toThrift() to toThrift(List<Integer> indexColumnUniqueIds)
Changed toPb() to toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)
Updated all callers to pass column IDs as parameters

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-03-12T13:37:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

qidaye · 2025-03-12T13:37:27Z

run buildall

doris-robot · 2025-03-12T13:55:49Z

TPC-H: Total hot run time: 32220 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6fdd2b80788f9e2859967be83e1ffe9bde4e227, data reload: false

------ Round 1 ----------------------------------
q1	17667	5192	5043	5043
q2	2033	289	168	168
q3	10419	1325	698	698
q4	10223	989	523	523
q5	7542	2380	2744	2380
q6	189	162	133	133
q7	930	730	616	616
q8	9306	1226	1088	1088
q9	4950	4850	4610	4610
q10	6809	2308	1879	1879
q11	476	272	256	256
q12	347	357	226	226
q13	17774	3685	3114	3114
q14	235	243	206	206
q15	548	491	508	491
q16	642	615	572	572
q17	585	876	349	349
q18	7071	6563	6269	6269
q19	1207	952	547	547
q20	320	316	202	202
q21	2750	2117	1870	1870
q22	1078	1016	980	980
Total cold run time: 103101 ms
Total hot run time: 32220 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5193	5153	5158	5153
q2	238	321	237	237
q3	2118	2689	2269	2269
q4	1440	1809	1405	1405
q5	4266	4101	4147	4101
q6	204	163	126	126
q7	1845	1917	1795	1795
q8	2602	2637	2623	2623
q9	7318	7187	7173	7173
q10	3023	3225	2744	2744
q11	566	519	497	497
q12	662	781	646	646
q13	3493	3909	3251	3251
q14	298	319	269	269
q15	543	513	496	496
q16	655	698	664	664
q17	1125	1628	1322	1322
q18	7833	7606	7340	7340
q19	813	812	850	812
q20	1987	2092	1845	1845
q21	5399	5143	4928	4928
q22	1094	992	994	992
Total cold run time: 52715 ms
Total hot run time: 50688 ms

airborne12

need ut

doris-robot · 2025-03-12T14:07:23Z

TPC-DS: Total hot run time: 185002 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6fdd2b80788f9e2859967be83e1ffe9bde4e227, data reload: false

query1	999	405	373	373
query2	6516	1858	1843	1843
query3	6796	220	214	214
query4	26190	23283	23364	23283
query5	4354	662	460	460
query6	297	186	181	181
query7	4596	501	300	300
query8	310	250	238	238
query9	8616	2626	2628	2626
query10	475	311	257	257
query11	15453	15146	14910	14910
query12	155	111	105	105
query13	1669	522	400	400
query14	9114	6264	6740	6264
query15	211	195	173	173
query16	7557	637	506	506
query17	1211	717	562	562
query18	1973	401	301	301
query19	195	188	168	168
query20	123	119	118	118
query21	211	124	106	106
query22	3960	4314	4044	4044
query23	33984	32922	32944	32922
query24	7727	2374	2369	2369
query25	517	452	398	398
query26	1227	263	150	150
query27	2266	497	319	319
query28	4044	2405	2395	2395
query29	726	564	461	461
query30	282	215	197	197
query31	941	871	774	774
query32	71	66	65	65
query33	582	352	293	293
query34	782	839	507	507
query35	796	812	748	748
query36	963	1009	925	925
query37	126	98	77	77
query38	4344	4236	4265	4236
query39	1433	1419	1376	1376
query40	204	113	129	113
query41	61	52	54	52
query42	120	100	97	97
query43	490	496	465	465
query44	1264	795	773	773
query45	173	174	167	167
query46	816	1016	616	616
query47	1765	1822	1711	1711
query48	367	415	308	308
query49	758	498	432	432
query50	687	724	410	410
query51	4227	4151	4147	4147
query52	110	104	99	99
query53	226	248	183	183
query54	497	492	417	417
query55	86	85	83	83
query56	263	264	280	264
query57	1146	1158	1062	1062
query58	240	262	247	247
query59	2606	2707	2749	2707
query60	278	273	268	268
query61	128	119	118	118
query62	791	740	665	665
query63	226	194	183	183
query64	4345	1062	659	659
query65	4425	4314	4318	4314
query66	1125	407	300	300
query67	15751	15569	15291	15291
query68	8868	877	507	507
query69	460	288	267	267
query70	1210	1126	1093	1093
query71	461	287	256	256
query72	5561	3558	3835	3558
query73	798	752	350	350
query74	8959	9197	8657	8657
query75	4039	3165	2708	2708
query76	3701	1189	755	755
query77	804	358	278	278
query78	10048	10026	9242	9242
query79	3300	816	577	577
query80	708	509	445	445
query81	492	256	226	226
query82	738	121	91	91
query83	207	171	155	155
query84	289	96	75	75
query85	767	351	303	303
query86	388	290	293	290
query87	4583	4393	4290	4290
query88	3599	2288	2294	2288
query89	406	312	279	279
query90	1911	213	219	213
query91	144	143	154	143
query92	69	63	54	54
query93	2269	1030	579	579
query94	655	419	287	287
query95	353	270	266	266
query96	480	565	275	275
query97	3398	3399	3267	3267
query98	227	203	196	196
query99	1447	1392	1237	1237
Total cold run time: 276306 ms
Total hot run time: 185002 ms

doris-robot · 2025-03-12T14:12:41Z

ClickBench: Total hot run time: 31.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6fdd2b80788f9e2859967be83e1ffe9bde4e227, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.06	0.06
query4	1.63	0.10	0.11
query5	0.58	0.54	0.55
query6	1.21	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.54	0.51
query10	0.56	0.62	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.62	0.61	0.61
query14	2.79	2.80	2.68
query15	0.90	0.84	0.84
query16	0.39	0.37	0.37
query17	1.07	1.05	1.03
query18	0.21	0.20	0.19
query19	1.94	1.82	1.98
query20	0.02	0.01	0.01
query21	15.36	0.88	0.56
query22	0.78	1.10	0.83
query23	14.84	1.37	0.67
query24	6.63	1.35	1.10
query25	0.54	0.27	0.11
query26	0.64	0.16	0.13
query27	0.05	0.06	0.05
query28	9.57	0.89	0.44
query29	12.54	4.00	3.33
query30	0.24	0.09	0.06
query31	2.82	0.58	0.39
query32	3.23	0.55	0.46
query33	2.98	2.98	3.11
query34	15.90	5.14	4.55
query35	4.63	4.54	4.54
query36	0.65	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.11 s
Total hot run time: 31.35 s

qidaye · 2025-03-12T14:17:52Z

run buildall

doris-robot · 2025-03-12T14:36:41Z

TPC-H: Total hot run time: 32469 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a40c15696e7f5d3007ae8875f40cb42d63a741f3, data reload: false

------ Round 1 ----------------------------------
q1	17612	5278	5096	5096
q2	2041	303	162	162
q3	10423	1311	690	690
q4	10219	1011	528	528
q5	7538	2425	2375	2375
q6	191	175	136	136
q7	916	738	605	605
q8	9309	1310	1096	1096
q9	5097	4893	4677	4677
q10	6831	2309	1896	1896
q11	483	276	271	271
q12	343	353	219	219
q13	17760	3673	3097	3097
q14	233	233	207	207
q15	544	494	507	494
q16	629	611	591	591
q17	579	866	348	348
q18	6833	6591	6322	6322
q19	1202	947	562	562
q20	337	322	193	193
q21	2805	2124	1916	1916
q22	1058	1041	988	988
Total cold run time: 102983 ms
Total hot run time: 32469 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5234	5148	5147	5147
q2	235	327	228	228
q3	2168	2680	2301	2301
q4	1466	1791	1389	1389
q5	4230	4087	4156	4087
q6	207	163	122	122
q7	1909	1907	1791	1791
q8	2637	2629	2604	2604
q9	7346	7266	7170	7170
q10	3015	3212	2744	2744
q11	584	505	491	491
q12	700	782	615	615
q13	3485	3876	3211	3211
q14	277	286	289	286
q15	568	481	489	481
q16	656	680	659	659
q17	1155	1602	1349	1349
q18	7887	7457	7574	7457
q19	832	813	964	813
q20	1981	2001	1862	1862
q21	5470	4944	4992	4944
q22	1104	1053	1007	1007
Total cold run time: 53146 ms
Total hot run time: 50758 ms

doris-robot · 2025-03-12T14:48:16Z

TPC-DS: Total hot run time: 185263 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a40c15696e7f5d3007ae8875f40cb42d63a741f3, data reload: false

query1	1001	391	372	372
query2	6528	1875	1862	1862
query3	6801	214	214	214
query4	26587	23350	23703	23350
query5	5056	664	477	477
query6	311	204	198	198
query7	4610	492	297	297
query8	303	251	245	245
query9	8650	2627	2622	2622
query10	497	311	262	262
query11	15700	15367	14853	14853
query12	172	113	107	107
query13	1656	522	412	412
query14	10916	6223	6280	6223
query15	193	188	168	168
query16	7652	676	468	468
query17	1439	703	565	565
query18	1997	418	313	313
query19	220	189	157	157
query20	121	119	116	116
query21	214	124	108	108
query22	4243	4442	4128	4128
query23	33874	32887	32928	32887
query24	7059	2381	2367	2367
query25	533	448	385	385
query26	1228	272	151	151
query27	2291	478	315	315
query28	4191	2415	2424	2415
query29	733	576	432	432
query30	287	212	191	191
query31	935	855	786	786
query32	73	70	65	65
query33	560	350	312	312
query34	777	841	499	499
query35	804	826	773	773
query36	962	975	890	890
query37	122	109	75	75
query38	4109	4066	4082	4066
query39	1525	1391	1576	1391
query40	210	116	104	104
query41	58	51	50	50
query42	118	99	106	99
query43	494	498	485	485
query44	1273	787	791	787
query45	178	167	165	165
query46	832	1029	624	624
query47	1762	1801	1710	1710
query48	373	411	296	296
query49	810	517	418	418
query50	688	733	417	417
query51	4152	4161	4173	4161
query52	111	110	95	95
query53	230	262	188	188
query54	493	498	428	428
query55	82	81	79	79
query56	258	257	247	247
query57	1111	1173	1066	1066
query58	243	235	236	235
query59	2440	2823	2592	2592
query60	273	281	258	258
query61	156	129	126	126
query62	781	722	670	670
query63	218	187	193	187
query64	4332	1018	662	662
query65	4451	4331	4364	4331
query66	1046	413	310	310
query67	15713	15587	15258	15258
query68	8850	875	505	505
query69	469	298	270	270
query70	1224	1106	1072	1072
query71	471	283	262	262
query72	5678	3579	3750	3579
query73	784	745	349	349
query74	9129	9134	8954	8954
query75	4211	3158	2707	2707
query76	3763	1178	755	755
query77	797	363	292	292
query78	10085	10243	9305	9305
query79	2946	813	580	580
query80	735	529	444	444
query81	462	258	220	220
query82	683	131	93	93
query83	201	189	156	156
query84	288	97	77	77
query85	761	363	313	313
query86	336	304	282	282
query87	4529	4511	4334	4334
query88	3106	2263	2284	2263
query89	436	314	285	285
query90	1977	214	229	214
query91	134	137	111	111
query92	81	58	55	55
query93	1161	1033	591	591
query94	674	410	305	305
query95	360	273	269	269
query96	485	555	276	276
query97	3259	3373	3302	3302
query98	220	210	211	210
query99	1446	1395	1268	1268
Total cold run time: 277557 ms
Total hot run time: 185263 ms

doris-robot · 2025-03-12T14:53:34Z

ClickBench: Total hot run time: 30.98 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a40c15696e7f5d3007ae8875f40cb42d63a741f3, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.57	0.55	0.55
query6	1.21	0.71	0.71
query7	0.02	0.03	0.02
query8	0.04	0.03	0.04
query9	0.59	0.52	0.51
query10	0.58	0.62	0.57
query11	0.15	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.61	0.60
query14	2.81	2.68	2.69
query15	0.92	0.85	0.87
query16	0.36	0.38	0.37
query17	1.02	0.99	1.01
query18	0.20	0.20	0.19
query19	1.89	1.73	1.94
query20	0.02	0.01	0.01
query21	15.35	0.92	0.58
query22	0.78	1.09	0.62
query23	15.06	1.38	0.58
query24	6.79	1.08	1.84
query25	0.63	0.16	0.23
query26	0.62	0.14	0.14
query27	0.05	0.06	0.05
query28	10.19	0.86	0.42
query29	12.54	3.98	3.34
query30	0.24	0.10	0.07
query31	2.81	0.57	0.38
query32	3.23	0.54	0.45
query33	3.03	3.04	3.00
query34	15.68	5.16	4.54
query35	4.55	4.54	4.55
query36	0.66	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.82 s
Total hot run time: 30.98 s

fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java

fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java

fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java

fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java

fe/fe-core/src/main/java/org/apache/doris/alter/CloudSchemaChangeJobV2.java

fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java

fe/fe-core/src/main/java/org/apache/doris/alter/IndexChangeJob.java

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

doris-robot · 2025-03-13T14:46:00Z

TPC-DS: Total hot run time: 192813 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 61e7fa159b52f2a061c23b7b640641a202c3012a, data reload: false

query1	1390	1007	996	996
query2	6412	2067	2012	2012
query3	11051	4521	4565	4521
query4	25510	23738	23503	23503
query5	4953	653	476	476
query6	297	191	190	190
query7	3988	494	299	299
query8	285	245	251	245
query9	8516	2597	2586	2586
query10	495	334	250	250
query11	15705	15091	15083	15083
query12	164	116	103	103
query13	1569	502	391	391
query14	9355	6265	6429	6265
query15	216	188	175	175
query16	7219	640	486	486
query17	1164	772	604	604
query18	1980	426	340	340
query19	202	202	170	170
query20	130	122	125	122
query21	212	139	112	112
query22	4652	4549	4410	4410
query23	34515	33535	33496	33496
query24	7979	2427	2474	2427
query25	549	558	390	390
query26	1171	280	161	161
query27	2538	490	328	328
query28	4505	2436	2412	2412
query29	758	573	433	433
query30	266	225	196	196
query31	926	887	785	785
query32	71	62	62	62
query33	554	368	305	305
query34	799	843	498	498
query35	846	856	788	788
query36	1016	1025	926	926
query37	118	106	85	85
query38	4307	4208	4172	4172
query39	1494	1432	1432	1432
query40	212	116	110	110
query41	55	54	50	50
query42	120	105	110	105
query43	537	532	503	503
query44	1303	797	822	797
query45	178	172	168	168
query46	860	1048	651	651
query47	1816	1896	1803	1803
query48	388	425	316	316
query49	772	530	431	431
query50	757	762	438	438
query51	4328	4328	4275	4275
query52	112	109	100	100
query53	240	264	195	195
query54	491	503	412	412
query55	81	81	80	80
query56	270	261	245	245
query57	1147	1208	1117	1117
query58	255	249	245	245
query59	2944	3233	2807	2807
query60	288	297	267	267
query61	126	124	115	115
query62	793	771	648	648
query63	236	191	205	191
query64	4094	1041	737	737
query65	4590	4456	4472	4456
query66	1030	399	308	308
query67	16407	15444	15419	15419
query68	9317	855	501	501
query69	457	292	264	264
query70	1198	1111	1070	1070
query71	444	286	267	267
query72	5078	3600	3791	3600
query73	861	748	356	356
query74	9044	9024	8857	8857
query75	3975	3143	2679	2679
query76	4929	1197	748	748
query77	910	345	277	277
query78	9859	10085	9332	9332
query79	4916	803	574	574
query80	587	516	507	507
query81	452	255	220	220
query82	390	126	93	93
query83	268	167	150	150
query84	279	93	72	72
query85	745	353	309	309
query86	323	300	294	294
query87	4361	4393	4332	4332
query88	2891	2229	2216	2216
query89	434	313	279	279
query90	2109	275	216	216
query91	136	134	111	111
query92	71	58	57	57
query93	3294	1013	577	577
query94	666	409	298	298
query95	360	275	262	262
query96	480	560	275	275
query97	3330	3405	3295	3295
query98	228	210	205	205
query99	1353	1385	1296	1296
Total cold run time: 286052 ms
Total hot run time: 192813 ms

doris-robot · 2025-03-13T14:51:17Z

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 61e7fa159b52f2a061c23b7b640641a202c3012a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.56	0.54	0.55
query6	1.19	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.53
query10	0.60	0.60	0.59
query11	0.16	0.10	0.11
query12	0.16	0.11	0.12
query13	0.62	0.61	0.61
query14	2.72	2.80	2.81
query15	0.92	0.83	0.84
query16	0.38	0.37	0.38
query17	1.00	1.02	1.07
query18	0.21	0.20	0.19
query19	1.87	1.81	1.99
query20	0.01	0.01	0.01
query21	15.38	0.90	0.54
query22	0.75	1.14	0.74
query23	14.86	1.38	0.65
query24	7.27	1.69	0.45
query25	0.49	0.19	0.19
query26	0.66	0.16	0.13
query27	0.05	0.05	0.04
query28	9.59	0.91	0.44
query29	12.64	3.95	3.29
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.22	0.56	0.47
query33	2.98	2.99	3.04
query34	15.67	5.16	4.48
query35	4.54	4.57	4.53
query36	0.67	0.51	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.09	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.34 s
Total hot run time: 30.65 s

yujun777

LGTM

github-actions · 2025-03-13T22:55:54Z

PR approved by anyone and no changes requested.

airborne12

LGTM

github-actions · 2025-03-17T01:24:54Z

PR approved by at least one committer and no changes requested.

dataroaring

LGTM

…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters

) (#49283) bp #48988

…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters

) (#49300) bp #48988

…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters

wm1581066 added dev/3.0.x p0_r labels Mar 12, 2025

airborne12 reviewed Mar 12, 2025

View reviewed changes

[fix](Index)Add null check before using columnUniqueIds in Index

a40c156

qidaye force-pushed the fix_create_partition_fail branch from a6fdd2b to a40c156 Compare March 12, 2025 14:12

airborne12 reviewed Mar 13, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java Outdated Show resolved Hide resolved

yujun777 reviewed Mar 13, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java Outdated Show resolved Hide resolved

Add UT

0400781

airborne12 reviewed Mar 13, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java Outdated Show resolved Hide resolved

qidaye added 2 commits March 13, 2025 11:26

check duplicated column unique id

590631b

opt function name

3c126ae

airborne12 reviewed Mar 13, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java Outdated Show resolved Hide resolved

Get index columnUniqueIds when using

4b0bb6d

qidaye requested review from dataroaring, gavinchou and w41ter as code owners March 13, 2025 08:20

qidaye added 2 commits March 13, 2025 18:34

opt

f8a9c9d

opt

7f335e1

csun5285 reviewed Mar 13, 2025

View reviewed changes

qidaye added 2 commits March 13, 2025 21:21

fix

b5097db

opt function name

0fafddf

yujun777 approved these changes Mar 13, 2025

View reviewed changes

github-actions bot added the reviewed label Mar 13, 2025

airborne12 approved these changes Mar 17, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 17, 2025

dataroaring approved these changes Mar 20, 2025

View reviewed changes

dataroaring merged commit 8fcdde1 into apache:master Mar 20, 2025
30 of 31 checks passed

github-actions bot added the dev/3.0.x-conflict label Mar 20, 2025

qidaye added the dev/2.1.x label Mar 20, 2025

qidaye deleted the fix_create_partition_fail branch March 20, 2025 02:50

github-actions bot added the dev/2.1.x-conflict label Mar 20, 2025

qidaye mentioned this pull request Mar 20, 2025

[fix](Index)Make column unique ids in index dynamically computed (#48988) #49283

Merged

qidaye mentioned this pull request Mar 20, 2025

[fix](Index)Make column unique ids in index dynamically computed (#48988) #49300

Merged

dataroaring pushed a commit that referenced this pull request Mar 20, 2025

[fix](Index)Make column unique ids in index dynamically computed (#48988

6ad6d66

) (#49283) bp #48988

dataroaring added dev/3.0.5-merged and removed dev/3.0.x dev/3.0.x-conflict labels Mar 20, 2025

dataroaring pushed a commit that referenced this pull request Mar 20, 2025

[fix](Index)Make column unique ids in index dynamically computed (#48988

c896e97

) (#49283) bp #48988

yiguolei pushed a commit that referenced this pull request Mar 21, 2025

[fix](Index)Make column unique ids in index dynamically computed (#48988

7182456

) (#49300) bp #48988

yiguolei added dev/2.1.9-merged and removed dev/2.1.x dev/2.1.x-conflict labels Mar 21, 2025

[fix](Index)Make column unique ids in index dynamically computed #48988

[fix](Index)Make column unique ids in index dynamically computed #48988

Uh oh!

Conversation

qidaye commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 12, 2025

Uh oh!

qidaye commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

qidaye commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Mar 13, 2025

Uh oh!

doris-robot commented Mar 13, 2025

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 17, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

qidaye commented Mar 12, 2025 •

edited

Loading