[fix](schema-change) Nested types should only support enlarging varchar length with light schema change #49452

Hastyshell · 2025-03-25T06:13:19Z

What problem does this PR solve?

Problem Summary:

Currently nested types only support light schema change for internal fields, for string types, only varchar can do light schema change.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…length with light schema change

Thearas · 2025-03-25T06:13:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Hastyshell · 2025-03-25T07:47:25Z

run buildall

doris-robot · 2025-03-25T08:27:42Z

TPC-H: Total hot run time: 34128 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e4317a165c889e840235661ca6853f92d6548da, data reload: false

------ Round 1 ----------------------------------
q1	25830	5160	5068	5068
q2	2062	284	157	157
q3	10410	1253	677	677
q4	10233	985	536	536
q5	7526	2396	2349	2349
q6	184	168	134	134
q7	911	734	625	625
q8	9339	1279	1092	1092
q9	6888	5161	5152	5152
q10	6875	2311	1890	1890
q11	475	282	260	260
q12	360	362	221	221
q13	17769	3681	3118	3118
q14	242	231	214	214
q15	536	489	494	489
q16	649	620	590	590
q17	571	849	339	339
q18	7484	7063	7027	7027
q19	1506	974	575	575
q20	310	327	192	192
q21	3967	2638	2427	2427
q22	1066	1039	996	996
Total cold run time: 115193 ms
Total hot run time: 34128 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5285	5080	5225	5080
q2	242	332	229	229
q3	2182	2661	2346	2346
q4	1436	1873	1406	1406
q5	4574	4504	4522	4504
q6	211	174	135	135
q7	2044	1976	1868	1868
q8	2641	2681	2616	2616
q9	7442	7204	7179	7179
q10	3027	3240	2721	2721
q11	576	507	496	496
q12	695	784	617	617
q13	3485	3903	3397	3397
q14	312	311	267	267
q15	513	482	476	476
q16	641	694	645	645
q17	1162	1522	1489	1489
q18	7892	7507	7435	7435
q19	794	813	908	813
q20	1977	2001	1833	1833
q21	5397	4898	4883	4883
q22	1144	1060	1008	1008
Total cold run time: 53672 ms
Total hot run time: 51443 ms

doris-robot · 2025-03-25T08:39:40Z

TPC-DS: Total hot run time: 193353 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e4317a165c889e840235661ca6853f92d6548da, data reload: false

query1	1413	1076	1044	1044
query2	6170	1892	1873	1873
query3	11195	4711	4637	4637
query4	24987	23807	23308	23308
query5	4674	651	485	485
query6	318	204	205	204
query7	3992	501	274	274
query8	312	260	243	243
query9	8529	2622	2619	2619
query10	490	305	251	251
query11	15381	15198	14838	14838
query12	156	108	106	106
query13	1560	515	389	389
query14	9066	6288	6123	6123
query15	199	187	165	165
query16	7004	652	487	487
query17	1157	692	543	543
query18	1897	405	307	307
query19	198	195	162	162
query20	124	121	118	118
query21	206	137	109	109
query22	4657	4425	4453	4425
query23	34275	33530	33618	33530
query24	8515	2492	2472	2472
query25	516	470	402	402
query26	1248	273	154	154
query27	2939	500	325	325
query28	4512	2458	2433	2433
query29	683	559	445	445
query30	280	224	200	200
query31	924	854	810	810
query32	81	66	63	63
query33	544	371	327	327
query34	846	847	528	528
query35	803	851	780	780
query36	975	1032	905	905
query37	127	111	80	80
query38	4240	4092	4171	4092
query39	1502	1481	1466	1466
query40	221	125	112	112
query41	54	53	51	51
query42	122	110	110	110
query43	497	549	491	491
query44	1383	828	827	827
query45	181	191	169	169
query46	888	1046	665	665
query47	1832	1921	1822	1822
query48	395	423	305	305
query49	755	507	423	423
query50	692	749	425	425
query51	4322	4307	4300	4300
query52	111	109	105	105
query53	226	256	198	198
query54	506	499	447	447
query55	85	84	81	81
query56	279	285	268	268
query57	1195	1176	1146	1146
query58	261	276	252	252
query59	2546	2821	2719	2719
query60	314	298	282	282
query61	152	145	147	145
query62	816	771	693	693
query63	226	197	189	189
query64	4308	1169	685	685
query65	4435	4361	4490	4361
query66	1088	403	296	296
query67	16256	15645	15349	15349
query68	9824	891	511	511
query69	491	298	269	269
query70	1231	1100	1110	1100
query71	459	290	301	290
query72	5083	4662	4575	4575
query73	765	571	351	351
query74	9138	8888	8999	8888
query75	4398	3219	2828	2828
query76	5348	1198	765	765
query77	983	383	289	289
query78	9901	10210	9293	9293
query79	2969	826	572	572
query80	618	519	438	438
query81	475	257	221	221
query82	298	125	99	99
query83	199	189	152	152
query84	287	98	86	86
query85	832	349	303	303
query86	338	311	292	292
query87	4453	4445	4385	4385
query88	2846	2273	2261	2261
query89	428	310	286	286
query90	2188	219	221	219
query91	137	144	112	112
query92	74	59	57	57
query93	2253	1076	587	587
query94	680	423	315	315
query95	359	277	264	264
query96	486	559	276	276
query97	3089	3159	3164	3159
query98	235	211	198	198
query99	1369	1408	1274	1274
Total cold run time: 283075 ms
Total hot run time: 193353 ms

doris-robot · 2025-03-25T08:44:59Z

ClickBench: Total hot run time: 31.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e4317a165c889e840235661ca6853f92d6548da, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.10
query3	0.25	0.19	0.19
query4	1.59	0.18	0.20
query5	0.59	0.58	0.59
query6	1.25	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.52
query10	0.58	0.58	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.11
query13	0.62	0.60	0.59
query14	2.68	2.70	2.73
query15	0.93	0.85	0.86
query16	0.39	0.38	0.39
query17	1.05	1.03	1.03
query18	0.21	0.20	0.20
query19	1.89	1.85	1.89
query20	0.01	0.00	0.01
query21	15.35	0.94	0.55
query22	0.75	1.22	0.74
query23	14.84	1.43	0.67
query24	7.64	1.33	1.16
query25	0.49	0.25	0.06
query26	0.56	0.16	0.15
query27	0.05	0.04	0.04
query28	9.28	0.81	0.44
query29	12.57	3.94	3.30
query30	0.25	0.10	0.07
query31	2.81	0.60	0.38
query32	3.25	0.57	0.47
query33	3.02	3.05	3.09
query34	15.78	5.12	4.58
query35	4.57	4.52	4.59
query36	0.65	0.49	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.04	0.02
Total cold run time: 105.49 s
Total hot run time: 31.71 s

fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java

regression-test/suites/schema_change_p0/test_type_length_change.groovy

dataroaring

LGTM

github-actions · 2025-03-26T07:47:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-26T07:47:19Z

PR approved by anyone and no changes requested.

Hastyshell · 2025-03-26T08:08:37Z

run buildall

dataroaring

LGTM

github-actions · 2025-03-26T08:11:17Z

PR approved by at least one committer and no changes requested.

Hastyshell · 2025-03-26T08:26:33Z

run buildall

doris-robot · 2025-03-26T08:45:59Z

TPC-H: Total hot run time: 34008 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c9986c18eab3de04363424bc4ca386ed4edb66e, data reload: false

------ Round 1 ----------------------------------
q1	25989	5415	5077	5077
q2	2067	292	165	165
q3	10405	1261	682	682
q4	10241	994	547	547
q5	7545	2440	2314	2314
q6	185	165	131	131
q7	930	752	606	606
q8	9308	1300	1044	1044
q9	6885	5086	5112	5086
q10	6858	2325	1902	1902
q11	484	271	280	271
q12	352	356	227	227
q13	17768	3752	3091	3091
q14	225	234	206	206
q15	543	485	498	485
q16	646	611	584	584
q17	595	858	337	337
q18	7692	7135	7081	7081
q19	1948	981	587	587
q20	333	337	189	189
q21	3847	3395	2401	2401
q22	1096	999	995	995
Total cold run time: 115942 ms
Total hot run time: 34008 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5321	5124	5153	5124
q2	242	331	234	234
q3	2162	2718	2306	2306
q4	1435	1830	1451	1451
q5	4471	4466	4419	4419
q6	218	170	129	129
q7	2005	1904	1766	1766
q8	2576	2574	2589	2574
q9	7314	7078	7186	7078
q10	2999	3193	2783	2783
q11	566	542	517	517
q12	722	804	605	605
q13	3526	3886	3411	3411
q14	300	296	263	263
q15	525	494	474	474
q16	630	701	657	657
q17	1179	1588	1377	1377
q18	7932	7664	7549	7549
q19	810	785	863	785
q20	1918	1961	1935	1935
q21	5382	4876	4944	4876
q22	1075	1077	1046	1046
Total cold run time: 53308 ms
Total hot run time: 51359 ms

doris-robot · 2025-03-26T08:57:54Z

TPC-DS: Total hot run time: 192312 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c9986c18eab3de04363424bc4ca386ed4edb66e, data reload: false

query1	1390	1036	1024	1024
query2	6285	1919	1890	1890
query3	11128	4705	4706	4705
query4	25813	23889	22935	22935
query5	5077	682	495	495
query6	306	205	192	192
query7	3982	484	272	272
query8	311	255	238	238
query9	8534	2605	2593	2593
query10	501	299	256	256
query11	15517	15137	15055	15055
query12	164	114	111	111
query13	1559	531	395	395
query14	9221	6235	6068	6068
query15	199	195	171	171
query16	7593	614	492	492
query17	1160	786	601	601
query18	2038	441	336	336
query19	206	251	155	155
query20	130	122	112	112
query21	207	119	107	107
query22	4661	4516	4265	4265
query23	34300	33585	33470	33470
query24	7890	2397	2405	2397
query25	502	469	415	415
query26	1172	269	143	143
query27	2810	506	329	329
query28	4924	2455	2419	2419
query29	688	570	447	447
query30	289	230	192	192
query31	887	849	804	804
query32	81	66	65	65
query33	529	411	306	306
query34	779	839	525	525
query35	842	878	788	788
query36	992	1032	932	932
query37	124	103	77	77
query38	4119	4190	4303	4190
query39	1518	1446	1418	1418
query40	212	118	103	103
query41	53	55	52	52
query42	120	107	114	107
query43	505	501	487	487
query44	1321	801	807	801
query45	184	178	168	168
query46	850	1051	648	648
query47	1815	1895	1801	1801
query48	387	422	304	304
query49	764	541	426	426
query50	706	752	417	417
query51	4272	4356	4327	4327
query52	110	110	100	100
query53	229	259	186	186
query54	503	482	404	404
query55	81	86	83	83
query56	274	285	268	268
query57	1168	1216	1117	1117
query58	256	241	278	241
query59	2610	2921	2709	2709
query60	322	282	258	258
query61	140	137	135	135
query62	823	729	681	681
query63	222	179	182	179
query64	3999	1055	702	702
query65	4452	4367	4399	4367
query66	1126	400	301	301
query67	16289	15961	15251	15251
query68	9641	876	509	509
query69	475	295	248	248
query70	1181	1084	1099	1084
query71	464	298	263	263
query72	5167	4805	4681	4681
query73	706	587	343	343
query74	9012	9144	8786	8786
query75	4299	3244	2708	2708
query76	4380	1183	753	753
query77	1005	381	273	273
query78	9957	10328	9265	9265
query79	1653	816	560	560
query80	754	507	443	443
query81	470	276	226	226
query82	430	123	94	94
query83	206	170	164	164
query84	285	98	70	70
query85	751	350	310	310
query86	336	280	302	280
query87	4456	4632	4292	4292
query88	2842	2227	2220	2220
query89	377	322	277	277
query90	1958	204	207	204
query91	138	141	109	109
query92	73	59	55	55
query93	1188	1032	588	588
query94	670	397	311	311
query95	359	269	329	269
query96	498	554	275	275
query97	3154	3230	3165	3165
query98	217	208	208	208
query99	1457	1396	1271	1271
Total cold run time: 280780 ms
Total hot run time: 192312 ms

doris-robot · 2025-03-26T09:03:15Z

ClickBench: Total hot run time: 31.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c9986c18eab3de04363424bc4ca386ed4edb66e, data reload: false

query1	0.05	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.58	0.60
query6	1.18	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.54	0.51
query10	0.57	0.60	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.82	2.72	2.84
query15	0.93	0.86	0.84
query16	0.39	0.39	0.37
query17	1.04	1.04	1.05
query18	0.22	0.20	0.19
query19	1.93	1.99	1.84
query20	0.01	0.01	0.01
query21	15.36	0.93	0.56
query22	0.99	1.08	0.98
query23	15.07	1.39	0.62
query24	7.38	1.63	0.57
query25	0.52	0.14	0.19
query26	0.61	0.15	0.13
query27	0.05	0.05	0.05
query28	9.49	0.95	0.47
query29	12.54	4.10	3.43
query30	0.24	0.09	0.06
query31	2.84	0.60	0.40
query32	3.22	0.54	0.47
query33	3.07	3.10	3.07
query34	15.85	5.15	4.56
query35	4.56	4.56	4.52
query36	0.67	0.49	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 106.16 s
Total hot run time: 31.61 s

dataroaring

LGTM

github-actions · 2025-03-26T11:31:48Z

PR approved by at least one committer and no changes requested.

…port length growing (#48025) ### What problem does this PR solve? cherry-pick from #46639 #48607 #49452 should be picked soon to resovle problem introduced by the pr Issue Number: close #xxx Co-authored-by: Siyang Tang <tangsiyang2001@foxmail.com>

…ar length with light schema change (apache#49452) Introduce: apache#48607, apache#46639 Problem Summary: Currently nested types only support light schema change for internal fields, for string types, only varchar can do light schema change.

…ar length with light schema change (#49452) (#49592) ### What problem does this PR solve? cherry-pick: #49452 Problem Summary: Currently nested types only support light schema change for internal fields, for string types, only varchar can do light schema change.

…ar length with light schema change (apache#49452) Introduce: apache#48607, apache#46639 Problem Summary: Currently nested types only support light schema change for internal fields, for string types, only varchar can do light schema change.

[fix](schema-change) Nested type should only support enlarge varchar …

65fa0fb

…length with light schema change

add test cases

692e174

Hastyshell force-pushed the fix-nested-type-light-sc branch from b0ba517 to 692e174 Compare March 25, 2025 07:42

Hastyshell added 2 commits March 25, 2025 15:44

Update ColumnType.java

8216a80

Update ColumnType.java

2e4317a

Hastyshell marked this pull request as ready for review March 25, 2025 07:47

Hastyshell mentioned this pull request Mar 25, 2025

[improve](schema-change) support nested type with varchar type to support length growing #48025

Merged

16 tasks

amorynan reviewed Mar 26, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java Show resolved Hide resolved

gavinchou added p0_b dev/3.0.x labels Mar 26, 2025

dataroaring reviewed Mar 26, 2025

View reviewed changes

regression-test/suites/schema_change_p0/test_type_length_change.groovy Show resolved Hide resolved

dataroaring previously approved these changes Mar 26, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2025

github-actions bot added the reviewed label Mar 26, 2025

Hastyshell dismissed dataroaring’s stale review via cf4fc01 March 26, 2025 08:08

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 26, 2025

dataroaring previously approved these changes Mar 26, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2025

luwei16 approved these changes Mar 26, 2025

View reviewed changes

gavinchou previously approved these changes Mar 26, 2025

View reviewed changes

avoid test case conflict with 3.0

3c9986c

Hastyshell dismissed stale reviews from gavinchou and dataroaring via 3c9986c March 26, 2025 08:26

Hastyshell force-pushed the fix-nested-type-light-sc branch from cf4fc01 to 3c9986c Compare March 26, 2025 08:26

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 26, 2025

dataroaring approved these changes Mar 26, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2025

dataroaring merged commit 8b268dc into apache:master Mar 26, 2025
27 of 28 checks passed

github-actions bot added the dev/3.0.x-conflict label Mar 26, 2025

Hastyshell mentioned this pull request Mar 27, 2025

[fix](schema-change) Nested types should only support enlarging varchar length with light schema change (#49452) #49592

Merged

16 tasks

Hastyshell removed the dev/3.0.x-conflict label Mar 27, 2025

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Mar 28, 2025

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

[fix](schema-change) Nested types should only support enlarging varchar length with light schema change #49452

[fix](schema-change) Nested types should only support enlarging varchar length with light schema change #49452

Uh oh!

Conversation

Hastyshell commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Mar 25, 2025

Uh oh!

Hastyshell commented Mar 25, 2025

Uh oh!

doris-robot commented Mar 25, 2025

Uh oh!

doris-robot commented Mar 25, 2025

Uh oh!

doris-robot commented Mar 25, 2025

Uh oh!

Uh oh!

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

Hastyshell commented Mar 26, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

Hastyshell commented Mar 26, 2025

Uh oh!

doris-robot commented Mar 26, 2025

Uh oh!

doris-robot commented Mar 26, 2025

Uh oh!

doris-robot commented Mar 26, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Hastyshell commented Mar 25, 2025 •

edited

Loading