Skip to content

[fix](variant) Fix metadata compatibility for FE #53748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 23, 2025
Merged

[fix](variant) Fix metadata compatibility for FE #53748

morrySnow merged 1 commit into from
Jul 23, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 23, 2025 •
edited
Loading

What problem does this PR solve?

  • In version 3.1, we used VariantType, which inherits from ScalarType, as the metadata storage for newly created variants, differing from the previously used ScalarType.
  • To ensure compatibility with previous metadata, convert VariantType to ScalarType before use.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@csun5285 csun5285 requested a review from morrySnow as a code owner July 23, 2025 02:52
@Thearas
Copy link
Contributor

Thearas commented Jul 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-H: Total hot run time: 32798 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28f5c12698d3e67ae17b9733fb40ef3d66c4cc36, data reload: false

------ Round 1 ----------------------------------
q1	17613	5532	5365	5365
q2	2059	295	188	188
q3	10614	1243	758	758
q4	10329	876	470	470
q5	9398	2374	2177	2177
q6	192	159	130	130
q7	902	752	626	626
q8	9326	1447	1211	1211
q9	5308	4934	4943	4934
q10	6780	2293	1834	1834
q11	470	275	266	266
q12	340	361	209	209
q13	17770	3592	3018	3018
q14	226	233	211	211
q15	517	463	475	463
q16	421	428	393	393
q17	634	864	365	365
q18	6984	6591	6467	6467
q19	1210	955	527	527
q20	333	366	210	210
q21	2959	2268	1999	1999
q22	1097	1053	977	977
Total cold run time: 105482 ms
Total hot run time: 32798 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5528	5493	5489	5489
q2	244	336	244	244
q3	2308	2679	2331	2331
q4	1360	1812	1339	1339
q5	4446	4982	4987	4982
q6	165	162	125	125
q7	2076	1977	1835	1835
q8	2629	2869	2714	2714
q9	7253	7214	7219	7214
q10	3058	3307	2724	2724
q11	586	518	501	501
q12	689	772	613	613
q13	3443	3808	3214	3214
q14	296	296	263	263
q15	534	478	483	478
q16	439	481	437	437
q17	1245	1755	1288	1288
q18	7591	7505	7325	7325
q19	821	1184	1095	1095
q20	1996	2073	1899	1899
q21	5515	5030	4497	4497
q22	1103	1088	1037	1037
Total cold run time: 53325 ms
Total hot run time: 51644 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-DS: Total hot run time: 198052 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28f5c12698d3e67ae17b9733fb40ef3d66c4cc36, data reload: false

query1	1306	927	900	900
query2	6203	1970	1886	1886
query3	10983	4472	4576	4472
query4	33024	24007	23606	23606
query5	3724	602	449	449
query6	266	185	179	179
query7	3986	487	315	315
query8	282	248	244	244
query9	9589	2587	2583	2583
query10	482	333	266	266
query11	17990	15421	15247	15247
query12	162	107	104	104
query13	1554	553	420	420
query14	9215	6676	7460	6676
query15	219	208	195	195
query16	7852	675	506	506
query17	1601	747	623	623
query18	2060	419	334	334
query19	226	210	175	175
query20	128	125	119	119
query21	207	131	111	111
query22	4514	4690	4404	4404
query23	35228	34936	34335	34335
query24	8007	2697	2760	2697
query25	501	494	430	430
query26	1176	292	190	190
query27	2304	489	365	365
query28	5389	2245	2182	2182
query29	648	600	471	471
query30	246	201	172	172
query31	1039	914	843	843
query32	74	64	61	61
query33	520	367	306	306
query34	759	889	526	526
query35	796	821	735	735
query36	1013	1066	969	969
query37	109	90	69	69
query38	4006	4088	4051	4051
query39	1552	1501	1483	1483
query40	207	113	104	104
query41	50	52	53	52
query42	128	104	102	102
query43	524	531	494	494
query44	1363	851	846	846
query45	191	185	172	172
query46	919	1081	725	725
query47	1992	2019	1898	1898
query48	419	452	358	358
query49	782	490	412	412
query50	700	697	436	436
query51	7295	7432	7275	7275
query52	105	102	90	90
query53	238	265	189	189
query54	552	555	491	491
query55	85	80	84	80
query56	280	282	265	265
query57	1290	1323	1196	1196
query58	237	220	246	220
query59	3063	3219	3270	3219
query60	283	283	264	264
query61	121	141	115	115
query62	795	767	722	722
query63	244	206	203	203
query64	3876	1008	695	695
query65	3498	3327	3294	3294
query66	978	411	311	311
query67	16579	15829	15421	15421
query68	5679	822	538	538
query69	506	311	276	276
query70	1165	1128	1174	1128
query71	391	303	263	263
query72	5854	3780	3774	3774
query73	641	748	348	348
query74	10591	9379	9397	9379
query75	3199	3187	2670	2670
query76	3105	1171	774	774
query77	506	374	284	284
query78	10468	10514	9666	9666
query79	3270	913	592	592
query80	711	529	447	447
query81	519	261	226	226
query82	643	127	93	93
query83	161	162	144	144
query84	247	106	83	83
query85	799	367	316	316
query86	402	321	299	299
query87	4448	4414	4288	4288
query88	4980	2449	2405	2405
query89	428	333	296	296
query90	1780	190	189	189
query91	138	140	111	111
query92	67	56	53	53
query93	2112	909	544	544
query94	695	412	294	294
query95	334	277	273	273
query96	493	608	289	289
query97	3179	3258	3166	3166
query98	229	217	207	207
query99	1676	1419	1289	1289
Total cold run time: 295150 ms
Total hot run time: 198052 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

ClickBench: Total hot run time: 28.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 28f5c12698d3e67ae17b9733fb40ef3d66c4cc36, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.54	0.53	0.51
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.50	0.52
query10	0.54	0.55	0.59
query11	0.14	0.11	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.60
query14	0.78	0.79	0.80
query15	0.85	0.85	0.82
query16	0.39	0.37	0.38
query17	1.07	1.03	1.04
query18	0.23	0.22	0.20
query19	1.92	1.78	1.76
query20	0.01	0.01	0.01
query21	15.38	0.95	0.58
query22	0.74	0.70	0.64
query23	15.24	1.39	0.57
query24	3.03	0.89	1.98
query25	0.23	0.19	0.07
query26	0.26	0.13	0.14
query27	0.05	0.08	0.04
query28	14.26	0.98	0.44
query29	12.58	4.02	3.34
query30	0.25	0.09	0.05
query31	2.85	0.60	0.38
query32	3.27	0.55	0.46
query33	3.00	2.99	3.07
query34	16.91	5.16	4.53
query35	4.53	4.52	4.54
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.64 s
Total hot run time: 28.76 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit 9d9a79c into apache:branch-3.1 Jul 23, 2025
23 checks passed
eldenmoon
eldenmoon reviewed Jul 23, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

csun5285 added a commit to csun5285/doris that referenced this pull request Jul 30, 2025
@csun5285
[fix](variant) Fix metadata compatibility for FE (apache#53748)
582949f 
### What problem does this PR solve?

- In version 3.1, we used 'VariantType', which inherits from
'ScalarType', as the metadata storage for newly created variants,
differing from the previously used 'ScalarType'.
- To ensure compatibility with previous metadata, convert 'VariantType'
to 'ScalarType' before use.
csun5285 added a commit to csun5285/doris that referenced this pull request Jul 30, 2025
@csun5285
branch-3.1: [enhance](variant) typed_paths_to sparse column (apache#5…
7a3c340 
…2927)

1. support when the sub-columns in the specified variant are of a fixed
type, these sub-columns can be included in sparse columns.
2. add many cases and ut

branch-3.1: [fix](variant type) fix add variant column, index (apache#53092)

[fix](variant) Fix metadata compatibility for FE (apache#53748)

- In version 3.1, we used 'VariantType', which inherits from
'ScalarType', as the metadata storage for newly created variants,
differing from the previously used 'ScalarType'.
- To ensure compatibility with previous metadata, convert 'VariantType'
to 'ScalarType' before use.

fix

[fix](variant) fix array contains null

fix case

fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmoon eldenmoon eldenmoon left review comments

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

dev/3.1.0-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@csun5285 @Thearas @doris-robot @hello-stephen @eldenmoon @morrySnow