Skip to content

branch-3.1: [enhance](variant) typed_paths_to sparse column #52927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 4 commits into from
Jul 10, 2025
Merged

branch-3.1: [enhance](variant) typed_paths_to sparse column #52927

morrySnow merged 4 commits into from
Jul 10, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 8, 2025 •
edited by eldenmoon
Loading

What problem does this PR solve?

  1. support when the sub-columns in the specified variant are of a fixed type, these sub-columns can be included in sparse columns.
  2. add many cases and ut

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@csun5285 csun5285 requested a review from morrySnow as a code owner July 8, 2025 07:18
@Thearas
Copy link
Contributor

Thearas commented Jul 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 8, 2025

run buildall

@csun5285 csun5285 changed the title [enhance](variant) typed_paths_to sparse column branch-3.1: [enhance](variant) typed_paths_to sparse column Jul 8, 2025
@csun5285
Copy link
Contributor Author

csun5285 commented Jul 8, 2025

run buildall

@HappenLee HappenLee mentioned this pull request Jul 8, 2025
Merged
@doris-robot
Copy link

doris-robot commented Jul 8, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.81% (1191/1421)
Line Coverage 67.88% (21018/30965)
Region Coverage 67.46% (10436/15469)
Branch Coverage 57.00% (5505/9658)

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

BE UT Coverage Report

Increment line coverage 76.18% (307/403) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.30% (12491/27574)
Line Coverage 36.29% (110917/305611)
Region Coverage 35.39% (57396/162187)
Branch Coverage 32.51% (31166/95876)

csun5285 added 2 commits July 9, 2025 16:12
@csun5285
[enhance](variant) typed_paths_to sparse column
f7d29be 
commit 1

commit 1

f2

test

tmp

commit 2

remove unused code

fix

fix

tmp commit

fix

fix

fix commment1

fix comment2

fix

fix

[fix](inverted index) pick index on fe

[fix](variant) fix decimal in sparse column and format code

fix

fix

ut

fix license

fix p0
@csun5285
format
5435495
@csun5285 csun5285 force-pushed the feat_typed_path_to_sparse_3.1 branch from 38a516d to 5435495 Compare July 9, 2025 08:12
@csun5285
Copy link
Contributor Author

csun5285 commented Jul 9, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 9, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.83% (1192/1422)
Line Coverage 67.99% (21050/30961)
Region Coverage 67.52% (10450/15478)
Branch Coverage 57.14% (5523/9666)

eldenmoon
eldenmoon reviewed Jul 9, 2025
View reviewed changes
eldenmoon
eldenmoon reviewed Jul 9, 2025
View reviewed changes
@eldenmoon
Copy link
Member

eldenmoon commented Jul 9, 2025

increment line coverage need 85%

@doris-robot
Copy link

doris-robot commented Jul 9, 2025

BE UT Coverage Report

Increment line coverage 76.18% (307/403) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.19% (12501/27661)
Line Coverage 36.08% (111031/307703)
Region Coverage 35.22% (57448/163118)
Branch Coverage 32.33% (31193/96484)

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 9, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.83% (1192/1422)
Line Coverage 67.95% (21039/30961)
Region Coverage 67.54% (10454/15478)
Branch Coverage 57.11% (5520/9666)

@doris-robot
Copy link

doris-robot commented Jul 9, 2025

BE UT Coverage Report

Increment line coverage 79.79% (304/381) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.19% (12500/27661)
Line Coverage 36.08% (111020/307689)
Region Coverage 35.21% (57437/163114)
Branch Coverage 32.32% (31186/96482)

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 10, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 10, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.83% (1192/1422)
Line Coverage 67.97% (21045/30961)
Region Coverage 67.52% (10451/15478)
Branch Coverage 57.07% (5516/9666)

@doris-robot
Copy link

doris-robot commented Jul 10, 2025

TPC-H: Total hot run time: 39755 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dafea2f231732d2a4b1bdc72b90e9916b0519c1d, data reload: false

------ Round 1 ----------------------------------
q1	17799	6867	6605	6605
q2	2075	168	158	158
q3	10620	1159	1184	1159
q4	10363	780	807	780
q5	7751	2911	2812	2812
q6	215	135	135	135
q7	989	625	609	609
q8	9345	1999	2043	1999
q9	6671	6376	6392	6376
q10	6966	2244	2314	2244
q11	464	260	262	260
q12	395	212	216	212
q13	17777	2945	3018	2945
q14	240	206	206	206
q15	508	464	476	464
q16	473	378	365	365
q17	995	605	515	515
q18	7474	6539	6668	6539
q19	1359	1095	1030	1030
q20	468	199	196	196
q21	3938	3170	3178	3170
q22	1074	995	976	976
Total cold run time: 107959 ms
Total hot run time: 39755 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6639	6560	6562	6560
q2	326	227	231	227
q3	2872	2946	2916	2916
q4	2066	1813	1841	1813
q5	5736	5711	5720	5711
q6	211	128	134	128
q7	2239	1782	1802	1782
q8	3385	3548	3549	3548
q9	8786	8872	8872	8872
q10	3532	3500	3512	3500
q11	593	499	488	488
q12	801	607	617	607
q13	10738	3204	3153	3153
q14	300	260	271	260
q15	512	455	464	455
q16	475	436	428	428
q17	1850	1628	1623	1623
q18	8254	7764	7708	7708
q19	1714	1655	1527	1527
q20	2077	1829	1820	1820
q21	5234	5080	5054	5054
q22	1100	1052	1042	1042
Total cold run time: 69440 ms
Total hot run time: 59222 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2025

TPC-DS: Total hot run time: 197088 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dafea2f231732d2a4b1bdc72b90e9916b0519c1d, data reload: false

query1	1300	911	910	910
query2	6361	1925	1852	1852
query3	10801	4312	4281	4281
query4	33237	23363	23828	23363
query5	4359	457	469	457
query6	290	182	194	182
query7	4034	327	328	327
query8	290	236	233	233
query9	9510	2631	2607	2607
query10	488	256	257	256
query11	18120	15227	15492	15227
query12	167	102	104	102
query13	1553	421	424	421
query14	10106	6981	6932	6932
query15	246	185	191	185
query16	7905	449	525	449
query17	1595	609	616	609
query18	2140	339	322	322
query19	242	166	181	166
query20	124	121	116	116
query21	200	112	104	104
query22	4590	4433	4397	4397
query23	36037	34263	33934	33934
query24	12171	2988	3048	2988
query25	608	438	440	438
query26	1263	180	180	180
query27	2736	361	372	361
query28	7729	2304	2250	2250
query29	722	473	465	465
query30	257	168	166	166
query31	1025	809	869	809
query32	99	65	60	60
query33	788	317	315	315
query34	1175	504	521	504
query35	866	750	749	749
query36	1116	949	971	949
query37	212	73	72	72
query38	4146	3959	3990	3959
query39	1497	1478	1547	1478
query40	257	104	102	102
query41	53	52	48	48
query42	117	112	104	104
query43	540	491	472	472
query44	1346	841	832	832
query45	193	169	168	168
query46	1189	764	749	749
query47	1981	1925	1906	1906
query48	446	336	360	336
query49	1120	402	400	400
query50	852	440	431	431
query51	7462	7204	7313	7204
query52	104	92	101	92
query53	263	195	192	192
query54	1422	492	474	474
query55	85	89	84	84
query56	279	258	253	253
query57	1332	1175	1203	1175
query58	255	214	216	214
query59	3293	3058	3097	3058
query60	293	268	281	268
query61	125	113	111	111
query62	854	691	675	675
query63	225	197	194	194
query64	5025	673	639	639
query65	3326	3275	3282	3275
query66	1265	310	308	308
query67	16300	15443	15430	15430
query68	5533	582	577	577
query69	441	267	264	264
query70	1161	1087	1121	1087
query71	327	274	284	274
query72	5934	4156	4066	4066
query73	755	358	360	358
query74	9463	9105	9057	9057
query75	3362	2644	2684	2644
query76	3241	1107	1158	1107
query77	374	277	279	277
query78	10555	9553	9540	9540
query79	2434	619	631	619
query80	1118	430	438	430
query81	546	220	225	220
query82	896	95	90	90
query83	230	150	141	141
query84	231	77	76	76
query85	1420	323	304	304
query86	437	285	290	285
query87	4401	4222	4246	4222
query88	4286	2435	2406	2406
query89	422	301	292	292
query90	2164	188	191	188
query91	136	109	107	107
query92	67	50	52	50
query93	2334	559	569	559
query94	1044	302	307	302
query95	371	259	266	259
query96	610	287	291	287
query97	3280	3139	3161	3139
query98	234	209	203	203
query99	1501	1293	1289	1289
Total cold run time: 309415 ms
Total hot run time: 197088 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2025

ClickBench: Total hot run time: 31.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dafea2f231732d2a4b1bdc72b90e9916b0519c1d, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.05	0.05
query3	0.24	0.06	0.06
query4	1.65	0.08	0.09
query5	0.52	0.50	0.52
query6	1.12	0.75	0.74
query7	0.02	0.02	0.02
query8	0.06	0.06	0.05
query9	0.54	0.50	0.50
query10	0.57	0.55	0.58
query11	0.16	0.11	0.12
query12	0.16	0.13	0.13
query13	0.62	0.59	0.60
query14	0.79	0.78	0.84
query15	0.85	0.86	0.84
query16	0.38	0.38	0.37
query17	1.09	1.09	1.02
query18	0.19	0.19	0.19
query19	1.89	1.83	1.79
query20	0.02	0.02	0.01
query21	15.39	0.67	0.66
query22	3.52	6.60	2.32
query23	18.32	1.36	1.33
query24	2.23	0.24	0.22
query25	0.15	0.08	0.09
query26	0.28	0.18	0.18
query27	0.09	0.08	0.08
query28	13.20	0.61	0.58
query29	12.66	3.37	3.33
query30	0.25	0.06	0.06
query31	2.86	0.40	0.39
query32	3.23	0.49	0.48
query33	2.97	3.01	3.02
query34	17.27	4.48	4.55
query35	4.55	4.54	4.54
query36	0.66	0.47	0.50
query37	0.20	0.17	0.16
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.17	0.13	0.14
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.44 s
Total hot run time: 31.24 s

eldenmoon
eldenmoon approved these changes Jul 10, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow morrySnow merged commit 8d79e6e into apache:branch-3.1 Jul 10, 2025
20 of 22 checks passed
csun5285 added a commit to csun5285/doris that referenced this pull request Jul 30, 2025
@csun5285
branch-3.1: [enhance](variant) typed_paths_to sparse column (apache#5…
1b99ce3 
…2927)

1. support when the sub-columns in the specified variant are of a fixed
type, these sub-columns can be included in sparse columns.
2. add many cases and ut
csun5285 added a commit to csun5285/doris that referenced this pull request Jul 30, 2025
@csun5285
branch-3.1: [enhance](variant) typed_paths_to sparse column (apache#5…
7a3c340 
…2927)

1. support when the sub-columns in the specified variant are of a fixed
type, these sub-columns can be included in sparse columns.
2. add many cases and ut

branch-3.1: [fix](variant type) fix add variant column, index (apache#53092)

[fix](variant) Fix metadata compatibility for FE (apache#53748)

- In version 3.1, we used 'VariantType', which inherits from
'ScalarType', as the metadata storage for newly created variants,
differing from the previously used 'ScalarType'.
- To ensure compatibility with previous metadata, convert 'VariantType'
to 'ScalarType' before use.

fix

[fix](variant) fix array contains null

fix case

fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmoon eldenmoon eldenmoon approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

dev/3.1.0-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@csun5285 @Thearas @doris-robot @hello-stephen @eldenmoon @morrySnow