Skip to content

[fix](variant) building index on the variant column is prohibited #49159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eldenmoon merged 2 commits into from
Apr 7, 2025
Merged

[fix](variant) building index on the variant column is prohibited #49159

eldenmoon merged 2 commits into from
Apr 7, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Mar 17, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:

Index files are not created for variant columns during the build index process, causing link file operations to fail

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented Mar 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-H: Total hot run time: 32119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17be23e3d43816035ef079a1a7061d8146502f80, data reload: false

------ Round 1 ----------------------------------
q1	24122	5055	5022	5022
q2	2050	298	181	181
q3	10392	1243	663	663
q4	10214	1030	531	531
q5	7531	2329	2328	2328
q6	186	161	130	130
q7	916	729	605	605
q8	9297	1282	1086	1086
q9	4914	4818	4628	4628
q10	6803	2286	1867	1867
q11	467	286	248	248
q12	351	352	220	220
q13	17762	3710	3065	3065
q14	230	232	202	202
q15	527	478	480	478
q16	633	614	595	595
q17	595	841	348	348
q18	7013	6502	6267	6267
q19	1206	947	550	550
q20	323	328	205	205
q21	2829	2170	1937	1937
q22	1042	1006	963	963
Total cold run time: 109403 ms
Total hot run time: 32119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5124	5154	5103	5103
q2	233	319	229	229
q3	2215	2656	2247	2247
q4	1420	1799	1368	1368
q5	4171	4124	4326	4124
q6	208	172	126	126
q7	1997	1909	1737	1737
q8	2559	2554	2482	2482
q9	7216	7125	7225	7125
q10	3006	3185	2687	2687
q11	568	493	498	493
q12	712	803	621	621
q13	3647	3935	3211	3211
q14	278	313	269	269
q15	532	512	474	474
q16	626	697	654	654
q17	1131	1561	1354	1354
q18	7857	7524	7586	7524
q19	867	897	1039	897
q20	1993	2051	1863	1863
q21	5498	4877	4852	4852
q22	1082	1085	1044	1044
Total cold run time: 52940 ms
Total hot run time: 50484 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-DS: Total hot run time: 191774 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17be23e3d43816035ef079a1a7061d8146502f80, data reload: false

query1	1405	1045	1032	1032
query2	6182	1894	1893	1893
query3	11231	4458	4510	4458
query4	54377	24729	23232	23232
query5	5237	525	470	470
query6	403	212	192	192
query7	5265	505	285	285
query8	323	232	233	232
query9	7007	2509	2518	2509
query10	402	306	241	241
query11	15387	15136	14878	14878
query12	158	118	104	104
query13	1207	524	414	414
query14	10440	7144	6486	6486
query15	198	203	200	200
query16	7018	688	434	434
query17	1119	735	578	578
query18	1560	418	328	328
query19	207	202	169	169
query20	136	130	116	116
query21	209	130	106	106
query22	4407	4652	4313	4313
query23	34050	33340	33302	33302
query24	5680	2440	2453	2440
query25	451	459	408	408
query26	798	270	154	154
query27	1948	506	332	332
query28	2961	2448	2452	2448
query29	561	565	448	448
query30	276	234	200	200
query31	864	865	780	780
query32	77	64	64	64
query33	484	381	322	322
query34	756	845	515	515
query35	807	861	781	781
query36	968	994	911	911
query37	123	103	79	79
query38	4251	4134	4298	4134
query39	1514	1443	1435	1435
query40	217	133	103	103
query41	53	51	49	49
query42	116	109	103	103
query43	502	517	497	497
query44	1347	798	798	798
query45	179	176	168	168
query46	847	1041	655	655
query47	1869	1881	1782	1782
query48	379	424	311	311
query49	726	519	432	432
query50	682	759	420	420
query51	4256	4344	4271	4271
query52	109	105	100	100
query53	240	261	190	190
query54	514	516	414	414
query55	85	79	85	79
query56	298	276	279	276
query57	1182	1178	1115	1115
query58	267	241	253	241
query59	2636	2771	2620	2620
query60	291	287	262	262
query61	120	123	119	119
query62	740	742	722	722
query63	230	194	195	194
query64	2108	1027	708	708
query65	4561	4520	4451	4451
query66	753	418	301	301
query67	15815	15624	15329	15329
query68	6789	875	492	492
query69	535	301	275	275
query70	1193	1124	1081	1081
query71	487	311	273	273
query72	5719	3593	3804	3593
query73	1070	748	356	356
query74	8934	9173	8933	8933
query75	3719	3141	2705	2705
query76	4155	1189	776	776
query77	607	392	279	279
query78	10109	10057	9271	9271
query79	3028	843	586	586
query80	754	533	455	455
query81	490	260	221	221
query82	781	128	96	96
query83	308	175	150	150
query84	286	105	75	75
query85	781	361	304	304
query86	403	302	274	274
query87	4322	4555	4389	4389
query88	3600	2231	2221	2221
query89	423	311	272	272
query90	1753	213	221	213
query91	135	141	111	111
query92	71	59	60	59
query93	2543	1072	573	573
query94	674	411	303	303
query95	360	275	275	275
query96	503	572	271	271
query97	3379	3420	3281	3281
query98	231	211	202	202
query99	1471	1399	1276	1276
Total cold run time: 301182 ms
Total hot run time: 191774 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17be23e3d43816035ef079a1a7061d8146502f80, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.24	0.19	0.20
query4	1.59	0.20	0.20
query5	0.59	0.58	0.61
query6	1.20	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.53
query10	0.57	0.57	0.57
query11	0.16	0.11	0.10
query12	0.14	0.12	0.12
query13	0.61	0.60	0.61
query14	2.68	2.71	2.80
query15	0.92	0.86	0.84
query16	0.39	0.39	0.40
query17	1.02	1.02	1.02
query18	0.20	0.19	0.19
query19	1.92	1.96	1.85
query20	0.01	0.01	0.01
query21	15.35	0.95	0.56
query22	0.75	1.23	0.96
query23	14.69	1.43	0.70
query24	6.87	3.14	0.44
query25	0.30	0.17	0.08
query26	0.59	0.16	0.14
query27	0.05	0.05	0.05
query28	9.31	0.83	0.43
query29	12.57	3.94	3.28
query30	0.25	0.08	0.06
query31	2.82	0.60	0.40
query32	3.22	0.54	0.48
query33	3.02	3.08	3.09
query34	15.67	5.14	4.50
query35	4.57	4.49	4.52
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.24 s
Total hot run time: 31.23 s

@csun5285
Copy link
Contributor Author

csun5285 commented Mar 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-H: Total hot run time: 32149 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce10159e59835648b4ad2343054e3749b2ad4d43, data reload: false

------ Round 1 ----------------------------------
q1	24248	5067	5009	5009
q2	2043	309	178	178
q3	10375	1233	678	678
q4	10221	994	532	532
q5	7541	2368	2290	2290
q6	188	162	131	131
q7	888	726	600	600
q8	9305	1219	1028	1028
q9	5045	4891	4904	4891
q10	6809	2286	1881	1881
q11	491	265	258	258
q12	348	348	208	208
q13	17777	3643	3047	3047
q14	229	218	200	200
q15	539	475	487	475
q16	648	622	568	568
q17	577	853	340	340
q18	6778	6316	6256	6256
q19	1217	942	537	537
q20	308	334	193	193
q21	2733	2098	1906	1906
q22	1026	1003	943	943
Total cold run time: 109334 ms
Total hot run time: 32149 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5137	5115	5144	5115
q2	242	324	230	230
q3	2142	2700	2275	2275
q4	1384	1797	1359	1359
q5	4206	4094	4282	4094
q6	213	168	127	127
q7	1972	1893	1766	1766
q8	2589	2584	2565	2565
q9	7185	7146	7188	7146
q10	3018	3151	2837	2837
q11	559	497	474	474
q12	659	724	625	625
q13	3487	3884	3372	3372
q14	276	285	265	265
q15	527	479	485	479
q16	613	675	631	631
q17	1119	1557	1393	1393
q18	7770	7526	7441	7441
q19	839	854	835	835
q20	1978	2005	1891	1891
q21	5464	4891	4668	4668
q22	1105	1047	1015	1015
Total cold run time: 52484 ms
Total hot run time: 50603 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-DS: Total hot run time: 192868 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce10159e59835648b4ad2343054e3749b2ad4d43, data reload: false

query1	1404	1038	1030	1030
query2	6191	1956	1932	1932
query3	11034	4582	4540	4540
query4	55254	24428	23253	23253
query5	4855	572	475	475
query6	334	210	193	193
query7	4870	514	292	292
query8	306	254	238	238
query9	5356	2615	2623	2615
query10	430	304	256	256
query11	15116	15081	14880	14880
query12	157	113	107	107
query13	1048	521	411	411
query14	10166	6790	6885	6790
query15	240	182	178	178
query16	7107	650	492	492
query17	1113	699	568	568
query18	1575	424	312	312
query19	201	195	194	194
query20	131	128	118	118
query21	210	128	109	109
query22	4320	4595	4375	4375
query23	34055	33396	33374	33374
query24	5774	2416	2390	2390
query25	484	480	412	412
query26	693	293	153	153
query27	1795	499	342	342
query28	2784	2483	2471	2471
query29	567	569	433	433
query30	270	239	194	194
query31	882	899	800	800
query32	78	62	64	62
query33	427	369	307	307
query34	777	866	531	531
query35	799	879	800	800
query36	960	1010	887	887
query37	123	108	69	69
query38	4251	4274	4195	4195
query39	1495	1489	1476	1476
query40	211	119	100	100
query41	52	54	49	49
query42	120	105	103	103
query43	501	505	493	493
query44	1319	836	804	804
query45	181	177	174	174
query46	860	1041	636	636
query47	1836	1872	1818	1818
query48	392	438	324	324
query49	705	512	471	471
query50	724	805	424	424
query51	4254	4359	4237	4237
query52	115	114	104	104
query53	229	260	196	196
query54	483	520	427	427
query55	89	79	82	79
query56	281	274	275	274
query57	1185	1190	1107	1107
query58	251	248	241	241
query59	2795	2840	2740	2740
query60	284	270	255	255
query61	121	134	119	119
query62	752	725	658	658
query63	222	195	196	195
query64	1539	1084	710	710
query65	4520	4485	4438	4438
query66	690	395	291	291
query67	16088	15643	15345	15345
query68	7511	871	508	508
query69	575	306	251	251
query70	1193	1092	1123	1092
query71	472	292	267	267
query72	5738	3662	3899	3662
query73	1353	746	351	351
query74	9031	9198	8921	8921
query75	3363	3174	2695	2695
query76	3891	1200	739	739
query77	603	432	274	274
query78	9885	10154	9282	9282
query79	2260	839	578	578
query80	617	538	455	455
query81	485	255	221	221
query82	645	133	94	94
query83	170	168	156	156
query84	284	110	76	76
query85	756	357	300	300
query86	364	277	295	277
query87	4565	4489	4460	4460
query88	3711	2252	2273	2252
query89	418	310	283	283
query90	1752	209	214	209
query91	139	141	108	108
query92	70	60	53	53
query93	1775	1074	588	588
query94	696	404	300	300
query95	346	272	255	255
query96	486	564	283	283
query97	3342	3375	3276	3276
query98	225	211	196	196
query99	1343	1371	1340	1340
Total cold run time: 296495 ms
Total hot run time: 192868 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

ClickBench: Total hot run time: 31.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce10159e59835648b4ad2343054e3749b2ad4d43, data reload: false

query1	0.05	0.04	0.03
query2	0.13	0.11	0.10
query3	0.25	0.19	0.19
query4	1.59	0.19	0.18
query5	0.62	0.56	0.59
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.52	0.50
query10	0.59	0.60	0.56
query11	0.15	0.11	0.10
query12	0.14	0.12	0.12
query13	0.62	0.60	0.60
query14	2.81	2.70	2.68
query15	0.93	0.85	0.85
query16	0.38	0.39	0.38
query17	1.01	1.04	1.02
query18	0.22	0.20	0.20
query19	1.89	1.89	1.84
query20	0.01	0.01	0.01
query21	15.35	0.89	0.55
query22	0.76	1.08	0.67
query23	15.06	1.37	0.62
query24	7.31	1.41	1.27
query25	0.52	0.20	0.06
query26	0.54	0.16	0.12
query27	0.06	0.05	0.04
query28	9.87	0.85	0.43
query29	12.64	3.99	3.30
query30	0.24	0.08	0.06
query31	2.82	0.58	0.38
query32	3.24	0.54	0.46
query33	3.04	3.04	3.09
query34	15.86	5.10	4.51
query35	4.52	4.54	4.53
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 31.54 s

qidaye
qidaye approved these changes Mar 20, 2025
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 20, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 20, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 20, 2025

PR approved by anyone and no changes requested.

eldenmoon
eldenmoon approved these changes Mar 20, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit ca674b2 into apache:master Apr 7, 2025
28 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 7, 2025
@csun5285
[fix](variant) building index on the variant column is prohibited (#4…
83cdcf9 
…9159)

Index files are not created for variant columns during the build index
process, causing link file operations to fail
@github-actions github-actions bot mentioned this pull request Apr 7, 2025
Merged
dataroaring pushed a commit that referenced this pull request Apr 9, 2025
@github-actions @csun5285
branch-3.0: [fix](variant) building index on the variant column is pr…
62bfeac 
…ohibited #49159 (#49844)

Cherry-picked from #49159

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
csun5285 added a commit to csun5285/doris that referenced this pull request Apr 14, 2025
@csun5285
[fix](variant) building index on the variant column is prohibited (ap…
b953be5 
…ache#49159)

Index files are not created for variant columns during the build index
process, causing link file operations to fail
@csun5285 csun5285 mentioned this pull request Apr 14, 2025
Merged
yiguolei pushed a commit that referenced this pull request Apr 16, 2025
@csun5285
[fix](variant) building index on the variant column is prohibited (#4…
2ef136c 
…9159) (#50028)

pick from master #49159
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@csun5285 @koarz
[fix](variant) building index on the variant column is prohibited (ap…
d3b2350 
…ache#49159)

Index files are not created for variant columns during the build index
process, causing link file operations to fail
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qidaye qidaye qidaye approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.5-merged kind/behavior-changed reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@csun5285 @Thearas @doris-robot @qidaye @eldenmoon @yiguolei @dataroaring