Skip to content

[fix] (inverted index) Disallow variant columns from using inverted index format v1 #49823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 4 commits into from
Apr 8, 2025
Merged

[fix] (inverted index) Disallow variant columns from using inverted index format v1 #49823

airborne12 merged 4 commits into from
Apr 8, 2025

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Apr 7, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:

Disallow variant columns from using inverted index format v1

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285 csun5285 force-pushed the enable_v1_for_variant branch from c225a3f to bbbc307 Compare April 7, 2025 06:03
@csun5285 csun5285 force-pushed the enable_v1_for_variant branch from bbbc307 to 646d804 Compare April 7, 2025 06:04
@csun5285
Copy link
Contributor Author

csun5285 commented Apr 7, 2025

run buildall

@csun5285
Copy link
Contributor Author

csun5285 commented Apr 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

TPC-H: Total hot run time: 35108 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 646d8042c3a8a9681717107355b7402d648f0663, data reload: false

------ Round 1 ----------------------------------
q1	25675	5067	5022	5022
q2	2070	289	202	202
q3	10379	1232	707	707
q4	10219	1030	544	544
q5	7519	2316	2357	2316
q6	185	166	133	133
q7	922	769	615	615
q8	9329	1324	1133	1133
q9	6898	5226	5176	5176
q10	6824	2279	1903	1903
q11	476	289	285	285
q12	354	368	228	228
q13	17784	3696	3119	3119
q14	229	231	203	203
q15	518	495	485	485
q16	639	622	586	586
q17	571	863	357	357
q18	7562	7339	7084	7084
q19	1222	952	562	562
q20	348	337	227	227
q21	4034	3535	3220	3220
q22	1095	1006	1001	1001
Total cold run time: 114852 ms
Total hot run time: 35108 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5227	5130	5156	5130
q2	238	319	225	225
q3	2173	2649	2262	2262
q4	1452	1876	1517	1517
q5	4425	4424	4374	4374
q6	220	189	130	130
q7	2014	1943	1823	1823
q8	2566	2559	2538	2538
q9	7400	7260	7226	7226
q10	3063	3220	2805	2805
q11	594	508	493	493
q12	702	775	622	622
q13	3522	3949	3399	3399
q14	278	295	264	264
q15	528	471	485	471
q16	638	708	642	642
q17	1145	1540	1388	1388
q18	7691	7634	7503	7503
q19	811	801	810	801
q20	1948	2048	1859	1859
q21	5379	4797	4783	4783
q22	1126	1078	1061	1061
Total cold run time: 53140 ms
Total hot run time: 51316 ms

@csun5285
Copy link
Contributor Author

csun5285 commented Apr 7, 2025

run buildall

@csun5285
Copy link
Contributor Author

csun5285 commented Apr 7, 2025

run buildall

gavinchou
gavinchou previously approved these changes Apr 7, 2025
View reviewed changes
@gavinchou
Copy link
Contributor

gavinchou commented Apr 7, 2025

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 7, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2025

PR approved by anyone and no changes requested.

eldenmoon
eldenmoon previously approved these changes Apr 7, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csun5285 csun5285 dismissed stale reviews from eldenmoon and gavinchou via 116186d April 7, 2025 10:26
@csun5285
Copy link
Contributor Author

csun5285 commented Apr 7, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 7, 2025
@doris-robot
Copy link

doris-robot commented Apr 7, 2025

TPC-H: Total hot run time: 34225 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 116186dbbc7623e5682c3d0f2770064731b0fac9, data reload: false

------ Round 1 ----------------------------------
q1	25949	5207	5042	5042
q2	2064	285	187	187
q3	10638	1243	707	707
q4	10287	1015	541	541
q5	8821	2331	2402	2331
q6	189	165	135	135
q7	902	743	612	612
q8	9323	1318	1145	1145
q9	7419	5125	5106	5106
q10	6822	2314	1865	1865
q11	482	282	270	270
q12	342	363	220	220
q13	17760	3670	3081	3081
q14	228	233	224	224
q15	532	490	486	486
q16	639	611	585	585
q17	574	853	358	358
q18	7662	7162	7177	7162
q19	1679	963	551	551
q20	330	318	220	220
q21	4141	3431	2435	2435
q22	1082	1016	962	962
Total cold run time: 117865 ms
Total hot run time: 34225 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5219	5128	5180	5128
q2	244	338	236	236
q3	2153	2684	2263	2263
q4	1433	1958	1543	1543
q5	4576	4435	4293	4293
q6	208	167	128	128
q7	1985	1885	1744	1744
q8	2597	2520	2541	2520
q9	7256	7184	7089	7089
q10	2985	3174	2759	2759
q11	566	501	493	493
q12	650	757	627	627
q13	3502	3821	3316	3316
q14	283	305	277	277
q15	509	469	470	469
q16	655	680	655	655
q17	1155	1511	1391	1391
q18	7797	7446	7309	7309
q19	808	797	926	797
q20	1887	1947	1829	1829
q21	5159	4623	4630	4623
q22	1067	1062	975	975
Total cold run time: 52694 ms
Total hot run time: 50464 ms

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

TPC-DS: Total hot run time: 184763 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 116186dbbc7623e5682c3d0f2770064731b0fac9, data reload: false

query1	990	468	465	465
query2	6565	1900	1884	1884
query3	6738	223	220	220
query4	25879	23380	23031	23031
query5	4337	635	483	483
query6	315	223	198	198
query7	4627	478	279	279
query8	296	250	245	245
query9	8619	2591	2564	2564
query10	487	319	258	258
query11	15389	15118	14776	14776
query12	162	118	108	108
query13	1661	527	405	405
query14	9413	6169	6172	6169
query15	200	189	173	173
query16	7268	641	468	468
query17	1197	726	595	595
query18	2061	406	316	316
query19	196	183	160	160
query20	124	121	122	121
query21	211	123	105	105
query22	4354	4543	4064	4064
query23	33726	32859	32899	32859
query24	8430	2344	2378	2344
query25	564	453	403	403
query26	1223	265	144	144
query27	2741	502	320	320
query28	4336	2425	2402	2402
query29	753	587	418	418
query30	292	221	195	195
query31	957	854	739	739
query32	70	61	65	61
query33	572	356	303	303
query34	789	855	521	521
query35	787	809	739	739
query36	951	975	880	880
query37	119	97	76	76
query38	4259	4076	3959	3959
query39	1439	1380	1398	1380
query40	206	116	109	109
query41	54	55	73	55
query42	117	104	107	104
query43	472	489	461	461
query44	1301	776	783	776
query45	175	180	167	167
query46	826	1016	613	613
query47	1783	1807	1762	1762
query48	381	411	311	311
query49	802	512	414	414
query50	651	671	398	398
query51	4219	4167	4159	4159
query52	108	104	105	104
query53	234	248	177	177
query54	586	565	507	507
query55	86	86	84	84
query56	312	323	289	289
query57	1103	1146	1092	1092
query58	281	261	264	261
query59	2623	2655	2471	2471
query60	323	326	300	300
query61	129	127	140	127
query62	775	754	683	683
query63	226	182	185	182
query64	4362	1059	686	686
query65	4296	4162	4208	4162
query66	1147	424	312	312
query67	15807	15273	15241	15241
query68	8901	841	518	518
query69	469	296	256	256
query70	1197	1106	1062	1062
query71	444	317	305	305
query72	5498	4603	4597	4597
query73	696	555	333	333
query74	8959	9101	8575	8575
query75	4217	3170	2686	2686
query76	3620	1185	754	754
query77	786	366	280	280
query78	9954	10117	9155	9155
query79	2140	814	566	566
query80	595	518	426	426
query81	481	255	226	226
query82	430	130	101	101
query83	294	258	238	238
query84	301	102	85	85
query85	797	432	307	307
query86	330	286	334	286
query87	4365	4526	4245	4245
query88	2847	2205	2199	2199
query89	390	311	277	277
query90	1869	210	212	210
query91	139	142	111	111
query92	75	63	57	57
query93	1127	939	594	594
query94	690	409	304	304
query95	368	345	280	280
query96	490	562	285	285
query97	3166	3239	3146	3146
query98	234	210	207	207
query99	1430	1381	1255	1255
Total cold run time: 273781 ms
Total hot run time: 184763 ms

@doris-robot
Copy link

doris-robot commented Apr 7, 2025

ClickBench: Total hot run time: 30.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 116186dbbc7623e5682c3d0f2770064731b0fac9, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.11	0.11
query3	0.25	0.19	0.20
query4	1.59	0.20	0.11
query5	0.56	0.55	0.56
query6	1.18	0.72	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.52
query10	0.61	0.59	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.59	0.61
query14	2.78	2.82	2.70
query15	0.94	0.85	0.84
query16	0.39	0.38	0.41
query17	1.03	0.99	1.04
query18	0.22	0.19	0.19
query19	2.09	1.85	1.81
query20	0.01	0.02	0.01
query21	15.34	0.93	0.55
query22	0.75	1.09	0.64
query23	15.09	1.40	0.62
query24	6.89	2.13	0.64
query25	0.46	0.11	0.08
query26	0.68	0.16	0.13
query27	0.06	0.05	0.05
query28	9.17	0.90	0.44
query29	12.55	3.94	3.29
query30	0.25	0.09	0.06
query31	2.84	0.59	0.38
query32	3.23	0.55	0.46
query33	3.11	3.06	3.04
query34	15.75	5.04	4.46
query35	4.53	4.55	4.47
query36	0.67	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.13	0.12
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.24 s
Total hot run time: 30.68 s

eldenmoon
eldenmoon approved these changes Apr 8, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Apr 8, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 8, 2025
airborne12
airborne12 approved these changes Apr 8, 2025
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 2f2cdef into apache:master Apr 8, 2025
26 of 28 checks passed
csun5285 added a commit to csun5285/doris that referenced this pull request Apr 9, 2025
@csun5285
[fix] (inverted index) Disallow variant columns from using inverted i…
559eb65 
…ndex format v1 (apache#49823)

Problem Summary:

Disallow variant columns from using inverted index format v1
@csun5285 csun5285 mentioned this pull request Apr 9, 2025
Merged
csun5285 added a commit to csun5285/doris that referenced this pull request Apr 9, 2025
@csun5285
[fix] (inverted index) Disallow variant columns from using inverted i…
30ef05a 
…ndex format v1 (apache#49823)

Problem Summary:

Disallow variant columns from using inverted index format v1

fix confilct

fix
dataroaring pushed a commit that referenced this pull request Apr 9, 2025
@csun5285
[fix] (inverted index) Disallow variant columns from using inverted i…
d2cae4d 
…ndex format v1 (#49823) (#49890)

pick from master #49823
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@csun5285 @koarz
[fix] (inverted index) Disallow variant columns from using inverted i…
230aa8f 
…ndex format v1 (apache#49823)

### What problem does this PR solve?

Problem Summary:

Disallow variant columns from using inverted index format v1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

@gavinchou gavinchou gavinchou left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@csun5285 @hello-stephen @doris-robot @gavinchou @airborne12 @eldenmoon @dataroaring