Skip to content

branch-3.0: [fix](index build) Correct inverted index behavior after dynamically adding a column #48389 #48547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Mar 8, 2025
Merged

branch-3.0: [fix](index build) Correct inverted index behavior after dynamically adding a column #48389 #48547

airborne12 merged 1 commit into from
Mar 8, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Mar 3, 2025

Cherry-picked from #48389

@github-actions github-actions bot requested a review from dataroaring as a code owner March 3, 2025 07:29
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Mar 3, 2025
@dataroaring dataroaring reopened this Mar 3, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 3, 2025

run buildall

dataroaring
dataroaring approved these changes Mar 4, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 4, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Mar 4, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented Mar 4, 2025

PR approved by anyone and no changes requested.

@airborne12 airborne12 force-pushed the auto-pick-48389-branch-3.0 branch from e04f534 to 2dbfe9d Compare March 5, 2025 01:27
@airborne12
Copy link
Member

airborne12 commented Mar 5, 2025

run buildall

@airborne12 airborne12 force-pushed the auto-pick-48389-branch-3.0 branch from 2dbfe9d to e795fd0 Compare March 5, 2025 10:53
@airborne12
Copy link
Member

airborne12 commented Mar 5, 2025

run buildall

@airborne12
[fix](index build) Correct inverted index behavior after dynamically …
01753a4 
…adding a column (#48389)

Problem Summary:
This pull request includes changes to improve the handling of inverted
indexes and to add new test cases for index changes on new columns. The
most important changes include fixing the placement of inverted index
writers, adding new test cases for inverted index changes, and modifying
the test output file to reflect the new test cases.
@airborne12 airborne12 force-pushed the auto-pick-48389-branch-3.0 branch from e795fd0 to 01753a4 Compare March 7, 2025 01:34
@airborne12
Copy link
Member

airborne12 commented Mar 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 7, 2025

TPC-H: Total hot run time: 40124 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01753a497f8c35d5a3eb5bc15683c43934b6cd7a, data reload: false

------ Round 1 ----------------------------------
q1	17589	6844	6732	6732
q2	2058	168	161	161
q3	10652	1088	1216	1088
q4	10548	753	723	723
q5	7760	2859	2804	2804
q6	220	133	126	126
q7	972	628	611	611
q8	9509	1927	2034	1927
q9	6694	6466	6444	6444
q10	6956	2272	2259	2259
q11	461	264	268	264
q12	395	215	207	207
q13	17764	2991	2995	2991
q14	244	209	207	207
q15	507	468	460	460
q16	677	600	605	600
q17	988	559	538	538
q18	7192	6642	6675	6642
q19	1397	1058	949	949
q20	486	210	204	204
q21	4010	3192	3253	3192
q22	1100	1006	995	995
Total cold run time: 108179 ms
Total hot run time: 40124 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6567	6587	6549	6549
q2	327	231	228	228
q3	2893	2734	2862	2734
q4	2033	1816	1831	1816
q5	5790	5742	5778	5742
q6	206	128	125	125
q7	2233	1799	1819	1799
q8	3397	3552	3498	3498
q9	8961	8790	8903	8790
q10	3552	3501	3530	3501
q11	581	500	491	491
q12	833	598	597	597
q13	8386	3189	3127	3127
q14	323	285	267	267
q15	522	471	474	471
q16	704	645	645	645
q17	1859	1641	1604	1604
q18	8173	7927	7744	7744
q19	1650	1512	1579	1512
q20	2051	1867	1872	1867
q21	5555	5460	5338	5338
q22	1119	1075	1015	1015
Total cold run time: 67715 ms
Total hot run time: 59460 ms

@doris-robot
Copy link

doris-robot commented Mar 7, 2025

TPC-DS: Total hot run time: 195930 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01753a497f8c35d5a3eb5bc15683c43934b6cd7a, data reload: false

query1	1277	915	902	902
query2	6233	2076	2094	2076
query3	10803	4428	4136	4136
query4	61076	29119	23163	23163
query5	5193	477	457	457
query6	406	185	177	177
query7	5520	329	306	306
query8	315	229	221	221
query9	8495	2579	2571	2571
query10	463	286	263	263
query11	17517	15026	15629	15026
query12	154	104	106	104
query13	1513	448	426	426
query14	10875	7114	6693	6693
query15	205	186	177	177
query16	7082	523	508	508
query17	1134	614	584	584
query18	1894	324	314	314
query19	212	163	161	161
query20	117	111	124	111
query21	206	102	101	101
query22	4768	4931	4683	4683
query23	34497	34124	33983	33983
query24	6130	2900	2889	2889
query25	515	426	395	395
query26	646	164	162	162
query27	1731	361	357	357
query28	4043	2421	2413	2413
query29	684	440	431	431
query30	252	167	157	157
query31	998	842	822	822
query32	65	52	54	52
query33	451	275	282	275
query34	942	498	505	498
query35	843	764	710	710
query36	1085	982	973	973
query37	111	64	65	64
query38	4120	4011	3954	3954
query39	1555	1466	1549	1466
query40	208	103	103	103
query41	51	53	50	50
query42	119	104	103	103
query43	550	510	487	487
query44	1177	828	822	822
query45	183	174	177	174
query46	1161	750	711	711
query47	2043	1948	1978	1948
query48	482	390	399	390
query49	763	425	408	408
query50	843	448	425	425
query51	7397	7172	7352	7172
query52	108	94	96	94
query53	265	197	188	188
query54	567	478	468	468
query55	87	85	84	84
query56	276	264	252	252
query57	1298	1195	1140	1140
query58	217	209	218	209
query59	3245	2983	2922	2922
query60	296	252	252	252
query61	104	141	106	106
query62	758	679	678	678
query63	220	186	181	181
query64	1374	664	623	623
query65	3222	3157	3198	3157
query66	689	287	295	287
query67	15982	15639	15510	15510
query68	4431	576	569	569
query69	421	272	263	263
query70	1129	1101	1108	1101
query71	319	263	249	249
query72	6277	3781	4021	3781
query73	762	345	347	345
query74	10416	8954	9171	8954
query75	3347	2617	2676	2617
query76	1953	1103	1143	1103
query77	468	260	261	260
query78	10622	9733	9586	9586
query79	1266	573	588	573
query80	769	431	413	413
query81	514	239	231	231
query82	1303	87	92	87
query83	166	140	140	140
query84	289	81	82	81
query85	831	319	296	296
query86	312	285	291	285
query87	4393	4305	4237	4237
query88	3617	2342	2342	2342
query89	411	295	297	295
query90	2032	184	181	181
query91	180	149	150	149
query92	61	49	51	49
query93	1277	538	541	538
query94	739	281	290	281
query95	352	257	255	255
query96	600	275	278	275
query97	3348	3217	3210	3210
query98	212	203	196	196
query99	1771	1285	1283	1283
Total cold run time: 312399 ms
Total hot run time: 195930 ms

@doris-robot
Copy link

doris-robot commented Mar 7, 2025

ClickBench: Total hot run time: 33.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01753a497f8c35d5a3eb5bc15683c43934b6cd7a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.51	0.50	0.52
query6	1.14	0.73	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.51
query10	0.53	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.58	0.60
query14	2.74	2.72	2.85
query15	0.90	0.83	0.82
query16	0.36	0.38	0.37
query17	1.07	1.03	1.05
query18	0.25	0.23	0.22
query19	1.90	1.89	1.98
query20	0.02	0.01	0.01
query21	15.36	0.58	0.57
query22	2.74	2.61	2.44
query23	16.93	1.06	0.81
query24	3.18	1.43	1.40
query25	0.23	0.18	0.13
query26	0.41	0.14	0.13
query27	0.03	0.04	0.03
query28	9.56	0.49	0.46
query29	12.59	3.25	3.29
query30	0.24	0.06	0.06
query31	2.86	0.40	0.40
query32	3.22	0.46	0.47
query33	3.02	3.04	3.03
query34	16.98	4.43	4.48
query35	4.50	4.51	4.51
query36	0.66	0.48	0.48
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.89 s
Total hot run time: 33.25 s

@airborne12 airborne12 merged commit cd7e03f into branch-3.0 Mar 8, 2025
24 of 27 checks passed
@airborne12 airborne12 deleted the auto-pick-48389-branch-3.0 branch March 8, 2025 04:12
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @airborne12 @doris-robot @dataroaring