[fix](index size) discard index size when meta size is invalid #46549

airborne12 · 2025-01-07T09:45:00Z

What problem does this PR solve?

Issue Number: close #45817

Related PR: #xxx

Problem Summary:
Try to discard and get real inverted index size when rowset meta index size is invalid.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-07T09:45:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2025-01-07T09:45:09Z

run buildall

doris-robot · 2025-01-07T10:05:24Z

TPC-H: Total hot run time: 32827 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91a83ec2493de2fb3583cf57c96d8a23bfc6f303, data reload: false

------ Round 1 ----------------------------------
q1	17599	7081	6016	6016
q2	2044	305	168	168
q3	10415	1244	720	720
q4	10220	852	438	438
q5	7459	2182	1959	1959
q6	211	180	146	146
q7	904	756	611	611
q8	9243	1351	1136	1136
q9	5282	4955	4924	4924
q10	6766	2299	1899	1899
q11	495	285	260	260
q12	357	374	221	221
q13	18235	3746	3221	3221
q14	255	253	224	224
q15	574	524	504	504
q16	633	629	594	594
q17	574	870	331	331
q18	7033	6584	6467	6467
q19	1234	970	527	527
q20	306	325	190	190
q21	2856	2167	1962	1962
q22	369	332	309	309
Total cold run time: 103064 ms
Total hot run time: 32827 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6202	6225	6226	6225
q2	233	324	231	231
q3	2214	2647	2355	2355
q4	1370	1848	1363	1363
q5	4315	4728	4862	4728
q6	182	179	145	145
q7	2099	2041	1804	1804
q8	2604	2765	2694	2694
q9	7262	7216	7229	7216
q10	3024	3327	2822	2822
q11	599	508	515	508
q12	665	749	591	591
q13	3438	3858	3207	3207
q14	320	302	275	275
q15	553	505	496	496
q16	657	694	636	636
q17	1207	1733	1259	1259
q18	7650	7408	6958	6958
q19	799	1021	1064	1021
q20	1905	1988	1819	1819
q21	5336	5122	4821	4821
q22	609	624	579	579
Total cold run time: 53243 ms
Total hot run time: 51753 ms

doris-robot · 2025-01-07T10:17:23Z

TPC-DS: Total hot run time: 190256 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91a83ec2493de2fb3583cf57c96d8a23bfc6f303, data reload: false

query1	967	390	364	364
query2	6512	2441	2301	2301
query3	6711	209	209	209
query4	33776	23668	23546	23546
query5	4359	614	465	465
query6	277	203	184	184
query7	4638	501	308	308
query8	311	250	232	232
query9	9535	2756	2738	2738
query10	473	317	266	266
query11	18074	15494	15250	15250
query12	166	111	115	111
query13	1669	553	441	441
query14	10192	7329	7973	7329
query15	256	192	194	192
query16	7931	580	452	452
query17	1663	762	580	580
query18	1931	370	298	298
query19	208	169	152	152
query20	119	118	111	111
query21	231	117	104	104
query22	4293	4268	4192	4192
query23	34315	33266	33416	33266
query24	6585	2243	2345	2243
query25	472	443	381	381
query26	761	267	153	153
query27	2067	452	334	334
query28	5353	2458	2442	2442
query29	600	528	414	414
query30	224	181	147	147
query31	981	896	825	825
query32	82	61	60	60
query33	518	383	286	286
query34	763	842	518	518
query35	799	820	752	752
query36	994	1031	965	965
query37	121	96	75	75
query38	4277	4260	4279	4260
query39	1504	1440	1409	1409
query40	204	118	96	96
query41	49	45	47	45
query42	119	108	104	104
query43	517	510	496	496
query44	1292	796	817	796
query45	182	175	164	164
query46	872	1039	652	652
query47	1890	1924	1891	1891
query48	392	399	323	323
query49	752	478	401	401
query50	624	632	399	399
query51	6846	7035	6718	6718
query52	104	101	96	96
query53	219	250	193	193
query54	475	474	390	390
query55	81	78	77	77
query56	253	251	248	248
query57	1213	1213	1132	1132
query58	251	232	234	232
query59	2879	3174	2876	2876
query60	269	284	242	242
query61	111	106	102	102
query62	922	779	761	761
query63	233	204	195	195
query64	3265	987	642	642
query65	3279	3199	3244	3199
query66	880	421	327	327
query67	15963	16061	15502	15502
query68	7915	701	513	513
query69	484	289	253	253
query70	1207	1151	1080	1080
query71	422	276	270	270
query72	6249	3853	3892	3853
query73	660	747	377	377
query74	9668	9115	8821	8821
query75	3885	3137	2679	2679
query76	3586	1189	791	791
query77	769	454	279	279
query78	10046	10154	9485	9485
query79	3623	819	591	591
query80	714	515	427	427
query81	472	273	240	240
query82	624	149	121	121
query83	193	168	147	147
query84	284	97	76	76
query85	744	365	307	307
query86	357	328	281	281
query87	4662	4500	4370	4370
query88	4220	2231	2204	2204
query89	412	323	336	323
query90	1943	190	197	190
query91	132	136	103	103
query92	61	63	56	56
query93	1630	866	522	522
query94	647	396	258	258
query95	332	260	250	250
query96	487	606	274	274
query97	2937	2998	2783	2783
query98	236	198	200	198
query99	1698	1546	1480	1480
Total cold run time: 290143 ms
Total hot run time: 190256 ms

doris-robot · 2025-01-07T10:22:46Z

ClickBench: Total hot run time: 31.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91a83ec2493de2fb3583cf57c96d8a23bfc6f303, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.11	0.11
query5	0.43	0.40	0.43
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.49	0.52
query10	0.55	0.55	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.10
query13	0.61	0.62	0.61
query14	2.73	2.75	2.73
query15	0.90	0.82	0.83
query16	0.38	0.38	0.38
query17	1.05	1.09	1.07
query18	0.24	0.20	0.21
query19	1.90	1.84	1.91
query20	0.01	0.01	0.01
query21	15.36	0.99	0.56
query22	0.77	0.82	0.59
query23	15.37	1.41	0.57
query24	3.02	1.57	1.37
query25	0.15	0.24	0.09
query26	0.16	0.14	0.14
query27	0.05	0.07	0.05
query28	14.15	1.50	1.05
query29	12.53	3.81	3.22
query30	0.25	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.46
query33	3.10	3.18	3.07
query34	16.74	5.16	4.55
query35	4.49	4.46	4.53
query36	0.84	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.34 s
Total hot run time: 31.65 s

doris-robot · 2025-01-07T10:25:35Z

TeamCity be ut coverage result:
Function Coverage: 38.86% (10130/26066)
Line Coverage: 29.92% (85734/286545)
Region Coverage: 29.01% (43732/150749)
Branch Coverage: 25.55% (22324/87366)
Coverage Report: http://coverage.selectdb-in.cc/coverage/91a83ec2493de2fb3583cf57c96d8a23bfc6f303_91a83ec2493de2fb3583cf57c96d8a23bfc6f303/report/index.html

github-actions · 2025-01-07T10:43:39Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-07T10:43:41Z

PR approved by anyone and no changes requested.

airborne12 · 2025-01-08T08:52:07Z

run buildall

airborne12 · 2025-01-08T09:50:26Z

run buildall

doris-robot · 2025-01-08T10:41:38Z

TPC-H: Total hot run time: 32855 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2801327cee3f7239bec0056c9be662abacfbf46d, data reload: false

------ Round 1 ----------------------------------
q1	17634	6177	6025	6025
q2	2046	307	169	169
q3	10420	1316	750	750
q4	10199	867	439	439
q5	7515	2238	1957	1957
q6	201	182	150	150
q7	924	814	616	616
q8	9251	1409	1199	1199
q9	5401	4928	4943	4928
q10	6761	2315	1840	1840
q11	493	272	262	262
q12	347	361	229	229
q13	17784	3737	3105	3105
q14	239	227	214	214
q15	578	527	491	491
q16	618	630	586	586
q17	588	862	330	330
q18	7111	6502	6494	6494
q19	1694	992	575	575
q20	303	319	196	196
q21	2863	2292	1990	1990
q22	358	336	310	310
Total cold run time: 103328 ms
Total hot run time: 32855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6256	6209	6233	6209
q2	240	328	242	242
q3	2271	2649	2318	2318
q4	1431	1865	1370	1370
q5	4345	4753	4860	4753
q6	187	179	148	148
q7	2118	2024	1884	1884
q8	2607	2843	2691	2691
q9	7323	7247	7174	7174
q10	3119	3326	2801	2801
q11	580	536	511	511
q12	678	776	605	605
q13	3516	3900	3230	3230
q14	287	299	277	277
q15	569	516	505	505
q16	636	665	633	633
q17	1216	1737	1260	1260
q18	7779	7263	7149	7149
q19	835	1129	1070	1070
q20	1925	2034	1857	1857
q21	5475	5063	4899	4899
q22	620	612	576	576
Total cold run time: 54013 ms
Total hot run time: 52162 ms

doris-robot · 2025-01-08T10:53:37Z

TPC-DS: Total hot run time: 188630 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2801327cee3f7239bec0056c9be662abacfbf46d, data reload: false

query1	975	376	369	369
query2	6513	2521	2413	2413
query3	6714	218	222	218
query4	33878	24195	23286	23286
query5	4370	610	435	435
query6	274	207	198	198
query7	4623	495	308	308
query8	309	254	243	243
query9	9569	2768	2752	2752
query10	455	321	245	245
query11	17935	15269	15283	15269
query12	158	109	108	108
query13	1686	549	413	413
query14	10700	7109	7108	7108
query15	225	203	187	187
query16	8111	626	473	473
query17	1592	752	603	603
query18	2100	395	290	290
query19	212	173	161	161
query20	120	139	113	113
query21	211	121	100	100
query22	4133	4356	4238	4238
query23	34135	32572	33256	32572
query24	6350	2261	2230	2230
query25	470	451	386	386
query26	801	279	152	152
query27	1979	469	336	336
query28	5226	2464	2463	2463
query29	611	540	421	421
query30	224	181	144	144
query31	944	880	800	800
query32	72	59	62	59
query33	526	349	305	305
query34	755	834	514	514
query35	796	808	715	715
query36	1020	1038	933	933
query37	121	97	74	74
query38	4262	4256	4091	4091
query39	1474	1400	1395	1395
query40	208	115	96	96
query41	50	44	46	44
query42	123	100	100	100
query43	514	548	471	471
query44	1315	815	827	815
query45	177	174	160	160
query46	859	1044	642	642
query47	1833	1879	1786	1786
query48	389	400	316	316
query49	726	495	372	372
query50	619	670	393	393
query51	6848	6907	6999	6907
query52	108	101	89	89
query53	218	245	186	186
query54	482	470	411	411
query55	83	78	81	78
query56	244	281	238	238
query57	1167	1185	1066	1066
query58	225	228	231	228
query59	3019	3294	3018	3018
query60	287	265	254	254
query61	113	105	110	105
query62	826	745	710	710
query63	229	203	195	195
query64	3359	1006	644	644
query65	3293	3142	3184	3142
query66	856	421	317	317
query67	16042	15666	15276	15276
query68	8504	696	511	511
query69	486	292	260	260
query70	1244	1126	1055	1055
query71	431	277	261	261
query72	6318	3812	3917	3812
query73	651	752	361	361
query74	10055	8965	8957	8957
query75	4609	3146	2644	2644
query76	4489	1184	764	764
query77	856	373	273	273
query78	10034	9996	9451	9451
query79	4367	824	587	587
query80	747	534	474	474
query81	492	265	227	227
query82	460	146	129	129
query83	201	167	146	146
query84	279	93	73	73
query85	753	367	308	308
query86	356	286	312	286
query87	4307	4292	4411	4292
query88	3498	2239	2208	2208
query89	436	332	305	305
query90	2039	187	184	184
query91	130	140	109	109
query92	66	57	51	51
query93	2528	844	517	517
query94	675	379	284	284
query95	335	263	256	256
query96	494	620	282	282
query97	2871	2908	2805	2805
query98	221	209	199	199
query99	1617	1509	1380	1380
Total cold run time: 293390 ms
Total hot run time: 188630 ms

doris-robot · 2025-01-08T10:57:08Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10130/26061)
Line Coverage: 29.93% (85763/286506)
Region Coverage: 29.04% (43771/150719)
Branch Coverage: 25.57% (22340/87354)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2801327cee3f7239bec0056c9be662abacfbf46d_2801327cee3f7239bec0056c9be662abacfbf46d/report/index.html

doris-robot · 2025-01-08T10:58:57Z

ClickBench: Total hot run time: 31.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2801327cee3f7239bec0056c9be662abacfbf46d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.44	0.40	0.41
query6	1.16	0.66	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.49
query10	0.55	0.57	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.59
query14	2.72	2.76	2.82
query15	0.89	0.82	0.82
query16	0.38	0.37	0.38
query17	1.05	1.00	1.02
query18	0.24	0.21	0.20
query19	1.90	1.89	2.04
query20	0.01	0.01	0.01
query21	15.36	0.94	0.59
query22	0.74	0.73	0.59
query23	15.43	1.48	0.60
query24	3.01	1.01	1.07
query25	0.33	0.14	0.05
query26	0.20	0.14	0.13
query27	0.05	0.05	0.06
query28	13.76	1.57	1.05
query29	12.59	3.91	3.27
query30	0.25	0.09	0.07
query31	2.84	0.60	0.38
query32	3.22	0.57	0.46
query33	3.07	3.08	3.09
query34	16.96	5.20	4.51
query35	4.56	4.50	4.53
query36	0.62	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.32 s
Total hot run time: 31.32 s

qidaye

LGTM

github-actions · 2025-01-08T11:46:55Z

PR approved by at least one committer and no changes requested.

csun5285

LGTM

…valid #46549 (#46717) cherry pick from #46549

gavinchou previously approved these changes Jan 7, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 7, 2025

github-actions bot added the reviewed label Jan 7, 2025

airborne12 dismissed gavinchou’s stale review via 309ecc3 January 7, 2025 11:46

airborne12 force-pushed the fix-ordered branch from 91a83ec to 309ecc3 Compare January 7, 2025 11:46

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 7, 2025

airborne12 force-pushed the fix-ordered branch 2 times, most recently from 8248767 to fd0ecee Compare January 8, 2025 08:10

airborne12 closed this Jan 8, 2025

airborne12 force-pushed the fix-ordered branch from fd0ecee to 744691a Compare January 8, 2025 08:45

[fix](ordered compaction) discard index size when meta size is invalid

61081b4

airborne12 reopened this Jan 8, 2025

airborne12 changed the title ~~[fix](ordered compaction) Discard index size when meta size is invalid~~ [fix](ordered compaction) discard index size when meta size is invalid Jan 8, 2025

airborne12 changed the title ~~[fix](ordered compaction) discard index size when meta size is invalid~~ [fix](index size) discard index size when meta size is invalid Jan 8, 2025

dataroaring added the p0_b label Jan 8, 2025

fix compile

2801327

qidaye approved these changes Jan 8, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 8, 2025

csun5285 approved these changes Jan 9, 2025

View reviewed changes

gavinchou approved these changes Jan 9, 2025

View reviewed changes

airborne12 merged commit 4624dea into apache:master Jan 9, 2025
26 of 28 checks passed

airborne12 deleted the fix-ordered branch January 9, 2025 10:34

airborne12 added dev/2.1.x dev/3.0.x labels Jan 9, 2025

github-actions bot added dev/2.1.x-conflict dev/3.0.x-conflict labels Jan 9, 2025

airborne12 mentioned this pull request Jan 9, 2025

branch-3.0: [fix](index size) discard index size when meta size is invalid #46549 #46717

Merged

airborne12 added a commit that referenced this pull request Jan 13, 2025

branch-3.0: [fix](index size) discard index size when meta size is in…

e174ddc

…valid #46549 (#46717) cherry pick from #46549

airborne12 added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict dev/2.1.x dev/2.1.x-conflict labels Jan 13, 2025

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

[fix](index size) discard index size when meta size is invalid #46549

[fix](index size) discard index size when meta size is invalid #46549

Uh oh!

Conversation

airborne12 commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 7, 2025

Uh oh!

airborne12 commented Jan 7, 2025

Uh oh!

doris-robot commented Jan 7, 2025

Uh oh!

doris-robot commented Jan 7, 2025

Uh oh!

doris-robot commented Jan 7, 2025

Uh oh!

doris-robot commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

airborne12 commented Jan 8, 2025

Uh oh!

airborne12 commented Jan 8, 2025

Uh oh!

doris-robot commented Jan 8, 2025

Uh oh!

doris-robot commented Jan 8, 2025

Uh oh!

doris-robot commented Jan 8, 2025

Uh oh!

doris-robot commented Jan 8, 2025

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 8, 2025

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

airborne12 commented Jan 7, 2025 •

edited

Loading