Skip to content

[fix] (move-memtable) fix errors caused by checks in the inverted index file writer #37621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qidaye merged 1 commit into from
Jul 11, 2024
Merged

[fix] (move-memtable) fix errors caused by checks in the inverted index file writer #37621

qidaye merged 1 commit into from
Jul 11, 2024

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 10, 2024 •
edited
Loading

Proposed changes

image

  1. thread A flush segment 0, flush inverted index 0
  2. thread B flush segment 1
  3. thread A add segment 0, dst be will check inverted index file size == segment file size, it will fail at this point.

Issue Number: close #xxx

check the inverted index file size == the segment file size during close()

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 40568 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e51c53a751da9c3e03ca4b1aae6c582260e5f0a4, data reload: false

------ Round 1 ----------------------------------
q1	17611	4297	4288	4288
q2	2020	196	188	188
q3	10455	1206	1076	1076
q4	10182	883	893	883
q5	7485	2657	2620	2620
q6	221	140	139	139
q7	942	614	611	611
q8	9249	2072	2045	2045
q9	8853	6495	6458	6458
q10	8989	3702	3696	3696
q11	474	232	231	231
q12	415	222	227	222
q13	17774	2990	2948	2948
q14	268	234	226	226
q15	522	488	476	476
q16	482	376	366	366
q17	964	676	768	676
q18	8021	7475	7387	7387
q19	7876	1428	1552	1428
q20	655	320	313	313
q21	4862	3950	3986	3950
q22	403	341	343	341
Total cold run time: 118723 ms
Total hot run time: 40568 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4401	4290	4272	4272
q2	366	268	280	268
q3	3114	2938	2960	2938
q4	1988	1724	1706	1706
q5	5479	5495	5534	5495
q6	224	133	134	133
q7	2231	1871	1829	1829
q8	3287	3389	3393	3389
q9	8648	8849	8720	8720
q10	3966	3750	3912	3750
q11	584	494	482	482
q12	810	649	640	640
q13	17251	3211	3189	3189
q14	295	283	279	279
q15	520	491	495	491
q16	473	435	442	435
q17	1818	1514	1514	1514
q18	8144	7929	7944	7929
q19	1745	1577	1651	1577
q20	2217	1858	1850	1850
q21	5290	4721	4920	4721
q22	612	561	537	537
Total cold run time: 73463 ms
Total hot run time: 56144 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 173965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e51c53a751da9c3e03ca4b1aae6c582260e5f0a4, data reload: false

query1	918	380	361	361
query2	6385	2506	2276	2276
query3	6633	204	222	204
query4	28293	17449	17189	17189
query5	3783	468	470	468
query6	268	178	168	168
query7	4584	292	284	284
query8	320	291	290	290
query9	8600	2378	2374	2374
query10	427	289	275	275
query11	10589	10011	9924	9924
query12	116	87	80	80
query13	1634	370	372	370
query14	10174	7731	7545	7545
query15	238	187	192	187
query16	7802	305	304	304
query17	1811	542	516	516
query18	1943	267	267	267
query19	191	147	149	147
query20	90	80	83	80
query21	204	126	123	123
query22	4328	4157	4154	4154
query23	34051	33610	33461	33461
query24	10784	2889	2828	2828
query25	590	407	429	407
query26	701	154	156	154
query27	2275	290	282	282
query28	6339	2149	2105	2105
query29	924	651	630	630
query30	261	157	162	157
query31	967	756	765	756
query32	103	55	58	55
query33	643	298	293	293
query34	934	499	503	499
query35	706	643	596	596
query36	1134	982	998	982
query37	140	81	85	81
query38	2958	2818	2746	2746
query39	887	816	804	804
query40	202	118	117	117
query41	53	50	55	50
query42	118	97	102	97
query43	616	564	589	564
query44	1103	728	743	728
query45	193	161	159	159
query46	1091	751	729	729
query47	1843	1764	1787	1764
query48	370	310	302	302
query49	837	407	420	407
query50	786	389	394	389
query51	6850	6776	6788	6776
query52	114	92	93	92
query53	368	296	295	295
query54	874	438	443	438
query55	75	74	76	74
query56	293	261	260	260
query57	1174	1034	1028	1028
query58	250	240	246	240
query59	3514	3359	3237	3237
query60	346	278	278	278
query61	121	96	93	93
query62	777	650	651	650
query63	321	302	291	291
query64	9172	2195	1629	1629
query65	3182	3121	3110	3110
query66	743	333	337	333
query67	15298	14980	15063	14980
query68	4560	555	564	555
query69	594	450	371	371
query70	1139	1122	1078	1078
query71	402	280	338	280
query72	7309	5172	5355	5172
query73	764	326	326	326
query74	5943	5517	5641	5517
query75	3386	2665	2681	2665
query76	2178	974	934	934
query77	620	290	298	290
query78	9576	9018	8955	8955
query79	2966	527	539	527
query80	2614	471	467	467
query81	596	218	220	218
query82	865	142	136	136
query83	301	171	171	171
query84	266	94	91	91
query85	1925	294	298	294
query86	453	333	315	315
query87	3296	3124	3158	3124
query88	3898	2448	2471	2448
query89	491	409	399	399
query90	1698	190	196	190
query91	131	105	106	105
query92	58	50	51	50
query93	3710	517	501	501
query94	1043	213	217	213
query95	400	320	327	320
query96	622	284	282	282
query97	3216	2990	2998	2990
query98	315	199	200	199
query99	1734	1250	1288	1250
Total cold run time: 281481 ms
Total hot run time: 173965 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 31.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e51c53a751da9c3e03ca4b1aae6c582260e5f0a4, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.06
query5	0.48	0.50	0.47
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.13	0.12
query13	0.60	0.59	0.58
query14	0.77	0.78	0.80
query15	0.86	0.81	0.83
query16	0.36	0.38	0.37
query17	1.02	1.05	1.02
query18	0.23	0.22	0.22
query19	1.78	1.74	1.73
query20	0.01	0.01	0.01
query21	15.44	0.80	0.66
query22	4.18	7.09	2.22
query23	18.28	1.35	1.28
query24	2.08	0.23	0.23
query25	0.15	0.09	0.08
query26	0.29	0.22	0.21
query27	0.45	0.23	0.23
query28	13.31	1.01	0.99
query29	12.68	3.39	3.43
query30	0.25	0.07	0.05
query31	2.87	0.39	0.39
query32	3.28	0.47	0.47
query33	2.88	2.94	2.85
query34	16.88	4.34	4.32
query35	4.37	4.42	4.44
query36	0.65	0.47	0.47
query37	0.18	0.15	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.49 s
Total hot run time: 31.03 s

airborne12
airborne12 approved these changes Jul 11, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 11, 2024

PR approved by anyone and no changes requested.

@csun5285 csun5285 changed the title [fix] (move memtable) fix errors caused by checks in the inverted index file writer [fix] (move-memtable) fix errors caused by checks in the inverted index file writer Jul 11, 2024
@qidaye qidaye merged commit 89845dc into apache:master Jul 11, 2024
@xiaokang xiaokang added doing and removed approved Indicates a PR has been approved by one committer. labels Jul 15, 2024
This was referenced Jul 17, 2024
Closed
Merged
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@csun5285 @seawinde
[fix] (move-memtable) fix errors caused by checks in the inverted ind…
6ad4b88 
…ex file writer (apache#37621)

## Proposed changes

![image](https://github.com/apache/doris/assets/76934516/5fee1263-e4b1-4f09-b38c-7c49cba3aa07)

1. thread A flush segment 0, flush inverted index 0
2. thread B flush segment 1
3. thread A add segment 0, dst be will check inverted index file size ==
segment file size, it will fail at this point.

check the inverted index file size == the segment file size during
`close()`
airborne12 pushed a commit that referenced this pull request Jul 17, 2024
@csun5285
[Pick 3.0](move-memtable) support for inverted index file (#37975)
a925586 
## Proposed changes

pick from master #35891
pick from master #37621
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@csun5285 @dataroaring
[fix] (move-memtable) fix errors caused by checks in the inverted ind…
ab4bdff 
…ex file writer (#37621)

## Proposed changes

![image](https://github.com/apache/doris/assets/76934516/5fee1263-e4b1-4f09-b38c-7c49cba3aa07)

1. thread A flush segment 0, flush inverted index 0
2. thread B flush segment 1
3. thread A add segment 0, dst be will check inverted index file size ==
segment file size, it will fail at this point.

check the inverted index file size == the segment file size during
`close()`
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@qidaye qidaye qidaye approved these changes

+1 more reviewer

@kaijchen kaijchen kaijchen approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@csun5285 @doris-robot @airborne12 @kaijchen @qidaye @xiaokang