Skip to content

[fix](inverted index)Make build index operation only affect base index #36869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qidaye merged 1 commit into from
Jun 28, 2024
Merged

[fix](inverted index)Make build index operation only affect base index #36869

qidaye merged 1 commit into from
Jun 28, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Jun 26, 2024

Proposed changes

When building index on table with mv, we should only send index build jobs to base index.
Otherwise the build process may be stuck.
This PR make the build index operation only be allowed in base index and add a test case.

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jun 26, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-H: Total hot run time: 39878 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e72eb242d0fb5cfac2cba711834ad25a82b82dbf, data reload: false

------ Round 1 ----------------------------------
q1	17617	4436	4259	4259
q2	1998	191	187	187
q3	10471	1148	1050	1050
q4	10188	795	795	795
q5	7459	2697	2585	2585
q6	216	135	135	135
q7	938	600	603	600
q8	9227	2063	2056	2056
q9	8883	6491	6454	6454
q10	8876	3763	3730	3730
q11	466	250	255	250
q12	451	231	229	229
q13	17761	2979	2976	2976
q14	285	237	224	224
q15	514	475	476	475
q16	496	392	402	392
q17	983	726	677	677
q18	7958	7541	7525	7525
q19	6898	1448	1491	1448
q20	666	318	337	318
q21	4872	3168	3949	3168
q22	396	346	345	345
Total cold run time: 117619 ms
Total hot run time: 39878 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4494	4312	4247	4247
q2	373	271	255	255
q3	2970	2924	2913	2913
q4	2050	1713	1767	1713
q5	5606	5557	5504	5504
q6	227	130	131	130
q7	2248	1941	1884	1884
q8	3233	3483	3443	3443
q9	8858	9050	8973	8973
q10	4338	3918	4024	3918
q11	615	525	526	525
q12	876	674	710	674
q13	18020	3379	3193	3193
q14	305	297	331	297
q15	525	478	485	478
q16	503	454	456	454
q17	1884	1534	1508	1508
q18	9360	8083	7742	7742
q19	1858	1636	1688	1636
q20	4213	1890	1873	1873
q21	13826	5054	5223	5054
q22	644	560	550	550
Total cold run time: 87026 ms
Total hot run time: 56964 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-DS: Total hot run time: 173689 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e72eb242d0fb5cfac2cba711834ad25a82b82dbf, data reload: false

query1	908	390	385	385
query2	6453	2353	2310	2310
query3	6623	203	197	197
query4	21215	17599	17378	17378
query5	3589	453	465	453
query6	238	172	149	149
query7	4606	311	292	292
query8	300	287	266	266
query9	8545	2488	2441	2441
query10	563	316	288	288
query11	10692	10096	10075	10075
query12	110	90	80	80
query13	1628	380	380	380
query14	8392	7935	6864	6864
query15	229	187	184	184
query16	7754	281	271	271
query17	1729	561	532	532
query18	1958	276	283	276
query19	202	155	162	155
query20	92	86	91	86
query21	219	132	126	126
query22	4298	4122	3976	3976
query23	34092	33773	33575	33575
query24	7686	2960	2859	2859
query25	593	373	375	373
query26	708	155	152	152
query27	2227	324	334	324
query28	5829	2184	2153	2153
query29	869	641	635	635
query30	269	161	152	152
query31	999	753	757	753
query32	96	53	55	53
query33	565	279	287	279
query34	919	477	495	477
query35	757	637	610	610
query36	1119	960	983	960
query37	144	71	68	68
query38	2945	2888	2852	2852
query39	930	831	852	831
query40	202	129	137	129
query41	55	59	51	51
query42	102	94	95	94
query43	586	540	523	523
query44	1078	742	726	726
query45	194	161	163	161
query46	1076	734	704	704
query47	1871	1777	1795	1777
query48	364	302	298	298
query49	841	402	445	402
query50	767	404	396	396
query51	6931	6774	6663	6663
query52	109	91	95	91
query53	358	296	283	283
query54	569	440	436	436
query55	73	73	73	73
query56	281	262	253	253
query57	1158	1079	1062	1062
query58	248	240	243	240
query59	3330	3271	3133	3133
query60	313	266	289	266
query61	94	86	92	86
query62	608	471	472	471
query63	313	285	283	283
query64	8489	2236	1734	1734
query65	3166	3134	3101	3101
query66	747	325	327	325
query67	15306	15030	15039	15030
query68	4510	534	531	531
query69	485	337	327	327
query70	1188	1158	1163	1158
query71	371	270	275	270
query72	7080	5200	4983	4983
query73	751	333	326	326
query74	6009	5627	5620	5620
query75	3374	2651	2671	2651
query76	2181	906	948	906
query77	413	293	301	293
query78	10224	9898	9759	9759
query79	2688	520	511	511
query80	1087	459	479	459
query81	594	218	219	218
query82	1443	109	106	106
query83	264	179	174	174
query84	236	92	87	87
query85	1328	292	276	276
query86	464	313	319	313
query87	3341	3148	3117	3117
query88	4106	2451	2452	2451
query89	473	376	378	376
query90	1719	188	193	188
query91	129	98	101	98
query92	62	48	48	48
query93	2523	514	505	505
query94	1142	190	190	190
query95	411	322	317	317
query96	606	283	269	269
query97	3216	3090	3075	3075
query98	207	207	193	193
query99	1182	833	842	833
Total cold run time: 264126 ms
Total hot run time: 173689 ms

@qidaye
Copy link
Contributor Author

qidaye commented Jun 26, 2024

run feut

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

ClickBench: Total hot run time: 31.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e72eb242d0fb5cfac2cba711834ad25a82b82dbf, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.06
query5	0.52	0.48	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.54	0.53	0.54
query11	0.15	0.11	0.12
query12	0.14	0.11	0.12
query13	0.59	0.59	0.60
query14	0.78	0.79	0.81
query15	0.82	0.80	0.82
query16	0.35	0.37	0.37
query17	1.05	1.04	1.06
query18	0.20	0.27	0.23
query19	1.84	1.69	1.76
query20	0.01	0.01	0.00
query21	15.46	0.65	0.66
query22	3.98	7.24	2.33
query23	18.23	1.40	1.27
query24	2.08	0.23	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.18
query27	0.08	0.09	0.08
query28	13.25	1.02	1.00
query29	12.64	3.36	3.40
query30	0.26	0.07	0.05
query31	2.85	0.40	0.38
query32	3.29	0.48	0.47
query33	2.92	2.92	2.88
query34	17.17	4.41	4.41
query35	4.50	4.50	4.49
query36	0.66	0.45	0.48
query37	0.19	0.16	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.16	0.15	0.14
query41	0.10	0.05	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.28 s
Total hot run time: 31.05 s

airborne12
airborne12 approved these changes Jun 28, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by anyone and no changes requested.

zzzxl1993
zzzxl1993 approved these changes Jun 28, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 783d4b8 into apache:master Jun 28, 2024
@qidaye qidaye deleted the fix_build_index_with_mv branch June 28, 2024 03:07
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jun 28, 2024
@qidaye
[fix](inverted index)Make build index operation only affect base index (
5611565 
apache#36869)

When building index on table with mv, we should only send index build
jobs to base index.
Otherwise the build process may be stuck.
This PR make the build index operation only be allowed in base index and
add a test case.
@qidaye qidaye mentioned this pull request Jun 28, 2024
Merged
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Jun 28, 2024
@qidaye
[fix](inverted index)Make build index operation only affect base index (
1b6e8de 
apache#36869)

When building index on table with mv, we should only send index build
jobs to base index.
Otherwise the build process may be stuck.
This PR make the build index operation only be allowed in base index and
add a test case.
@qidaye qidaye mentioned this pull request Jun 28, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@qidaye @dataroaring
[fix](inverted index)Make build index operation only affect base index (
405ecde 
#36869)

When building index on table with mv, we should only send index build
jobs to base index.
Otherwise the build process may be stuck.
This PR make the build index operation only be allowed in base index and
add a test case.
xiaokang pushed a commit that referenced this pull request Jun 28, 2024
@qidaye
[fix](inverted index)Make build index operation only affect base index (
4bec08c 
#36869) (#36988)

backport #36869
xiaokang pushed a commit that referenced this pull request Jun 28, 2024
@qidaye
[fix](inverted index)Make build index operation only affect base index (
6b3cb56 
#36869) (#36991)
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Closed
1 task
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@qidaye
[fix](inverted index)Make build index operation only affect base index (
55eb2f5 
apache#36869) (apache#36991)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@qidaye @doris-robot @airborne12 @zzzxl1993 @xiaokang @dataroaring