Skip to content

[fix](inverted index)Make build index operation only affect base index (#36869) #36991

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 1 commit into from
Jun 28, 2024
Merged

[fix](inverted index)Make build index operation only affect base index (#36869) #36991

xiaokang merged 1 commit into from
Jun 28, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Jun 28, 2024

backport #36869

@qidaye
[fix](inverted index)Make build index operation only affect base index (
1b6e8de 
apache#36869)

When building index on table with mv, we should only send index build
jobs to base index.
Otherwise the build process may be stuck.
This PR make the build index operation only be allowed in base index and
add a test case.
@doris-robot
Copy link

doris-robot commented Jun 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

TPC-H: Total hot run time: 49649 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b6e8de0b6ced398936a6e1e4f3f0f37ddada88b, data reload: false

------ Round 1 ----------------------------------
q1	17803	4442	4348	4348
q2	2069	156	149	149
q3	10301	1932	1952	1932
q4	10364	1250	1357	1250
q5	8472	3900	3907	3900
q6	232	125	126	125
q7	2050	1617	1625	1617
q8	9547	2744	2698	2698
q9	14065	10294	10117	10117
q10	8628	3521	3494	3494
q11	413	255	249	249
q12	480	301	296	296
q13	18344	3913	4001	3913
q14	367	320	339	320
q15	502	466	458	458
q16	680	565	568	565
q17	1132	937	907	907
q18	7250	6932	6995	6932
q19	1778	1648	1558	1558
q20	509	311	302	302
q21	4465	4129	4075	4075
q22	523	459	444	444
Total cold run time: 119974 ms
Total hot run time: 49649 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4317	4291	4270	4270
q2	318	234	227	227
q3	4164	4152	4128	4128
q4	2738	2752	2755	2752
q5	7136	7113	7113	7113
q6	240	120	117	117
q7	3271	2887	2847	2847
q8	4363	4467	4472	4467
q9	16908	16713	16788	16713
q10	4198	4221	4295	4221
q11	744	685	697	685
q12	1045	853	868	853
q13	7186	3718	3730	3718
q14	446	430	420	420
q15	506	460	458	458
q16	743	683	673	673
q17	3839	3893	3834	3834
q18	8821	8679	8734	8679
q19	1743	1672	1689	1672
q20	2378	2153	2123	2123
q21	8524	8565	8507	8507
q22	1058	987	954	954
Total cold run time: 84686 ms
Total hot run time: 79431 ms

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

TPC-DS: Total hot run time: 203239 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b6e8de0b6ced398936a6e1e4f3f0f37ddada88b, data reload: false

query1	935	423	381	381
query2	6528	2858	2688	2688
query3	6921	208	205	205
query4	20102	18047	17960	17960
query5	19728	6521	6573	6521
query6	295	217	234	217
query7	4151	302	298	298
query8	498	394	435	394
query9	3137	2669	2609	2609
query10	430	285	299	285
query11	11351	10579	10707	10579
query12	133	80	72	72
query13	5595	691	676	676
query14	17913	13237	13220	13220
query15	353	240	256	240
query16	6478	288	271	271
query17	1508	1489	878	878
query18	2275	425	412	412
query19	211	148	150	148
query20	79	81	82	81
query21	190	95	93	93
query22	5330	5086	5106	5086
query23	32537	31810	32213	31810
query24	6961	6516	6527	6516
query25	540	436	427	427
query26	620	168	165	165
query27	1993	307	295	295
query28	6186	2385	2322	2322
query29	2922	2852	2665	2665
query30	243	170	171	170
query31	919	730	763	730
query32	66	66	61	61
query33	411	268	255	255
query34	848	478	467	467
query35	1128	907	864	864
query36	1304	1204	1199	1199
query37	88	58	59	58
query38	3075	2903	2933	2903
query39	1390	1319	1301	1301
query40	208	90	94	90
query41	46	46	44	44
query42	88	82	86	82
query43	679	631	771	631
query44	1165	721	733	721
query45	249	233	238	233
query46	1217	966	954	954
query47	1805	1742	1670	1670
query48	1026	714	691	691
query49	616	382	362	362
query50	863	606	598	598
query51	4782	4639	4617	4617
query52	103	79	82	79
query53	445	329	311	311
query54	2659	2458	2456	2456
query55	83	82	81	81
query56	238	212	219	212
query57	1262	1076	1163	1076
query58	228	188	181	181
query59	4377	4027	3766	3766
query60	213	192	199	192
query61	102	97	97	97
query62	795	537	459	459
query63	485	339	335	335
query64	2608	1523	1532	1523
query65	3616	3547	3583	3547
query66	826	391	389	389
query67	15815	16756	15812	15812
query68	8444	654	684	654
query69	570	382	361	361
query70	1576	1337	1582	1337
query71	405	316	317	316
query72	6548	3518	3492	3492
query73	739	313	307	307
query74	6268	5782	5774	5774
query75	5032	3727	3679	3679
query76	5091	1125	1179	1125
query77	736	257	248	248
query78	12563	11642	11553	11553
query79	9151	637	652	637
query80	1511	408	412	408
query81	507	231	237	231
query82	1685	97	99	97
query83	176	133	132	132
query84	259	72	72	72
query85	877	322	317	317
query86	339	309	318	309
query87	3194	2974	3037	2974
query88	5034	2328	2313	2313
query89	404	301	290	290
query90	1925	213	210	210
query91	195	143	141	141
query92	60	54	54	54
query93	6250	579	553	553
query94	727	208	209	208
query95	1141	1074	1044	1044
query96	642	332	327	327
query97	6433	6375	6365	6365
query98	186	173	176	173
query99	2954	807	810	807
Total cold run time: 313593 ms
Total hot run time: 203239 ms

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b6e8de0b6ced398936a6e1e4f3f0f37ddada88b, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.25	0.05	0.06
query4	1.77	0.07	0.06
query5	0.56	0.52	0.53
query6	1.28	0.61	0.62
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.48	0.49
query10	0.53	0.53	0.52
query11	0.12	0.09	0.08
query12	0.12	0.10	0.09
query13	0.62	0.61	0.61
query14	0.77	0.79	0.80
query15	0.77	0.77	0.76
query16	0.35	0.38	0.36
query17	1.04	0.98	1.01
query18	0.21	0.27	0.24
query19	1.94	1.84	1.84
query20	0.01	0.01	0.01
query21	15.49	0.56	0.57
query22	1.96	2.03	1.71
query23	17.11	1.16	0.99
query24	5.87	1.60	0.94
query25	0.38	0.07	0.07
query26	0.71	0.15	0.15
query27	0.04	0.04	0.03
query28	7.03	0.77	0.71
query29	12.65	2.23	2.31
query30	0.63	0.58	0.52
query31	2.81	0.40	0.38
query32	3.36	0.49	0.50
query33	3.09	3.08	3.07
query34	15.26	4.80	4.80
query35	4.87	4.86	4.82
query36	1.07	1.02	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 103.74 s
Total hot run time: 30.8 s

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1b6e8de0b6ced398936a6e1e4f3f0f37ddada88b with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@qidaye
Copy link
Contributor Author

qidaye commented Jun 28, 2024

run external

@xiaokang xiaokang merged commit 6b3cb56 into apache:branch-2.0 Jun 28, 2024
@qidaye qidaye deleted the pick_build_index_with_mv_2.0 branch June 28, 2024 10:03
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@qidaye
[fix](inverted index)Make build index operation only affect base index (
55eb2f5 
apache#36869) (apache#36991)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qidaye @doris-robot @xiaokang