Skip to content

[fix](move-memtable) fix timeout to get tablet schema #33256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Apr 4, 2024
Merged

[fix](move-memtable) fix timeout to get tablet schema #33256

liaoxin01 merged 1 commit into from
Apr 4, 2024

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Apr 4, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

  1. LoadStreamMap use shared IndexToTabletSchema and IndexToEnableMoW.
  2. rename LoadStreamSubPool to LoadStreamMapPool.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Apr 4, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@liaoxin01 liaoxin01 force-pushed the fix_move_mem branch 2 times, most recently from 936ee37 to 2ecfe82 Compare April 4, 2024 10:28
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Apr 4, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

github-actions[bot]
github-actions bot reviewed Apr 4, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/vec/exec/load_stream_stub_map_test.cpp
// under the License.
#include "vec/sink/load_stream_stub_pool.h"

#include <gtest/gtest.h>
Copy link
Contributor

@github-actions github-actions bot Apr 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]

#include <gtest/gtest.h>
         ^

@doris-robot
Copy link

doris-robot commented Apr 4, 2024

TeamCity be ut coverage result:
Function Coverage: 35.64% (8882/24922)
Line Coverage: 27.37% (72919/266428)
Region Coverage: 26.55% (37711/142035)
Branch Coverage: 23.35% (19218/82296)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8cfa07811053651718bcc6279116bb5c4bb61117_8cfa07811053651718bcc6279116bb5c4bb61117/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 4, 2024

TPC-H: Total hot run time: 39016 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8cfa07811053651718bcc6279116bb5c4bb61117, data reload: false

------ Round 1 ----------------------------------
q1	17678	4145	4084	4084
q2	2024	185	182	182
q3	10485	1170	1288	1170
q4	10859	860	978	860
q5	8028	3088	3016	3016
q6	225	136	139	136
q7	1122	648	641	641
q8	9716	2029	2086	2029
q9	6812	6346	6296	6296
q10	8533	3524	3514	3514
q11	411	243	232	232
q12	377	209	202	202
q13	17777	2894	2901	2894
q14	271	240	229	229
q15	520	492	468	468
q16	479	401	376	376
q17	962	932	922	922
q18	7247	6571	6466	6466
q19	1583	1536	1547	1536
q20	571	318	308	308
q21	3512	3158	3172	3158
q22	367	297	308	297
Total cold run time: 109559 ms
Total hot run time: 39016 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4015	4014	4050	4014
q2	331	223	223	223
q3	2969	2968	2961	2961
q4	1898	1882	1853	1853
q5	5279	5227	5235	5227
q6	208	127	124	124
q7	2260	1803	1827	1803
q8	3220	3289	3286	3286
q9	8505	8497	8527	8497
q10	3766	3870	3816	3816
q11	533	450	431	431
q12	708	590	533	533
q13	11690	2921	2895	2895
q14	281	255	264	255
q15	513	463	459	459
q16	446	395	381	381
q17	1731	1707	1701	1701
q18	7704	7270	7262	7262
q19	1647	1655	1646	1646
q20	1951	1719	1714	1714
q21	5012	4816	4740	4740
q22	487	415	419	415
Total cold run time: 65154 ms
Total hot run time: 54236 ms

@doris-robot
Copy link

doris-robot commented Apr 4, 2024

TPC-DS: Total hot run time: 181173 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8cfa07811053651718bcc6279116bb5c4bb61117, data reload: false

query1	894	1126	1126	1126
query2	6478	1864	1726	1726
query3	6667	208	220	208
query4	25006	21571	21379	21379
query5	4220	394	405	394
query6	276	188	192	188
query7	4615	296	300	296
query8	226	175	179	175
query9	8484	2252	2284	2252
query10	579	249	243	243
query11	14927	14479	14546	14479
query12	140	90	87	87
query13	1638	386	381	381
query14	8611	6771	6753	6753
query15	204	171	186	171
query16	7164	269	261	261
query17	1034	600	573	573
query18	1924	282	279	279
query19	204	157	154	154
query20	93	91	89	89
query21	202	123	122	122
query22	4974	4826	4859	4826
query23	33918	32946	32935	32935
query24	11113	3099	3083	3083
query25	704	397	401	397
query26	1898	159	161	159
query27	3033	320	325	320
query28	6756	1868	1848	1848
query29	1378	587	590	587
query30	306	165	164	164
query31	942	715	712	712
query32	99	54	57	54
query33	665	253	246	246
query34	955	471	483	471
query35	821	714	678	678
query36	991	869	868	868
query37	284	73	73	73
query38	3464	3396	3390	3390
query39	1586	1532	1518	1518
query40	290	133	131	131
query41	50	47	46	46
query42	112	112	96	96
query43	457	417	430	417
query44	1133	717	714	714
query45	287	260	250	250
query46	1069	795	791	791
query47	1924	1797	1818	1797
query48	368	306	301	301
query49	1162	367	363	363
query50	795	388	395	388
query51	6799	6643	6624	6624
query52	108	92	93	92
query53	360	279	288	279
query54	293	251	234	234
query55	82	75	74	74
query56	251	233	228	228
query57	1205	1120	1126	1120
query58	250	213	207	207
query59	2520	2348	2535	2348
query60	255	242	247	242
query61	114	124	111	111
query62	704	459	436	436
query63	312	283	287	283
query64	6478	3224	3241	3224
query65	3110	3029	3037	3029
query66	1437	328	322	322
query67	15798	14946	14796	14796
query68	8767	577	584	577
query69	545	308	304	304
query70	1282	1117	1109	1109
query71	482	279	275	275
query72	6335	2579	2414	2414
query73	807	321	319	319
query74	6632	6379	6428	6379
query75	3513	2325	2324	2324
query76	5649	1150	1224	1150
query77	672	247	246	246
query78	10826	10094	10162	10094
query79	9868	524	528	524
query80	1384	423	418	418
query81	496	233	232	232
query82	475	96	92	92
query83	222	166	171	166
query84	264	89	88	88
query85	1202	301	281	281
query86	370	293	285	285
query87	3715	3503	3521	3503
query88	3745	2297	2294	2294
query89	547	366	379	366
query90	2060	177	179	177
query91	137	104	107	104
query92	61	47	48	47
query93	6629	555	532	532
query94	1309	188	186	186
query95	458	323	322	322
query96	619	270	271	270
query97	2639	2469	2468	2468
query98	231	226	209	209
query99	1318	858	810	810
Total cold run time: 298455 ms
Total hot run time: 181173 ms

@doris-robot
Copy link

doris-robot commented Apr 4, 2024

ClickBench: Total hot run time: 29.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8cfa07811053651718bcc6279116bb5c4bb61117, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.67	0.07	0.06
query5	0.48	0.48	0.49
query6	1.15	0.66	0.66
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.51	0.52
query10	0.56	0.58	0.58
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.77	0.78	0.80
query15	0.86	0.83	0.85
query16	0.35	0.36	0.36
query17	0.97	1.03	0.99
query18	0.25	0.25	0.24
query19	1.90	1.73	1.69
query20	0.01	0.02	0.01
query21	15.42	0.67	0.64
query22	4.25	7.90	1.39
query23	18.14	1.25	1.26
query24	1.51	0.20	0.19
query25	0.15	0.08	0.08
query26	0.27	0.16	0.17
query27	0.08	0.07	0.09
query28	13.87	0.96	0.95
query29	12.53	3.25	3.26
query30	0.29	0.08	0.08
query31	2.80	0.38	0.38
query32	3.30	0.48	0.48
query33	2.87	2.85	2.93
query34	15.49	4.35	4.32
query35	4.37	4.37	4.37
query36	0.66	0.48	0.47
query37	0.19	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.64 s
Total hot run time: 29.74 s

@doris-robot
Copy link

doris-robot commented Apr 4, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8cfa07811053651718bcc6279116bb5c4bb61117 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       17.3 seconds inserted 10000000 Rows, about 578K ops/s

dataroaring
dataroaring approved these changes Apr 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2024

PR approved by anyone and no changes requested.

@liaoxin01 liaoxin01 mentioned this pull request Apr 4, 2024
Merged
kaijchen
kaijchen approved these changes Apr 4, 2024
View reviewed changes
Copy link
Member

@kaijchen kaijchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 605aa8d into apache:master Apr 4, 2024
@liaoxin01 liaoxin01 deleted the fix_move_mem branch April 4, 2024 13:25
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Apr 4, 2024
@liaoxin01
[fix](move-memtable) fix timeout to get tablet schema (apache#33256)
fc1d224
@liaoxin01 liaoxin01 mentioned this pull request Apr 4, 2024
Merged
liaoxin01 added a commit that referenced this pull request Apr 4, 2024
@liaoxin01
[fix](move-memtable) fix timeout to get tablet schema (#33256) (#33260)
950ca68
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@kaijchen kaijchen kaijchen approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liaoxin01 @doris-robot @kaijchen @dataroaring