Skip to content

Conversation

@nextdreamblue
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

restore job run() must add synchronized for #27027

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39357 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 31d78af2e5b922c72b9fa96bb13a260b75c54ead, data reload: false

------ Round 1 ----------------------------------
q1	17949	5646	5850	5646
q2	2053	136	133	133
q3	10892	1124	1166	1124
q4	10535	783	817	783
q5	7834	3180	3150	3150
q6	204	124	126	124
q7	894	491	475	475
q8	9240	1970	2000	1970
q9	7315	6391	6408	6391
q10	8240	3113	3104	3104
q11	407	198	212	198
q12	365	193	199	193
q13	18087	3435	3397	3397
q14	244	222	222	222
q15	578	519	514	514
q16	433	393	364	364
q17	948	494	530	494
q18	7573	6824	6787	6787
q19	1614	1403	1295	1295
q20	584	312	295	295
q21	2829	2398	2391	2391
q22	358	307	316	307
Total cold run time: 109176 ms
Total hot run time: 39357 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6013	5641	5748	5641
q2	354	220	215	215
q3	3352	3030	3082	3030
q4	2172	2084	2088	2084
q5	5999	5850	5924	5850
q6	210	128	123	123
q7	2398	2013	1936	1936
q8	3498	3814	3669	3669
q9	9703	25584	9504	9504
q10	4692	3757	3840	3757
q11	595	462	455	455
q12	1191	669	649	649
q13	17152	3279	3242	3242
q14	329	279	286	279
q15	585	509	508	508
q16	536	493	475	475
q17	6049	2044	1879	1879
q18	9483	9368	13873	9368
q19	13566	1609	1609	1609
q20	3869	1942	1928	1928
q21	14558	5641	5596	5596
q22	939	621	615	615
Total cold run time: 107243 ms
Total hot run time: 62412 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178539 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 31d78af2e5b922c72b9fa96bb13a260b75c54ead, data reload: false

query1	930	340	326	326
query2	6716	1802	1787	1787
query3	6724	208	208	208
query4	32033	22218	22201	22201
query5	6943	576	570	570
query6	261	177	183	177
query7	4606	278	277	277
query8	264	191	217	191
query9	9142	2732	2865	2732
query10	467	246	253	246
query11	16942	15318	15239	15239
query12	116	73	73	73
query13	1703	384	377	377
query14	11292	7462	7365	7365
query15	207	173	178	173
query16	4612	258	260	258
query17	997	497	485	485
query18	1790	269	265	265
query19	183	152	147	147
query20	82	80	76	76
query21	208	124	134	124
query22	4870	4776	4801	4776
query23	31726	30960	31159	30960
query24	11536	2827	2779	2779
query25	583	318	333	318
query26	1680	150	155	150
query27	3245	277	275	275
query28	7342	1870	1862	1862
query29	1556	690	690	690
query30	278	136	140	136
query31	937	749	771	749
query32	80	64	62	62
query33	701	236	249	236
query34	1151	450	457	450
query35	952	764	759	759
query36	1409	1322	1240	1240
query37	99	75	74	74
query38	3391	3277	3275	3275
query39	1312	1278	1278	1278
query40	212	93	95	93
query41	38	35	34	34
query42	100	94	97	94
query43	516	511	505	505
query44	1141	699	726	699
query45	206	194	183	183
query46	1068	652	651	651
query47	1645	1581	1623	1581
query48	413	324	312	312
query49	1157	310	314	310
query50	700	322	322	322
query51	5325	5206	5238	5206
query52	92	95	92	92
query53	368	281	287	281
query54	908	452	461	452
query55	95	91	85	85
query56	195	210	192	192
query57	973	951	948	948
query58	207	186	194	186
query59	2867	2600	2552	2552
query60	247	236	224	224
query61	86	80	83	80
query62	671	411	405	405
query63	296	288	290	288
query64	5153	1707	1672	1672
query65	3372	3269	3262	3262
query66	1270	319	338	319
query67	15493	15260	15032	15032
query68	11882	512	501	501
query69	622	339	343	339
query70	1700	1630	1505	1505
query71	493	238	248	238
query72	5085	2879	2821	2821
query73	2716	317	315	315
query74	6980	6415	6458	6415
query75	4806	2290	2289	2289
query76	5837	971	1134	971
query77	678	282	267	267
query78	9088	8896	8659	8659
query79	1027	528	529	528
query80	541	329	340	329
query81	462	197	204	197
query82	208	85	91	85
query83	141	115	118	115
query84	281	70	74	70
query85	1066	339	321	321
query86	399	366	409	366
query87	3562	3373	3474	3373
query88	2970	2220	2224	2220
query89	436	403	390	390
query90	2006	197	206	197
query91	164	129	132	129
query92	58	57	57	57
query93	1415	438	409	409
query94	1250	181	183	181
query95	519	485	461	461
query96	614	322	310	310
query97	4281	4159	4176	4159
query98	203	181	184	181
query99	1002	715	782	715
Total cold run time: 296372 ms
Total hot run time: 178539 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.06 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 22.3 seconds inserted 10000000 Rows, about 448K ops/s
storage size: 17183684031 Bytes

@doris-robot
Copy link

ClickBench: Total hot run time: 31.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 31d78af2e5b922c72b9fa96bb13a260b75c54ead, data reload: false

query1	0.06	0.05	0.05
query2	0.06	0.02	0.02
query3	0.25	0.13	0.11
query4	1.77	0.12	0.11
query5	0.55	0.52	0.53
query6	1.38	0.66	0.64
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.57	0.50	0.49
query10	0.56	0.56	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.60	0.60	0.59
query14	0.77	0.78	0.79
query15	0.79	0.79	0.78
query16	0.38	0.40	0.38
query17	1.03	0.98	1.02
query18	0.25	0.25	0.27
query19	1.82	1.82	1.79
query20	0.01	0.01	0.02
query21	15.41	0.57	0.56
query22	2.88	2.84	1.78
query23	17.24	0.77	0.82
query24	2.63	1.23	1.53
query25	0.51	0.14	0.18
query26	0.56	0.14	0.14
query27	0.07	0.06	0.05
query28	11.15	0.80	0.80
query29	12.56	3.29	3.28
query30	0.58	0.49	0.49
query31	2.83	0.35	0.35
query32	3.45	0.50	0.49
query33	3.21	3.22	3.26
query34	15.73	4.26	4.26
query35	4.21	4.29	4.26
query36	1.13	1.08	1.08
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.15	0.14	0.14
query41	0.06	0.01	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.66 s
Total hot run time: 31.44 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 31d78af2e5b922c72b9fa96bb13a260b75c54ead with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.9 seconds inserted 10000000 Rows, about 671K ops/s

@nextdreamblue
Copy link
Contributor Author

run pipelinex_p0

@morningman morningman self-assigned this Mar 15, 2024
@github-actions
Copy link
Contributor

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Sep 12, 2024
@github-actions github-actions bot closed this Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants