Skip to content

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Nov 8, 2024

Cherry-picked from #42959

## Proposed changes

Currently, when retrying a broker load, it will use a different
`load_id` with the same `job_id`.
The `total_scan_nums` in progress is accumulated by `job_id`.
This will cause the `total_scan_nums` progress to be multiple of the
actual scan nums.

For example, suppose a 10 instance broker load gets retried:

```
initial (0/10) 0% -> retry (0/20) 0% -> finish (10/20) 50%
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40485 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9385b4914caaa0b6acb22841b8c11fda82892ba, data reload: false

------ Round 1 ----------------------------------
q1	17839	7460	7314	7314
q2	2080	172	161	161
q3	10687	1049	1154	1049
q4	10554	745	692	692
q5	7757	2802	2790	2790
q6	238	148	149	148
q7	968	625	617	617
q8	9375	2001	2033	2001
q9	6631	6441	6470	6441
q10	6997	2314	2303	2303
q11	452	261	266	261
q12	400	210	216	210
q13	17785	2953	2949	2949
q14	239	205	214	205
q15	573	515	503	503
q16	663	595	599	595
q17	975	626	540	540
q18	7306	6504	6540	6504
q19	1395	1016	993	993
q20	476	208	193	193
q21	4004	3130	3044	3044
q22	1064	1023	972	972
Total cold run time: 108458 ms
Total hot run time: 40485 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7293	7272	7233	7233
q2	410	314	308	308
q3	2881	2868	2865	2865
q4	1998	1772	1788	1772
q5	5673	5684	5712	5684
q6	226	143	143	143
q7	2183	1772	1789	1772
q8	3342	3588	3530	3530
q9	8717	8875	8863	8863
q10	3559	3514	3545	3514
q11	587	500	490	490
q12	786	571	576	571
q13	16433	3195	3165	3165
q14	306	270	267	267
q15	596	539	545	539
q16	738	662	672	662
q17	1858	1638	1629	1629
q18	8275	7895	7628	7628
q19	5963	1512	1525	1512
q20	2062	1861	1863	1861
q21	5383	5219	5248	5219
q22	1160	1040	1013	1013
Total cold run time: 80429 ms
Total hot run time: 60240 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192469 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9385b4914caaa0b6acb22841b8c11fda82892ba, data reload: false

query1	2493	2268	2229	2229
query2	6236	2096	1989	1989
query3	14774	11038	231	231
query4	32909	23305	23373	23305
query5	3670	448	460	448
query6	290	187	196	187
query7	3979	329	310	310
query8	301	240	237	237
query9	9506	2702	2691	2691
query10	455	265	272	265
query11	17849	14980	15063	14980
query12	151	101	103	101
query13	1554	432	418	418
query14	9827	7222	7513	7222
query15	236	174	178	174
query16	7518	522	503	503
query17	1663	609	594	594
query18	2238	622	603	603
query19	276	201	190	190
query20	125	119	119	119
query21	217	107	108	107
query22	4566	4279	4170	4170
query23	34467	33675	33445	33445
query24	11723	3331	3410	3331
query25	609	407	410	407
query26	739	186	194	186
query27	1984	300	309	300
query28	6523	2522	2463	2463
query29	715	454	447	447
query30	385	312	306	306
query31	1052	765	781	765
query32	98	53	59	53
query33	668	297	281	281
query34	895	500	509	500
query35	826	725	701	701
query36	1178	919	921	919
query37	121	78	76	76
query38	4069	3791	3837	3791
query39	1467	1416	1407	1407
query40	202	103	99	99
query41	54	52	50	50
query42	106	99	94	94
query43	527	484	520	484
query44	1176	794	794	794
query45	184	167	169	167
query46	1129	701	722	701
query47	1893	1797	1839	1797
query48	462	396	373	373
query49	913	395	377	377
query50	805	403	403	403
query51	7183	7052	6989	6989
query52	100	86	92	86
query53	255	187	190	187
query54	1259	447	447	447
query55	72	77	79	77
query56	247	235	235	235
query57	1176	1093	1087	1087
query58	223	213	204	204
query59	3200	2881	2882	2881
query60	276	250	250	250
query61	133	109	107	107
query62	873	648	670	648
query63	212	188	188	188
query64	3801	665	620	620
query65	3205	3162	3164	3162
query66	848	307	305	305
query67	15777	15312	15396	15312
query68	4490	583	572	572
query69	439	263	265	263
query70	1179	1135	1147	1135
query71	396	281	256	256
query72	6671	3937	3940	3937
query73	760	347	342	342
query74	10093	8945	8938	8938
query75	3314	2619	2639	2619
query76	2228	938	928	928
query77	377	270	271	270
query78	11064	9614	9405	9405
query79	9374	593	604	593
query80	1976	416	421	416
query81	593	240	244	240
query82	1614	117	109	109
query83	314	164	155	155
query84	295	74	79	74
query85	1863	304	286	286
query86	498	291	295	291
query87	4516	4386	4105	4105
query88	5502	2372	2381	2372
query89	555	295	289	289
query90	1976	188	175	175
query91	174	144	143	143
query92	67	48	53	48
query93	8113	531	528	528
query94	872	294	284	284
query95	341	251	252	251
query96	614	278	286	278
query97	3360	3119	3150	3119
query98	227	198	194	194
query99	1925	1279	1286	1279
Total cold run time: 320423 ms
Total hot run time: 192469 ms

@dataroaring dataroaring reopened this Nov 10, 2024
@dataroaring dataroaring merged commit 041f01c into branch-3.0 Nov 10, 2024
@dataroaring dataroaring deleted the auto-pick-42959-branch-3.0 branch November 10, 2024 13:56
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants