Skip to content

[improve](cloud-mow) Make mow stream load retry times more reasonable #47666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 21, 2025
Merged

[improve](cloud-mow) Make mow stream load retry times more reasonable #47666

dataroaring merged 1 commit into from
Feb 21, 2025

Conversation

@hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Feb 10, 2025

What problem does this PR solve?

pr #46365 adjust timeout about mow load, thers is no need so much retry times, can reduce to 5.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Feb 10, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TPC-H: Total hot run time: 31533 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a77d94da1c338c6f364cd4c96fe139de7f9a73dd, data reload: false

------ Round 1 ----------------------------------
q1	17591	5198	5081	5081
q2	2043	304	175	175
q3	10463	1226	743	743
q4	10206	1008	550	550
q5	7516	2329	2345	2329
q6	185	168	131	131
q7	898	754	587	587
q8	9300	1298	1062	1062
q9	4802	4797	4550	4550
q10	6819	2338	1904	1904
q11	468	272	263	263
q12	349	356	224	224
q13	17770	3666	3103	3103
q14	222	228	207	207
q15	493	464	467	464
q16	628	616	595	595
q17	564	869	332	332
q18	6642	6333	6219	6219
q19	1204	938	548	548
q20	313	321	194	194
q21	2803	2151	1959	1959
q22	366	332	313	313
Total cold run time: 101645 ms
Total hot run time: 31533 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5103	5096	5134	5096
q2	237	324	232	232
q3	2189	2660	2297	2297
q4	1458	1794	1412	1412
q5	4207	4149	4176	4149
q6	225	171	128	128
q7	1907	1850	1667	1667
q8	2587	2633	2567	2567
q9	7154	7169	7127	7127
q10	3006	3203	2763	2763
q11	583	510	493	493
q12	703	807	653	653
q13	3519	4110	3281	3281
q14	273	301	280	280
q15	520	469	465	465
q16	642	671	654	654
q17	1147	1586	1372	1372
q18	7681	7462	7328	7328
q19	810	767	848	767
q20	1983	1995	1848	1848
q21	5335	4933	4769	4769
q22	632	581	563	563
Total cold run time: 51901 ms
Total hot run time: 49911 ms

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TPC-DS: Total hot run time: 190586 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a77d94da1c338c6f364cd4c96fe139de7f9a73dd, data reload: false

query1	1308	947	937	937
query2	6335	1920	1910	1910
query3	10956	4440	4321	4321
query4	53855	24699	23481	23481
query5	5102	555	486	486
query6	345	200	179	179
query7	4898	519	291	291
query8	320	245	228	228
query9	5785	2495	2515	2495
query10	430	316	263	263
query11	15095	15091	15010	15010
query12	157	109	107	107
query13	1060	512	409	409
query14	10134	6801	6803	6801
query15	203	203	187	187
query16	6937	671	469	469
query17	1090	745	616	616
query18	1513	419	312	312
query19	203	195	169	169
query20	135	125	161	125
query21	211	131	108	108
query22	4425	4525	4415	4415
query23	34076	33297	33219	33219
query24	5687	2437	2471	2437
query25	481	465	396	396
query26	660	266	154	154
query27	1820	502	343	343
query28	2719	2436	2388	2388
query29	556	573	435	435
query30	226	184	156	156
query31	899	872	787	787
query32	73	64	60	60
query33	449	347	316	316
query34	756	889	541	541
query35	784	825	753	753
query36	969	999	898	898
query37	121	109	77	77
query38	4418	4322	4247	4247
query39	1503	1467	1448	1448
query40	207	119	133	119
query41	51	51	48	48
query42	121	101	107	101
query43	513	508	487	487
query44	1289	820	816	816
query45	178	174	169	169
query46	893	1065	669	669
query47	1859	1862	1823	1823
query48	392	414	314	314
query49	694	533	439	439
query50	709	751	451	451
query51	4267	4296	4219	4219
query52	110	114	94	94
query53	233	268	190	190
query54	490	496	468	468
query55	84	81	79	79
query56	258	270	267	267
query57	1136	1169	1097	1097
query58	252	247	259	247
query59	2747	2912	2673	2673
query60	285	267	270	267
query61	126	119	122	119
query62	766	748	676	676
query63	230	200	194	194
query64	1392	1090	671	671
query65	3320	3217	3126	3126
query66	739	393	298	298
query67	15866	15582	15449	15449
query68	5575	790	529	529
query69	556	291	259	259
query70	1186	1090	1146	1090
query71	451	301	272	272
query72	6432	3649	3679	3649
query73	1253	744	362	362
query74	8924	9130	8743	8743
query75	3207	3142	2692	2692
query76	3908	1181	734	734
query77	526	376	288	288
query78	9955	9986	9302	9302
query79	2717	785	603	603
query80	704	527	440	440
query81	485	288	241	241
query82	549	151	117	117
query83	169	170	148	148
query84	289	93	83	83
query85	768	347	399	347
query86	376	305	284	284
query87	4483	4401	4412	4401
query88	3475	2213	2198	2198
query89	407	323	289	289
query90	1762	199	195	195
query91	135	141	108	108
query92	66	57	55	55
query93	2112	1010	584	584
query94	745	407	307	307
query95	338	265	255	255
query96	476	564	271	271
query97	2838	2857	2741	2741
query98	238	206	203	203
query99	1310	1412	1252	1252
Total cold run time: 292316 ms
Total hot run time: 190586 ms

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

ClickBench: Total hot run time: 30.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a77d94da1c338c6f364cd4c96fe139de7f9a73dd, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.42	0.41	0.39
query6	1.17	0.67	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.50
query10	0.57	0.57	0.57
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.59	0.59
query14	2.80	2.72	2.69
query15	0.93	0.85	0.85
query16	0.38	0.38	0.38
query17	1.00	1.04	0.99
query18	0.21	0.20	0.20
query19	1.90	1.79	2.04
query20	0.01	0.01	0.01
query21	15.35	0.88	0.55
query22	0.75	1.05	0.74
query23	14.96	1.31	0.65
query24	6.75	1.84	1.00
query25	0.54	0.17	0.12
query26	0.58	0.16	0.14
query27	0.06	0.04	0.04
query28	9.86	0.84	0.44
query29	12.55	3.99	3.29
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.23	0.56	0.46
query33	3.13	3.03	3.11
query34	15.78	5.14	4.52
query35	4.56	4.52	4.52
query36	0.66	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.28 s
Total hot run time: 30.85 s

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TeamCity be ut coverage result:
Function Coverage: 42.65% (11181/26215)
Line Coverage: 32.68% (94089/287930)
Region Coverage: 31.83% (48257/151606)
Branch Coverage: 27.73% (24359/87838)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a77d94da1c338c6f364cd4c96fe139de7f9a73dd_a77d94da1c338c6f364cd4c96fe139de7f9a73dd/report/index.html

zhannngchen
zhannngchen approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6c1778f into apache:master Feb 21, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Feb 21, 2025
@hust-hhb
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
cb6b7e4 
…#47666)

pr #46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
@github-actions github-actions bot mentioned this pull request Feb 21, 2025
Merged
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
@hust-hhb @seawinde
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
60f48df 
…apache#47666)

pr apache#46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
@hust-hhb hust-hhb mentioned this pull request Mar 14, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Apr 1, 2025
@hust-hhb
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
367d4e4 
…#49089)

pick:#47666
dataroaring pushed a commit that referenced this pull request Apr 30, 2025
@github-actions @hust-hhb
branch-3.0: [improve](cloud-mow) Make mow stream load retry times mor…
c331ae9 
…e reasonable #47666 (#48194)

Cherry-picked from #47666

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhb @koarz
[improve](cloud-mow) Make mow stream load retry times more reasonable (…
f05aabd 
…apache#47666)

pr apache#46365 adjust timeout about mow load, thers is no need so much retry
times, can reduce to 5.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hust-hhb @hello-stephen @doris-robot @zhannngchen @dataroaring