Skip to content

[improve](routine load) introduce routine load task min timeout #46292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Jan 6, 2025
Merged

[improve](routine load) introduce routine load task min timeout #46292

liaoxin01 merged 1 commit into from
Jan 6, 2025

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jan 2, 2025

What problem does this PR solve?

For certain scenarios, it is possible to set the max_match_interval of the routine load to be very small, such as setting it to 1, which can result in a very short timeout period. In some scenarios (such as high pressure and using MOW table), it may lead to a continuous non consumption for timeout. Therefore, a minimum timeout is configured to solve a problem.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Jan 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-H: Total hot run time: 33012 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b70e75c4676435265fbdd8477fb3eddc3e84eb2a, data reload: false

------ Round 1 ----------------------------------
q1	17590	6198	6069	6069
q2	2045	301	175	175
q3	10415	1270	732	732
q4	10212	863	438	438
q5	7540	2210	1969	1969
q6	208	179	148	148
q7	899	747	631	631
q8	9235	1347	1227	1227
q9	5259	4898	5015	4898
q10	6741	2310	1891	1891
q11	489	275	268	268
q12	354	373	224	224
q13	17753	3686	3061	3061
q14	241	228	228	228
q15	552	519	506	506
q16	649	607	593	593
q17	584	847	345	345
q18	7284	6516	6507	6507
q19	1740	965	571	571
q20	312	324	191	191
q21	2880	2245	2031	2031
q22	359	325	309	309
Total cold run time: 103341 ms
Total hot run time: 33012 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6254	6284	6273	6273
q2	247	328	229	229
q3	2235	2656	2250	2250
q4	1415	1827	1327	1327
q5	4418	4736	4880	4736
q6	191	179	144	144
q7	2106	2035	1798	1798
q8	2908	2767	2668	2668
q9	7353	7277	7167	7167
q10	3068	3322	2803	2803
q11	593	531	488	488
q12	659	719	628	628
q13	3451	3783	3250	3250
q14	287	316	275	275
q15	579	509	499	499
q16	652	681	660	660
q17	1199	1739	1261	1261
q18	7741	7464	7249	7249
q19	805	1068	1098	1068
q20	1924	1936	1880	1880
q21	5382	5043	4783	4783
q22	600	610	586	586
Total cold run time: 54067 ms
Total hot run time: 52022 ms

liaoxin01
liaoxin01 approved these changes Jan 2, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-DS: Total hot run time: 189809 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b70e75c4676435265fbdd8477fb3eddc3e84eb2a, data reload: false

query1	958	397	366	366
query2	6521	2422	2419	2419
query3	6703	216	215	215
query4	33800	23665	23280	23280
query5	4325	641	463	463
query6	281	198	190	190
query7	4628	510	315	315
query8	302	243	243	243
query9	9555	2640	2618	2618
query10	470	310	250	250
query11	18555	15333	15139	15139
query12	165	107	107	107
query13	1688	544	404	404
query14	11081	7032	7623	7032
query15	251	194	188	188
query16	7764	588	430	430
query17	1586	766	590	590
query18	1736	398	336	336
query19	226	181	147	147
query20	124	111	107	107
query21	203	117	100	100
query22	4307	4317	4290	4290
query23	35574	33496	33271	33271
query24	6314	2288	2221	2221
query25	476	448	381	381
query26	1199	271	148	148
query27	2006	451	330	330
query28	5097	2407	2417	2407
query29	667	533	412	412
query30	231	183	149	149
query31	1020	903	818	818
query32	74	60	57	57
query33	496	358	286	286
query34	761	836	494	494
query35	815	803	730	730
query36	999	1041	933	933
query37	116	102	74	74
query38	4407	4163	4189	4163
query39	1520	1435	1418	1418
query40	210	116	109	109
query41	56	52	44	44
query42	120	105	102	102
query43	533	542	509	509
query44	1305	809	802	802
query45	188	179	165	165
query46	866	1035	639	639
query47	1914	1958	1909	1909
query48	380	417	315	315
query49	763	470	390	390
query50	621	652	383	383
query51	7204	7153	6947	6947
query52	100	101	91	91
query53	227	263	189	189
query54	460	473	394	394
query55	76	79	78	78
query56	264	250	237	237
query57	1194	1182	1107	1107
query58	253	228	235	228
query59	3279	3283	3231	3231
query60	273	274	250	250
query61	111	108	110	108
query62	888	792	733	733
query63	222	197	190	190
query64	4245	1002	662	662
query65	3282	3186	3257	3186
query66	886	415	313	313
query67	15837	15767	15562	15562
query68	5495	724	523	523
query69	432	289	269	269
query70	1219	1136	1164	1136
query71	355	284	254	254
query72	6584	3867	3730	3730
query73	647	745	361	361
query74	9847	9030	8855	8855
query75	3467	3182	2650	2650
query76	4496	1191	764	764
query77	593	343	274	274
query78	10132	10265	9507	9507
query79	3148	787	586	586
query80	715	504	416	416
query81	475	278	227	227
query82	589	152	130	130
query83	199	176	150	150
query84	338	108	69	69
query85	717	352	313	313
query86	332	326	297	297
query87	4535	4540	4310	4310
query88	4181	2219	2142	2142
query89	413	334	298	298
query90	1850	184	189	184
query91	133	135	105	105
query92	68	54	53	53
query93	995	858	539	539
query94	675	381	288	288
query95	332	256	252	252
query96	496	597	269	269
query97	2929	2991	2834	2834
query98	231	202	198	198
query99	1691	1609	1432	1432
Total cold run time: 291067 ms
Total hot run time: 189809 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b70e75c4676435265fbdd8477fb3eddc3e84eb2a, data reload: false

query1	0.04	0.04	0.05
query2	0.08	0.04	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.11
query5	0.42	0.40	0.41
query6	1.14	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.51	0.51
query10	0.56	0.57	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.74	2.82	2.75
query15	0.91	0.82	0.84
query16	0.38	0.37	0.38
query17	1.07	0.97	0.97
query18	0.22	0.21	0.19
query19	1.94	1.86	2.02
query20	0.02	0.01	0.02
query21	15.36	0.98	0.60
query22	0.76	0.74	0.77
query23	15.20	1.45	0.52
query24	2.87	0.56	1.99
query25	0.27	0.12	0.17
query26	0.22	0.14	0.14
query27	0.06	0.07	0.06
query28	14.24	1.48	1.04
query29	12.58	3.95	3.25
query30	0.26	0.09	0.07
query31	2.83	0.60	0.37
query32	3.23	0.56	0.46
query33	3.18	3.14	3.06
query34	16.71	5.04	4.49
query35	4.46	4.42	4.48
query36	0.65	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.25 s
Total hot run time: 30.88 s

MoanasDaddyXu
MoanasDaddyXu approved these changes Jan 6, 2025
View reviewed changes
Copy link
Contributor

@MoanasDaddyXu MoanasDaddyXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit fd1e246 into apache:master Jan 6, 2025
24 of 26 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 6, 2025
@sollhui
[improve](routine load) introduce routine load task min timeout (#46292)
39ba9aa 
For certain scenarios, it is possible to set the max_match_interval of
the routine load to be very small, such as setting it to 1, which can
result in a very short timeout period. In some scenarios (such as high
pressure and using MOW table), it may lead to a continuous non
consumption for timeout. Therefore, a minimum timeout is configured to
solve a problem.
@github-actions github-actions bot mentioned this pull request Jan 6, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jan 13, 2025
@github-actions @sollhui
branch-3.0: [improve](routine load) introduce routine load task min t…
de72a8e 
…imeout #46292 (#46439)

Cherry-picked from #46292

Co-authored-by: hui lai <laihui@selectdb.com>
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

+1 more reviewer

@MoanasDaddyXu MoanasDaddyXu MoanasDaddyXu approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged kind/behavior-changed reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@sollhui @Thearas @doris-robot @liaoxin01 @MoanasDaddyXu @dataroaring