Skip to content

Conversation

@HappenLee
Copy link
Contributor

@HappenLee HappenLee commented May 8, 2024

Proposed changes

  1. remove the unless timer in pipeline fragment context
  2. remove the unless code in local shuffle
  3. change tuple to struct

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@HappenLee
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.68% (8982/25171)
Line Coverage: 27.35% (74215/271377)
Region Coverage: 26.59% (38373/144299)
Branch Coverage: 23.41% (19574/83606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5d4db5668a00087cf357d954f8e47af3b7f583d2_5d4db5668a00087cf357d954f8e47af3b7f583d2/report/index.html

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.68% (8981/25172)
Line Coverage: 27.35% (74217/271391)
Region Coverage: 26.59% (38370/144302)
Branch Coverage: 23.40% (19567/83608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5d56c9be713530b4635083c8225d3f2c10d389f9_5d56c9be713530b4635083c8225d3f2c10d389f9/report/index.html

@HappenLee HappenLee marked this pull request as ready for review May 9, 2024 01:45
@HappenLee
Copy link
Contributor Author

run be_ut

@HappenLee
Copy link
Contributor Author

run buildall

@HappenLee HappenLee force-pushed the timer branch 2 times, most recently from 484f4a2 to e3ed928 Compare May 9, 2024 09:25
@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.67% (8980/25174)
Line Coverage: 27.33% (74188/271418)
Region Coverage: 26.57% (38354/144326)
Branch Coverage: 23.39% (19559/83632)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e3ed928afb6a82448ab5119e2628c8ffcc02450d_e3ed928afb6a82448ab5119e2628c8ffcc02450d/report/index.html

Gabriel39
Gabriel39 previously approved these changes May 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 9, 2024

PR approved by anyone and no changes requested.

@HappenLee
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 10, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae6b858c46ed205c6322f638b7890c9dd4ff71f1, data reload: false

------ Round 1 ----------------------------------
q1	18014	4504	4385	4385
q2	2556	192	201	192
q3	11447	1252	1233	1233
q4	10384	937	736	736
q5	7520	2790	2708	2708
q6	223	135	137	135
q7	1018	588	565	565
q8	9235	2103	2072	2072
q9	9080	6577	6494	6494
q10	8962	3736	3748	3736
q11	455	241	243	241
q12	446	216	218	216
q13	17760	2978	2970	2970
q14	279	218	208	208
q15	518	467	478	467
q16	510	386	380	380
q17	975	668	715	668
q18	8175	7503	7403	7403
q19	6271	1468	1536	1468
q20	658	303	317	303
q21	5007	3974	3954	3954
q22	353	286	292	286
Total cold run time: 119846 ms
Total hot run time: 40820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4267	4212	4212
q2	380	281	281	281
q3	2955	2710	2735	2710
q4	1888	1634	1643	1634
q5	5341	5255	5276	5255
q6	208	124	124	124
q7	2261	1880	1919	1880
q8	3213	3353	3359	3353
q9	8333	8336	8410	8336
q10	3922	3740	3718	3718
q11	599	492	483	483
q12	758	599	589	589
q13	16532	2958	3005	2958
q14	303	269	275	269
q15	520	476	480	476
q16	462	419	403	403
q17	1766	1492	1470	1470
q18	7592	7558	7422	7422
q19	1668	1576	1764	1576
q20	1968	1763	1778	1763
q21	4915	4969	4806	4806
q22	582	507	484	484
Total cold run time: 70525 ms
Total hot run time: 54202 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.69% (8981/25166)
Line Coverage: 27.34% (74204/271390)
Region Coverage: 26.58% (38365/144317)
Branch Coverage: 23.40% (19564/83618)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ae6b858c46ed205c6322f638b7890c9dd4ff71f1_ae6b858c46ed205c6322f638b7890c9dd4ff71f1/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 187374 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae6b858c46ed205c6322f638b7890c9dd4ff71f1, data reload: false

query1	910	376	356	356
query2	6966	2389	2429	2389
query3	6644	212	211	211
query4	25098	21228	21330	21228
query5	4139	412	422	412
query6	259	178	173	173
query7	4579	288	286	286
query8	251	203	188	188
query9	8493	2421	2423	2421
query10	430	249	260	249
query11	14772	14153	14346	14153
query12	128	87	89	87
query13	1660	361	356	356
query14	10152	7570	7553	7553
query15	216	177	173	173
query16	7797	267	260	260
query17	1864	567	546	546
query18	1959	276	270	270
query19	195	147	148	147
query20	92	83	85	83
query21	190	127	123	123
query22	5214	4987	5018	4987
query23	35308	33407	33731	33407
query24	12028	2949	2902	2902
query25	564	365	363	363
query26	1414	150	151	150
query27	2866	320	334	320
query28	7429	2066	2043	2043
query29	865	598	607	598
query30	293	153	154	153
query31	1011	774	773	773
query32	90	51	54	51
query33	735	249	245	245
query34	1025	481	480	480
query35	807	693	668	668
query36	1076	928	926	926
query37	228	68	65	65
query38	2883	2749	2809	2749
query39	1642	1552	1566	1552
query40	272	124	125	124
query41	41	39	38	38
query42	104	96	96	96
query43	571	518	567	518
query44	1162	715	735	715
query45	267	251	248	248
query46	1074	696	721	696
query47	1967	1905	1908	1905
query48	372	299	303	299
query49	1210	415	389	389
query50	800	384	401	384
query51	6724	6700	6728	6700
query52	99	88	87	87
query53	348	274	272	272
query54	915	429	431	429
query55	73	72	74	72
query56	238	217	218	217
query57	1252	1178	1160	1160
query58	222	194	192	192
query59	3388	3094	3022	3022
query60	263	232	226	226
query61	92	91	106	91
query62	676	491	480	480
query63	306	286	287	286
query64	10024	7492	7448	7448
query65	3113	3085	3075	3075
query66	1378	346	343	343
query67	15681	15039	15148	15039
query68	4695	528	529	528
query69	500	315	313	313
query70	1182	1154	1252	1154
query71	423	276	263	263
query72	7044	2525	2331	2331
query73	720	323	328	323
query74	6692	6106	6198	6106
query75	3369	2684	2631	2631
query76	2843	1052	1024	1024
query77	601	259	269	259
query78	10746	10427	10203	10203
query79	2552	505	507	505
query80	1984	439	426	426
query81	534	217	211	211
query82	755	96	95	95
query83	319	164	167	164
query84	265	84	83	83
query85	2027	271	257	257
query86	501	303	309	303
query87	3343	3074	3165	3074
query88	4301	2433	2424	2424
query89	486	380	374	374
query90	2048	184	181	181
query91	125	95	99	95
query92	65	49	48	48
query93	3647	512	488	488
query94	1228	215	181	181
query95	394	295	309	295
query96	605	264	267	264
query97	3240	3009	3019	3009
query98	238	221	215	215
query99	1231	900	905	900
Total cold run time: 293607 ms
Total hot run time: 187374 ms

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41248 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 871f026570359fa2866e18531573565fb978d9eb, data reload: false

------ Round 1 ----------------------------------
q1	17603	4639	4301	4301
q2	2025	197	187	187
q3	10470	1255	1259	1255
q4	10204	817	771	771
q5	7486	2711	2737	2711
q6	235	132	136	132
q7	987	560	537	537
q8	9221	2175	2133	2133
q9	9511	6799	6809	6799
q10	9188	3920	3857	3857
q11	451	248	240	240
q12	500	227	230	227
q13	17228	3167	3254	3167
q14	275	224	238	224
q15	516	461	455	455
q16	470	384	395	384
q17	983	656	819	656
q18	8390	7961	7789	7789
q19	5173	1546	1555	1546
q20	664	321	313	313
q21	5306	4233	3279	3279
q22	351	286	285	285
Total cold run time: 117237 ms
Total hot run time: 41248 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4532	4436	4399	4399
q2	375	282	282	282
q3	3233	2980	2994	2980
q4	1858	1601	1599	1599
q5	5491	5486	5570	5486
q6	213	125	124	124
q7	1774	1431	1452	1431
q8	3259	3423	3484	3423
q9	8726	8765	8816	8765
q10	4115	3749	3891	3749
q11	601	491	510	491
q12	812	631	627	627
q13	16666	3252	3049	3049
q14	279	262	265	262
q15	520	490	474	474
q16	463	419	433	419
q17	1777	1501	1487	1487
q18	7943	7759	7595	7595
q19	5476	1574	1531	1531
q20	2047	1770	1769	1769
q21	12143	5364	5243	5243
q22	614	539	588	539
Total cold run time: 82917 ms
Total hot run time: 55724 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.68% (8981/25173)
Line Coverage: 27.33% (74189/271487)
Region Coverage: 26.57% (38356/144354)
Branch Coverage: 23.39% (19558/83634)
Coverage Report: http://coverage.selectdb-in.cc/coverage/871f026570359fa2866e18531573565fb978d9eb_871f026570359fa2866e18531573565fb978d9eb/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 187471 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 871f026570359fa2866e18531573565fb978d9eb, data reload: false

query1	2114	367	366	366
query2	7878	2356	2249	2249
query3	7536	217	227	217
query4	23156	21907	21520	21520
query5	4061	412	402	402
query6	253	179	178	178
query7	4221	295	290	290
query8	233	192	191	191
query9	8248	2444	2442	2442
query10	423	246	240	240
query11	14810	14261	14139	14139
query12	139	89	82	82
query13	1491	378	362	362
query14	10430	7724	8601	7724
query15	266	179	182	179
query16	8191	287	268	268
query17	1906	571	577	571
query18	2114	285	280	280
query19	324	152	152	152
query20	92	84	84	84
query21	201	126	130	126
query22	5113	4832	4849	4832
query23	34340	33836	33453	33453
query24	10401	2888	2875	2875
query25	581	377	370	370
query26	698	162	154	154
query27	2094	317	320	317
query28	5920	2074	2067	2067
query29	904	623	621	621
query30	270	155	155	155
query31	985	763	754	754
query32	95	53	59	53
query33	650	258	271	258
query34	906	476	481	476
query35	804	689	671	671
query36	1052	941	952	941
query37	104	67	68	67
query38	2920	2780	2779	2779
query39	1613	1607	1565	1565
query40	200	131	126	126
query41	44	41	41	41
query42	103	94	106	94
query43	578	510	540	510
query44	1091	716	738	716
query45	263	312	241	241
query46	1087	756	721	721
query47	2001	1893	1853	1853
query48	374	312	295	295
query49	969	388	399	388
query50	759	391	381	381
query51	6761	6590	6693	6590
query52	103	90	89	89
query53	344	279	289	279
query54	852	450	451	450
query55	72	72	74	72
query56	249	225	227	225
query57	1221	1145	1181	1145
query58	213	204	203	203
query59	3341	3396	3167	3167
query60	261	230	233	230
query61	91	90	87	87
query62	699	488	486	486
query63	331	287	281	281
query64	8537	7402	7363	7363
query65	3113	3058	3105	3058
query66	814	330	337	330
query67	15751	14975	15029	14975
query68	8890	551	551	551
query69	552	304	321	304
query70	1399	1147	1069	1069
query71	514	264	258	258
query72	8050	2527	2388	2388
query73	1532	326	324	324
query74	6587	6117	6095	6095
query75	4375	2593	2628	2593
query76	5121	1068	992	992
query77	669	266	260	260
query78	10852	10246	10177	10177
query79	7748	513	501	501
query80	1439	437	443	437
query81	502	230	226	226
query82	646	92	100	92
query83	236	168	167	167
query84	263	86	87	86
query85	1111	274	268	268
query86	404	294	301	294
query87	3307	3084	3124	3084
query88	4116	2424	2410	2410
query89	526	370	374	370
query90	2064	187	186	186
query91	129	97	101	97
query92	55	47	50	47
query93	5823	498	505	498
query94	1061	178	186	178
query95	392	298	299	298
query96	605	282	270	270
query97	3148	2994	2973	2973
query98	239	218	257	218
query99	1257	920	894	894
Total cold run time: 301371 ms
Total hot run time: 187471 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 10, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@HappenLee HappenLee merged commit a25d488 into apache:master May 10, 2024
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
COUNTER_UPDATE(_yield_counts, 1);
break;
}
SCOPED_RAW_TIMER(&time_spent);
Copy link

@mengzhisy mengzhisy Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this line removed? It seems to have broken the dynamic priority level adjustment of PipelineTask in task_queue. @HappenLee @Gabriel39

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mengzhisy Sorry for that. I think it is a mistake and I fix it in #49939

Gabriel39 added a commit that referenced this pull request Apr 10, 2025
Execution timer is needed to make a task yield after a time slice.
#34557 removed the timer by mistake.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
Execution timer is needed to make a task yield after a time slice.
apache#34557 removed the timer by mistake.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants