ll_schedule: add a DSP load tracker to ll_tasks_execute #5407

jsarha · 2022-02-23T16:29:40Z

Add a lightweight load calculatio to ll_tasks_execute() and prints out
the average every 2^10 runs (roughly once per second on most systems).

Example log output:
--cut--
[ 6950848.421715] ( 809129.187500) c0 ll-schedule ./schedule/ll_schedule.c:139 INFO ll avg 19720/38400
[ 8868838.449667] ( 679100.000000) c0 ll-schedule ./schedule/ll_schedule.c:139 INFO ll avg 19658/38400
[ 10802857.643650] ( 565147.187500) c0 ll-schedule ./schedule/ll_schedule.c:139 INFO ll avg 19822/38400
--cut--

Signed-off-by: Kai Vehmanen kai.vehmanen@linux.intel.com

src/schedule/ll_schedule.c

lgirdwood · 2022-02-25T15:27:35Z

@jsarha any update ?

marcinszkudlinski · 2022-03-07T09:05:10Z

src/schedule/ll_schedule.c

 }

+/* perf measurement windows size 2^x */
+#define CHECKS_WINDOW_SIZE	10


It looks to me as a debug feature,
once per second doesn't look very heavy, anyway i suggest putting it as an option in kconfig

@jsarha can we give it the same Kconfig menu as the same feature that we have in the Zephyr LL scheduler i.e. the same Kconfig should enable them.
Can we also make the output trace message the same as Zephyr version too (using same units and strings) so that it's easy for developers to compare performance between xtos and Zephyr when optimising. Thanks.

I have commit in my own repo for adding kconfig under Debug menu, but is that what you wanted? Or should it be without menu presence, and maybe in the schedule directory with changes?

I think we need to met a very aggressive power limitations and need to count every single cycle, so I think that if performance log is the only purpose for calculating CPU load, it should be off when there's no need for it (like in production). Unless there are other purposes?

There is now a Kconfig option, are you are happy with it? Can I resolve this conversation?

src/schedule/ll_schedule.c

jsarha · 2022-03-16T08:58:50Z

I think this now does more or less the same thing as the Zephyr side implementation. Is there something more needed here before I turn this to be ready for review? Maybe squash everything to a single commit?

jsarha · 2022-03-16T09:07:34Z

This is how the output on xtos side looks now:

[    24574359.440169] (     1697423.750000) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0x9e17ec18 agent-work <c63c4e75-8f61-4420-9319-1395932efa9e> avg 113, max 116
[    24924414.009593] (      350054.562500) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0xbe1c0100 pipe-task <f11818eb-e92e-4082-82a3-dc54c604ebb3> avg 1023, max 1031
[    24924931.769989] (         517.760376) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0xbe1c0180 pipe-task <f11818eb-e92e-4082-82a3-dc54c604ebb3> avg 9643, max 9775
[    26622359.567122] (     1697427.750000) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0x9e17ec18 agent-work <c63c4e75-8f61-4420-9319-1395932efa9e> avg 113, max 116
[    26972413.667796] (      350054.093750) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0xbe1c0100 pipe-task <f11818eb-e92e-4082-82a3-dc54c604ebb3> avg 1023, max 1030
[    26972927.834442] (         514.166626) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0xbe1c0180 pipe-task <f11818eb-e92e-4082-82a3-dc54c604ebb3> avg 9644, max 9776
[    28670359.173242] (     1697431.375000) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0x9e17ec18 agent-work <c63c4e75-8f61-4420-9319-1395932efa9e> avg 113, max 117
[    29020413.898916] (      350054.718750) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0xbe1c0100 pipe-task <f11818eb-e92e-4082-82a3-dc54c604ebb3> avg 1023, max 1032
[    29020940.982228] (         527.083313) c0 ll-schedule        ./schedule/ll_schedule.c:172  INFO task 0xbe1c0180 pipe-task <f11818eb-e92e-4082-82a3-dc54c604ebb3> avg 9643, max 9787

lgirdwood · 2022-03-16T13:23:40Z

@jsarha if this aligns the output with xtos and zephyr then we should use this as a starting point and add future performance logging in tandem for both in teh same PRs. @marcinszkudlinski @mwasko ok plan for you ?

mwasko · 2022-03-17T13:15:56Z

src/schedule/ll_schedule.c

-	if (load->cycles_peak < diff)
-		load->cycles_peak = diff;
+	if (task->cycles_max < diff)
+		task->cycles_max = diff;


Why max instead of peak? Max suggests range and here we clearly want to track 'peak' value in given timespan

*_peak was my name for variable before the per task implementation. With the per task implementation I decided to reuse the same struct task members as Zephyr side implementation does. The struct task members have different names.

mwasko

Please check comment.

mwasko · 2022-03-17T13:29:26Z

@jsarha if this aligns the output with xtos and zephyr then we should use this as a starting point and add future performance logging in tandem for both in teh same PRs. @marcinszkudlinski @mwasko ok plan for you ?

@lgirdwood, @jsarha the LL average and peak performance logging is SOF audio specific since we need to track MCPS consumption in a sys tick processing period that Zephyr is not aware of. We can of course use Zephyr Thread statistics to obtain the data in each sys tick but the logging and tracking sys tick average / peak will most likely remain a part of SOF itself.

jsarha · 2022-03-17T15:36:54Z

Maybe for simplicity I squash all changes to one, with the kconfig menu option under Debug menu. The evolution process is not that relevant, but it can be still be found here: https://github.com/jsarha/sof/tree/xtos-dsp-load-evolution

lgirdwood

I'm happy as long as we can compare data between xtos and Zephyr.

src/schedule/ll_schedule.c

Calculates average and records the maximum execution cycles taken by each execution round and prints the results out once every 1024 round, e.g. roughly once per second with a typical 1ms tick configuration. The output is mimicked from the Zephyr side scheduler output. There is a Kconfig menu option under debug menu to disable this feature. Currently the default is on. Signed-off-by: Jyri Sarha <jyri.sarha@intel.com>

jsarha · 2022-03-22T21:25:29Z

Please check comment.

@mwasko is been a while and I do not any more follow what this comment was referring to.

lgirdwood

@mwasko I'm good for this, to help with current issues. We can always build on the data output and align as we go forward.

dbaluta reviewed Feb 23, 2022

View reviewed changes

src/schedule/ll_schedule.c Outdated Show resolved Hide resolved

src/schedule/ll_schedule.c Show resolved Hide resolved

lgirdwood added this to the v2.1 milestone Feb 25, 2022

marcinszkudlinski reviewed Mar 7, 2022

View reviewed changes

mwasko reviewed Mar 9, 2022

View reviewed changes

src/schedule/ll_schedule.c Outdated Show resolved Hide resolved

lgirdwood marked this pull request as ready for review March 16, 2022 13:23

lgirdwood requested review from lbetlej, mmaka1, mrajwa and plbossart as code owners March 16, 2022 13:23

mwasko reviewed Mar 17, 2022

View reviewed changes

mwasko requested changes Mar 17, 2022

View reviewed changes

jsarha force-pushed the xtos-dsp-load branch from c414dbc to 6a5e367 Compare March 17, 2022 15:34

lgirdwood reviewed Mar 18, 2022

View reviewed changes

src/schedule/ll_schedule.c Outdated Show resolved Hide resolved

jsarha force-pushed the xtos-dsp-load branch from 6a5e367 to 7e68b7a Compare March 22, 2022 14:35

lgirdwood approved these changes Mar 23, 2022

View reviewed changes

mwasko approved these changes Mar 23, 2022

View reviewed changes

lgirdwood merged commit ece6d76 into thesofproject:main Mar 23, 2022

johnylin76 mentioned this pull request Mar 25, 2022

[FEATURE] Checkers with limits of DSP load tracker #5518

Closed

ll_schedule: add a DSP load tracker to ll_tasks_execute #5407

ll_schedule: add a DSP load tracker to ll_tasks_execute #5407

Uh oh!

Conversation

jsarha commented Feb 23, 2022

Uh oh!

Uh oh!

Uh oh!

lgirdwood commented Feb 25, 2022

Uh oh!

marcinszkudlinski Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

lgirdwood Mar 8, 2022

Choose a reason for hiding this comment

Uh oh!

jsarha Mar 17, 2022

Choose a reason for hiding this comment

Uh oh!

marcinszkudlinski Mar 17, 2022

Choose a reason for hiding this comment

Uh oh!

jsarha Mar 18, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsarha commented Mar 16, 2022

Uh oh!

jsarha commented Mar 16, 2022 • edited by lgirdwood Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lgirdwood commented Mar 16, 2022

Uh oh!

mwasko Mar 17, 2022

Choose a reason for hiding this comment

Uh oh!

jsarha Mar 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwasko left a comment

Choose a reason for hiding this comment

Uh oh!

mwasko commented Mar 17, 2022

Uh oh!

jsarha commented Mar 17, 2022

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsarha commented Mar 22, 2022

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jsarha commented Mar 16, 2022 •

edited by lgirdwood

Loading

jsarha Mar 17, 2022 •

edited

Loading