DSP Ops tester - Take 3 #4874

ranj063 · 2024-03-25T17:41:04Z

This PR adds the API for enabling testing the DSP ops in the driver for testing the FW boot flow, PM flow etc. It introduces a new kernel config to enable the testing debugfs interface. When enabled, normal audio operations will be disabled.

For now the following ops are supported for testing:

boot_fw : To boot a firmware file pass from userspace by setting the fw_filename and fw_path debugfs files
dsp_power_state: To set the DSP state. CUrrent only D3 is supported with this op. SUpport for other states will be added soon.
fw_state: To check the current fw boot state

plbossart

I am not on-board with the directions, the simple "echo 1 > boot" or echo 3 > dsp_state is too simple for scripting. These transitions take time, so there's no way for a script to know how long it needs to wait and if the transition was successful.

I would recommend writing a 2-page summary of what we want to achieve, requirements included, otherwise we'll have lots of code review without first agreeing on what we want to achieve.

sound/soc/sof/core.c

plbossart · 2024-03-25T19:23:23Z

sound/soc/sof/core.c

I would put this DSP_OPS_TEST as a new bitfield in sof_debug

Document the DSPless VS ops_test mode and print if the DSPless mode is ignored due to the ops_test taking precedence.
Or should DSPless mode be the one which should have the higher priority?

sound/soc/sof/debug-dsp-ops.c

plbossart · 2024-03-25T19:28:30Z

sound/soc/sof/debug-dsp-ops.c

this doesn't seem related to the boot but more a shutdown...

plbossart · 2024-03-25T19:29:07Z

sound/soc/sof/debug-dsp-ops.c

Yikes. power up the DSP and dsp_runtime_resume doesn't seem like the same thing at all to me.

plbossart · 2024-03-25T19:29:56Z

sound/soc/sof/debug-dsp-ops.c

No control on whether we boot from IMR?

Always a fresh boot with the boot option. I will need to add another option to do IMR boot later

The IMR boot is the target, I don't think anyone is interested in a fresh boot, historically this created all kinds of issues in stress tests.

ok, fixed now to do IMR boot as default

plbossart · 2024-03-25T19:31:49Z

sound/soc/sof/debug-dsp-ops.c

The problem I have with this API is that it's not quite clear how a script would know that the dsp_power_state has been reached.
How long should the script wait and does it get any error codes?

This seems to simple to be useful IMHO.

@plbossart i have exposed both the current fw boot state and the dsp power state that the script can poll to see if the desired state has been reached. How long to poll something that we can set based on the current kernel poll wait times. As for error codes, Im afraid I have no way to pass that on to userspace but we have everything we need in the dmesg isnt it?

Well, it's rather problematic if you don't have an error code for an API, and rely on something else to find if the routine worked or not... If you start fuzzing or doing all kinds of stress tests, you may have difficulties finding out when exactly the problem occured. it's just very hard to parse such messages. Exhibit A is the SOF CI, where we need to have a start marker and we stop on dev_err but that leaves lots of issues undetected...

It's also problematic if we have to duplicate in scripts the timeouts we already use for the kernel.

I do think we ought to work first on the design/architecture before going into the code review space. It's a very large effort we are starting with scripts that will have to be maintained. We need to do this right from the beginning.

plbossart

There are multiple comments not addressed, not sure if they were missed or github messes with the comments.
The ordering of the patches is weird, we should first have code moves then the addition of new functionality.

plbossart · 2024-03-25T19:22:47Z

sound/soc/sof/Kconfig

We should make it a SND_SOC_SOF_DEBUG_DSP_OPS_TEST_ENABLE, to let distros opt-out, and then set the actual boolean with a kernel parameter.

it's too much work for everyone to have yet another build.

plbossart · 2024-03-25T22:27:59Z

sound/soc/sof/debug-dsp-ops.c

Well, it's rather problematic if you don't have an error code for an API, and rely on something else to find if the routine worked or not... If you start fuzzing or doing all kinds of stress tests, you may have difficulties finding out when exactly the problem occured. it's just very hard to parse such messages. Exhibit A is the SOF CI, where we need to have a start marker and we stop on dev_err but that leaves lots of issues undetected...

It's also problematic if we have to duplicate in scripts the timeouts we already use for the kernel.

I do think we ought to work first on the design/architecture before going into the code review space. It's a very large effort we are starting with scripts that will have to be maintained. We need to do this right from the beginning.

sound/soc/sof/debug-dsp-ops.c

ujfalusi · 2024-06-11T08:43:25Z

I am not on-board with the directions, the simple "echo 1 > boot" or echo 3 > dsp_state is too simple for scripting. These transitions take time, so there's no way for a script to know how long it needs to wait and if the transition was successful.

to note:
boot: echo on > "/sys/bus/pci/devices/0000:00:1f.3/power/control"
power off: echo auto > "/sys/bus/pci/devices/0000:00:1f.3/power/control"

I would recommend writing a 2-page summary of what we want to achieve, requirements included, otherwise we'll have lots of code review without first agreeing on what we want to achieve.

ujfalusi

@ranj063, I'm still not sure about the use of this.
We will introduce a new flow which will be tested and it is different than what we normally use, and we draw conclusion based on the test results?

I also think that if this mode is selected then we should not load topology, not create sound card. I see this as a really dangerous feature which can introduce unexpected side-effects on the normal use of the stack.

ujfalusi · 2024-06-11T08:45:46Z

sound/soc/sof/core.c

Document the DSPless VS ops_test mode and print if the DSPless mode is ignored due to the ops_test taking precedence.
Or should DSPless mode be the one which should have the higher priority?

sound/soc/sof/debug.c

sound/soc/sof/amd/acp-loader.c

ujfalusi · 2024-06-11T08:58:11Z

sound/soc/sof/amd/acp-loader.c

What about the fw_data_bin? acp needs two files to be loaded...

ahh I missed this. I don't know how to handle this for ACP with 2 files. Let me fix the ACP loader to ignore the pass argument for now

sound/soc/sof/debug-dsp-ops.c

ujfalusi · 2024-06-11T10:10:29Z

sound/soc/sof/debug-dsp-ops.c

how do you start the trace after powering on the DSP? Usually we need to send IPC messages. Does the sof_resume() will be called to take care of it?

Our regular sequence for starting trace after boot is: run_firmware() -> trace_init().

So, in the test script, we'd do the same. boot the firmware and then call init_trace. And then do whatever else we want to do say for example load libaries, topologies etc

Can you share the test script? It is hard to visualize how this is used.

Oooohh, so that's why you haven't been reviewing the test script @ujfalusi ? Mere lack of marketing? The PR is here:

tools: Add a new script for firmware boot stress testing sof-test#1180

sound/soc/sof/core.c

sound/soc/sof/intel/hda-dsp.c

ranj063 · 2024-06-11T20:26:27Z

@ranj063, I'm still not sure about the use of this. We will introduce a new flow which will be tested and it is different than what we normally use, and we draw conclusion based on the test results?

I also think that if this mode is selected then we should not load topology, not create sound card. I see this as a really dangerous feature which can introduce unexpected side-effects on the normal use of the stack.

Yes, agreed. Thats what the mode does. There's no audio functionality when this mode is selected

sound/soc/sof/sof-priv.h

sound/soc/sof/core.c

ujfalusi · 2024-06-12T13:31:13Z

sound/soc/sof/intel/hda-dsp.c

the 'during suspend' is redundant, no?
Is it even correct in case we remove the drivers and the power off fails?

sorry not following the "is it correct?" comment? Are you asking if this is correct to call this during driver remove or are you asking is it OK to fail?

sound/soc/sof/debug-dsp-ops.c

ujfalusi · 2024-06-12T14:45:52Z

sound/soc/sof/debug-dsp-ops.c

I cannot put the 'dsp_ops' to anywhere to be honest, strange name.

what would you suggest I call it instead of "dsp_ops"?

"dsp_test" | "dsp_test_mode" | "test_mode" ?
Probably the "test_mode" is better as it is not really testing the DSP as such.

But I'm bad at naming.

This will hopefully be used to find more bugs in the firmware/software than bugs in the DSP hardware! So how about fw_debug_ops? Then you can all the now redundant fw_ prefixes in fw_debug_ops/fw_* below.

ok renamed to fw_debug_ops. But I will leave the fw_ prefix for now because there will also be tplg_filename/ path etc.

ujfalusi · 2024-06-12T14:50:24Z

sound/soc/sof/ipc4-loader.c

On module remove you are going to leak these, we rely on devm to handle these (see sof_ipc4_exit())

ujfalusi · 2024-06-12T14:51:45Z

sound/soc/sof/ipc4-loader.c

As for IPC3, I was merely asking if you have checked how this works with ipc_type=0.

ujfalusi · 2024-06-12T14:52:58Z

sound/soc/sof/debug-dsp-ops.c

I still don't get the meaning of this comment....

ujfalusi

@ranj063, it looks much cleaner now for sure.

I still have reservation on the usefulness of such interface. We test isolated and narrowed down cases which will never going to be possible to use in real use.

Probably it is beneficial, and I just cannot see that. The added code and complexity is minimal, which is a good sign.

sound/soc/sof/sof-priv.h

sound/soc/sof/core.c

sound/soc/sof/debug-dsp-ops.c

ujfalusi · 2024-06-13T08:03:22Z

sound/soc/sof/debug-dsp-ops.c

"dsp_test" | "dsp_test_mode" | "test_mode" ?
Probably the "test_mode" is better as it is not really testing the DSP as such.

But I'm bad at naming.

ujfalusi · 2024-06-13T08:11:33Z

sound/soc/sof/debug-dsp-ops.c

Should we tell the user that what is tried is going to be a NOP and the previously loaded kernel is going to remain?

ujfalusi · 2024-06-13T08:28:45Z

sound/soc/sof/sof-priv.h

and overrides the DSPless mode.

ujfalusi · 2024-06-13T08:33:21Z

sound/soc/sof/debug-dsp-ops.c

What is "In progress"?

ujfalusi · 2024-06-13T08:34:10Z

sound/soc/sof/debug-dsp-ops.c

I kind of like the numeric values as well (got used to them in the kernel log)...
Not sure if it makes sense to include it in the print (6: BOOT_COMPLETE)

ujfalusi · 2024-06-13T08:36:29Z

sound/soc/sof/debug-dsp-ops.c

No, what I mean is that we have this information already, but it is just returning the raw u32.
I don't know if it is used by anyone.

ujfalusi · 2024-06-13T08:39:11Z

sound/soc/sof/debug-dsp-ops.c

Can you share the test script? It is hard to visualize how this is used.

sound/soc/sof/ops.h

plbossart · 2024-06-17T16:49:51Z

sound/soc/sof/core.c

i would really flip the logic and only take the DSP test mode when the user didn't explicitly bypass the DSP.

plbossart · 2024-06-17T16:51:19Z

sound/soc/sof/sof-priv.h

I really believe this is only relevant IF the DSPless mode is NOT enabled. It's more logical to me...

ok I've flipped the logic to have DSPLESS mode take priority over ops testing

sound/soc/sof/debug-dsp-ops.c

plbossart · 2024-06-17T16:56:33Z

sound/soc/sof/debug-dsp-ops.c

We need to be clear on what we mean by 'new firmware'. if someone overrides an existing firmware in /lib/firmware, they will face all kinds of caching issues. We cannot really add a new firmware with SSH and hope that it's used immediately, that's really asking for trouble.

The assumption should be that you test the firmware that was available at boot.

plbossart · 2024-06-17T16:58:00Z

sound/soc/sof/debug-dsp-ops.c

The IMR boot is the target, I don't think anyone is interested in a fresh boot, historically this created all kinds of issues in stress tests.

plbossart · 2024-06-17T16:59:20Z

sound/soc/sof/debug-dsp-ops.c

I was also unable to understand the comment.

Modify the signature of the load_firmware op to take the firmware filename as an argument. The firmware filename should be include the full path of the firmware file to be loaded. This is in preparation to add a debug op that will be used to boot a firmware with the filename passed from userspace. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Before the first firmware boot, set the initial state of the DSP to D3. The DSP D3 state is not the same as the SOF PCI device D3. Instead, it represents that the DSP is in the powered off state. After firmware boot is completed successfully, set the state to D0 to indicate that the DSP has been powered on. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

This will allow for the set_power_state op to be used for switching the DSP to the D3 state. The logic for powering up the DSP when setting the state to D0 also needs to be moved and will be done in a follow up. But for now, this will allow stress testing firmware boot. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

marc-hb · 2024-06-18T21:40:08Z

(can't reply to the original anymore because Github does not like force pushes)

The IMR boot is the target, I don't think anyone is interested in a fresh boot, historically this created all kinds of issues in stress tests.

I don't understand this. AFAIK, the purpose of this PR is to debug boot issues. We know booting from IMR versus not have had different sets of issues in the past. So shouldn't this PR allow testing both? I don't known which default would be best but I feel like both should be possible.

plbossart

Looks ok to me, I am just not sure what the next steps are?

sound/soc/sof/core.c

ranj063 · 2024-06-19T16:33:37Z

Looks ok to me, I am just not sure what the next steps are?

@plbossart the first tiny next step is to have the fw boot script run with our CI to do the stress test. I also want to keep building the test script library to add more things like PM flows, DMA flows, topology loading, IPC etc

Introduce a new debug option to enable testing of DSP operations. With this ioption enabled, normal audio operations will be disabled. The API for the testing interface will be introduced in the follow up patches. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Add debugfs entries for testing DSP ops. This patch only adds the firmware filename and path entries to pass them from userspace. These will be saved in the newly added field for saving the test profile in struct snd_sof_dev. These will be used for testing DSP ops such as booting firmware and setting the DSP power state in the following patches. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Add a new op to test firmware boot. Make sure to the DSP is powered off before attempting to boot a new firmware. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Add a new op to test DSP power state changes. For now only D3 is supported. More states will be added in follow up patches. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Add a read-only debugfs entry to read the current FW boot state. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Unload any existing firmware and set the fw_state to PREPARE in preparation for a subsequent firmware boot. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

These will be used to debugging purposes when testing the different DSP ops. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

ranj063 · 2024-06-19T19:00:38Z

@plbossart just a minor change to add the fw filename and fw_path string lengths

ranj063 · 2024-07-23T17:38:47Z

@ujfalusi any objection to merging this? This will help me build upon it and add more test cases.

ranj063 requested review from bardliao, dbaluta, kv2019i, lgirdwood, plbossart and ujfalusi as code owners March 25, 2024 17:41

ranj063 force-pushed the sof-dev-033124 branch 2 times, most recently from 61b5f1f to e15cf2c Compare March 25, 2024 18:34

plbossart reviewed Mar 25, 2024

View reviewed changes

ranj063 force-pushed the sof-dev-033124 branch from e15cf2c to 4f1fa70 Compare April 24, 2024 16:22

ranj063 mentioned this pull request Apr 24, 2024

tools: Add a new script for firmware boot stress testing thesofproject/sof-test#1180

Merged

ranj063 force-pushed the sof-dev-033124 branch from 4f1fa70 to ecb73cb Compare June 10, 2024 16:39

plbossart reviewed Jun 10, 2024

View reviewed changes

ujfalusi reviewed Jun 11, 2024

View reviewed changes

ranj063 force-pushed the sof-dev-033124 branch from ecb73cb to cd8d129 Compare June 11, 2024 20:46

ranj063 requested review from plbossart and ujfalusi June 11, 2024 20:46

ujfalusi reviewed Jun 12, 2024

View reviewed changes

ranj063 force-pushed the sof-dev-033124 branch from cd8d129 to 50118f7 Compare June 12, 2024 17:51

ujfalusi reviewed Jun 13, 2024

View reviewed changes

sound/soc/sof/ops.h Outdated Show resolved Hide resolved

plbossart reviewed Jun 17, 2024

View reviewed changes

ranj063 added 3 commits June 18, 2024 09:29

ranj063 force-pushed the sof-dev-033124 branch from 50118f7 to e9ef08b Compare June 18, 2024 18:07

ranj063 requested a review from plbossart June 18, 2024 18:08

plbossart previously approved these changes Jun 19, 2024

View reviewed changes

sound/soc/sof/core.c Outdated Show resolved Hide resolved

ranj063 dismissed plbossart’s stale review via 7d4c924 June 19, 2024 18:59

ranj063 force-pushed the sof-dev-033124 branch from e9ef08b to 7d4c924 Compare June 19, 2024 18:59

ranj063 requested a review from plbossart June 19, 2024 18:59

ranj063 added 7 commits June 19, 2024 11:59

ASoC: SOF: debug-dsp-ops: Add a new op to test firmware boot

9d75165

Add a new op to test firmware boot. Make sure to the DSP is powered off before attempting to boot a new firmware. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

ASoC: SOF: debug_dsp_ops: Add op to test DSP power state changes

d18aca6

Add a new op to test DSP power state changes. For now only D3 is supported. More states will be added in follow up patches. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

ASoC: SOF: debug-dsp-ops: Add a debugfs entry to read current FW status

1972c98

Add a read-only debugfs entry to read the current FW boot state. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

ASoC: SOF: debug-dsp-ops: Add op for unloading firmware

95be13c

Unload any existing firmware and set the fw_state to PREPARE in preparation for a subsequent firmware boot. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

ASoC: SOF: debug-dsp-ops: Add ops for trace init/free

bbac831

These will be used to debugging purposes when testing the different DSP ops. Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

ranj063 force-pushed the sof-dev-033124 branch from 7d4c924 to bbac831 Compare June 19, 2024 19:00

plbossart approved these changes Jun 20, 2024

View reviewed changes

lgirdwood approved these changes Jun 21, 2024

View reviewed changes

plbossart requested review from marc-hb and ujfalusi June 21, 2024 11:00

plbossart merged commit 7df4fc1 into thesofproject:topic/sof-dev Jul 30, 2024

ranj063 deleted the sof-dev-033124 branch July 30, 2024 17:03

marc-hb mentioned this pull request Aug 2, 2024

use prep_c for xtensa to align with other architectures zephyrproject-rtos/zephyr#75665

Merged

DSP Ops tester - Take 3 #4874

DSP Ops tester - Take 3 #4874

Uh oh!

Conversation

ranj063 commented Mar 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

plbossart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plbossart left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ujfalusi commented Jun 11, 2024

Uh oh!

ujfalusi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ranj063 commented Jun 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

ranj063 commented Mar 25, 2024 •

edited

Loading