[SDW] DSP panic w/ sof-test suspend-resume on Dell SKU 09C6 (CML single amp)

[plbossart]

I was running some tests on SoundWire and started getting a DSP panic during suspend/resume tests

log.txt

kernel: b7cd5a3 ASoC: SOF: ops: print out the polling register
SOF: b5b244f11 (HEAD -> main, origin/main) math: remove path.h dependency
sof-test: f93a3c8 (HEAD -> main, origin/main) multiple-pipeline: remove initial sleep and clarify second sleep
topology TPLG=/lib/firmware/intel/sof-tplg/sof-cml-rt711-rt1308-mono-rt715.tplg
run-all-tests.sh -l1

@bardliao can you try to see if you can reproduce this? The device is no longer in CI/daily tests :-(

[plbossart]

This seems to be a kernel regression, if I go back to the sound/upstream-20210426 tag things are fine.

[plbossart]

sof-dev-rebase-20210426 is broken, bisecting now

[plbossart]

patch to sof-test to speed-up the tests.
sof-test.diff.txt

This seems to occur after either a playback/capture or module load/unload. If I do a suspend only things are fine. I don't know which exact sequence causes the issue.

[plbossart]

git bisect tests on topic/sof-dev-rebase branch

339397348db61ffe3ee9e2202c66ffae694d1977 is the first bad commit
commit 339397348db61ffe3ee9e2202c66ffae694d1977
Author: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
Date:   Fri Mar 12 10:40:59 2021 -0800

    ASoC: SOF: Intel: hda: make sure DAI widget is set up before IPC
    
    With the implementation of the dynamic pipeline feature, widgets
    will only be setup when a PCM is opened during the
    hw_params ioctl. The BE hw_params callback is responsible for
    sending the DAI_CONFIG for the DAI widgets in the DSP.
    With dynamic pipelines, the DAI widgets will need to set up
    first before sending the DAI_CONFIG IPC in the BE hw_params.
    
    Update the BE hw_params/hw_free callbacks for all ALH, HDA and SSP
    DAIs to set up/free the DAI widget before/after DAI_CONFIG IPC.
    
    Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
    Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
    Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
    Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>

 sound/soc/sof/intel/hda-dai.c | 160 ++++++++++++++++++++++++------------------
 sound/soc/sof/intel/hda.c     | 144 ++++++++++++++++++++++++-------------
 sound/soc/sof/intel/hda.h     |   4 ++
 sound/soc/sof/sof-audio.h     |   1 -
 sound/soc/sof/topology.c      |   3 -
 5 files changed, 194 insertions(+), 118 deletions(-)

git bisect start
# good: [a49194360d81ee84d39672e2c6ce8ed62cb3d95d] CI: GitHub: set up build test for SOF
git bisect good a49194360d81ee84d39672e2c6ce8ed62cb3d95d
# good: [f221de624b0b3956287b707911d37be7443edd15] ipc: debug: Add shared memory heap to memory scan
git bisect good f221de624b0b3956287b707911d37be7443edd15
# bad: [f1560a519e633b6b8b6c1455f332750a14ef91c8] ASoC: rt715-sdca: fix clock stop prepare timeout issue
git bisect bad f1560a519e633b6b8b6c1455f332750a14ef91c8
# good: [e901af346028f25037331fe29d462c1f6b43041e] ASoC: SOF: restore kcontrols for widget during set up
git bisect good e901af346028f25037331fe29d462c1f6b43041e
# bad: [c06aca9b8b7602bb3008b9bc3b6a231e3d471afc] ASoC: SOF: Add support for dynamic pipelines
git bisect bad c06aca9b8b7602bb3008b9bc3b6a231e3d471afc
# good: [d4650862e582fd37f4eefd1befd106beb73057f5] ASoC: SOF: Introduce widget use_count
git bisect good d4650862e582fd37f4eefd1befd106beb73057f5
# bad: [339397348db61ffe3ee9e2202c66ffae694d1977] ASoC: SOF: Intel: hda: make sure DAI widget is set up before IPC
git bisect bad 339397348db61ffe3ee9e2202c66ffae694d1977
# first bad commit: [339397348db61ffe3ee9e2202c66ffae694d1977] ASoC: SOF: Intel: hda: make sure DAI widget is set up before IPC

@bardliao @RanderWang @ranj063 @fredoh9 FYI

[plbossart]

Removing PulseAudio removes the IPC error and DSP panic. We're probably missing something in the resume sequence after the dynamic pipeline rework.

[plbossart]

Well, the "good news" is that I can reproduce this problem on a TGL Dell device (SKU 0A3E), using SOF v1.7 however since it's using an Intel signed key.

Same DSP panic with PulseAudio doing weird things. It's not a case of broken hardware but more a broken state machine.

@ranj063 I think given the correlation with dynamic pipeline changes you may want to setup your device and take a look.

[plbossart]

TGL dmesg log for reference
dmesg.txt

[bardliao]

Well, the "good news" is that I can reproduce this problem on a TGL Dell device (SKU 0A3E)

I can also see the issue on TGL Dell device (SKU 0A3E)
dmesg.txt