[BUG] Volume DSP Panic on IPC response timeout, DSP Panic notification

[wszypelt]

Describe the bug

ERROR:Exception DspPanicError occurred at 'C:\work\tools\cavs_scripts-py\utilities\ipc_driver.py':171 DSP Panic on IPC response timeout! 0x00000005 0x0DEAD006 0x00004000
ERROR:Exception DspPanicError occurred at 'C:\work\tools\cavs_scripts-py\utilities\ipc_driver.py':171 DSP Panic notification received: SOF_IPC_PANIC_EXCEPTION 0x00000005 0x0DEAD006 0x00004000

Bug occurs with a playback iteration greater than 1

Bug was found in:
TestVolumeSimple8000Hz16b16b1ch
TestVolumeSimple8000Hz24b32b4ch
TestVolumeSimple48000Hz32b32b8ch
TestVolumeChangeInMiddle8000Hz32b32b4ch
TestVolumeChangeInMiddle48000Hz32b32b4ch
TestVolumeMultipleChanges8000Hz24b32b2ch
TestVolumeMultipleChanges48000Hz24b32b2ch
TestVolumeMute8000Hz32b32b8ch
TestVolumeMute48000Hz32b32b8ch
TestVolumeInitialRampSimple8000Hz24b32b4ch
TestVolumeInitialRampSimple48000Hz32b32b2ch
TestDiffVolumeChangeInChannels8000Hz24b32b4ch
TestDiffVolumeChangeInChannels48000Hz32b32b4ch

Topology

TestVolumeSimple

                +--------------------------------+
    +------+    | +---+  +- -+  +---+  +-------+ |
    | Host |----->|Buf|->|Vol|->|Buf|->|SSP Dai|---+
    +------+    | +---+  +-- +  +---+  +-------+ | |
                +--------------------------------+ |
                                                   |
                +------------------+               |
    +------+    | +---+  +-------+ |               |
    | Host |<-----|Buf|<-|SSP Dai|<----------------+
    +------+    | +---+  +-------+ |
                +------------------+

Description:
Simultaneous playback and capture on external loopback (same SSP) with signal amplitude verification.
Playback pipeline contains volume component.
Volume gain is explicitly set to 1 before playback starts.

To Reproduce
Run tests with diagnostic driver
with argument: --playback_iterations=2

03_00_TestVolumeSimple8000Hz16b16b1ch

Reproduction Rate
100%

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • SOF BRANCH: main HASH: ea72e3e
2. Name of the platform(s) on which the bug is observed.
   • Platform: TGL RVP

the last working and not working commit: ebc222d

Logs
DSPPanicVolume.zip

[lgirdwood]

@ranj063 module adapter related ?

[kfrydryx]

Still reproducing. Bisect points to module adapter - first bad commit f7663de.

[lgirdwood]

@wszypelt @kfrydryx is this only reproducing with the loopback ? What happens if gain is NOT set prior to test. The DEAD code is an assert().
@lyakh can you take a look at this, thanks !

[lyakh]

@wszypelt can this still be reproduced? But this test isn't in the standard CI, so we don't see it in PR testing?

[wszypelt]

@lyakh @kfrydryx @lgirdwood
Issue still reproducing, with playback iteration > 1. In CI we test all tests with playback_iteration = 1. Therefore, it does not show this issue. We perform stress tests every two weeks.

[lgirdwood]

@lyakh @kfrydryx @lgirdwood
Issue still reproducing, with playback iteration > 1. In CI we test all tests with playback_iteration = 1. Therefore, it does not show this issue. We perform stress tests every two weeks.

Ok, so the Linux CI will test multiple times, but I dont think it uses the loopback mode above. @fredoh9 can you confirm if we have a similar loopback test in the Jenkins CI ?