[TGL][BUG] "error : DSP panic!" occured when do multi-pipeline stress test.

[sinahuang]

Describe the bug
"DMIC: ASoC: trigger FE cmd: 1 failed: -110" occured when do multi-pipeline stress test. After issue occured, all pipleline can't work. Can be recovered by rebooting system.

To Reproduce

1. Sudo reboot
2. Run "./multiplex_pipeline-test.sh pcm0p pcm2p pcm6p pcm1c pcm3c".

scripts:

multiplex_pipeline-test.txt

Reproduce rate
Test for 300 times, failed at 17th time.

Expected result
Multi-pipeline should work normally.

Actual result
"DMIC: ASoC: trigger FE cmd: 1 failed: -110" occured when do multi-pipeline stress test.

[  458.545838] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x34001e successful
[  458.545844] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60040000: GLB_STREAM_MSG: TRIG_START
[  458.545916] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x2034001e successful
[  458.546224] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60040000: GLB_STREAM_MSG: TRIG_START
[  458.546306] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60040000: GLB_STREAM_MSG: TRIG_START
[  458.552455] sof-audio-pci 0000:00:1f.3: error : DSP panic!
[  458.552461] sof-audio-pci 0000:00:1f.3: panic: dsp_oops_offset 788480 offset 788480
[  458.552470] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[  458.552569] sof-audio-pci 0000:00:1f.3: error: can't enter idle
[  458.552571] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[  458.552572] sof-audio-pci 0000:00:1f.3: error: panic at src/lib/agent.c:51
[  458.552574] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
[  458.552576] sof-audio-pci 0000:00:1f.3: EXCCAUSE 0x0000003f EXCVADDR 0x00000000 PS       0x00060925 SAR     0x00000000
[  458.552578] sof-audio-pci 0000:00:1f.3: EPC1     0x00000000 EPC2     0xbe03fd98 EPC3     0x00000000 EPC4    0x00000000
[  458.552579] sof-audio-pci 0000:00:1f.3: EPC5     0xbe03fd98 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  458.552581] sof-audio-pci 0000:00:1f.3: EPS2     0x00060520 EPS3     0x00000000 EPS4     0x00000000 EPS5    0x00060520
[  458.552582] sof-audio-pci 0000:00:1f.3: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00000000 INTERRU 0x00010222
[  458.552584] sof-audio-pci 0000:00:1f.3: stack dump from 0xbe071120
[  458.552587] sof-audio-pci 0000:00:1f.3: 0xbe071120: be071150 00000001 00078600 00000001
[  458.552589] sof-audio-pci 0000:00:1f.3: 0xbe071124: 00000000 00000000 00000033 00000000
[  458.552590] sof-audio-pci 0000:00:1f.3: 0xbe071128: 95279400 77f0f1cd 9958f828 ffff94da
[  458.552592] sof-audio-pci 0000:00:1f.3: 0xbe07112c: 9958f828 ffff94da 0000008b 00000000
[  458.552594] sof-audio-pci 0000:00:1f.3: 0xbe071130: bd4c11cc ffffffff 93692600 ffff94da
[  458.552596] sof-audio-pci 0000:00:1f.3: 0xbe071134: 95279400 77f0f1cd 9925ccc0 ffff94da
[  458.552597] sof-audio-pci 0000:00:1f.3: 0xbe071138: 9925ccf8 ffff94da bd488e70 ffffffff
[  458.552599] sof-audio-pci 0000:00:1f.3: 0xbe07113c: 00000000 00000000 00000000 00000000
[  458.552600] sof-audio-pci 0000:00:1f.3: error: waking up any trace sleepers

[  459.053474] sof-audio-pci 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[  459.053476] sof-audio-pci 0000:00:1f.3: error: waking up any trace sleepers
[  459.053480]  DMIC: ASoC: trigger FE cmd: 1 failed: -110
[  459.053486] sof-audio-pci 0000:00:1f.3: pcm: trigger stream 3 dir 1 cmd 0
[  459.053491] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[  459.557722] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x60050000 size 12
[  459.557727] sof-audio-pci 0000:00:1f.3: info: preventing DSP entering D3 state to preserve context
[  459.557736] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[  459.557848] sof-audio-pci 0000:00:1f.3: error: can't enter idle
[  459.557849] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[  459.557851] sof-audio-pci 0000:00:1f.3: error: panic at src/lib/agent.c:51

Test recipe

Kernel:https://github.com/thesofproject/linux/commits/topic/sof-dev commit:d05959
FW: https://github.com/thesofproject/sof/commits/master commit: 8e5ab9c
Tplg: Same with FW branch, sof-tgl-rt711-i2s-rt1308.tplg
Platform:TGL-U RVP with codec ALC711 in SDW mode and RT1308 in I2S mode

multipipeline_test_27_dmesg.log
Because of FW issue #2507, so can't cat the sof trace.

[sinahuang]

This issue happened after blocked issue #2492 fixed with FW commit:30492d7, from my bi-sect result might be caused by 30492d7 or c596217
@tlauda Please help to check it.

[sinahuang]

Tested with master(commit:8e5ab9c2)+sof-dev(commit: 0f90705), issue can be reproduced.
Test for 300 times, failed at 37th time.
multipipeline_test_37_logger_error.log
multipipeline_test_37_all_logger.log
multipipeline_test_37_dmesg.log

[tlauda]

@sinahuang It looks like DMIC clocking problem. Are you sure your HW is correctly configured/reworked? We don't see such behavior on our TGL RVPs.
@emilchudzik Could you help with our rework list and BIOS settings?
@singalsu Any ideas?

[sinahuang]

My HW configure/reworks as below:

1. Reworked SDW on ALC711 and I2S on 1308 on TGL-RVP .

2. BIOS settings:
   Set Audio Link Mode to Advanced Link Config
   Enable: HDA link, DMIC #0, DMIC DMA trace output can contain repeated trace data blocks #1 , SSP Topic/module - Test Ignore #2, SDW DMA trace output can contain repeated trace data blocks #1 (ALC1308 is on SSP Topic/module - Test Ignore #2, ALC711 is on SDW DMA trace output can contain repeated trace data blocks #1）
   Enter HD Audio DSP Features Configuration, Enable I2S and select I2S Codec: Realtek ALC1308

[singalsu]

@tlauda Those RORs should not happen. Apparently some slowness with DMA capture apparently does it. The other source for ROR is wrong configuration of decimation but I doubt it as happened since it worked until 37th time. I wonder if in addition to clearing the status bit should the FIFO be emptied by setting FINIT for OUTCTLRx? That would flush all the samples in FIFO and create discontinuity into audio.

[tlauda]

@singalsu I think it won't help in the long run. It looks like there is something fundamentally wrong with the DMIC clocking.
@sinahuang Do you have any other TGL RVP? Maybe there is something wrong with the HW on this one, since we don't observe such issue on our boards.

[sinahuang]

Used the other TGL RVP with codec ALC711 in SDW mode and RT1308 in I2S mode, issue still can be reproduced.

Test for 300 times, failed at 28th time.
multipipeline_test_28_dmesg.log

[tlauda]

@sinahuang If it only reproduces with multiple pipelines, then please verify with higher clock. Please build FW with disabled low power clock: Platform -> [ ] Enable low power ring oscillator as the default clock.

[sinahuang]

Checked on FW with disabled low power clock: Platform -> [ ] Enable low power ring oscillator as the default clock, Issue still can be reproduced.

[tlauda]

@sinahuang Could you check again? I've rebuilt the master with this option disabled and the test has passed 300 times on your platform.