Arecord tampering causes DSP crashes

[cujomalainey]

I am able to cause the DSP to crash pretty easily with a simple script.
while arecord -D hw:0,99 -f S32_LE -d 1; do :; done

Then by tampering with loop by using C-z (to suspend) or C-c (to interrupt) you can crash the dsp pretty easily.

Example of DSP panic by interrupting loop

[  103.538428] sof-audio sof-audio: error : DSP panic!
[  103.538479] sof-audio sof-audio: status: fw entered - code 00000005
[  103.538645] sof-audio sof-audio: error: can't enter idle
[  103.538648] sof-audio sof-audio: error: trace point 00004000
[  103.538652] sof-audio sof-audio: error: DSP Firmware Oops
[  103.538655] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[  103.538658] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[  103.538661] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[  103.538665] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  103.538668] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[  103.538671] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[  103.538674] sof-audio sof-audio: stack dump from 0xbe07fec0
[  103.538678] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[  103.538681] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[  103.538684] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe00361c 0x00000000
[  103.538687] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[  103.538690] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[  103.538693] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[  103.538697] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[  103.538700] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[  103.538703] sof-audio sof-audio: error: waking up any trace sleepers

a run where I simply suspend the stream (no resume)

[   82.612077] sof-audio sof-audio: firmware boot complete
[   90.108473] sof-audio sof-audio: error : DSP panic!
[   90.108538] sof-audio sof-audio: status: fw entered - code 00000005
[   90.108731] sof-audio sof-audio: error: can't enter idle
[   90.108734] sof-audio sof-audio: error: trace point 00004000
[   90.108737] sof-audio sof-audio: error: DSP Firmware Oops
[   90.108740] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   90.108744] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   90.108747] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   90.108751] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   90.108754] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   90.108757] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   90.108760] sof-audio sof-audio: stack dump from 0xbe07fec0
[   90.108764] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   90.108767] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   90.108770] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   90.108773] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   90.108776] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   90.108780] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   90.108783] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   90.108786] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   90.108789] sof-audio sof-audio: error: waking up any trace sleepers

Another run where I suspend then resume arecord (note, did not check if crash happened before resume)

[1]+  Stopped                 arecord -D hw:0,99 -f S32_LE -d 1
localhost ~ # fg
arecord -D hw:0,99 -f S32_LE -d 1
overrun!!! (at least 1473.533 ms long)
arecord: pcm_read:2143: read error: Connection timed out

the kernel output

[   24.127078] sof-audio sof-audio: firmware boot complete
[   27.873506] sof-audio sof-audio: error : DSP panic!
[   27.873538] sof-audio sof-audio: status: fw entered - code 00000005
[   27.873701] sof-audio sof-audio: error: can't enter idle
[   27.873704] sof-audio sof-audio: error: trace point 00004000
[   27.873708] sof-audio sof-audio: error: DSP Firmware Oops
[   27.873711] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   27.873715] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   27.873718] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   27.873721] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   27.873725] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   27.873728] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   27.873731] sof-audio sof-audio: stack dump from 0xbe07fec0
[   27.873735] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   27.873738] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   27.873741] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   27.873745] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   27.873748] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   27.873751] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   27.873754] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   27.873758] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   27.873761] sof-audio sof-audio: error: waking up any trace sleepers
[   28.847341] sof-audio sof-audio: error: ipc timed out for 0x60040000 size 0xc
[   28.847369] sof-audio sof-audio: status: fw entered - code 00000005
[   28.847508] sof-audio sof-audio: error: can't enter idle
[   28.847514] sof-audio sof-audio: error: trace point 00004000
[   28.847520] sof-audio sof-audio: error: DSP Firmware Oops
[   28.847526] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   28.847533] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   28.847559] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   28.847562] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   28.847566] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   28.847569] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   28.847573] sof-audio sof-audio: stack dump from 0xbe07fec0
[   28.847576] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   28.847579] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   28.847583] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   28.847586] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   28.847589] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   28.847593] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   28.847596] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   28.847600] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   28.847603] sof-audio sof-audio: error: waking up any trace sleepers
[   28.847616]  DMIC01: ASoC: trigger FE failed -110
[   29.151353] sof-audio sof-audio: error: ipc timed out for 0x60050000 size 0xc
[   29.151381] sof-audio sof-audio: status: fw entered - code 00000005
[   29.151523] sof-audio sof-audio: error: can't enter idle
[   29.151529] sof-audio sof-audio: error: trace point 00004000
[   29.151535] sof-audio sof-audio: error: DSP Firmware Oops
[   29.151541] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   29.151547] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   29.151554] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   29.151560] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   29.151566] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   29.151587] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   29.151590] sof-audio sof-audio: stack dump from 0xbe07fec0
[   29.151594] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   29.151597] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   29.151600] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   29.151603] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   29.151606] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   29.151610] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   29.151613] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   29.151616] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   29.151619] sof-audio sof-audio: error: waking up any trace sleepers
[   29.151632]  DMIC01: ASoC: trigger FE failed -110

These results are all from V0.5 drop, will continue testing as I move around the tree

[ranj063]

@singalsu is this related to the crash that you are seeing after 46 iterations of dmic capture ?

[xiulipan]

@cujomalainey
Seems duplicate to #489
And like some DMIC issues

[cujomalainey]

@xiulipan you linked to this issue, do you mean #487? I think thats very possible

[xiulipan]

@cujomalainey
Yes, #487.
I think this panic will only happen with DMIC record.
And it is a software panic report by sa_agent.

[lgirdwood]

@singalsu Looks like another vector that may help you identifying root cause. @ZhendanYang can you try this test with SSP only to confirm it's DMIC issue.

[cujomalainey]

@lgirdwood I don't thinks its DMIC specific, I was able to produce something similar by running a similar script and tampering with it but instead using speaker-test. Here is the DSP crash from that.

[  341.973324] sof-audio sof-audio: error : DSP panic!
[  341.973356] sof-audio sof-audio: status: fw entered - code 00000005
[  341.973581] sof-audio sof-audio: error: can't enter idle
[  341.973583] sof-audio sof-audio: error: trace point 00004000
[  341.973587] sof-audio sof-audio: error: DSP Firmware Oops
[  341.973590] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[  341.973593] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[  341.973596] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[  341.973599] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  341.973602] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[  341.973605] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[  341.973607] sof-audio sof-audio: stack dump from 0xbe07fec0
[  341.973611] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[  341.973614] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[  341.973617] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe00361c 0x00dbb993
[  341.973620] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe059d74 0xbe059d50
[  341.973623] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[  341.973626] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[  341.973629] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[  341.973631] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[  341.973634] sof-audio sof-audio: error: waking up any trace sleepers
[  343.712330] sof-audio sof-audio: error: ipc timed out for 0x60040000 size 0xc
[  343.712358] sof-audio sof-audio: status: fw entered - code 00000005
[  343.712523] sof-audio sof-audio: error: can't enter idle
[  343.712526] sof-audio sof-audio: error: trace point 00004000
[  343.712529] sof-audio sof-audio: error: DSP Firmware Oops
[  343.712532] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[  343.712535] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[  343.712538] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[  343.712541] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  343.712544] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[  343.712547] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[  343.712549] sof-audio sof-audio: stack dump from 0xbe07fec0
[  343.712552] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[  343.712555] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[  343.712558] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe00361c 0x00dbb993
[  343.712561] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe059d74 0xbe059d50
[  343.712564] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[  343.712567] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[  343.712570] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[  343.712573] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[  343.712576] sof-audio sof-audio: error: waking up any trace sleepers
[  343.712582]  Speakers: ASoC: trigger FE failed -110

[lgirdwood]

@cujomalainey ok, must be something in the pipeline or DMA related.
@ZhendanYang can you try modifying the script and use speaker-test and check this applies to both HDA and SSP endpoints. This will tell us if it's pipeline or DMA specific (since both use different DMA)

[ZhendanYang]

On up2, tests are blocked by #443 . Please @keqiaozhang if it's a GLK issue.

[mmaka1]

@cujomalainey @lgirdwood I am able to repro DSP crash easily with the arecord loop, however I get 'out of memory', not the original 'can't enter idle'. With the #495 applied, there is no repro on my board. No other issues like the 'idle' one appeared.
Are you still observing this with 079558d?

[cujomalainey]

I cannot reproduce this currently at master. I'm not seeing any errors in the trace buffer. These is the only errors I'm seeing in the kernel logs.

[  322.234723] sof-audio sof-audio: XRUN for unknown stream, msg_id 0
and
[  362.033758] sof-audio sof-audio: error: DSP trace buffer overflow 16 bytes. Total messages 479