[BUG] [TGL Chrome] Fw panic when we overwrite the firmware & topology binary to the file system first time and boot it

[RDharageswari]

Describe the bug
Fw panic when we copy the firmware & topology binary to the file system first time and boot it

To Reproduce
Steps to reproduce the behavior: (e.g. list commands or actions used to reproduce the bug)

Reproduction Rate
whenever we copy the firmware & topology for the first time on a board.
The problem goes away if we reboot

Expected behavior
No firmware panic and the sound card should bind successfully

Impact
showstopper

Environment
tgl-13 release.

Screenshots or console output
[ 7.550234] sof-audio-pci 0000:00:1f.3: error : DSP panic!
[ 7.550243] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 7.550456] sof-audio-pci 0000:00:1f.3: error: runtime exception
[ 7.550458] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 7.550459] sof-audio-pci 0000:00:1f.3: error: panic at :0
[ 7.550461] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
[ 7.550463] sof-audio-pci 0000:00:1f.3: error: Exception Cause: LoadStorePIFD ataErrorCause, Synchronous PIF data error during LoadStore access
[ 7.550465] sof-audio-pci 0000:00:1f.3: EXCCAUSE 0x0000000d EXCVADDR 0xbe1d00 00 PS 0x00060925 SAR 0x00000000
[ 7.550466] sof-audio-pci 0000:00:1f.3: EPC1 0xbe02e3b7 EPC2 0xbe0424 0a EPC3 0x00000000 EPC4 0x00000000
[ 7.550468] sof-audio-pci 0000:00:1f.3: EPC5 0x00000000 EPC6 0x000000 00 EPC7 0x00000000 DEPC 0x00000000
[ 7.550470] sof-audio-pci 0000:00:1f.3: EPS2 0x00060b20 EPS3 0x000000 00 EPS4 0x00000000 EPS5 0x00000000
[ 7.550471] sof-audio-pci 0000:00:1f.3: EPS6 0x00000000 EPS7 0x000000 00 INTENABL 0x00000000 INTERRU 0x00000222
[ 7.550473] sof-audio-pci 0000:00:1f.3: stack dump from 0xbe131360
[ 7.550476] sof-audio-pci 0000:00:1f.3: 0xbe131360: 3fe6b541 a0000000 3fe5ad5 0 3eb5aa0d
[ 7.550478] sof-audio-pci 0000:00:1f.3: 0xbe131364: 00000000 3ff99999 9980000 0 be136fb0
[ 7.550479] sof-audio-pci 0000:00:1f.3: 0xbe131368: 00000000 00000000 0000000 0 00000000
[ 7.550481] sof-audio-pci 0000:00:1f.3: 0xbe13136c: b66db3a0 ffff89ff 45573dc 0 ffffb2e5
[ 7.550483] sof-audio-pci 0000:00:1f.3: 0xbe131370: 8ec1bb40 ffffffff 8f21b70 0 ffffffff
[ 7.550484] sof-audio-pci 0000:00:1f.3: 0xbe131374: 5c59e028 ffff89ff 000c080 0 00000000
[ 7.550486] sof-audio-pci 0000:00:1f.3: 0xbe131378: 00000000 00000000 45573e5 0 ffffb2e5
[ 7.550488] sof-audio-pci 0000:00:1f.3: 0xbe13137c: 45573e78 ffffb2e5 c055216 3 ffffffff
[ 7.812424] NET: Registered protocol family 38
[ 7.878571] x86/PAT: flashrom:2707 conflicting memory types fe000000-10000000 0 uncached-minus<->uncached
[ 7.878575] x86/PAT: reserve_memtype failed [mem 0xfe000000-0xffffffff], trac k uncached-minus, req uncached-minus
[ 7.889983] Bluetooth: rfcomm_init_sockets() RFCOMM socket layer initialized
[ 7.890009] Bluetooth: rfcomm_init() RFCOMM ver 1.11
[ 7.908778] NET: Registered protocol family 40
[ 7.932593] Bluetooth: msft_le_set_advertisement_filter_enable_cb() hci0: MSF T filter_enable is already on
[ 7.957788] input: cros_fp_input as /devices/virtual/input/input13
[ 8.051731] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x40070000 s ize 12
[ 8.051740] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 8.051947] sof-audio-pci 0000:00:1f.3: error: runtime exception
[ 8.051948] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 8.051950] sof-audio-pci 0000:00:1f.3: error: panic at :0
[ 8.051951] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
[ 8.051953] sof-audio-pci 0000:00:1f.3: error: Exception Cause: LoadStorePIFD ataErrorCause, Synchronous PIF data error during LoadStore access
[ 8.051954] sof-audio-pci 0000:00:1f.3: EXCCAUSE 0x0000000d EXCVADDR 0xbe1d00 00 PS 0x00060925 SAR 0x00000000
[ 8.051956] sof-audio-pci 0000:00:1f.3: EPC1 0xbe02e3b7 EPC2 0xbe0424 0a EPC3 0x00000000 EPC4 0x00000000
[ 8.051957] sof-audio-pci 0000:00:1f.3: EPC5 0x00000000 EPC6 0x000000 00 EPC7 0x00000000 DEPC 0x00000000
[ 8.051965] sof-audio-pci 0000:00:1f.3: EPS2 0x00060b20 EPS3 0x000000 00 EPS4 0x00000000 EPS5 0x00000000
[ 8.051966] sof-audio-pci 0000:00:1f.3: EPS6 0x00000000 EPS7 0x000000 00 INTENABL 0x00000000 INTERRU 0x00000222
[ 8.051968] sof-audio-pci 0000:00:1f.3: stack dump from 0xbe131360
[ 8.051970] sof-audio-pci 0000:00:1f.3: 0xbe131360: 3fe6b541 a0000000 3fe5ad5 0 3eb5aa0d
[ 8.051971] sof-audio-pci 0000:00:1f.3: 0xbe131364: 00000000 3ff99999 9980000 0 be136fb0
[ 8.051972] sof-audio-pci 0000:00:1f.3: 0xbe131368: 00000000 00000000 0000000 0 00000000
[ 8.051979] sof-audio-pci 0000:00:1f.3: 0xbe13136c: 406535c0 ffffb2e5 0000328 2 00000000
[ 8.051980] sof-audio-pci 0000:00:1f.3: 0xbe131370: 406535a0 ffffb2e5 2883140 0 49f1c1f5
[ 8.051985] sof-audio-pci 0000:00:1f.3: 0xbe131374: b49380d8 ffff89ff 5c59e02 8 ffff89ff
[ 8.051986] sof-audio-pci 0000:00:1f.3: 0xbe131378: 00000000 00000000 406536f 0 ffffb2e5
[ 8.051987] sof-audio-pci 0000:00:1f.3: 0xbe13137c: b4938000 ffff89ff 8dd6932 5 ffffffff
[ 8.052014] sof-audio-pci 0000:00:1f.3: error: hda irq intsts 0x00000000 intl ctl 0xc0000000 rirb 00
[ 8.052015] sof-audio-pci 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspi s 0x00000000
[ 8.052033] sof-audio-pci 0000:00:1f.3: error: host status 0x00000000 dsp sta tus 0x00000000 mask 0x00000003
[ 8.052035] sof-audio-pci 0000:00:1f.3: error: core 1 enable ipc failure -110
[ 8.052040] sof-audio-pci 0000:00:1f.3: error: enable core: -110
[ 8.052055] sof-audio-pci 0000:00:1f.3: ASoC: failed to load widget PCM99C

[RDharageswari]

FW_panic_while_boot.txt

[lgirdwood]

@RDharageswari can you confirm that you copy files (how ? over NFS, scp ?) and then immediately reboot (how) ?
This could also be kernel issue or FS cache. Have you tried copy files then unload/reload modules ?
@ujfalusi anything in kernel flow for FW loading that would break here ?

[bkokoszx]

@lgirdwood @RDharageswari
I was able to reproduce this issue. I remove tplg and fw files, then I copy tplg/fw using scp and reboot by sudo reboot. Sometimes after reboot I get dsp panic in dmesg.

Once I also had a case, that just after reboot I did not have panic, but then I tried to run:

aplay -Dhw:0,0 tmp.wav -c2 -r48000 -fs24_le -v

and after that I got similar panic.

[ujfalusi]

Can it be not related to firmware copying and it is just a coincident? Or is it more likely to happen after delete fw/tplg, copy new fw/tplg, then reboot?
Fwiw, in the log the panic happens before sending (SOF_IPC_GLB_PM_MSG|SOF_IPC_PM_CORE_ENABLE) message (ipc timed out for 0x40070000 size 12), it is hard to say if it is for the pipeline->core or for the swidget->core and also from where it happens.

Can it be reproduced if only the firmware or the tplg file is deleted/copyied ?

After the first DSP crash you can reboot repeatably without any issue?

Can it be caused by unclean shutdown of the DSP in a way that same stale data remain from the previous boot and when we load the new tplg, the firmware tries to interpret it and fails?

Can you try to delete the fw/tplg files and then copy the same ones back? Does that also triggers a crash?
Can you try to update the fw/tplg and then shut down and then power on? Does the crash remain?
Can you reboot repeatedly without touching the fw/tplg and see if there is really no DSP crash?

I have been doing locally: update the fw/tplg, unload all drivers, load the drivers again and now the new firmware is in use. Can you try the same, so update the fw/tplg then remove/insert the modules and see if the crash happens also like that?

[ranj063]

@RDharageswari can you confirm this is tgl-013 specific problem?

[RDharageswari]

I have seen this problem only with tgl-13. Did not hit it before.