[BUG][TGL] failed to ready HDA controller gctl 0x0 (sof_probe_work failed)

[sathya-nujella]

DUT: TGL Chrome DUT with Max98373_RT5682 I2S
BIOS: Coreboot
Kernel: Google - Kernel 5.4
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/refs/heads/chromeos-5.4

Issue:
Sound card creation failed.

TEST SCENARIO:
During reboot tests, randomly seeing this issue.
Able to reproduce the issue (no consistent method, just reboot some times catch this issue).

From logs:
1) error from dmesg:
[ 8.912119] sof-audio-pci 0000:00:1f.3: error: failed to ready HDA controller gctl 0x0
[ 8.922070] sof-audio-pci 0000:00:1f.3: error: get caps error
[ 8.929600] sof-audio-pci 0000:00:1f.3: display power disable
[ 8.929622] sof-audio-pci 0000:00:1f.3: error: failed to probe DSP -5
[ 8.937423] sof-audio-pci 0000:00:1f.3: error: sof_probe_work failed err: -5

3. lspci -vvvv //1f.3 part
   00:1f.3 Multimedia audio controller: Intel Corporation Device a0c8 (rev 20)
   Subsystem: Intel Corporation Device 7270
   Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
   Interrupt: pin A routed to IRQ 16
   Region 0: Memory at 7ff48000 (64-bit, non-prefetchable) [size=16K]
   Region 4: Memory at 7fe00000 (64-bit, non-prefetchable) [size=1M]
   Capabilities: [50] Power Management version 3
   Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
   Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [80] Vendor Specific Information: Len=14 <?>
   Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
   Address: 0000000000000000 Data: 0000
   Kernel driver in use: sof-audio-pci
   Kernel modules: snd_hda_intel, snd_sof_pci

[kv2019i]

There are a couple of past bugs with similar fingerprint:
#1399
#1556
#1735

Most have such low reproduce rates, that no exact patch could be pointed to for making the problem go away.

One concrete thing to try is this:
#1735 (comment)

[kv2019i]

Findings today:

• with snd-hda-intel, bug is harder to trigger (on same system)
• when bug occurs, unloading and reloading does not help
• when bug occurs, unloading and reloading snd-hda-intel instead of SOF, does not help

Yet to be tested:

• try to reproduce with force-nocodec mode
• unloading driver always before reboot
• increase delay to wait HDA to come from reset

[plbossart]

Right, we faced this before and never reached a conclusion as to what cause the capability parsing to fail. I had a discussion with hardware folks and there was no smoking gun.

A work-around was added in upstream commit f09e9c7 ('ASoC: SOF: Intel: hda-ctrl: add reset cycle before parsing capabilities')

@sathya-nujella can you check if this was back-ported and try to revert it if yes?

[ranj063]

Right, we faced this before and never reached a conclusion as to what cause the capability parsing to fail. I had a discussion with hardware folks and there was no smoking gun.

A work-around was added in upstream commit f09e9c7 ('ASoC: SOF: Intel: hda-ctrl: add reset cycle before parsing capabilities')

@sathya-nujella can you check if this was back-ported and try to revert it if yes?

@plbossart I already suggested this last night and it didnt help

[kv2019i]

More tests:

• try to reproduce with force-nocodec mode -> still happens
• increase delay to wait HDA to come from reset -> done, did not help

Ongoing:

• unloading driver always before reboot -> under progress, @sathya-nujella running, no errors yet but very limited iterations

[plbossart]

@ranj063 if the unloading driver before reboot solves the issue, wouldn't this point to a .shutdown() callback that's not doing the same thing as a .remove()?

[ranj063]

@ranj063 if the unloading driver before reboot solves the issue, wouldn't this point to a .shutdown() callback that's not doing the same thing as a .remove()?

@plbossart I'd think so. I'll work with @sathya-nujella this morning to try it out.

[sathya-nujella]

Thank you @ranj063 . In today's debug with Ranjani's support tried, warm reboot tests.
With her suggestion, found out that after reverting this SOF patch, issue not reproduced in 100-iterations.
As it is random issue, I will continue more tests. But clearly reverting that SOF patch shows improvement.

Patch reverted: thesofproject/sof@75561a9

SOF issue reported based on this: thesofproject/sof#3633

[mengdonglin]

Let's track the issue in thesofproject/sof#3633 atm