[BUG] Elkhart Lake CRB dsp init failed

[mattialis]

Describe the bug
Hi, I'm trying an Elkhart lake CRB devkit, with slimbootloader and ubuntu 22.04.5 (kernel 6.8). This board uses an I2S codec (rt5660). When it tries to load SOF firmware, it panics on init:

[ 21.606869] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if info 0x040100
[ 21.606897] sof-audio-pci-intel-tgl 0000:00:1f.3: Digital mics found on Skylake+ platform, using SOF driver
[ 21.607522] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100
[ 21.607764] sof-audio-pci-intel-tgl 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[ 21.615420] sof-audio-pci-intel-tgl 0000:00:1f.3: use msi interrupt mode
[ 25.636581] sof-audio-pci-intel-tgl 0000:00:1f.3: codec #2 probe error, ret: -5
[ 25.636790] sof-audio-pci-intel-tgl 0000:00:1f.3: NHLT_DEVICE_I2S detected, ssp_mask 0x1
[ 25.636796] sof-audio-pci-intel-tgl 0000:00:1f.3: no hda codecs found!
[ 27.943913] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware paths/files for ipc type 0:
[ 27.943927] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware file: intel/sof/sof-ehl.ri
[ 27.943933] sof-audio-pci-intel-tgl 0000:00:1f.3: Topology file: intel/sof-tplg/sof-ehl-rt5660.tplg
[ 27.945350] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:0:0-b678a
[ 27.945358] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:20:0 Kernel ABI 3:23:0
[ 27.945372] sof-audio-pci-intel-tgl 0000:00:1f.3: unknown sof_ext_man header type 3 size 0x30
[ 28.850584] sof-audio-pci-intel-tgl 0000:00:1f.3: cl_dsp_init: timeout with rom_status_reg (0x80000) read
[ 28.850607] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 28.850613] sof-audio-pci-intel-tgl 0000:00:1f.3: Boot iteration failed: 3/3
[ 28.850618] sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[ 28.850635] sof-audio-pci-intel-tgl 0000:00:1f.3: 0x06000021: module: ROM, state: CSE_IPC_RESET_PHASE_1, waiting for: CSE_CSR, running
[ 28.850666] sof-audio-pci-intel-tgl 0000:00:1f.3: extended rom status: 0x6000021 0x0 0x0 0x0 0x0 0x0 0x25401c8 0x0
[ 28.850671] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 28.850694] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp init failed after 3 attempts with err: -110
[ 28.850790] sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to start DSP
[ 28.850795] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to boot DSP firmware -110
[ 28.905151] sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda_dsp_core_reset_enter: timeout on HDA_DSP_REG_ADSPCS read
[ 28.905168] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp core reset failed: core_mask 1
[ 28.905644] sof-audio-pci-intel-tgl 0000:00:1f.3: error: sof_probe_work failed err: -110

To Reproduce
Simply startup ubuntu 22.04.5 on this board, with enabled AUDIO DSP

Reproduction Rate
Always

Expected behavior
SOF DSP firmware intel/sof/sof-ehl.ri successfully startup DSP

Impact
No audio

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).

   • Kernel: the official ubuntu 22.04.5 kernel (I have tried with 6.8.0-59-generic, then 6.8.0-40-generic; also I built a kernel -40 based named -41 for my convenience where I print some debug info)

2. Name of the topology file

   • Topology: intel/sof-tplg/sof-ehl-rt5660.tplg

3. Name of the platform(s) on which the bug is observed.

   • Platform: intel CRB

[mattialis]

I tried also with latest ubuntu 25.04 (kernel Linux ubuntu 6.14.0-15-generic #15-Ubuntu SMP PREEMPT_DYNAMIC Sun Apr 6 15:05:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux )

[ 38.959335] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100
[ 40.531350] sof-audio-pci-intel-tgl 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[ 40.539302] sof-audio-pci-intel-tgl 0000:00:1f.3: use msi interrupt mode
[ 40.551342] sof-audio-pci-intel-tgl 0000:00:1f.3: no hda codecs found!
[ 40.551357] sof-audio-pci-intel-tgl 0000:00:1f.3: NHLT device BT(0) detected, ssp_mask 0x20
[ 40.551364] sof-audio-pci-intel-tgl 0000:00:1f.3: BT link detected in NHLT tables: 0x20
[ 40.551370] sof-audio-pci-intel-tgl 0000:00:1f.3: DMICs detected in NHLT tables: 4
[ 40.551375] sof-audio-pci-intel-tgl 0000:00:1f.3: NHLT device I2S(4) detected, ssp_mask 0x1
[ 40.697476] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware paths/files for ipc type 0:
[ 40.697493] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware file: intel/sof/sof-ehl.ri
[ 40.697499] sof-audio-pci-intel-tgl 0000:00:1f.3: Topology file: intel/sof-tplg/sof-ehl-rt5660.tplg
[ 40.702550] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864
[ 40.702561] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:1
[ 41.615101] sof-audio-pci-intel-tgl 0000:00:1f.3: cl_dsp_init: timeout with rom_status_reg (0x80000) read
[ 41.615131] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 41.615137] sof-audio-pci-intel-tgl 0000:00:1f.3: Boot iteration failed: 3/3
[ 41.615143] sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[ 41.615160] sof-audio-pci-intel-tgl 0000:00:1f.3: 0x06000021: module: ROM, state: CSE_IPC_RESET_PHASE_1, waiting for: CSE_CSR, running
[ 41.615192] sof-audio-pci-intel-tgl 0000:00:1f.3: extended rom status: 0x6000021 0x0 0x0 0x0 0x0 0x0 0x25401c8 0x0
[ 41.615197] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 41.615221] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp init failed after 3 attempts with err: -110

[lgirdwood]

@mattialis I suspect here that FW is still booting when kernel calls a time out (kernel has changed a bit since SOF v2.2), the status showing FW is still booting. Can you try and extend the timeout in the driver and see if that helps.

[mattialis]

Hi, as suggested I rebuilt my ubuntu 22 kernel enlarging trying and timeouts. I don't know if the increase is not enough, however this is the dmesg (the print with asterisk is mine)
[ 25.484919] sof-audio-pci-intel-tgl 0000:00:1f.3: NHLT_DEVICE_I2S detected, ssp_mask 0x1
[ 25.484925] sof-audio-pci-intel-tgl 0000:00:1f.3: no hda codecs found!
[ 27.219948] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware paths/files for ipc type 0:
[ 27.219972] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware file: intel/sof/sof-ehl.ri
[ 27.219978] sof-audio-pci-intel-tgl 0000:00:1f.3: Topology file: intel/sof-tplg/sof-ehl-rt5660.tplg
[ 27.222181] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:0:0-b678a
[ 27.222192] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:20:0 Kernel ABI 3:23:0
[ 27.222213] sof-audio-pci-intel-tgl 0000:00:1f.3: unknown sof_ext_man header type 3 size 0x30
[ 27.223688] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 28.226385] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 29.230675] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 30.233383] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 31.235516] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 31.892191] pci 0000:00:13.0: deferred probe pending: (reason unknown)
[ 32.238507] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 33.241380] **** before snd_sof_dsp_read_poll_timeout, target status 1 BOOT ATTEMPTS MAX 7 poll each 800 us timeout 1000
[ 34.241524] sof-audio-pci-intel-tgl 0000:00:1f.3: cl_dsp_init: timeout with rom_status_reg (0x80000) read
[ 34.241549] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 34.241555] sof-audio-pci-intel-tgl 0000:00:1f.3: Boot iteration failed: 7/7
[ 34.241560] sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[ 34.241577] sof-audio-pci-intel-tgl 0000:00:1f.3: 0x06000021: module: ROM, state: CSE_IPC_RESET_PHASE_1, waiting for: CSE_CSR, running
[ 34.241607] sof-audio-pci-intel-tgl 0000:00:1f.3: extended rom status: 0x6000021 0x0 0x0 0x0 0x0 0x0 0x25401c8 0x0
[ 34.241612] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 34.243680] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp init failed after 7 attempts with err: -110
[ 34.243801] sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to start DSP
[ 34.243807] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to boot DSP firmware -110
[ 34.294376] sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda_dsp_core_reset_enter: timeout on HDA_DSP_REG_ADSPCS read
[ 34.294394] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp core reset failed: core_mask 1
[ 34.294902] sof-audio-pci-intel-tgl 0000:00:1f.3: error: sof_probe_work failed err: -110

[lgirdwood]

@mattialis ok, so FW should have booted in < 6 seconds. Not sure what's different with your system compared to the EHL validation system (which I no longer have). EHL is an older platform and SOF was parked at version v2.2 for EHL, but older version for EHL were also supported.
Can you clone https://github.com/thesofproject/sof-bin and try older EHL FW versions (you can copy over the system FW file with teh older version from this repo and reload the driver).

[kv2019i]

@mattialis what is slimbootloader? (slimbootloader must be https://github.com/slimbootloader ). Is Intel CSME enabled normally? The error indicates audio DSP ROM is failing to engage with the CSME to complete authentication of the DSP FW image and the DSP boot fails.

[mattialis]

Hi, after contacting Intel premium support, I discovered that the original IFWI bios shipped with the reference board has this issue. Updating the IFWI with the latest the dsp init problem disappeared (also, stitching slimbootloader with this new IFWI works too)

Thank you for your time,
Mattia