[BUG] APL_UP2_NOCODEC_ZEPHYR with XCC failed to playback or capture

Describe the bug
In APL_UP2_NOCODEC device with the firmware built with XCC and Zephyr, playback or capture any audio will fail.
NOTE: If build the firmware with GCC or XTOS it will work well.
This is introduced after we re-enable Zephyr builds for APL with XCC in #4852.
I guess this is a Zephyr issue, but it's hard to find a commit of Zephyr that works well, so I didn't bisect the issue further.

To Reproduce
Playback or capture.

Reproduction Rate
Almost 100%

Environment
Kernel Branch: topic/sof-dev
Kernel Commit: thesofproject/linux@259db995
SOF Branch: main
SOF Commit: 8107d6a41b9b
SOF Compiler: XCC
Topology: /lib/firmware/intel/sof-tplg/sof-apl-nocodec.tplg
Platform: APL_UP2_NOCODEC_ZEPHYR

Screenshots or console output
dmesgs look different in different cases, here are some typical examples:

[  126.519287] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[  126.520176] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx succeeded: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[  126.520204] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx: 0x80010000: GLB_DAI_MSG: CONFIG
[  127.021044] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx timed out for 0x80010000 (msg/reply size: 216/12)
[  127.021250] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: preventing DSP entering D3 state to preserve context
[  127.021259] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump start ]------------
[  127.021277] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: hda irq intsts 0x00000000 intlctl 0xc0000001 rirb 00
[  127.021291] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: dsp irq ppsts 0x00000000 adspis 0x00000000
[  127.021305] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[  127.021317] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump end ]------------
[  127.021328] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump start ]------------
[  127.021339] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_COMPLETE (5)
[  127.021353] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: status: fw entered - code 00000005
[  127.021407] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: unexpected fault 0x00000000 trace 0x00004000
[  127.021420] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump end ]------------
[  127.021434] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed to set dai config for NoCodec-0
[  127.021446] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed to load dai config for DAI SSP0.OUT
[  127.021466] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[  127.525024] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx timed out for 0x30020000 (msg/reply size: 12/12)
[  127.525057] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: preventing DSP entering D3 state to preserve context
[  127.525065] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump start ]------------
[  127.525080] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: hda irq intsts 0x00000000 intlctl 0xc0000001 rirb 00
[  127.525092] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: dsp irq ppsts 0x00000000 adspis 0x00000000
[  127.525105] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[  127.525116] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump end ]------------
[  127.525125] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump start ]------------
[  127.525134] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_COMPLETE (5)
[  127.525148] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: status: fw entered - code 00000005
[  127.525201] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: unexpected fault 0x00000000 trace 0x00004000
[  127.525212] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump end ]------------
[  127.525225] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed to free widget SSP0.OUT
[  127.525240] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed setting up DAI widget SSP0.OUT
[  127.525252] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ASoC: error at snd_soc_dai_hw_params on SSP0 Pin: -110
[  127.525277] kernel:  NoCodec-0: ASoC: soc_pcm_hw_params() failed (-110)
[  127.525293] kernel:  Port0: ASoC: dpcm_fe_dai_hw_params failed (-110)
[  127.525314] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: free stream 0 dir 0
[  127.526893] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: close stream 0 dir 0

[  269.749342] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: open stream 10 dir 1
[  269.749355] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: period min 192 max 16384 bytes
[  269.749358] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: period count 2 max 16
[  269.749361] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: buffer max 65536 bytes
[  269.749588] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[  270.251512] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx timed out for 0x30010000 (msg/reply size: 96/20)
[  270.251543] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: preventing DSP entering D3 state to preserve context
[  270.251550] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump start ]------------
[  270.251564] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: hda irq intsts 0x00000000 intlctl 0xc0000001 rirb 00
[  270.251575] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: dsp irq ppsts 0x00000000 adspis 0x00000000
[  270.251587] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[  270.251597] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump end ]------------
[  270.251605] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump start ]------------
[  270.251613] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_COMPLETE (5)
[  270.251625] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: status: fw entered - code 00000005
[  270.251676] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: unexpected fault 0x00000000 trace 0x00004000
[  270.251687] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump end ]------------
[  270.251702] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed to load widget DMIC0.IN
[  270.251712] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed setting up DAI widget DMIC0.IN
[  270.251723] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ASoC: error at snd_soc_dai_hw_params on SSP3 Pin: -110
[  270.251776] kernel:  NoCodec-3: ASoC: soc_pcm_hw_params() failed (-110)
[  270.251810] kernel:  DMIC: ASoC: dpcm_fe_dai_hw_params failed (-110)
[  270.251840] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: free stream 10 dir 1
[  270.251976] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: open stream 11 dir 1
[  270.251990] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: period min 192 max 16384 bytes
[  270.251997] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: period count 2 max 16
[  270.252004] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: buffer max 65536 bytes
[  270.253288] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: close stream 10 dir 1
[  270.253569] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[  270.755234] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ipc tx timed out for 0x30010000 (msg/reply size: 96/20)
[  270.755257] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: preventing DSP entering D3 state to preserve context
[  270.755262] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump start ]------------
[  270.755274] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: hda irq intsts 0x00000000 intlctl 0xc0000001 rirb 00
[  270.755281] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: dsp irq ppsts 0x00000000 adspis 0x00000000
[  270.755290] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[  270.755296] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ IPC dump end ]------------
[  270.755301] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump start ]------------
[  270.755307] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_COMPLETE (5)
[  270.755315] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: status: fw entered - code 00000005
[  270.755361] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: unexpected fault 0x00000000 trace 0x00004000
[  270.755368] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump end ]------------
[  270.755379] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed to load widget DMIC1.IN
[  270.755386] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed setting up DAI widget DMIC1.IN
[  270.755393] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ASoC: error at snd_soc_dai_hw_params on SSP4 Pin: -110
[  270.755435] kernel:  NoCodec-4: ASoC: soc_pcm_hw_params() failed (-110)
[  270.755446] kernel:  DMIC16kHz: ASoC: dpcm_fe_dai_hw_params failed (-110)
[  270.755465] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: free stream 11 dir 1
[  270.756425] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: pcm: close stream 11 dir 1

In today's daily test, the firmware with Zephyr failed to boot in APL_UP2_NOCODEC.

Environment
Kernel Branch: topic/sof-dev
Kernel Commit: thesofproject/linux@41d1493c
SOF Branch: main
SOF Commit: b82e0efc7557
SOF Compiler: XCC
Zephyr Branch: main
Zephyr commit: zephyrproject-rtos/zephyr@726656d

dmesg:

[    7.786486] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: DSP core(s) enabled? 0 : core_mask 2
[    7.802973] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: FW Poll Status: reg[0x80000]=0x5000001 successful
[    7.820440] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: FW Poll Status: reg[0x80000]=0x5 successful
[    7.820467] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: FW Poll Status: reg[0x160]=0x140000 successful
[    7.820479] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: Firmware download successful, booting...
[    8.525995] kernel: i915 0000:00:02.0: [drm] failed to retrieve link info, disabling eDP
[    8.574997] kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    8.611633] kernel: ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[    8.614657] kernel: input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input2
[    9.261629] kernel: loop0: detected capacity change from 0 to 8
[    9.825445] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: firmware boot failure
[    9.825524] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump start ]------------
[    9.825587] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_IN_PROGRESS (2)
[    9.825653] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: status: fw entered - code 00000005
[    9.825732] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: invalid header size 0x1010202. FW oops is bogus
[    9.825798] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: unexpected fault 0x00000000 trace 0x00000000
[    9.825862] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump end ]------------
[    9.825923] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state change: 2 -> 3
[    9.825932] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: error: failed to boot DSP firmware -5
[    9.826978] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: FW Poll Status: reg[0x4]=0x1010303 successful
[    9.827000] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: FW Poll Status: reg[0x4]=0x303 successful
[    9.827010] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: DSP core(s) enabled? 0 : core_mask 3
[    9.827692] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state change: 3 -> 0
[    9.827726] kernel: sof-audio-pci-intel-apl: probe of 0000:00:0e.0 failed with error -5

[marc-hb]

@lyakh , @kv2019i is this not a duplicate?

[kv2019i]

This is fixed and can be closed, right? @lyakh @YongAnLu00

Not reproducible, let's close it.