[BUG] FW boot failure with SOF main when trace mailbox CONFIG_TRACEM is enabled

[ranj063]

Describe the bug
FW fails to boot when built with trace mailbox enabled.

What have you tried to diagnose or workaround this issue?
Bisection points to 7df3674
Reverting the commit fixes the problem.

To Reproduce
Build FW with Trace->Trace Mailbox enabled in config

Reproduction Rate
100%

Expected behavior
No issues during FW boot.

Impact
show-stopper

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • Kernel: thesofproject/linux@a2f0699
   • SOF: 6b650bd

Error log:
[ 8.414944] sof-audio-pci-intel-tgl 0000:00:1f.3: error: firmware boot failure
[ 8.414988] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 8.415028] sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[ 8.415098] sof-audio-pci-intel-tgl 0000:00:1f.3: error: extended rom status: 0x5 0x0 0x4000 0x0 0x0 0x0 0x256051f 0x0
[ 8.415138] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 8.415169] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to boot DSP firmware -5

[ranj063]

@keyonjie FYI

[keyonjie]

Thanks for reporting @ranj063

After debugging, just root caused this to be deadlock in the trace->lock, will submit a fix for it soon.

Good finding here, maybe we should add Not-default kconfig validation coverage to our CI also. @marc-hb @keqiaozhang

[keyonjie]

@ranj063 the PR here: #4700

[marc-hb]

Simpler fix at #4760 (two reverts + 1 line change)

[marc-hb]

Good finding here, maybe we should add Not-default kconfig validation coverage to our CI also. @marc-hb @keqiaozhang

CONFIG_TRACEM is likely to have a severe performance impact.

[marc-hb]

BTW I rely on CONFIG_TRACEM for my transition to Zephyr "strategy", see (now reverted) commit 0def905