[BUG] Signature issue on TGL UP Extreme i11 - openssl3

[plbossart]

Describe the bug

The firmware generated locally on my NUC does not boot on UpExtreme 11. The daily-build firmware and the 2.1 version boot fine, so it's not a hardware/CSME issue, rather an undocumented or missing environment for the signature to work.

Standard Ubuntu 22.04 on my side. Same scripts work fine on Up Extreme (WHL)

To Reproduce
Steps to reproduce the behavior: (e.g. list commands or actions used to reproduce the bug)

scripts/xtensa-build-all.sh -d tgl (debug version)
scripts/xtensa-build-all.sh tgl

Reproduction Rate

100%

Expected behavior

no errors on boot

Impact

showstopper

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).

   • Kernel: 9eaf67f8c40cc (sof/topic/sof-dev) soundwire: intel: set dev_num_ida_min

   • SOF: e9442e04c (HEAD -> main, origin/main) module_adpater: change the buffer size of waves_codec

2. Name of the topology file

   • Topology: irrelevant

3. Name of the platform(s) on which the bug is observed.

   • Platform: TGL UpExtreme i11

Screenshots or console output

[    7.008951] snd_sof_intel_hda_common:hda_cl_copy_fw: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x80000]=0x80000012 timedout
[    7.008957] sof-audio-pci-intel-tgl 0000:00:1f.3: hda_cl_copy_fw: timeout with rom_status_reg (0x80000) read
[    7.008969] snd_sof_intel_hda_common:hda_dsp_stream_trigger: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x160]=0x140000 successful
[    7.008973] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[    7.008974] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware download failed
[    7.008975] sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (2)
[    7.008978] snd_sof_intel_hda_common:hda_dsp_get_status: sof-audio-pci-intel-tgl 0000:00:1f.3: unknown ROM status value 80000012
[    7.008991] sof-audio-pci-intel-tgl 0000:00:1f.3: extended rom status:  0x80000012 0x2c 0x0 0x0 0x0 0x0 0x2510113 0x0
[    7.008992] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[    7.009016] sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to start DSP
[    7.009017] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to boot DSP firmware -110
[    7.009020] snd_sof:sof_set_fw_state: sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state change: 2 -> 3
[    7.061361] snd_sof_intel_hda_common:hda_dsp_core_reset_enter: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x4]=0x1d003c timedout
[    7.061368] sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda_dsp_core_reset_enter: timeout on HDA_DSP_REG_ADSPCS read
[    7.061370] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp core reset failed: core_mask 1
[    7.062148] snd_sof:sof_set_fw_state: sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state change: 3 -> 0
[    7.062173] sof-audio-pci-intel-tgl 0000:00:1f.3: error: sof_probe_work failed err: -110

[lgirdwood]

@plbossart can you attach your FW. Would be nice to diff it's headers against v2.1.
Btw, I assume you are signing with community key.? Do both v2.1 community and Intel keys work for you ?

[lgirdwood]

One more thing, is your rimage submodule up to date ? (and in $PATH)

[plbossart]

git log --oneline
b4886bebb (HEAD -> main, origin/main) module_adapter: add zephyr logging support

cd rimage/
git log --oneline
9d45332 (HEAD) Write firmware file micro version to manifest for cAVS platforms

rimage is not in $PATH. It was not needed before and must be set by scripts.

[plbossart]

compilation log
log.txt

firmware
sof-tgl.ri.gz

[marc-hb]

This reminds me this bug fix: 95d887251ee40397300

[fredoh9]

Interesting, I compared with mine

1. SHA1 => same
2. build log => not so much different
3. fw binaries => a few bytes are different after "--erase_vars" but not sure that is crucial (thanks @marc-hb )

./sof_ri_info.py ~/Downloads/sof-tgl.ri --erase_vars ~/Downloads/sof-tgl.ri_no_vars

[marc-hb]

fw binaries => a few bytes are different after "--erase_vars" but not sure that is crucial (thanks @marc-hb )

I implemented --erase-vars so the image after --erase_vars is 100% identical when using the same toolchain (which you obviously did, otherwise the differences would be much bigger). So I do find these few bytes difference worrying, can you please share the hexdiff?

Also, please share the manifest differences as shown by diff -u <(sof/tools/sof_ri_info.py image1.ri) <(sof/tools/sof_ri_info.py image2.ri). Some randomness is expected because rimage uses a salt (that's why --erase_vars exists), other differences are not.

BTW --erase-vars is used in CI to test every PR catch any __TIMESTAMP__ (in addition to checkpatch)

[fredoh9]

Don't know the root cause, but from the build log, Source content hash is different.

This is mine,

-- GIT_TAG / GIT_LOG_HASH : v2.0-rc1-997-gb4886bebbe49 / b4886bebbe49
-- Source content hash: 1c511e77. Note: by design, source hash is broken by config changes. See thesofproject/sof#3890.

This is Pierre's,

-- GIT_TAG / GIT_LOG_HASH : v2.0-rc1-997-gb4886bebbe49 / b4886bebb
-- Source content hash: 67531a8c. Note: by design, source hash is broken by config changes. See thesofproject/sof#3890.

[plbossart]

@keqiaozhang reported a similar issue worked-around with the "scripts/xtensa-build-all.sh -d tgl" option. In my case the debug option doesn't solve anything.

[marc-hb]

Don't know the root cause, but from the build log, Source content hash is different.

This is the very first difference and the one we must focus on first. All other differences could be impacted by this. The source hash is not affected by any of these:

• The toolchain does not affect the source hash
• The rimage version does not affect the source hash
• The .config does not affect the source hash (it should but it does not, see bug [BUG] logger "ABI" dictionary check ignores configuration and other changes sof#3890)
• The -d option does not affect the source hash.

So there is really, absolutely no reason for @plbossart's source hash to be different. I have the same 1c511e77 source hash as @fredoh9 with any toolchain.

[fredoh9]

@keqiaozhang reported a similar issue worked-around with the "scripts/xtensa-build-all.sh -d tgl" option. In my case the debug option doesn't solve anything.

if -d make difference, this is more serious bug.
Good thing is it worked for both for me, it didn't work for both for Pierre. I like the results, consistency, at least.

[marc-hb]

@plbossart can you please first make sure that git status --ignored is clean (sorry for asking this but we're grasping at straws now), then run the following commands and report which ones don't match

wc build_tgl_?cc/source_hash/*

  1408   1408  57728 build_tgl_gcc/source_hash/tracked_file_hash_list
  1408   1408  61688 build_tgl_gcc/source_hash/tracked_file_list
  2816   2816 119416 total

md5sum build_tgl_?cc/source_hash/*

0c24b8d8a641391d053cba84ec86f20f  build_tgl_gcc/source_hash/tracked_file_hash_list
72028df5e905505089a10e92e4d94781  build_tgl_gcc/source_hash/tracked_file_list

git ls-files src/ scripts/ | md5sum

72028df5e905505089a10e92e4d94781  -

git hash-object src/probe/probe.c

265c6fe9fea227847ba9094acd663e0a421f565a

[fredoh9]

@marc-hb, I built without -d, I have 100% same with yours above