[BUG] compare-linux-win CI build check fails due to zephyr_version.h difference

[kv2019i]

Status update:

Starting with daily build 795, https://github.com/thesofproject/sof/actions/runs/12940934767
the Linux + mnft (manifest) builds started to lose git tags in the git describe output that goes into their BUILD_VERSION. For instance v4.0.0-2813-g42701fdb2729 became 42701fdb2729. The Windows builds kept it and the Linux "zmain" (Zephyr main branch) kept it too.
This correlate with a Ubuntu VM upgrade, see below.

---

Describe the bug
Starting today 27th Jan 2025, multiple pull-requests are showing a failure in build reproducibility between Linux and Windows builds with
error:

https://github.com/thesofproject/sof/actions/runs/12947318306/job/36114439673?pr=9794

 Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/stripped-zephyr.elf and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/stripped-zephyr.elf differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr.lst and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr.lst differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr_version.h and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr_version.h differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/stripped-zephyr.elf and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/stripped-zephyr.elf differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr.lst and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr.lst differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr_version.h and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr_version.h differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8ulp/stripped-zephyr.elf and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8ulp/stripped-zephyr.elf differ

This is shown in multiple open PRs:

• Scripts: Fix path in script sof-testbench-build-profile.sh #9794 , ipc4: mixin: Add "mix with gain" HiFi5 impl #9795 , rimage: manifest: uuid: llext: Get loadable module UUID from uuid-registry and unify module manifest uuid type #9793
• ... so does not seem to be related to the pull-requests.

To add to the mystery, I can't see the failure in any of the pull requests failed last week.

To Reproduce
Submit a pull request to SOF repository.

Reproduction Rate
100%

Expected behavior
Linux and Windows host builds should create identical binaries.

Impact
Failure seen on every CI request, can mask other errors.

[kv2019i]

@thesofproject/nxp Rings any bells to you? I checked multiple PRs merged last week, but I couldn't find a PR where this failure would have been seen (already pre merge). I could have missed some.

[dbaluta]

@LaurentiuM1234 @marc-hb didn't we have this issue couple of months ago? Will need to check on that.

[dbaluta]

@kv2019i looks like this is not only for imx builds.

 Files linux-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/stripped-zephyr.elf and windows-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/stripped-zephyr.elf differ
Files linux-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr.lst and windows-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr.lst differ
Files linux-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr_version.h and windows-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr_version.h differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/boot.mod and windows-build -d mtl/build-sof-staging/sof-info/mtl/boot.mod differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/stripped-main.elf and windows-build -d mtl/build-sof-staging/sof-info/mtl/stripped-main.elf differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/stripped-zephyr.elf and windows-build -d mtl/build-sof-staging/sof-info/mtl/stripped-zephyr.elf differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/zephyr.lst and windows-build -d mtl/build-sof-staging/sof-info/mtl/zephyr.lst differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h and windows-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h differ

[dbaluta]

@kv2019i Relevant issue in the past #9034 was solved by opting out on adding the source code on the assembly listing file.

[LaurentiuM1234]

FYI test also failed when trying to add build job for imx95 #9546 a while ago. If I recall correctly, one of the compilers would add some comments on one of the OSes but not the other. Superficially looking at the 2 .lst's from #9794 it would seem like a dif. issue though.

[marc-hb]

Files linux-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h and windows-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h differ

This is the easiest bit to focus on first.

You can downloads artefacts from every PR page, please compare these.

Relevant issue in the past #9034 was solved by opting out on adding the source code on the assembly listing file.

zephyr_version.h looks like a very different problem this time.

[marc-hb]

You can downloads artefacts from every PR page,

Actually, daily tests are more convenient for this: because zero PR interference.

Started failing in daily 795 , 4 days ago: https://github.com/thesofproject/sof/actions/runs/12940934767

please compare these.

Don't wait because artefacts expire after some time.

[marc-hb]

There's been a progressive rollout of a new Windows image ( Version: 20250113.1.0 -> 20250120.2.0) around the same time but it does not correlate with the failure. EDIT: it's unrelated. The problem is on the Linux side.

For instance, the IMX build got the newer Windows image in 794 but 794 was still all green:
https://github.com/thesofproject/sof/actions/runs/12898787138/job/35966473504

As you can tell from the version numbers, minor upgrades like these happen all the time.

The first step is really to download and compare zephyr_version.h.