[BUG] Cmake runs initially without CONFIG vars set

[cujomalainey]

Describe the bug
When we build SOF for some reason the initial call to cmake does not include any configs. You can see the travis log below that make *defconfig is not called till after the first call to cmake. This is causing problems with refactoring as some of the variables are resolving to null and causing bugs.

Assuming there is a reason for this initial no config call, then we are calling into areas without configs like arch/xtensa which need those configs to run properly, which means we should be returning. Maybe we are just missing a check at the top of arch/xtensa to return if a xtensa config is not set?

To Reproduce
run xtensa-build-all.sh with a message to long your platform config. notice empty line in first run.

Reproduction Rate
How often does the issue happen ? i.e. 1/10 (once in ten attempts), 1/1000 or all the time.
100%

Expected behavior
Cmake should not run without the config

Impact
Blocks refactoring for fuzzer

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • Kernel: N/A
   • SOF: e7a339b
2. Name of the topology file
   • Topology: N/A
3. Name of the platform(s) on which the bug is observed.
   • Platform: N/A

Screenshots or console output

$ ./scripts/docker-run.sh ./scripts/xtensa-build-all.sh -r $PLATFORM
Build in build_tgl_gcc
+ cmake -DTOOLCHAIN=xtensa-cnl-elf -DROOT_DIR=/home/sof/work/sof.git/../xtensa-root/xtensa-cnl-elf -DMEU_OPENSSL= '' -DRIMAGE_PRIVATE_KEY=/home/sof/work/sof.git/keys/otc_private_key_3k.pem ..
-- Preparing Xtensa toolchain
-- The C compiler identification is GNU 9.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /home/sof/work/xtensa-cnl-elf/bin/xtensa-cnl-elf-gcc
-- Linking git pre-commit hook
-- Linking git post-commit hook
-- Found Git: /usr/bin/git (found version "2.17.1") 
-- GIT_TAG / GIT_LOG_HASH : v1.5-rc1-898-ge7a339bd4cc7 / e7a339bd4
-- Source content hash: f770dbfc
-- Generating /home/sof/work/sof.git/build_tgl_gcc/generated/include/version.h
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3.0") 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/sof/work/sof.git/build_tgl_gcc
+ make tgplp_defconfig
Scanning dependencies of target tgplp_defconfig
Applying olddefconfig with tgplp_defconfig
Using existing configuration '.config' as base
Configuration written to '.config'
Built target tgplp_defconfig
-- Preparing Xtensa toolchain
-- GIT_TAG / GIT_LOG_HASH : v1.5-rc1-898-ge7a339bd4cc7 / e7a339bd4
-- Source content hash: f770dbfc
-- Up-to-date /home/sof/work/sof.git/build_tgl_gcc/generated/include/version.h
-- Configuring done
-- Generating done
-- Build files have been written to: /home/sof/work/sof.git/build_tgl_gcc
Scanning dependencies of target overrideconfig
Using existing configuration '.config' as base
Configuration written to '.config'
Built target overrideconfig
-- Preparing Xtensa toolchain
-- GIT_TAG / GIT_LOG_HASH : v1.5-rc1-898-ge7a339bd4cc7 / e7a339bd4
-- Source content hash: f770dbfc
-- Up-to-date /home/sof/work/sof.git/build_tgl_gcc/generated/include/version.h
-- Configuring done
-- Generating done
-- Build files have been written to: /home/sof/work/sof.git/build_tgl_gcc
Scanning dependencies of target genconfig

[lgirdwood]

@jajanusz @marc-hb any inputs here ?

[marc-hb]

I often invoke cmake and make/ninja directly (not the least when... fixing them) and from a (lack of) user friendliness perspective I always found the intermediate and extra defconfig step super annoying. Meta-build systems that build entire system images/distributions are very likely incompatible with such an intermediate step.

Just for the record this extra defconfig step is official and documented:
https://thesofproject.github.io/latest/getting_started/build-guide/build-from-scratch.html#incremental-builds

What's even more annoying is I found no way to clean the build directory without losing my .config! Must be saved and restored every time, really painful.

So before even looking at the details of the specific issues listed by @cujomalainey I would absolutely love if the defconfig step were integrated in the CMake step. This is what Zephyr does BTW: https://docs.zephyrproject.org/2.3.0/guides/kconfig/setting.html#setting-symbols-in-configuration-files

Note this would of course require passing config information at cmake time, either with cmake -Dsomething=aplconfig or by switching to a source directory with some desired .conf files. Zephyr supports the combination of both.

@jajanusz was the current design a conscious choice or just the easiest way for you to implement it at the time?

cc: kconfig master @plbossart , @xiulipan

[cujomalainey]

Ditto to marc. I think that would prevent my issues I am seeing. Right now all I am trying to do is add a linker script to the host build and I tried extracting sof_add_ld_script (name might not be exact, going off memory) from xtensa so I could share it with the host and it broke because a bunch of vars were evaluating to nothing causing arg count failures due to missing configs.

[xiulipan]

@marc-hb I think this issue is not related with kconfig, it is totally a cmake config issue. I think we can include the make tgplp_defconfig in cmake stage.

[marc-hb]

I think this issue is not related with kconfig

The .config file is a kconfig file so it's not clear what you meant here.

I just checked and as expected make foo_defconfig VERBOSE=1 invokes scripts/kconfig/olddefconfig.py

[cujomalainey]

just an fyi, tried adding a return else case to the platform select in arch/xtensa/cmakelist and it doesnt work.

./scripts/xtensa-build-all.sh: line 182: type: xtensa-bxt-elf-gcc: not found
Build in build_apl_gcc
+ cmake -DTOOLCHAIN=xtensa-apl-elf -DROOT_DIR=/home/sof/work/sof.git/../xtensa-root/xtensa-apl-elf -DMEU_OPENSSL= '' '' ..
-- Preparing Xtensa toolchain
-- The C compiler identification is GNU 9.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /home/sof/work/xtensa-apl-elf/bin/xtensa-apl-elf-gcc
-- Found Git: /usr/bin/git (found version "2.17.1") 
-- GIT_TAG / GIT_LOG_HASH : v1.5-rc1-917-gcc5db3e1b476 / cc5db3e1b476
-- Source content hash: dd56d502
-- Generating /home/sof/work/sof.git/build_apl_gcc/generated/include/version.h
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3.0") 
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    MEU_OPENSSL
    ROOT_DIR


-- Build files have been written to: /home/sof/work/sof.git/build_apl_gcc
+ make apollolake_defconfig
Scanning dependencies of target apollolake_defconfig
[100%] Applying olddefconfig with apollolake_defconfig
Using existing configuration '.config' as base
Configuration written to '.config'
[100%] Built target apollolake_defconfig
make: *** No rule to make target 'bin'.  Stop.

[cujomalainey]

@slawblauciak can we bump this to a P1/P2? This is blocking oss-fuzz from covering a large part of the firmware and also I think making integration of future testbench tools harder for @ShriramShastry