Skip to content

Conversation

@upshaw-alex
Copy link
Contributor

This PR:

  • Adds code coverage instrumentation to CMake builds with either Clang or GCC.
  • Extends run_all_tests.sh to automatically generate .html code coverage reports from runs with code coverage instrumentation present.
  • Updates CI workflows that now generate code coverage reports to save them as artifacts.
  • Updates README.md and index.md to mention these code coverage capabilities.
  • Updates Linux/Mac provision scripts to install the required dependencies for coverage.
    • I omitted updating the Windows provision script from this PR, since I wasn't sure what package manager we'd get the dependencies from on Windows.

You can see examples of the generated reports from the checks ran for this PR.

This PR also includes unrelated changes to:

@upshaw-alex upshaw-alex self-assigned this Dec 3, 2025
# libstdc++-10-dev was added for the sake of clang. see: https://stackoverflow.com/q/26333823/10278
sudo apt-get --assume-yes install \
build-essential \
clang \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've assumed (without verifying) that using clang, gcovr, and llvm isn't a concern licensing-wise since they're used as tools, not ingredients. Please let me know if there is any particular due diligence I should do to add these dependencies.

--overwrite python@3.12 \
clang-format@21 \
fontconfig \
gcovr \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added coverage dependencies to provision_mac.sh, and I don't think I've done anything more platform-dependent than existing code in CMakeLists.txt and run_all_tests.sh, but coverage on Mac is untested since the only Mac build in CI uses qmake. Feel free to let me know any thoughts on if I should replace or extend .github/workflows/mac2023.yml with a CMake build to run Mac coverage in CI.

if (PROJECT_IS_TOP_LEVEL)
set(MYAPP_CLANG_COVERAGE_FLAGS
" -fprofile-instr-generate \
-fcoverage-mapping \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case it's of interest, I'll mention I also tried enabling MC/DC coverage for Clang (adding compile flag -fcoverage-mcdc and giving -show-mcdc to llvm-cov), but oddly that caused the Android build to fail with clang++: error: unknown argument: '-fcoverage-mcdc'. I've noticed the Android build at one point uses a different version of Clang, and with different flags than the main Linux build. I'm afraid I may not immediately be able to prioritize troubleshooting this specific issue further, but where I left it was wondering how MYAPP_CLANG_COVERAGE_FLAGS are even in effect in the Android build, given it doesn't use MYAPP_TEMPLATE_COMPILERCHOICE_CLANG: 1.

@pestophagous
Copy link
Member

While testing this locally, I noticed an intriguing line of output from llvm-cov, and then I searched in the github CI action raw logs and I see that it happens in the cloud CI as well:

warning: 128 functions have mismatched data

For now, I do not consider this a blocker for merging this PR. (I am still reviewing the PR actively at this moment, so while nothing is blocked, nothing is yet ready-to-merge either.)

Anyone interested should probably follow these 2 llvm tickets:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants