Noise in coverage scores

[nymius]

Describe the issue

Coverage score is affected by non-library code (test code and testenv code), also, branch coverage is not exercised:

Commands & Reports

grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --keep-only '**/crates/**' --ignore '**/tests/**' --ignore '**/examples/**' -o target/coverage/html

grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --keep-only '**/crates/**' --ignore '**/crates/testenv/**' --ignore '**/tests/**' --ignore '**/examples/**' -o target/coverage/html

grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --keep-only '**/crates/**' --excl-start "#\[cfg\(test\)\]" --ignore '**/crates/testenv/**' --ignore '**/tests/**' --ignore '**/examples/**' -o target/coverage/html

cargo +nightly -q llvm-cov --doctests --branch --all --ignore-filename-regex "(example-crates/*|crates/testenv/*)" --all-features --open

¹

Possible solutions

• Include the following flags to the grcov command in the ci coverage job:

  • Add --ignore "**/crates/testenv/** flag: exclude testenv crate from coverage score.
  • Add --excl-start "#\[cfg\(test\)\]" flag: as long as BDK places unit test in module at the bottom of the files, this will exclude all test code from coverage score.
  • Add --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()" flag: ignore boilerplate code that can be confused as sources of branches.
  • Add --excl-line "^\s*\S+\s*(=>)?\s*unreachable!.*" flag: ignore unreachable code (not more than 3 lines in this repository codebase).

• Replace grcov by cargo-llvm-cov

Use case
Improve signal of test coverage score.

Additional context

I was experimenting with coverage tools in the ci last week, and discovered some issues with the local reporting of grcov. At the end I decided with cargo-llvm-cov for bdk-sp because it was the easiest way to avoid some false positives that I discovered with grcov, like misreporting lines under docstrings as not tested or marking test code as part of the overall coverage.

During today's call I mentioned the concern as bdk main repo is using grcov.
We discovered that crates/testenv wasn't being excluded from coveralls reports, and by later exploring them, I discovered other false positives, like test being counted towards the final coverage score.

I also tested against cargo-llvm-cov using nightly toolchain and the score was even lower.

Footnotes

1. There is an unstable flag, --mcdc (MCDC) which supersedes also unstable --branch flag, but as github fails to build with it, I think we can rely on --branch for now. ↩

[notmandatory]

Good research! I agree the cargo-llvm-cov looks like a better tool to use.

[oleonardolima]

Agree! I'm also on board to use cargo-llvm-cov instead (maybe benchmark against cargo-tarpaulin too ?).

Some of these issues (e.g test code being considered) were faced in fedimint in the past, but it wasn't able to migrate to it due to Nix issues, though we don't have this issue here :)

[nymius]

maybe benchmark against cargo-tarpaulin too ?

Good call! I will. Some concerns I have is cargo-tarpaulin has been using ptrace calls to map coverage, and just recently added support for llvm coverage.
ptrace provides information extracted at runtime, while llvm cov extracts information from static instrumentation of the code at the instruction level.
My guess is static analysis is more precise overall, but will have to look at it in exercise to get a real perspective.

[nymius]

Any vote against ignoring chain/src/example_utils.rs too? Seems to be used just to produce smaller doctests, and it is support code only.

[nymius]

cargo tarpaulin using ptrace engine (very slow, around 6 minutes each run):

cargo tarpaulin --workspace --exclude-files "*test*" --exclude-files "*example*" --exclude-files "*bench*" --out html

[nymius]

cargo tarpaulin using llvm engine (still felt slower than cargo llvm-cov, around 4 minutes each run):

cargo tarpaulin --engine llvm --workspace --exclude-files "*test*" --exclude-files "*example*" --exclude-files "*bench*" --out html

[nymius]

Repeated execution of cargo llvm-cov also ignoring example* files and extracting the time to run. Took 1 and a half minute to run on average:

 cargo +nightly llvm-cov -q --doctests --branch --all --ignore-filename-regex "(example*|crates/testenv/*)" --all-features --open