Code coverage for System.Private.CoreLib is broken in CI

[stephentoub]

It appears we're no longer able to get code coverage of System.Private.CoreLib, even when using CoreCLROverridePath, even when using set COMPlus_ZapDisable=1.

cc: @mellinoe, @danmosemsft

[stephentoub]

It looks like an issue with PDBs. I was able to work around this by doing build windowsmscorlib release after doing my full coreclr build; presumably something about the crossgen step done in the regular coreclr build is messing with the PDBs in a way that prevents OpenCover from successfully using the PDB, and build windowsmscorlib release skips crossgen.

Unfortunately, while this allows OpenCover to successfully run and index the data from System.Private.CoreLib, I then hit a StackOverflowException in ReportGenerator consuming it 😦

[stephentoub]

I built https://github.com/icsharpcode/NRefactory/ from source, overwrote the binaries in my ReportGenerator package directory (corefx\packages\reportgenerator\2.5.0\tools), and the stack overflow went away. So I'm unblocked for now, but this is not a good state.

[danmoseley]

Seems the latest release of NRefactory is 6.0.0 beta 1 (https://www.nuget.org/packages/ICSharpCode.NRefactory/6.0.0-beta1) ... if you see it work with binaries from that package, we could ask ReportGenerator folk to consider upgrading their dependency.

[stephentoub]

I tried that before I tried compiling from source... failed miserably.

[stephentoub]

Summarizing current workaround, to get code coverage for corelib, one needs to:

1. Build the coreclr repo
2. Build corelib without crossgen (build windowsmscorlib release)
3. Build the corefx repo with CoreCLROverridePath to target the local coreclr build
4. Build the https://github.com/icsharpcode/NRefactory/ repo
5. Overwrite the NRefactory binaries in the local directory containing the unzipped ReportGenerator binaries (e.g. corefx\packages\reportgenerator\2.5.0\tools) with those built from the NRefactory repo
6. Before running tests with /p:Coverage=true, set COMPlus_ZapDisable=1

[danmoseley]

I see ReportGenerator distributes NRefactory within its package -- so we don't need NRefactory to update their package necessarily, we just need ReportGenerator to include an updated one. I see ReportGenerator 3.0 has just come out, but it has the identical NRefactory that 2.5 has. Worth opening an issue on the ReportGenerator repo to ask?

[stephentoub]

we just need ReportGenerator to include an updated one

ReportGenerator just references the NRefactory NuGet package, no? If it actually distributes the binary, then yeah, we should just ask ReportGenerator to include the latest from source.

[danmoseley]

No, it distributes it.

[stephentoub]

danielpalme/ReportGenerator#105

[danmoseley]

I think 3.0.1 needs to go in two places

https://github.com/dotnet/buildtools/blob/master/src/Microsoft.DotNet.Build.Tasks/PackageFiles/CodeCoverage.targets#L9
https://github.com/dotnet/corefx/blob/7f1c480a6a280fbced29606120b507596bafbefc/external/test-runtime/XUnit.Runtime.depproj#L64

[danmoseley]

I am trying to get coverage to work without building CoreCLR myself, instead using the corelib pdb that is now gotten automatically. Note this is a Windows style PDB not a portable PDB. I cannot get Cecil to load this System.Private.Corelib.pdb - debugging Cecil, it seems to think it does not match the dll, because the guid does not match; the guid it gets from the pdb seems right per Symchk, the guid it reads from the dll header is unexpected. Symchk and VS consider the pdb to be a match. My next guess is to try Cecil 0.10, but it is not a drop in replacement: it needs changes in opencover.

[danmoseley]

Howver, I can verify that report generator works fine.

[danmoseley]

It would be interesting to try stepping over the guid check to see whether the corelib pdb works correctly.

[stephentoub]

I am trying to get coverage to work without building CoreCLR myself, instead using the corelib pdb that is now gotten automatically.

What source would that match to / use to generate the rendered code coverage report?