OpenCover code coverage broken dotnet msbuild

[ViktorHofer]

See the warnings. Testing with xunit 2.4.0. cc @pjanotti.
Relates to dotnet/corefx#31260

C:\git\corefx\src\System.Text.RegularExpressions\tests>dotnet msbuild /t:RebuildAndTest /p:Coverage=true
Microsoft (R) Build Engine version 15.8.160-preview+gaf9d27ba72 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  RemoteExecutorConsoleApp -> C:\git\corefx\bin\AnyOS.AnyCPU.Debug\RemoteExecutorConsoleApp\netstandard\RemoteExecutorConsoleApp.exe
  System.Text.RegularExpressions.Tests -> C:\git\corefx\bin\AnyOS.AnyCPU.Debug\System.Text.RegularExpressions.Tests\netcoreapp\System.Text.RegularExpressions.Tests.dll
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning : SymUnmanagedWriterException: Specified method is not supported. [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :    at Microsoft.DiaSymReader.SymUnmanagedWriterFactory.CreateWriter(ISymWriterMetadataProvider metadataProvider, SymUnmanagedWriterCreationOptions options) in /_/src/Microsoft.DiaSymReader/Writer/SymUnmanagedWriterFactory.cs:line 86 [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :    at Microsoft.DiaSymReader.Tools.PdbConverter.ConvertPortableToWindows(PEReader peReader, MetadataReader pdbReader, Stream targetPdbStream, PortablePdbConversionOptions options) in /_/src/Microsoft.DiaSymReader.Converter/PdbConverter.cs:line 169 [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :    at Microsoft.DiaSymReader.Tools.PdbConverter.ConvertPortableToWindows(PEReader peReader, Stream sourcePdbStream, Stream targetPdbStream, PortablePdbConversionOptions options) in /_/src/Microsoft.DiaSymReader.Converter/PdbConverter.cs:line 143 [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :    at Microsoft.DotNet.Build.Tasks.ConvertPortablePdbsToWindowsPdbs.ConvertPortableToWindows(ITaskItem file, PdbConverter converter, PortablePdbConversionOptions parsedConversionOptions) [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :    at Microsoft.DotNet.Build.Tasks.ConvertPortablePdbsToWindowsPdbs.Execute() [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning : NotSupportedException: Specified method is not supported. [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :    at Microsoft.DiaSymReader.SymUnmanagedWriterFactory.CreateWriter(ISymWriterMetadataProvider metadataProvider, SymUnmanagedWriterCreationOptions options) in /_/src/Microsoft.DiaSymReader/Writer/SymUnmanagedWriterFactory.cs:line 81 [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
C:\git\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64/shared/Microsoft.NETCore.App/10.10.10/System.Text.RegularExpressions.dll : warning :  [C:\git\corefx\src\System.Text.RegularExpressions\tests\System.Text.RegularExpressions.Tests.csproj]
  Replacing Portable PDB with Windows Pdb in place!
  Using C:\git\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\ as the test runtime folder.
  Executing in C:\git\corefx\bin\tests\System.Text.RegularExpressions.Tests\netcoreapp-Windows_NT-Debug-x64\
  ----- start  4:38:12.62 ===============  To repro directly: =====================================================
  pushd C:\git\corefx\bin\tests\System.Text.RegularExpressions.Tests\netcoreapp-Windows_NT-Debug-x64\
  call C:\git\corefx\packages/OpenCover\4.6.519\tools\OpenCover.Console.exe -oldStyle -filter:"+[System.Text.RegularExpressions]*" -excludebyfile:"*\Common\src\System\SR.*" -nodefaultfilters -excludebyattribute:*.ExcludeFromCodeCoverage* -skipautoprops -hideskipped:All -threshold:1 -returntargetcode -register:user -target:C:\git\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\\dotnet.exe -output:C:\git\corefx\bin/tests/coverage\System.Text.RegularExpressions.Tests.coverage.xml -targetargs:"xunit.console.dll System.Text.RegularExpressions.Tests.dll  -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonwindowstests  -notrait category=OuterLoop -notrait category=failing  -notrait Benchmark=true"
  popd
  ===========================================================================================================
  Executing: C:\git\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\dotnet.exe
  xUnit.net Console Runner v2.4.0 (64-bit .NET Core 4.6.26720.05)
    Discovering: System.Text.RegularExpressions.Tests
    Discovered:  System.Text.RegularExpressions.Tests
    Starting:    System.Text.RegularExpressions.Tests
    Finished:    System.Text.RegularExpressions.Tests
  === TEST EXECUTION SUMMARY ===
     System.Text.RegularExpressions.Tests  Total: 1498, Errors: 0, Failed: 0, Skipped: 0, Time: 5.963s
  Committing...
  No results, this could be for a number of reasons. The most common reasons are:
      1) missing PDBs for the assemblies that match the filter please review the
      output file and refer to the Usage guide (Usage.rtf) about filters.
      2) the profiler may not be registered correctly, please refer to the Usage
      guide and the -register switch.
  ----- end  4:38:20.71 ----- exit code 0 ----------------------------------------------------------
  Loading report 'C:\git\corefx\bin\tests\coverage\System.Text.RegularExpressions.Tests.coverage.xml'
   Preprocessing report
   Initiating parser for OpenCover
  Initializing report builders for report types: Html, Badges
  Analyzing 0 classes
   Creating summary
  Report generation took 0.4 seconds

C:\git\corefx\src\System.Text.RegularExpressions\tests>
``´

[ViktorHofer]

@pjanotti I want to fix this as fast as possible. I don't yet understand the error but let's talk (IM) later.

[pjanotti]

@ViktorHofer can you try "msbuild" from VS2017 dev command instead of "dotnet msbuild"? My first guess is that is related to where msbuild is pulling the dependencies for the task that converts portable PDBs to Windows PDB.

[ViktorHofer]

You are right, this doesn't happen when using msbuild from the Developer Command Prompt. Any idea how to fix this? We want to recommend to use dotnet msbuild soon...

[pjanotti]

@ViktorHofer I wouldn't block changing to dotnet msbuild on this, I would rather make coverage a special case (i.e.: make it require VS and msbuild). The simplest solution is for OpenCover to drop the requirement for Windows PDBs, but @tmat opened that PR some time ago, but, it was not merged yet - i.e.: do not count on it happening soon.

@tmat what is the advice in this case? Any workaround to have "dotnet msbuild" able to use the task that generates Windows PDBs from portable PDBs?

[ViktorHofer]

Thanks, I contacted Shaun on both Twitter and the PR. Let's see...

[tmat]

I don't know what in CoreFX repo today determines the kind of PDBs that are generated. I am surprised it is dependent on whether VS or dotnet msbuild is used. @weshaggard?

[pjanotti]

Clarification: it generates portable PDBs by default, however, OpenCover still requires Windows PDBs so for coverage runs our build tries to convert from portable PDBs to Windows PDBs. The conversion uses a task in buildtools that in the end requires DiaSymReader. The issue is that when this task uses "VS msbuild" the version of DiaSymReader is able to do the conversion, however, "dotnet msbuild" seems to load a different version of DiaSymReader and fails to do the conversion.

[weshaggard]

I know that @eerhardt went to great lengths to ensure we had a version of diasymreader in the shared framework of the SDK we use to build or projects so perhaps the dotnet you have on your path doesn't have that fix?

@ViktorHofer can you try using the dotnet we have under tools/dotnetcli directory and see if this works with that one?

[eerhardt]

Yes, you need dotnet --version with 2.1.401-preview-009081 or higher. Basically a nightly build of the 2.1.3-servicing shared framework, where 2.1.3 has not shipped yet.

[ViktorHofer]

Wonderful, that fixed it! Looking forward when the SDK update is released.

On another note, over the weekend I looked through some of our dependencies and updated 2-3 projects to netstandard2.0. Mono.Cecil and OpenCover are still using an older version (and also the old project SDK). There is already a PR for Mono.Cecil out: jbevain/cecil#527 but not yet acknowledged by the maintainer. For OpenCover we should decide if we want to stick with it if the project becomes stall. I know that we already integrated coverlet as an alternative option...