MSBuild amd64 fails when linking ucrtd.lib

[mharmer]

Steps to reproduce

Attached reproduce project here: msbuild_amd64_bug.zip

Directory contents:

- CompilerIdC.vcxproj
- CMakeCCompilerId.c

Command line:

"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Current\Bin\amd64\MSBuild.exe" CompilerIdC.vcxproj /p:Configuration=Debug /p:Platform=x64

Expected behavior

Build succeeds without warnings or errors.

Actual behavior

Build fails with 1 error:

Link:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.20.27404\bin\HostX64\x64\link.exe /ERRORREPORT:QUEUE /OUT:".\CompilerIdC.exe" /INCREMENTAL:NO /NO
  LOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /MANIFESTUAC:"leve
  l='asInvoker' uiAccess='false'" /manifest:embed /PDB:".\CompilerIdC.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:".\CompilerIdC.lib" /MACHINE:X64 Debug\CMa
  keCCompilerId.obj
LINK : fatal error LNK1104: cannot open file 'ucrtd.lib' [C:\dev\msbuild_amd64_bug\CompilerIdC.vcxproj]
Done Building Project "C:\dev\msbuild_amd64_bug\CompilerIdC.vcxproj" (default targets) -- FAILED.


Build FAILED.

"C:\dev\msbuild_amd64_bug\CompilerIdC.vcxproj" (default target) (1) ->
(Link target) ->
  LINK : fatal error LNK1104: cannot open file 'ucrtd.lib' [C:\dev\msbuild_amd64_bug\CompilerIdC.vcxproj]

    0 Warning(s)
    1 Error(s)

Environment data

msbuild /version output:

Microsoft (R) Build Engine version 16.0.443+g5775d0d6bb for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

16.0.443.30039

OS info: Version 1809, OS Build: 17763.316

Additional information

When building with the 32-bit version of MSBuild the build succeeds, this is the commandline used:

"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Current\Bin\MSBuild.exe" CompilerIdC.vcxproj /p:Configuration=Debug /p:Platform=x64

I think this is an issue with the path construction for some of the UniversalCRT properties to the Windows SDK folder:

1. For the issue seen under x64, the following properties are set (presumably) incorrectly as such:

UCRTContentRoot = C:\Program Files\Windows Kits\10\
UniversalCRT_IncludePath = C:\Program Files\Windows Kits\10\Include\10.0.17763.0\ucrt;
UniversalCRT_LibraryPath_arm = C:\Program Files\Windows Kits\10\lib\10.0.17763.0\ucrt\arm;
UniversalCRT_LibraryPath_arm64 = C:\Program Files\Windows Kits\10\lib\10.0.17763.0\ucrt\arm64;
UniversalCRT_LibraryPath_x64 = C:\Program Files\Windows Kits\10\lib\10.0.17763.0\ucrt\x64;
UniversalCRT_LibraryPath_x86 = C:\Program Files\Windows Kits\10\lib\10.0.17763.0\ucrt\x86;

2. In constrast, the x86 shows the following values for the same properties:

UCRTContentRoot = C:\Program Files (x86)\Windows Kits\10\
UniversalCRT_IncludePath = C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\ucrt;
UniversalCRT_LibraryPath_arm = C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\arm;
UniversalCRT_LibraryPath_arm64 = C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\arm64;
UniversalCRT_LibraryPath_x64 = C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64;
UniversalCRT_LibraryPath_x86 = C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x86;

[livarcocc]

Can you see what link.exe command is being used in the successful case?

[mharmer]

@livarcocc The link command appears to be the exact same between the unsuccessful and successful cases:

Working:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.20.27404\bin\HostX64\x64\link.exe /ERRORREPORT:QUEUE /OUT:".\CompilerIdC.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:".\CompilerIdC.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:".\CompilerIdC.lib" /MACHINE:X64 Debug\CMakeCCompilerId.obj

Not working:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.20.27404\bin\HostX64\x64\link.exe /ERRORREPORT:QUEUE /OUT:".\CompilerIdC.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:".\CompilerIdC.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:".\CompilerIdC.lib" /MACHINE:X64 Debug\CMakeCCompilerId.obj

To look further into this, I took a look at the link.read.1.tlog, the working output listed the path discovered for UCRTD.LIB as:
C:\PROGRAM FILES (X86)\WINDOWS KITS\10\LIB\10.0.17763.0\UCRT\X64\UCRTD.LIB

Whereas the unsuccessful run listed the path as:
C:\PROGRAM FILES\WINDOWS KITS\10\LIB\10.0.17763.0\UCRT\X64\UCRTD.LIB

(Note: the link.read.1.tlog initially doesn't get populated when link.exe fails, so to workaround this I manually constructed the folder structure (C:\PROGRAM FILES\WINDOWS KITS\10\LIB\10.0.17763.0\UCRT\X64) and dropped in the ucrtd.lib from the correct location to get the log file to show it - at which point link.exe also succeeded).

I suspect it must be an environment variable that is triggering this since the command line arguments match, possibly assigned in LIB?

[cbaxter]

Trying to work out why, but depending on the environment, I am seeing different value for the following registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Kits\Installed Roots@KitsRoot10

Since uCRT.props pulls from this location, the working environment has the following:

HKLM:SOFTWARE\Microsoft\Windows Kits\Installed Roots\KitsRoot10
C:\Program Files (x86)\Windows Kits\10\

Where the non-working environment has the following:

HKLM:SOFTWARE\Microsoft\Windows Kits\Installed Roots\KitsRoot10
C:\Program Files\Windows Kits\10\

Trying to work out why there is a difference and what feature selection(s) during install resulted in the unexpected value... so far the only notable difference is working environments originally installed the SDKs with VS2017 setup and the new environment installed the SDKs with VS2019 only...

Regardless, don't know why yet, so will continue to investigate.

[cbaxter]

Sadly, I have opted to just restore the original registry key for now...

If it helps, the only folder under the Program Files location is C:\Program Files\Microsoft SDKs\Azure (specifically .NET SDK and Emulator); so I am guessing one of the VS2019 Azure features (i.e., perhaps Azure Compute Emulator, Azure Storage Emulator, or related) overrode the registry key during install... (pure speculation tho).

[bartlomiejcieszkowski]

I had similiar issue, with VS2019 and cmake 3.12~

updated to cmake 3.15.2 and the issue is gone

The previous cmake version had failed with identifying CMAKE_C_COMPILER at the begginning

Do you still have repro when switching to latest cmake?

[bradking]

The reason it works in CMake 3.15 is because CMake reverted the change to use a 64-bit MSBuild due to this issue. It "works" by using the 32-bit MSBuild even on a 64-bit host. IIUC the reproduction instructions in the description of this issue do not require running CMake.

[bartlomiejcieszkowski]

i got here originally from the issue that was filed in cmake gitlab:
https://gitlab.kitware.com/cmake/cmake/issues/19037

my bad

checked on latest vs2019 and it seems that there is no repro:

Z:\Bart\Downloads\msbuild_amd64_bug>"Z:\Microsoft\VisualStudio2019\MSBuild\Current\Bin\amd64\MSBuild.exe" CompilerIdC.vcxproj /p:Configuration=Debug /p:Platform=x64
Microsoft (R) Build Engine version 16.2.37902+b5aaefc9f for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 8/19/2019 4:27:57 PM.
Project "Z:\Bart\Downloads\msbuild_amd64_bug\CompilerIdC.vcxproj" on node 1 (default targets).
PrepareForBuild:
  Creating directory "Debug\".
  Creating directory "Debug\CompilerIdC.tlog\".
InitializeBuildStatus:
  Creating "Debug\CompilerIdC.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:
  Z:\Microsoft\VisualStudio2019\VC\Tools\MSVC\14.22.27905\bin\HostX64\x64\CL.exe /c /nologo /W0 /WX- /diagnostics:column /Od /D _MBCS /Gm- /EHs
  c /RTC1 /MDd /GS /fp:precise /Qspectre /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"Debug\\" /Fd"Debug\vc142.pdb" /Gd /TC /FC /errorReport:queue C
  MakeCCompilerId.c
  CMakeCCompilerId.c
Link:
  Z:\Microsoft\VisualStudio2019\VC\Tools\MSVC\14.22.27905\bin\HostX64\x64\link.exe /ERRORREPORT:QUEUE /OUT:".\CompilerIdC.exe" /INCREMENTAL:NO
  /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbcc
  p32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:".\CompilerIdC.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYN
  AMICBASE /NXCOMPAT /IMPLIB:".\CompilerIdC.lib" /MACHINE:X64 Debug\CMakeCCompilerId.obj
  CompilerIdC.vcxproj -> Z:\Bart\Downloads\msbuild_amd64_bug\.\CompilerIdC.exe
PostBuildEvent:
  for %%i in (cl.exe) do @echo CMAKE_C_COMPILER=%%~$PATH:i
  :VCEnd
  CMAKE_C_COMPILER=Z:\Microsoft\VisualStudio2019\VC\Tools\MSVC\14.22.27905\bin\Hostx64\x64\cl.exe
FinalizeBuildStatus:
  Deleting file "Debug\CompilerIdC.tlog\unsuccessfulbuild".
  Touching "Debug\CompilerIdC.tlog\CompilerIdC.lastbuildstate".
Done Building Project "Z:\Bart\Downloads\msbuild_amd64_bug\CompilerIdC.vcxproj" (default targets).


Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.29

[jmwolfe]

Sadly, I have opted to just restore the original registry key for now...

If it helps, the only folder under the Program Files location is C:\Program Files\Microsoft SDKs\Azure (specifically .NET SDK and Emulator); so I am guessing one of the VS2019 Azure features (i.e., perhaps Azure Compute Emulator, Azure Storage Emulator, or related) overrode the registry key during install... (pure speculation tho).

I am having this problem as well. Compiles fine in Visual Studio, but in TeamCity, using the .NET Runner msbuild command, linker cannot find ucrt.lib.

I dug around and I found this registry key:

HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows Kits\Installed Roots\KitsRoot10

which IS set to C:\Program Files (x86)\Windows Kits\10\

Which means the registry key discussed above wasn't trounced by some VS component.

This traces back to C:\Program Files (x86)\Windows Kits\10\DesignTime\CommonConfiguration\Neutral\uCRT.props as was mentioned.. where this code runs:

<UCRTContentRoot Condition="'$(UCRTContentRoot)' == ''">$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Kits\Installed Roots@KitsRoot10)</UCRTContentRoot>
<UCRTContentRoot Condition="'$(UCRTContentRoot)' == ''">$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows Kits\Installed Roots@KitsRoot10)</UCRTContentRoot>

it appears that UCRTContentRoot is not set at all, so it takes the first registry key, which of course points to C:\Program Files... .

Not sure why there are no files installed by the SDK into C:\Program Files\Windows Kits\10.. but so far a random check of our servers shows none have this directory. The SDK shouldn't really set a registry key if it uses a path that doesn't exist.

Anyway, my workaround was to set UCRTContentRoot to the Program Files (x86) path in our CI runner and now it all Just works. I have opened a TeamCity issue to look into it; the runner should have set this correctly.

For anyone not using the TeamCity .NET runner, check to make sure you run vcvarsall.bat prior to running msbuild (or run from the Visual Studio developer command line). That may resolve your issue.

[bilbothebaggins]

I had a similar problem:
I found this from 25th Sep 2020 when searching for UCRTContentRoot: https://developercommunity.visualstudio.com/t/ucrt-doesnt-work-in-x64-msbuild/1184283

I've spoken to the Windows SDK team about this. In general, kit installers are not supposed to set 'HKLM\Software\Microsoft\Windows Kits\Installed Roots@KitsRoot10' to C:\Program Files\Windows Kits\10, it is always supposed to point to C:\Program Files (x86)\Windows Kits\10. However, there are Kits out there that make this mistake, and the registry key is never updated if it already exists prior to any kit installation. I believe whichever windows kit you've installed on that system first had this issue.

That said, these issues will never go away entirely since there will always be kits and machines floating around with this issue. I've updated ucrt.props to be more defensive about this by checking the Wow6432Node version first (which has not had this issue historically), and only if that isn't present to fall back to the usual registry key.

This fix will be present in the next released Windows 10 SDK. In the meantime, I recommend either deleting that reg key and reinstalling the Windows 10 SDK, or simply directly modifying HKLM\Software\Microsoft\Windows Kits\Installed Roots@KitsRoot10 to point to C:\Program Files (x86)\Windows Kits\10 (the same effects of the deleting the reg key and reinstalling, but less error prone).

Hope this helps!

Steve Wishnousky
Senior Software Engineer - Visual C++ Libraries
stwish@microsoft.com

Sep 25, 2020