[Arm64] Enable arm64 cli builds

[sdmaclea]

Adds arm64 builds

Depends on #8901 merging

@Petermarcu @johnbeisner @peterhuene @livarcocc

[sdmaclea]

cc @janvorli

[sdmaclea]

With a cherry-pick of #8881 to pickup arm64 runtime
Build with default configuration (copied from netci.groovy command) succeeded

./build.sh --linux-portable --skip-prereqs --architecture arm64 --targets Default /p:CLIBUILD_SKIP_TESTS=true

End of build looked like.

       GenerateNugetPackages:
         > [DotNetTool] pack    /home/sdmaclea/git/cli/src/Microsoft.DotNet.Cli.Utils/Microsoft.DotNet.Cli.Utils.csproj --configuration Debug --no-build --output /home/sdmaclea/git/cli/bin/2/linux-arm64/packages --version-suffix preview3-008416   
         > /home/sdmaclea/git/cli/.dotnet_stage0/arm64/dotnet pack    /home/sdmaclea/git/cli/src/Microsoft.DotNet.Cli.Utils/Microsoft.DotNet.Cli.Utils.csproj --configuration Debug --no-build --output /home/sdmaclea/git/cli/bin/2/linux-arm64/packages --version-suffix preview3-008416    
         Microsoft (R) Build Engine version 15.6.12.27473 for .NET Core
         Copyright (C) Microsoft Corporation. All rights reserved.
         
           Successfully created package '/home/sdmaclea/git/cli/bin/2/linux-arm64/packages/Microsoft.DotNet.Cli.Utils.2.1.300-preview3-008416.nupkg'.
         > [DotNetTool] pack    /home/sdmaclea/git/cli/src/Microsoft.DotNet.MSBuildSdkResolver/Microsoft.DotNet.MSBuildSdkResolver.csproj --configuration Debug --no-build --output /home/sdmaclea/git/cli/bin/2/linux-arm64/packages --version-suffix preview3-008416  --include-symbols 
         > /home/sdmaclea/git/cli/.dotnet_stage0/arm64/dotnet pack    /home/sdmaclea/git/cli/src/Microsoft.DotNet.MSBuildSdkResolver/Microsoft.DotNet.MSBuildSdkResolver.csproj --configuration Debug --no-build --output /home/sdmaclea/git/cli/bin/2/linux-arm64/packages --version-suffix preview3-008416  --include-symbols  
       RestoreSymbolsProject:
         > [DotNetTool] restore    "/home/sdmaclea/git/cli/build/package/Symbols.csproj"      
         > /home/sdmaclea/git/cli/.dotnet_stage0/arm64/dotnet restore    "/home/sdmaclea/git/cli/build/package/Symbols.csproj"       
           Restoring packages for /home/sdmaclea/git/cli/build/package/Symbols.csproj...
           Generating MSBuild file /home/sdmaclea/git/cli/bin/2/linux-arm64/obj/Symbols/Symbols.csproj.nuget.g.props.
           Generating MSBuild file /home/sdmaclea/git/cli/bin/2/linux-arm64/obj/Symbols/Symbols.csproj.nuget.g.targets.
           Restore completed in 96.27 ms for /home/sdmaclea/git/cli/build/package/Symbols.csproj.
       GenerateSymbolsNuget:
         > [DotNetTool] pack    "/home/sdmaclea/git/cli/build/package/Symbols.csproj"  --no-build --output /home/sdmaclea/git/cli/bin/2/linux-arm64/packages    
         > /home/sdmaclea/git/cli/.dotnet_stage0/arm64/dotnet pack    "/home/sdmaclea/git/cli/build/package/Symbols.csproj"  --no-build --output /home/sdmaclea/git/cli/bin/2/linux-arm64/packages     
         Microsoft (R) Build Engine version 15.6.12.27473 for .NET Core
         Copyright (C) Microsoft Corporation. All rights reserved.
         
           Successfully created package '/home/sdmaclea/git/cli/bin/2/linux-arm64/packages/runtime.linux-arm64.Microsoft.DotNet.SDK.2.1.300-preview3-008416.nupkg'.
         Moving file from "/home/sdmaclea/git/cli/bin/2/linux-arm64/packages/runtime.linux-arm64.Microsoft.DotNet.SDK.2.1.300-preview3-008416.nupkg" to "/home/sdmaclea/git/cli/bin/2/linux-arm64/packages/runtime.linux-arm64.Microsoft.DotNet.SDK.2.1.300-preview3-008416.symbols.nupkg".
       GenerateArchives:
         > /bin/tar -czf /home/sdmaclea/git/cli/bin/2/linux-arm64/packages/dotnet-sdk-internal-2.1.300-preview3-008416-linux-arm64.tar.gz --directory /home/sdmaclea/git/cli/bin/2/linux-arm64/intermediate/layouts/dotnet-sdk-internal/   "." 
       GenerateArchives:
         > /bin/tar -czf /home/sdmaclea/git/cli/bin/2/linux-arm64/packages/dotnet-sdk-2.1.300-preview3-008416-linux-arm64.tar.gz --directory /home/sdmaclea/git/cli/bin/2/linux-arm64/intermediate/layouts/dotnet-sdk/   --exclude *.resources.dll "." 
       GenerateArchives:
         > /bin/tar -czf /home/sdmaclea/git/cli/bin/2/linux-arm64/packages/dotnet-sdk-langpack-2.1.300-preview3-008416-linux-arm64.tar.gz --directory /home/sdmaclea/git/cli/bin/2/linux-arm64/intermediate/layouts/dotnet-sdk-langpack/   "." 
     1>Done Building Project "/home/sdmaclea/git/cli/build.proj" (default targets).

Build succeeded.

       "/home/sdmaclea/git/cli/build.proj" (default target) (1) ->
       (RestoreSrcPackages target) -> 
         /home/sdmaclea/git/cli/build/MSBuildExtensions.targets(2,3): warning MSB4011: "/home/sdmaclea/git/cli/build/BundledDotnetTools.props" cannot be imported again. It was already imported at "/home/sdmaclea/git/cli/Directory.Build.props (60,5)". This is most likely a build authoring error. This subsequent import will be ignored. [/home/sdmaclea/git/cli/src/redist/redist.csproj] [/home/sdmaclea/git/cli/build.proj]
         /home/sdmaclea/git/cli/build/MSBuildExtensions.targets(2,3): warning MSB4011: "/home/sdmaclea/git/cli/build/BundledDotnetTools.props" cannot be imported again. It was already imported at "/home/sdmaclea/git/cli/Directory.Build.props (60,5)". This is most likely a build authoring error. This subsequent import will be ignored. [/home/sdmaclea/git/cli/src/redist/redist.csproj] [/home/sdmaclea/git/cli/build.proj]
         /home/sdmaclea/git/cli/build/MSBuildExtensions.targets(2,3): warning MSB4011: "/home/sdmaclea/git/cli/build/BundledDotnetTools.props" cannot be imported again. It was already imported at "/home/sdmaclea/git/cli/Directory.Build.props (60,5)". This is most likely a build authoring error. This subsequent import will be ignored. [/home/sdmaclea/git/cli/src/redist/redist.csproj] [/home/sdmaclea/git/cli/build.proj]


         /home/sdmaclea/git/cli/build/MSBuildExtensions.targets(2,3): warning MSB4011: "/home/sdmaclea/git/cli/build/BundledDotnetTools.props" cannot be imported again. It was already imported at "/home/sdmaclea/git/cli/Directory.Build.props (60,5)". This is most likely a build authoring error. This subsequent import will be ignored. [/home/sdmaclea/git/cli/src/redist/redist.csproj]

    4 Warning(s)
    0 Error(s)

[sdmaclea]

This is drafted from the arm changes earlier this month. Looks like this enable CI builds, but I am not certain whether it enables official builds to myget

[livarcocc]

@johnbeisner to take a look. We will also need to setup up arm64 VSO build for builds. As for arm32, this is an unofficial build, as there are not tests running here.

@Petermarcu this is along the same lines for the runtime as well?

[sdmaclea]

unofficial build

arm64 is not quite stable enough to be an official release yet. CoreCLR tests are passing, but there are several CoreFX tests failing. core-setup and cli tests have not yet been run. Initial experience with trying the bootstrapped version of the cli has been unsatisfactory.

We will make more progress as we approach ZBB and post ZBB. I suspect arm64 will not be passing all its tests in time for the initial; official 2.1 release. I would hope we would be allowed to continue to work on arm64 issues and include in one of the servicing 2.1 releases. @dotnet/arm64-contrib

[Petermarcu]

@livarcocc for the runtime we're executing tests but if you need to do the same as you did for arm32 for now, that will be a fine place to start.

[sdmaclea]

@Petermarcu I didn't think I enabled tests for arm64 in core-setup or corefx CI. If they are running in either that would be good news.

[Petermarcu]

You are right, I don't see them running. I would expect that will come online shortly though, right?

[sdmaclea]

@Petermarcu I didn't try to enable the arm64 tests because I am not aware of the CI arm64 capacity or infrastructure. I think this is something it would be difficult for me to do without significant MS help.

@jashook has historically been the person to do something like this in the CoreCLR world. There are a few linux-arm64 machines used in CoreCLR CI. Capacity is limited so they are not run on most PR's.

It would seem ideal to run the tests periodically or at least for daily builds.

The second factor to consider is the passing rate. CoreFX tests are not all passing. We just got these running on linux-arm64 in the last week or so. Of the 230 tests, 206 are passing, 12 have a small set of test cases failing, 8 asserts, 2 segmentation faults, and 2 timeouts (presumably hangs). If we run tests, we need to publish the failures, but not break the build. Many of the failures are due to a missing JumpStub memory management feature which was enabled for amd64 12/2017, but not for arm64. I am working on this today. https://github.com/dotnet/coreclr/issues/17173

I have not attempted to run the core-setup or cli tests yet. Although I plan to in the next few weeks.

[sdmaclea]

@eerhardt Now that #8901 has merged this can be tested & merged too. "test ci please"

[livarcocc]

@dotnet-bot Test CentOS7.1 x64 Debug Build

[eerhardt]

@dotnet-bot test ci please

[eerhardt]

Leg is kicked off:

https://ci.dot.net/job/dotnet_cli/job/master/job/GenPRTest/job/debug_linux_arm64_prtest/

Again, waiting for a machine to run on. It is job #1, so the link to the job will be:
https://ci.dot.net/job/dotnet_cli/job/master/job/GenPRTest/job/debug_linux_arm64_prtest/1/

[eerhardt]

The Jenkins run failed due to

12:13:45          Downloading 'https://dotnetcli.azureedge.net/dotnet/Runtime/2.1.0-preview2-26314-02/dotnet-runtime-2.1.0-preview2-26314-02-linux-arm64.tar.gz' to 

That file not existing. We will need an official build of core-setup to run first which produces this file.

EDIT: actually, the version it is looking for is the 3 weeks ago version. @sdmaclea - can you merge master into your branch?

[sdmaclea]

Rebased to pickup master tip

[eerhardt]

test Linux arm64 Debug Build

[eerhardt]

The linux arm64 leg already passed.

@livarcocc @sdmaclea @Petermarcu - What do you think about enabling crossgen for linux-arm64? It should be pretty simple (says the person who doesn't have to do it)

Modify this line:

cli/src/redist/redist.csproj

Line 221 in 2a63f9e

Condition=" '$(DISABLE_CROSSGEN)' == '' And !$(Architecture.StartsWith('arm')) "

And then adjust the CrossgenPath as necessary to get the right crossgen tools/x64_arm64

cli/build/CrossGen.props

Line 4 in 661f004

<CrossgenPath>$(NuGetPackagesDir)/$(RuntimeNETCoreAppPackageName)/$(MicrosoftNETCoreAppPackageVersion)/tools/crossgen$(ExeExtension)</CrossgenPath>

[sdmaclea]

What do you think about enabling crossgen for linux-arm64?

I took a look at doing this, but the x64_arm64/crossgen is not currently published by the arm64 runtime, and it must be obtained by getting it from the coreclr package

transport.runtime.linux-arm64.microsoft.netcore.runtime.coreclr/$(version)/tools/x64_arm64/crossgen

I looked at adding a package reference, but build/DependencyVersions.props doesn't have the relevant coreclr version info. It is possible something could be hacked up, but I think someone needs to engineer this.

I'll open an issue. #8998

[sdmaclea]

I think this can be reviewed, approved, and merged.

[eerhardt]

I took a look at doing this, but the x64_arm64/crossgen is not currently published by the arm64 runtime, and it must be obtained by getting it from the coreclr package

We should be re-packaging x64_arm64/crossgen in runtime.linux-arm64.Microsoft.NETCore.App nupkg. We put the normal crossgen in there, and we should be doing the same for everything in the tools directory. This chunk should be doing it:

https://github.com/dotnet/core-setup/blob/e132169c6bd78f58a488cbd0de412433296f642b/src/pkg/projects/Microsoft.NETCore.App/src/Microsoft.NETCore.App.depproj#L149-L152

UPDATE: that chunk is only grabbing the files directly in the tools folder. It isn't recursively copying directories. I've opened https://github.com/dotnet/core-setup/issues/3987 for this.

[sdmaclea]

Can this be merged?