Enabling Linux ARM32 for .NET Core

[hqueue]

Host: Ubuntu 14.04 x64
Target: Ubuntu 14.04 or(and) 16.04 ARM, Ubuntu 14.04 ARM Softfp and Tizen

To enable core-setup for Linux ARM32, we are going to follow steps suggested in PR dotnet/core-setup#712.

ARM hardfp

Common

• stable default clang version for cross compilation (System.Xml.Serialization.XmlSerializer serializes 0x00 but crashes on deserialization #1411)
• Use same logic to prepare rootfs across dotnet (Improve JIT of static variables which value is already known #1432)

CoreCLR

• Update CoreCLR to generate nuget packages of CoreCLR (PR First step to create nuget packages for ARM32/Linux coreclr#8421)
• Remaining CoreCLR ARM32 issues tracked by issue dotnet/coreclr#8549. Still on going

CoreFX

• Add RID for Linux/ARM (Add RID for Linux/ARM32 (ubuntu) corefx#14161)
• Add Xenial (Ubuntu 16.04) support for build-rootfs.sh (issue https://github.com/dotnet/corefx/issues/14538)
• Make clang3.6 default toolset for cross arm32 build (issue https://github.com/dotnet/corefx/issues/15511)
• Enable CI support for Linux Arm32 (https://github.com/dotnet/coreclr/issues/9273, [ARM/CI] ARM CI script for arm and armel corefx#15900)
• Update CoreFX to generate nuget packages of CoreFX (issue dotnet/corefx#14298)
• Enable CoreFX Pipeline build support for Linux for Arm32 and publish packages.

core-setup
For core-setup, we will follow steps below.

• Add support for Linux Arm32 RID (PR Add new rids for Linux ARM32 core-setup#712)
• Preparing ROOTFS for cross-compilation Infrastructure (issue dotnet/core-setup#729, PR Introduce rootfs for cross-compilation core-setup#747)
• Enable cross compiling corehost for Linux/ARM (issue Cross compiling corehost for Linux ARM32 #2547, PR Enable cross-compilation of corehost for ARM core-setup#749, Add CROSS and ROOTFS_DIR environment for ARM core-setup#766, Select correct RID for package files when cross-building core-setup#812 )
• Update build script to enable cross building (PR Add new rids for Linux ARM32 core-setup#712)
• Update build script to enable downloading and packaging for Linux/ARM (PR Add new rids for Linux ARM32 core-setup#712)
• Build nuget packages for the host and Microsoft.NETCore.App metapackage (PR Add new rids for Linux ARM32 core-setup#712)
• Make clang3.6 default toolset for cross arm32 build (issue https://github.com/dotnet/core-setup/issues/1322)
• Enable CI support for Linux Arm32 (issue https://github.com/dotnet/core-setup/issues/790)(PR [ARM/CI] Add ARM CI script for cross build core-setup#1512)
• Enable Pipeline build support for Linux Arm32 (issue https://github.com/dotnet/core-setup/issues/789)
• Update Core-Setup readme.md to reflect the download location of Linux Arm32 archive binaries.

For example, we want to build core-setup for Linux/ARM using following command.

$ ./build.sh  --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs-coreclr/arm/

ASP.NET

• Build LibUV for Arm32 (Issue Need a build for Linux arm aspnet/libuv-build#19).

ARM softfp

We will use armel for arm-softfp through out the dotnet.

Common

• stable default clang version for cross compilation (System.Xml.Serialization.XmlSerializer serializes 0x00 but crashes on deserialization #1411)
• Use same logic to prepare rootfs across dotnet (Improve JIT of static variables which value is already known #1432)

Debian.8

CoreFX - Part1

• Add RID for armel (Add debian.8-armel and tizen.4.0.0-armel into nupkg corefx#14792)

CoreCLR

• Use armel for arm-softfp through out the dotnet. (https://github.com/dotnet/coreclr/issues/8770, Use armel instead of arm-softfp coreclr#8771)
• Generate packages for armel RID (Generate nuget package for armel (Tizen, Debian) coreclr#8827)
• Prepare ROOTFS for cross-compilation Infrastructure (already available)

CoreFX - Part2

• Use armel for arm-softfp through out the dotnet. (Use armel instead of arm-softfp corefx#14803)
• Prepare ROOTFS for cross-compilation Infrastructure (already available)

core-setup
For core-setup, we will follow steps below.

• Use armel for arm-softfp through out the dotnet (AppDomain.AssemblyResolve can be fired/not fired depending on the type of exception thrown by other extension points #1025)
• Generate nupkgs and tarball for debian.8-armel (AppDomain.AssemblyResolve can be fired/not fired depending on the type of exception thrown by other extension points #1025)

./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=debian.8-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel

Tizen 4.0.0

CoreFX - Part1

• Add RID for Tizen

CoreCLR

• Prepare ROOTFS for cross-compilation Infrastructure (Add Tizen rootfs for armel coreclr#8781)
• Enable PipeLine build support for Linux armel (Tizen) (dotnet/coreclr#9689)

CoreFX - Part2

• Prepare ROOTFS for cross-compilation Infrastructure (Add Tizen rootfs for armel corefx#14844)
• Prepare host packages for Tizen.4.0.0-armel (https://github.com/dotnet/corefx/issues/16175)
• Remove workaround to prevent armel package restore (https://github.com/dotnet/corefx/issues/16174)
• Enable CoreFX Pipeline build support for Linux for armel (Tizen) and publish packages (dotnet/corefx#16335)

core-setup
For core-setup, we will follow steps below.

• Prepare ROOTFS for cross-compilation Infrastructure (AppDomain.AssemblyResolve can be fired/not fired depending on the type of exception thrown by other extension points #1025)
• Generate nupkgs and tarball for tizen.4.0.0-armel (AppDomain.AssemblyResolve can be fired/not fired depending on the type of exception thrown by other extension points #1025)
• Enable Pipeline build support for Linux armel (Tizen) (System.Text.Json default DateTimeZoneHandling #1566)

dotnet-ci

• Add Tizen OS to Machine map ([ARM/CI] Add Tizen os to machine map dotnet-ci#603)

./build.sh -c Release --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=tizen.4.0.0-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel-tizen

Results

ubuntu.14.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds)
dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19)
dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz
ubuntu.16.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds)
dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19)
dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz
debian.8-armel
dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz
tizen.4.0.0-armel
dotnet-tizen.4.0.0-armel.1.2.0-beta-001273-00.tar.gz

[gkhanna79]

CC @schellap @ramarag @janvorli

[janvorli]

As I have mentioned in the discussion in dotnet/core-setup#724, I believe that we should be consistent in the way we do cross build over our repos. That means to use rootfs and to do the cross build using clang and not GCC. The same way we do it in CoreCLR and CoreFX.
I hope we can reuse the logic from the build.sh in CoreCLR here.

[gkhanna79]

@hqueue @hseok-oh Is Ubuntu 14.04 an intended target for Arm32?

[hqueue]

@gkhanna79 AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04.

[hseok-oh]

@hqueue @gkhanna79 RaspberryPI2 used ubuntu 14.04. (but no longer maintained)
https://wiki.ubuntu.com/ARM/RaspberryPi

[gkhanna79]

@hseok-oh If Ubuntu 14.04 is not maintained and supported, is there a specific reason to support it for Linux Arm32? If not, it will simplify bunch of things (e.g. libcoreclrtraceptprovider.so builds just fine for 16.04 but needs additional work for 14.04).

What do you think?

[gkhanna79]

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

[gkhanna79]

@kouvel This is the Linux Arm32 main work-list issue.

[gkhanna79]

CC @schellap

[hqueue]

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

@gkhanna79
I don't think it's related to Tizen OS.
The most possible explanation will be that ubuntu 14.04 is selected because of devices (e.g. ARM Emulator, Raspberry pi 2 and etc.) which were used to bring up CoreCLR ARM.
Please check out https://github.com/dotnet/coreclr/issues/3805 for ARM emulator which is used at the very early stage of CoreCLR ARM32 bring up.

[gkhanna79]

Thanks for the explanation @hqueue. For now, we can continue to build for both 14.04 and 16.04 and see how far we will need the 14.04 implementation.

With regards to the emulator, I noticed that in CoreCLR repo, we bootup the emulator and setup the rootfs within it and then perform the build. This is different from the build instructions we have at https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md where it comprises of two steps:

1. Build rootfs for an architecture
2. Perform a cross build of the repo

We strive hard to ensure that CI builds the same way as any developer would. Since the above steps are how any developer in the community would be expected to build for Arm32, what do you think of updating the CI script to do just that and not rely on emulator to be present for performing the build?

Secondly, the emulator should only be required to deploy binaries and execute tests. I notice that we only run 22 tests within the emulator, even though the repo has 11K+ tests. Do these 22 tests add any value from the CI perspective (e.g. here is a log of a recent PR that shows the 22 tests that ran - https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm_emulator_cross_debug_ubuntu_prtest/1080/consoleFull) since they dont seem to cover key scenarios where breaks maybe introduced?

[hqueue]

@gkhanna79 For the first question, I will call @sjsinju @wateret, since I'm not aware of the latest of ARM CI.

@sjsinju @wateret Can you please answer the first question above regarding CI and ARM emulator ?

For the second one, I also think tests are not sufficient to cover key sccenarios. We will look into them.

[wateret]

@gkhanna79
CI cross-builds the same way as any developers would. You can check out arm32_ci_script.sh#L227. The emulator is only for running tests. Of course, we still need pre-built rootfs to cross-build though.

One more thing that may be confusing is that the CI has arm-softfp rootfs only which cannot be created by 'any developers'. There is my work on CoreFX for enabling arm-hardfp(ubuntu 14.04). After it's done I will apply it to CoreCLR as well.