Optimizing RootFS generation

[gkhanna79]

@hqueue @hseok-oh @jyoungyun

I wanted to open a seperate issue to discuss this topic. Essentially, when we generate RootFS today for Ubuntu 14.04/16.04 Armhf, it takes about 17 mins on our official build machines, something that I would like to reduce without impacting correctness of cross-compilation.

One approach to this would be for us to create our Docker images with these RootFS already generated. However, it relies on the fact that the generated RootFS changes very infrequently - what are your thoughts on the topic? Does the cross toolset (persisted in RootFS) change very often?

If not, I can see us generating a Docker image that contains RootFS for Armhf, armel, x86, etc.

Let me know what you think.

CC @janvorli

[hqueue]

One approach to this would be for us to create our Docker images with these RootFS already generated. However, it relies on the fact that the generated RootFS changes very infrequently - what are your thoughts on the topic? Does the cross toolset (persisted in RootFS) change very often?

@gkhanna79 I agree with your opinion on creating Docker image with RootFS for Ubuntu 14.04/16.04 Armhf, because I don't think cross toolset changes often since they have been already released and I don't experience critical failure due to old cross toolset in Ubuntu 16.04 armhf.

However for armel, esp. Tizen, I'm not sure about this. I'm afraid we have to construct RootFS on demand with the latest version, because it is also being developed in open source community with non-minor changes. Furthermore RootFS for armel(Tizen) can be constructed within a minuite.
You can compare time required to consturct RootFS for Ubuntu.16.04-arm and Tizen.4.0.0-armel when you you look into corefx CI log at dotnet/corefx#15900. Therefore if this issue is about optimizing RootFS generation, I think it would be desirable to embed RootFS for arm(Ubuntu 14.04/16.04) and make RootFS for armel(Tizen 4.0.0) everytime for the time being.

• For armel(Tizen 4.0.0), it took only about 12 secs to construct RootFS, (https://ci.dot.net/job/dotnet_corefx/job/master/job/linuxarmemulator_softfp_cross_release_prtest/20/consoleFull)

00:01:08.937 + sudo ./cross/build-rootfs.sh armel tizen --skipunmount
...
00:01:19.724 + docker run -i --rm -v /mnt/j/workspace/dotnet_corefx/master/linuxarmemulator_softfp_cross_release_prtest:/opt/corefx -w /opt/corefx t2wish/dotnetcore:ubuntu1404_cross_prereqs_v2 /opt/corefx/build-native.sh -buildArch=armel -release -- cross verbose
...
00:09:40.795 Finished: SUCCESS

• For arm(Ubuntu 16.04), it took about 10 mins.(https://ci.dot.net/job/dotnet_corefx/job/master/job/linuxarmemulator_hardfp_cross_release_prtest/18/consoleFull)

00:01:23.867 + sudo ./cross/build-rootfs.sh arm xenial --skipunmount
...
00:11:40.467 + docker run -i --rm -v /mnt/j/workspace/dotnet_corefx/master/linuxarmemulator_hardfp_cross_release_prtest:/opt/corefx -w /opt/corefx chcosta/dotnetcore:ubuntu1404_cross_prereqs_v1 /opt/corefx/build-native.sh -buildArch=arm -release -- cross verbose
...
00:20:15.096 Finished: SUCCESS

/cc: @lemmaa @jyoungyun @hseok-oh @chunseoklee
What do you think for Tizen ? Do we need the latest RootFS for Tizen? or can we use a pre-constructed RootFS for a while and update RootFS sometimes when needed ?

[hseok-oh]

I agree with your opinion that uses docker image including RootFS already generated for Ubuntu. And I think it's better to generate RootFS for Tizen for each build request because it has small overhead to generate RootFS for Tizen.

[hqueue]

@gkhanna79 BTW if we are going to use same Docker image with RoofFS for core-setup, coreclr and corefx, we have to construct a single RootFS for all three repos and this is being acvhieved by using build-rootfs.sh in https://github.com/dotnet/core-setup/issues/1432 (not merged yet), dotnet/coreclr#9411 (not merged yet), dotnet/corefx#15755 (merged).

[hseok-oh]

We need to prepare to contribute rootfs generation. For example, I want to add patch process for ubuntu 14.04 to build libcoreclrtraceptprovider.so (#8684), but I'll get failure in CI without docker image update, and ARM test may fail if this change merged.

[gkhanna79]

I think it would be desirable to embed RootFS for arm(Ubuntu 14.04/16.04) and make RootFS for armel(Tizen 4.0.0) everytime for the time being

Excellent - for now, I am most interested in optimizing the time it takes for the armhf rootfs for 14.04/16.04 and given that Tizen one gets constructed soon and is under development, it sounds reasonable to construct it on the fly.

@hqueue Can you please get the Core-Setup and dotnet/coreclr#9411 issues merged so that I can look into Docker image update for this?

@hseok-oh I agree that you should be able to contribute to RootFS generation. The way I envision this to happen is by having the respective DockerFiles being made available in a public repo that you can contribute to, followed by some automatic validation of the updated image (though, in the short term, it maybe manual validation).

@MichaelSimons @dleeapho Can you please share your thoughts on how we can make this happen for 2.0?

[hqueue]

Can you please get the Core-Setup and dotnet/coreclr#9411 issues merged so that I can look into Docker image update for this?

@gkhanna79 Oh, I've made a mistake. PR for Core-Setup and CoreFX has been alreay merged.
And I'm waiting PR in CoreCLR (#9411) to be reviewed and merged, since I don't have merge permission.

We can use build-rootfs.sh in Core-Setup (dotnet/core-setup#1493) to construct a RootFS for all core-setup, coreclr and corefx, because build-rootfs.sh in Core-Setup and CoreCLR will have exactly same structure and the only difference is supported architecture, i.e. Core-Setup does not support arm64 yet.

[hqueue]

@gkhanna79 If we are going to embed RootFS in docker image, are we going to maintain seperate docker images for each rootfs ?

[gkhanna79]

@hqueue I would that we will have all the RootFS in a single Docker image in a well defined layout, something like:

\rootfs\<arch>\<platform>

where

<arch> can be arm, armel (when it is stabilized), x86.
<platform> can be Ubuntu 14.04, Ubuntu 16.04, Tizen

And when invoking the build, we will set ROOTFS_DIR environment variable to point to the right one that build-rootfs.sh can pickup from https://github.com/dotnet/coreclr/blob/master/cross/build-rootfs.sh#L123.

[seanshpark]

@gkhanna79 , sounds good :)

[seanshpark]

CC @parjong , @wateret

[hqueue]

@gkhanna79 Due to Jenkins was not working yesterday, I've triggered CI for dotnet/corefx#15900) now and it worked.

\rootfs\<arch>\<platform>

I also think the suggested layout will work with three scripts, i.e. build-rootfs.sh, CI script and build script.

BTW currently we are using different docker image for ubuntu.14.04-arm, ubuntu.16.04-arm and tizen.4.0.0-armel. Are you going to merge them too ? or just let each docker image have all three rootfs for arm?

[gkhanna79]

Thanks for the confirmation @seanshpark @hqueue.

We will continue to maintain Ubuntu 14.04 and Ubuntu 16.04 Docker images. For Tizen, I am not clear yet - what would you suggest? Will either 14.04 or 16.04 images work for Tizen build once we generate Tizen rootfs on them?

[jyoungyun]

You can generate Tizen rootfs in either 14.04 and 16.04 images if the libxml2-utils is installed in the docker image by https://github.com/dotnet/core-setup/blob/master/cross/armel/tizen-fetch.sh. However, in the case of Tizen, we will use the 14.04 image by default.

[hqueue]

@gkhanna79 As @jyoungyun said, there is no problem to build dotnet for Tizen in 14.04 and 16.04 if required packages are installed to construct rootfs. And I also prefer 14.04 over 16.04 if we have to choose one docker image.