From 67c533a7ecb3fcb16a22f3274ba62d4dba2fbcd1 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Thu, 11 Aug 2016 18:32:08 -0400 Subject: [PATCH] [xa-prep-tasks] Use the Mono bundle Context: https://github.com/xamarin/xamarin-android/commit/fbfd676c102c63e4e06e750857b178725e33450c Stage 3 of the cunning plan is to (attempt to) use the mono bundle introduced in commit fbfd676c. This "simple" desire (ha!) re-raises the architectural project dependency issue "solved" in fbfd676c, but first, a simple question: What should download the mono bundle? There are two plausible answers: 1. `make prepare` can (somehow) handle it. 2. MSBuild can (somehow) handle it. Both are plausible. The problem with using `make` targets (1) is there is increased potential for "duplication" -- duplication of the bundle filename, downloading it, and extracting it. Plus, `make` isn't "Windows friendly", in that GNU make isn't (normally) present with Visual Studio. (`NMAKE` is, but the Makefiles in this project are not compatible with `NMAKE`.) Which brings us to MSBuild (2): can it handle the task? To tackle that, we need to be able to have an MSBuild task project which has *no dependencies*, so that it can download and extract the mono bundle *before anything else runs*, as it may be downloading contents which mean that other projects don't *need* to run. The need for a "pre-bootstrap" task assembly -- called `xa-prep-tasks` -- thus "undoes" *some* of the logic regarding `libzip-windows.mdproj` and the `` task from fbfd676c: it isn't *possible* to rely on `libzip` from a "pre-build" state, as `libzip` is one of the things in the mono bundle, so now we need *two* "bootstrap" task assemblies: one without a `libzip` dependency -- `xa-prep-tasks.dll` -- and one *with* a `libzip` dependency -- `Xamarin.Android.Tools.BootstrapTasks.dll` Move tasks which don't currently require `libzip` -- or won't in the future, or laziness -- from `Xamarin.Android.Tools.BootstrapTasks.dll` and move them into `xa-prep-tasks.dll`. With that architectural compromise in place, add `xa-prep-tasks` as a `@(ProjectReference)` to various projects to help ensure it's built *first*, and rearchitect `bundle.mdproj` so that `xa-prep-tasks.targets` and `bundle.targets` can use the same targets to compute the bundle filename, now in `build-tools/bundle/bundle-path.targets`. Add a post-build step to `xa-prep-tasks.csproj` which downloads and extracts the expected mono bundle. One "problem" (feature?) is that the new `` task doesn't report errors as errors when unzip'ing the file. This turns out to be fine here because when downloading the mono bundle from Azure we don't get a 404 *anyway* -- Azure instead returns an XML document containing an error message (wat?!). We can thus ignore most error handling entirely...though we're *also* ignoring any checking for invalid downloads, which is something we should address in the future. Update the varioous project files so that they won't attempt to rebuild binaries that were present in the mono bundle. --- Makefile | 5 +- README.md | 10 +- Xamarin.Android.sln | 11 ++ .../android-toolchain.mdproj | 7 +- .../android-toolchain.targets | 6 +- build-tools/bundle/bundle-path.targets | 30 ++++ build-tools/bundle/bundle.mdproj | 5 + build-tools/bundle/bundle.targets | 70 ++------ .../libzip-windows/libzip-windows.mdproj | 3 +- build-tools/libzip/libzip.mdproj | 10 +- build-tools/libzip/libzip.projitems | 4 + build-tools/libzip/libzip.targets | 41 +++-- .../mono-runtimes/mono-runtimes.mdproj | 27 +--- .../mono-runtimes/mono-runtimes.targets | 68 +++++++- build-tools/scripts/Paths.targets | 12 ++ build-tools/scripts/RequiredPrograms.targets | 2 +- .../xa-prep-tasks/Properties/AssemblyInfo.cs | 27 ++++ .../DownloadUri.cs | 15 +- .../Git.cs | 4 +- .../GitBranch.cs | 4 +- .../GitCommitHash.cs | 4 +- .../SystemUnzip.cs | 153 ++++++++++++++++++ .../Which.cs | 21 ++- .../xa-prep-tasks/xa-prep-tasks.csproj | 49 ++++++ .../xa-prep-tasks/xa-prep-tasks.targets | 43 +++++ ...amarin.Android.Tools.BootstrapTasks.csproj | 5 - src/monodroid/monodroid.targets | 2 - 27 files changed, 497 insertions(+), 141 deletions(-) create mode 100644 build-tools/bundle/bundle-path.targets create mode 100644 build-tools/xa-prep-tasks/Properties/AssemblyInfo.cs rename {src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks => build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks}/DownloadUri.cs (78%) rename {src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks => build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks}/Git.cs (96%) rename {src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks => build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks}/GitBranch.cs (94%) rename {src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks => build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks}/GitCommitHash.cs (94%) create mode 100644 build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/SystemUnzip.cs rename {src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks => build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks}/Which.cs (77%) create mode 100644 build-tools/xa-prep-tasks/xa-prep-tasks.csproj create mode 100644 build-tools/xa-prep-tasks/xa-prep-tasks.targets diff --git a/Makefile b/Makefile index 5a7d72d6bfb..5cdc92fb48e 100644 --- a/Makefile +++ b/Makefile @@ -29,11 +29,14 @@ all-tests:: prepare:: prepare-external prepare-props +# $(call GetPath,path) +GetPath = $(shell $(MSBUILD) $(MSBUILD_FLAGS) /p:DoNotLoadOSProperties=True /nologo /v:minimal /t:Get$(1)FullPath build-tools/scripts/Paths.targets | tr -d '[[:space:]]' ) + prepare-external: git submodule update --init --recursive nuget restore $(SOLUTION) nuget restore Xamarin.Android-Tests.sln - (cd `$(MSBUILD) $(MSBUILD_FLAGS) /p:DoNotLoadOSProperties=True /nologo /v:minimal /t:GetJavaInteropFullPath build-tools/scripts/Paths.targets` && nuget restore) + (cd $(call GetPath,JavaInterop) && nuget restore) prepare-props: cp Configuration.Java.Interop.Override.props external/Java.Interop/Configuration.Override.props diff --git a/README.md b/README.md index 31a9f793da0..1f7592d8619 100644 --- a/README.md +++ b/README.md @@ -292,8 +292,14 @@ within `build-tools/mono-runtimes/obj/$(Configuration)/TARGET`. If you change sources within `external/mono`, a top-level `make`/`xbuild` invocation may not rebuild those mono native binaries. To explicitly rebuild -Mono for a given target, run `make` from the relevant directory. -For example, to rebuild Mono for armeabi-v7a: +*all* Mono runtimes, use the `ForceBuild` target: + + # Build and install all runtimes + $ xbuild /t:ForceBuild build-tools/mono-runtimes/mono-runtimes.mdproj + +To build Mono for a specific target, run `make` from the relevant directory +and invoke the `_InstallRuntimes` target. For example, to rebuild +Mono for armeabi-v7a: $ cd build-tools/mono-runtimes $ make -C obj/Debug/armeabi-v7a diff --git a/Xamarin.Android.sln b/Xamarin.Android.sln index 2a68f802337..03bb4f780aa 100644 --- a/Xamarin.Android.sln +++ b/Xamarin.Android.sln @@ -89,6 +89,8 @@ Project("{9344BDBB-3E7F-41FC-A0DD-8665D75EE146}") = "libzip-windows", "build-too EndProject Project("{9344BDBB-3E7F-41FC-A0DD-8665D75EE146}") = "bundle", "build-tools\bundle\bundle.mdproj", "{1640725C-4DB8-4D8D-BC96-74E688A06EEF}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "xa-prep-tasks", "build-tools\xa-prep-tasks\xa-prep-tasks.csproj", "{7CE69551-BD73-4726-ACAA-AAF89C84BAF8}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|AnyCPU = Debug|AnyCPU @@ -407,6 +409,14 @@ Global {1640725C-4DB8-4D8D-BC96-74E688A06EEF}.XAIntegrationDebug|AnyCPU.Build.0 = Debug|Any CPU {1640725C-4DB8-4D8D-BC96-74E688A06EEF}.XAIntegrationRelease|AnyCPU.ActiveCfg = Release|Any CPU {1640725C-4DB8-4D8D-BC96-74E688A06EEF}.XAIntegrationRelease|AnyCPU.Build.0 = Release|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.Debug|AnyCPU.ActiveCfg = Debug|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.Debug|AnyCPU.Build.0 = Debug|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.Release|AnyCPU.ActiveCfg = Release|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.Release|AnyCPU.Build.0 = Release|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.XAIntegrationDebug|AnyCPU.ActiveCfg = Debug|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.XAIntegrationDebug|AnyCPU.Build.0 = Debug|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.XAIntegrationRelease|AnyCPU.ActiveCfg = Debug|Any CPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8}.XAIntegrationRelease|AnyCPU.Build.0 = Debug|Any CPU EndGlobalSection GlobalSection(NestedProjects) = preSolution {8FF78EB6-6FC8-46A7-8A15-EBBA9045C5FA} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62} @@ -449,6 +459,7 @@ Global {C9FF2E4D-D927-479E-838B-647C16763F64} = {04E3E11E-B47D-4599-8AFC-50515A95E715} {0DE278D6-000F-4001-BB98-187C0AF58A61} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62} {1640725C-4DB8-4D8D-BC96-74E688A06EEF} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62} + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62} EndGlobalSection GlobalSection(MonoDevelopProperties) = preSolution Policies = $0 diff --git a/build-tools/android-toolchain/android-toolchain.mdproj b/build-tools/android-toolchain/android-toolchain.mdproj index f904c1a1304..b2742045076 100644 --- a/build-tools/android-toolchain/android-toolchain.mdproj +++ b/build-tools/android-toolchain/android-toolchain.mdproj @@ -26,12 +26,15 @@ + + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8} + xa-prep-tasks + False + {E8492EFB-D14A-4F32-AA28-88848322ECEA} Xamarin.Android.Tools.BootstrapTasks False - Content - PreserveNewest diff --git a/build-tools/android-toolchain/android-toolchain.targets b/build-tools/android-toolchain/android-toolchain.targets index 94cc1e91e98..aed831bd247 100644 --- a/build-tools/android-toolchain/android-toolchain.targets +++ b/build-tools/android-toolchain/android-toolchain.targets @@ -10,10 +10,8 @@ Properties="OutputPath=$(AndroidToolchainDirectory)" /> - - - - + + + + + + + + + + + + + + + + + + bundle-v2-$(Configuration)-$(HostOS)-libzip=$(_LibZipHash),llvm=$(_LlvmHash),mono=$(_MonoHash).zip + + + diff --git a/build-tools/bundle/bundle.mdproj b/build-tools/bundle/bundle.mdproj index eea19bb4499..18118232e81 100644 --- a/build-tools/bundle/bundle.mdproj +++ b/build-tools/bundle/bundle.mdproj @@ -22,6 +22,11 @@ + + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8} + xa-prep-tasks + False + {8FF78EB6-6FC8-46A7-8A15-EBBA9045C5FA} android-toolchain diff --git a/build-tools/bundle/bundle.targets b/build-tools/bundle/bundle.targets index 66cd997ebe5..be80ae5297f 100644 --- a/build-tools/bundle/bundle.targets +++ b/build-tools/bundle/bundle.targets @@ -1,78 +1,30 @@ - - + - - - - - - - - - - - - - - <_BundlePath>$(OutputPath)bundle-v2-$(Configuration)-$(HostOS)-libzip=$(_LibZipHash),llvm=$(_LlvmHash),mono=$(_MonoHash).zip - - + + - + DependsOnTargets="GetBundleFileName"> + - - <_Archive Include="@(_BclInstalledItem)" /> - <_Archive Include="$(OutputPath)lib\xbuild-frameworks\MonoAndroid\v1.0\Facades\*.dll*" /> - <_Archive Include="$(OutputPath)lib\xbuild-frameworks\MonoAndroid\v1.0\RedistList\FrameworkList.xml" /> - <_Archive Include="@(_InstallRuntimeOutput)" /> - <_Archive Include="@(_InstallUnstrippedRuntimeOutput)" /> - <_Archive Include="@(_InstallProfilerOutput)" /> - <_Archive Include="@(_InstallUnstrippedProfilerOutput)" /> - <_Archive Include="@(_InstallMonoPosixHelperOutput)" /> - <_Archive Include="@(_InstallUnstrippedMonoPosixHelperOutput)" /> - <_Archive Include="@(_RuntimeEglibHeaderOutput)" /> - <_Archive Include="@(_LibZipTarget->'$(OutputPath)lib\xbuild\Xamarin\Android\%(OutputLibrary)')" /> - <_Archive Include="@(_MonoConstsOutput)" /> - <_Archive Include="$(OutputPath)%(_MonoCrossRuntime.InstallPath)%(_MonoCrossRuntime.CrossMonoName)%(_MonoCrossRuntime.ExeSuffix)" - Condition=" '@(_MonoCrossRuntime)' != '' " - /> - <_Archive Include="@(_LlvmRuntime->'$(OutputPath)bin\llc%(ExeSuffix)')" - Condition=" '%(_LlvmRuntime.InstallBinaries)' == 'true' " - /> - <_Archive Include="@(_LlvmRuntime->'$(OutputPath)bin\opt%(ExeSuffix)')" - Condition=" '%(_LlvmRuntime.InstallBinaries)' == 'true' " - /> - + DependsOnTargets="GetMonoBundleItems;GetLibZipBundleItems"> + DependsOnTargets="GetBundleFileName;_GetArchiveItems" + Inputs="@(BundleItem)" + Outputs="$(OutputPath)$(XABundleFileName)"> diff --git a/build-tools/libzip-windows/libzip-windows.mdproj b/build-tools/libzip-windows/libzip-windows.mdproj index 004197132bb..8dec3429fa9 100644 --- a/build-tools/libzip-windows/libzip-windows.mdproj +++ b/build-tools/libzip-windows/libzip-windows.mdproj @@ -17,8 +17,7 @@ ResolveReferences; - _Configure; - _Make + _BuildUnlessCached diff --git a/build-tools/libzip/libzip.mdproj b/build-tools/libzip/libzip.mdproj index 91559fcd315..3c3286ba68a 100644 --- a/build-tools/libzip/libzip.mdproj +++ b/build-tools/libzip/libzip.mdproj @@ -17,13 +17,19 @@ ResolveReferences; - _Configure; - _Make + _BuildUnlessCached + + + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8} + xa-prep-tasks + False + + diff --git a/build-tools/libzip/libzip.projitems b/build-tools/libzip/libzip.projitems index 8dbfec42784..161b68e3ad6 100644 --- a/build-tools/libzip/libzip.projitems +++ b/build-tools/libzip/libzip.projitems @@ -14,4 +14,8 @@ libzip.so + + + + diff --git a/build-tools/libzip/libzip.targets b/build-tools/libzip/libzip.targets index 9410290413a..7fc900769a2 100644 --- a/build-tools/libzip/libzip.targets +++ b/build-tools/libzip/libzip.targets @@ -1,19 +1,14 @@ - - - - - - - - - - - - - - + + + + _SetCMakeListsTxtTimeToLastCommitTimestamp; + CheckForRequiredPrograms; + _Configure; + _Make + + - - + + + + + + + + + + diff --git a/build-tools/mono-runtimes/mono-runtimes.mdproj b/build-tools/mono-runtimes/mono-runtimes.mdproj index 769a2e8c47a..8d088d3e61d 100644 --- a/build-tools/mono-runtimes/mono-runtimes.mdproj +++ b/build-tools/mono-runtimes/mono-runtimes.mdproj @@ -6,30 +6,13 @@ GenericProject {C03E6CF1-7460-4CDC-A4AB-292BBC0F61F2} - - ..\..\bin\Debug - - - ..\..\bin\Release - - - - - ResolveReferences; - _BuildLlvm; - _InstallLlvm; - _Autogen; - _ConfigureRuntimes; - _BuildRuntimes; - _InstallRuntimes; - _InstallBcl; - _ConfigureCrossRuntimes; - _BuildCrossRuntimes; - _InstallCrossRuntimes; - - + + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8} + xa-prep-tasks + False + {8FF78EB6-6FC8-46A7-8A15-EBBA9045C5FA} android-toolchain diff --git a/build-tools/mono-runtimes/mono-runtimes.targets b/build-tools/mono-runtimes/mono-runtimes.targets index d2ac1af851d..afca8c016ec 100644 --- a/build-tools/mono-runtimes/mono-runtimes.targets +++ b/build-tools/mono-runtimes/mono-runtimes.targets @@ -1,5 +1,19 @@ + + ..\..\bin\Debug + + + ..\..\bin\Release + + + + + ResolveReferences; + _BuildUnlessCached + + + <_SourceTopDir>..\.. <_BclFrameworkDir>$(OutputPath)\lib\xbuild-frameworks\MonoAndroid\v1.0 @@ -9,6 +23,20 @@ <_MonoProfileDir>$(MonoSourceFullPath)\mcs\class\lib\monodroid + + + _BuildLlvm; + _InstallLlvm; + _Autogen; + _ConfigureRuntimes; + _BuildRuntimes; + _InstallRuntimes; + _InstallBcl; + _ConfigureCrossRuntimes; + _BuildCrossRuntimes; + _InstallCrossRuntimes; + + @@ -71,8 +99,11 @@ Command="touch -m -t `git log -1 --format=%25cd --date=format-local:%25Y%25m%25d%25H%25M.%25S` autogen.sh" WorkingDirectory="$(MonoSourceFullPath)" /> + - <_LlvmSourceFile Include="$(LlvmSourceFullPath)\lib\**\*.cpp" /> @@ -406,6 +437,41 @@ Files="$(OutputPath)%(_MonoCrossRuntime.InstallPath)%(_MonoCrossRuntime.CrossMonoName)%(_MonoCrossRuntime.ExeSuffix)" /> + + + + + + + + + + + + + + + + + + + + + + diff --git a/build-tools/scripts/Paths.targets b/build-tools/scripts/Paths.targets index c0c88bf4e39..e85354f4610 100644 --- a/build-tools/scripts/Paths.targets +++ b/build-tools/scripts/Paths.targets @@ -19,6 +19,18 @@ Importance="High" /> + + + + + + - + diff --git a/build-tools/xa-prep-tasks/Properties/AssemblyInfo.cs b/build-tools/xa-prep-tasks/Properties/AssemblyInfo.cs new file mode 100644 index 00000000000..96312264b62 --- /dev/null +++ b/build-tools/xa-prep-tasks/Properties/AssemblyInfo.cs @@ -0,0 +1,27 @@ +using System.Reflection; +using System.Runtime.CompilerServices; + +// Information about this assembly is defined by the following attributes. +// Change them to the values specific to your project. + +[assembly: AssemblyTitle ("xapreptasks")] +[assembly: AssemblyDescription ("")] +[assembly: AssemblyConfiguration ("")] +[assembly: AssemblyCompany ("Microsoft Corporation")] +[assembly: AssemblyProduct ("")] +[assembly: AssemblyCopyright ("Microsoft Corporation")] +[assembly: AssemblyTrademark ("Microsoft")] +[assembly: AssemblyCulture ("")] + +// The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}". +// The form "{Major}.{Minor}.*" will automatically update the build and revision, +// and "{Major}.{Minor}.{Build}.*" will update just the revision. + +[assembly: AssemblyVersion ("1.0.*")] + +// The following attributes are used to specify the signing key for the assembly, +// if desired. See the Mono documentation for more information about signing. + +//[assembly: AssemblyDelaySign(false)] +//[assembly: AssemblyKeyFile("")] + diff --git a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/DownloadUri.cs b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/DownloadUri.cs similarity index 78% rename from src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/DownloadUri.cs rename to build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/DownloadUri.cs index 4fd2bb963fb..f1efe95dae2 100644 --- a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/DownloadUri.cs +++ b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/DownloadUri.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.IO; using System.Net.Http; using System.Threading.Tasks; @@ -9,7 +9,7 @@ using TTask = System.Threading.Tasks.Task; using MTask = Microsoft.Build.Utilities.Task; -namespace Xamarin.Android.Tools.BootstrapTasks { +namespace Xamarin.Android.BuildTools.PrepTasks { public class DownloadUri : MTask { @@ -58,12 +58,17 @@ async TTask DownloadFile (HttpClient client, string uri, string destinationFile) Log.LogMessage (MessageImportance.Normal, $"Skipping uri '{uri}' as destination file already exists '{destinationFile}'."); return; } - Log.LogMessage (MessageImportance.Low, $"Downloading '{uri}'."); + var dp = Path.GetDirectoryName (destinationFile); + var dn = Path.GetFileName (destinationFile); + var tempPath = Path.Combine (dp, "." + dn + ".download"); + + Log.LogMessage (MessageImportance.Low, $"Downloading `{uri}` to `{tempPath}`."); using (var r = await client.GetAsync (uri)) - using (var o = File.OpenWrite (destinationFile)) { + using (var o = File.OpenWrite (tempPath)) { await r.Content.CopyToAsync (o); } + Log.LogMessage (MessageImportance.Low, $"mv '{tempPath}' '{destinationFile}'."); + File.Move (tempPath, destinationFile); } } } - diff --git a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/Git.cs b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/Git.cs similarity index 96% rename from src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/Git.cs rename to build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/Git.cs index b81a88e823b..a2ade4bc7ab 100644 --- a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/Git.cs +++ b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/Git.cs @@ -8,11 +8,9 @@ using Microsoft.Build.Framework; using Microsoft.Build.Utilities; -using Xamarin.Tools.Zip; - using IOFile = System.IO.File; -namespace Xamarin.Android.Tools.BootstrapTasks +namespace Xamarin.Android.BuildTools.PrepTasks { public class Git : ToolTask { diff --git a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/GitBranch.cs b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/GitBranch.cs similarity index 94% rename from src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/GitBranch.cs rename to build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/GitBranch.cs index 18c467cf67f..826b94031e8 100644 --- a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/GitBranch.cs +++ b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/GitBranch.cs @@ -8,11 +8,9 @@ using Microsoft.Build.Framework; using Microsoft.Build.Utilities; -using Xamarin.Tools.Zip; - using IOFile = System.IO.File; -namespace Xamarin.Android.Tools.BootstrapTasks +namespace Xamarin.Android.BuildTools.PrepTasks { public sealed class GitBranch : Git { diff --git a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/GitCommitHash.cs b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/GitCommitHash.cs similarity index 94% rename from src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/GitCommitHash.cs rename to build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/GitCommitHash.cs index 2a94226e8af..0dec568fcbd 100644 --- a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/GitCommitHash.cs +++ b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/GitCommitHash.cs @@ -5,11 +5,9 @@ using Microsoft.Build.Framework; using Microsoft.Build.Utilities; -using Xamarin.Tools.Zip; - using IOFile = System.IO.File; -namespace Xamarin.Android.Tools.BootstrapTasks +namespace Xamarin.Android.BuildTools.PrepTasks { public sealed class GitCommitHash : Git { diff --git a/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/SystemUnzip.cs b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/SystemUnzip.cs new file mode 100644 index 00000000000..0953c7dec1e --- /dev/null +++ b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/SystemUnzip.cs @@ -0,0 +1,153 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Text; +using System.Threading.Tasks; + +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; + +using TTask = System.Threading.Tasks.Task; +using MTask = Microsoft.Build.Utilities.Task; + +namespace Xamarin.Android.BuildTools.PrepTasks { + + public class SystemUnzip : MTask + { + [Required] + public ITaskItem[] SourceFiles { get; set; } + + public string SourceEntryGlob { get; set; } + + public string EntryNameEncoding { get; set; } + + public string HostOS { get; set; } + + [Required] + public ITaskItem DestinationFolder { get; set; } + + string[] SourceEntryGlobParts; + + public override bool Execute () + { + Log.LogMessage (MessageImportance.Low, $"{nameof (SystemUnzip)}:"); + Log.LogMessage (MessageImportance.Low, $" {nameof (DestinationFolder)}: {DestinationFolder.ItemSpec}"); + Log.LogMessage (MessageImportance.Low, $" {nameof (EntryNameEncoding)}: {EntryNameEncoding}"); + Log.LogMessage (MessageImportance.Low, $" {nameof (HostOS)}: {HostOS}"); + + Log.LogMessage (MessageImportance.Low, $" {nameof (SourceEntryGlob)}: {SourceEntryGlob}"); + Log.LogMessage (MessageImportance.Low, $" {nameof (SourceFiles)}:"); + for (int i = 0; i < SourceFiles.Length; ++i) { + var sf = SourceFiles [i].ItemSpec; + var rp = SourceFiles [i].GetMetadata ("DestDir"); + rp = string.IsNullOrEmpty (rp) + ? "" + : " [ " + rp + " ]"; + Log.LogMessage (MessageImportance.Low, " {0}{1}", sf, rp); + } + + if (File.Exists (DestinationFolder.ItemSpec)) { + Log.LogError ($"{nameof (DestinationFolder)} must be a directory!"); + return false; + } + + SourceEntryGlobParts = (SourceEntryGlob ?? "*").Split ('/', '\\'); + + Directory.CreateDirectory (DestinationFolder.ItemSpec); + + var tempDir = Path.Combine (Path.GetTempPath (), Path.GetRandomFileName ()); + Directory.CreateDirectory (tempDir); + Log.LogMessage (MessageImportance.Low, $" Extracting into temporary directory: {tempDir}"); + + var encoding = string.IsNullOrEmpty (EntryNameEncoding) + ? null + : Encoding.GetEncoding (EntryNameEncoding); + + var tasks = new TTask [SourceFiles.Length]; + for (int i = 0; i < SourceFiles.Length; ++i) { + var td = tempDir; + var sourceFile = SourceFiles [i].ItemSpec; + var relativeDestDir = SourceFiles [i].GetMetadata ("DestDir"); + var enc = encoding; + var destFolder = DestinationFolder.ItemSpec; + tasks [i] = TTask.Run (() => ExtractFile (td, sourceFile, relativeDestDir, destFolder, enc)); + } + + TTask.WaitAll (tasks); + + Directory.Delete (tempDir, recursive: true); + + return !Log.HasLoggedErrors; + } + + // Ignore CS1998 because there's no async System.IO APIs to use here. + // Instead, we're using Tasks so that we can extract multiple files + // in parallel via Task.Run() and Task.WaitAll(). +#pragma warning disable 1998 + async TTask ExtractFile (string tempDir, string sourceFile, string relativeDestDir, string destinationFolder, Encoding encoding) + { + var tempName = Path.GetRandomFileName (); + var nestedTemp = Path.Combine (tempDir, tempName); + Directory.CreateDirectory (nestedTemp); + + relativeDestDir = relativeDestDir?.Replace ('\\', Path.DirectorySeparatorChar); + + if (string.Equals (HostOS, "Windows", StringComparison.OrdinalIgnoreCase)) { + ZipFile.ExtractToDirectory (sourceFile, nestedTemp, encoding); + } else { + var start = new ProcessStartInfo ("unzip", $"\"{sourceFile}\" -d \"{nestedTemp}\"") { + CreateNoWindow = true, + UseShellExecute = false, + }; + Log.LogMessage (MessageImportance.Low, $"unzip \"{sourceFile}\" -d \"{nestedTemp}\""); + var p = Process.Start (start); + p.WaitForExit (); + } + + var entries = GetExtractedSourceEntries (nestedTemp); + + // "merge" directories from `name`/within `sourceFile` and `destinationFolder`. + // If we did e.g. `mv foo/lib destination/lib` *and* `destination/lib` *already exists*, + // we'd create `destination/lib/lib`, which isn't intended. + // If we did e.g. `mv foo/lib destination`, **mv**(1) may *overwrite* `destination/lib` + // if it already exists, which *also* isn't intended. + // If `destination/lib/example` and `sourceFile` contains a `lib/another` entry, + // then we want to create a `destination/lib/another` file. + foreach (var entry in entries) { + var name = Path.GetFileName (entry); + var destDir = string.IsNullOrEmpty (relativeDestDir) + ? destinationFolder + : Path.Combine (destinationFolder, relativeDestDir); + destDir = Path.Combine (destDir, name); + foreach (var file in Directory.EnumerateFiles (entry, "*", SearchOption.AllDirectories)) { + var relPath = file.Substring (entry.Length + 1); + var dest = Path.Combine (destDir, relPath); + Directory.CreateDirectory (Path.GetDirectoryName (dest)); + Log.LogMessage (MessageImportance.Low, $"mv '{file}' '{dest}'"); + if (Directory.Exists (entry)) + Process.Start ("/bin/mv", $@"""{file}"" ""{dest}""").WaitForExit (); + else { + if (File.Exists (dest)) + File.Delete (dest); + File.Move (file, dest); + } + } + } + } + + IEnumerable GetExtractedSourceEntries (string root) + { + var entries = Directory.EnumerateFileSystemEntries (root, SourceEntryGlobParts [0], SearchOption.TopDirectoryOnly); + for (int i = 1; i < SourceEntryGlobParts.Length; ++i) { + entries = entries + .SelectMany (e => Directory.EnumerateFileSystemEntries (e, SourceEntryGlobParts [i], SearchOption.TopDirectoryOnly)); + } + return entries; + } +#pragma warning restore 1998 + } +} + diff --git a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/Which.cs b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/Which.cs similarity index 77% rename from src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/Which.cs rename to build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/Which.cs index acc334f9fde..f18af287eab 100644 --- a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks/Which.cs +++ b/build-tools/xa-prep-tasks/Xamarin.Android.BuildTools.PrepTasks/Which.cs @@ -5,7 +5,7 @@ using Microsoft.Build.Framework; using Microsoft.Build.Utilities; -namespace Xamarin.Android.Tools.BootstrapTasks +namespace Xamarin.Android.BuildTools.PrepTasks { public class Which : Task { @@ -19,13 +19,18 @@ public class Which : Task [Output] public ITaskItem Location { get; set; } - static readonly string[] FileExtensions = new []{ - null, - ".bat", - ".cmd", - ".com", - ".exe", - }; + static readonly string[] FileExtensions; + + static Which () + { + var pathExt = Environment.GetEnvironmentVariable ("PATHEXT"); + var pathExts = pathExt?.Split (new char [] { Path.PathSeparator }, StringSplitOptions.RemoveEmptyEntries); + FileExtensions = new string [(pathExts?.Length ?? 0) + 1]; + FileExtensions [0] = null; + if (pathExts != null) { + Array.Copy (pathExts, 0, FileExtensions, 1, pathExt.Length); + } + } public override bool Execute () { diff --git a/build-tools/xa-prep-tasks/xa-prep-tasks.csproj b/build-tools/xa-prep-tasks/xa-prep-tasks.csproj new file mode 100644 index 00000000000..51d8804045c --- /dev/null +++ b/build-tools/xa-prep-tasks/xa-prep-tasks.csproj @@ -0,0 +1,49 @@ + + + + Debug + AnyCPU + {7CE69551-BD73-4726-ACAA-AAF89C84BAF8} + Library + Xamarin.Android.BuildTools.PrepTasks + xa-prep-tasks + v4.5 + + + + true + full + false + ..\..\bin\BuildDebug + DEBUG; + prompt + 4 + false + + + true + ..\..\bin\BuildRelease + prompt + 4 + false + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/build-tools/xa-prep-tasks/xa-prep-tasks.targets b/build-tools/xa-prep-tasks/xa-prep-tasks.targets new file mode 100644 index 00000000000..54358ef9f55 --- /dev/null +++ b/build-tools/xa-prep-tasks/xa-prep-tasks.targets @@ -0,0 +1,43 @@ + + + + + + + <_AzureBaseUri>https://xamjenkinsartifact.blob.core.windows.net/xamarin-android/xamarin-android/bin/ + + + + <_BundlePath>$(OutputPath)$(XABundleFileName) + + + + + + + + + + + + \ No newline at end of file diff --git a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks.csproj b/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks.csproj index f821f59d2dd..02525f44277 100644 --- a/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks.csproj +++ b/src/Xamarin.Android.Tools.BootstrapTasks/Xamarin.Android.Tools.BootstrapTasks.csproj @@ -38,14 +38,9 @@ - - - - - diff --git a/src/monodroid/monodroid.targets b/src/monodroid/monodroid.targets index 9e9b04e8cb4..ec50cd2d848 100644 --- a/src/monodroid/monodroid.targets +++ b/src/monodroid/monodroid.targets @@ -1,7 +1,5 @@ - -