Build scripts while copying native binaries #36482

sdmaclea · 2020-05-15T01:30:54Z

Stop building test scripts during the managed build phase
Build after copying the native targets
Only build scripts for the current platform.

Remove issues relative to building scripts on OSX

trylek

LGTM

Stop building test scripts during the managed build phase Build after copying the native targets Remove issues relative to building scripts on OSX

jashook

Lgtm thank you for the change

jashook · 2020-05-15T20:04:13Z

Looks like some tests are missing their cmd/sh files

jashook · 2020-05-15T20:04:34Z

Ah I think I know the problem, superpmi I think copied these files and expects them already to be on disk.

jashook · 2020-05-15T20:08:18Z

  <ItemGroup>
    <_SpmiTestProjects Include="..\Performance\CodeQuality\Bytemark\Bytemark.csproj" />
    <_SpmiTestProjects Include="..\Methodical\fp\exgen\10w5d_cs_do.csproj" />
    <_SpmiTestProjects Update="@(_SpmiTestProjects)">
      <Properties>OutputPath=$(OutputPath)\%(FileName)</Properties>
    </_SpmiTestProjects>
  </ItemGroup>

jashook · 2020-05-15T20:08:39Z

@sdmaclea this test I believe can just be marked as platform specific to avoid this problem

jashook · 2020-05-15T20:08:49Z

/cc @dotnet/jit-contrib

sdmaclea · 2020-05-15T21:45:05Z

I just pushed the fix for the superpmi build scripts. I spent the better part of a day trying to figure out how to add a project target that depended on our other targets. I think this solution is clean.

BruceForstall · 2020-05-15T22:31:59Z

Is there a dev innerloop impact with this change? E.g., when are the wrapper scripts generated? During test build time? Some kind of on-platform pre-test-run time?

sdmaclea · 2020-05-15T23:08:47Z

@BruceForstall There is not intended to be a significant impact to normal dev experience.

The intention is for the scripts to be generated at the same time as the native binaries are built, because both are target specific.

So the intended change is for the scripts to build unless skipnative is selected.
Previously they would build unless skipmanaged was selected.

* Build scripts while copying native binaries Stop building test scripts during the managed build phase Build after copying the native targets Fixes issues relative to building scripts on OSX * Fix superpmi script generation * Cleanup issues.targets * Keep 6 failing IJW tests

* Build all managed coreclr tests on OSX in CI (#36253) Remove concept of targetGeneric and targetSpecific Add OSX corelcr managed tests builds to all pipes Remove all other platform coreclr managed test build Remove property `TestUnsupportedOutsideWindows` Rename conditional `DisableProjectBuild` to `CLRTestTargetUnsupported` Refactor tests to allow all managed tests to be built on OSX. Split managed tests which based on target properties conditionally: + `<Compile Include/>` + `<DefineConstant/>` This creates a separate test for each target configuration permutation. Add `<CLRTestTargetUnsupported/>` conditions to select these at build and/or run time Append split tests with `_Target*` to identify intended test target For tests which depend on target specific internal details od System.Private.Corlib, expose a dummy implementation for unsupported targets. Clean up Remove <TraitTags/> Remove <CLRTestNeedTarget/> Remove managedOSXBuild Remove managedTestBuildOsGroup Remove managedTestBuildOsSubGroup Use issues.target to disable expected failures Some tests are configured to run on a targets with specific charecteristics. Use issues.targets to conditionally disable tests not intended to run on all targets. * Setup pr and batch runs for dev/infrsatructure (#36218) (#36299) Co-authored-by: Jarret Shook <jashoo@microsoft.com> * Add alias CoreClrTestBuildHost (#36256) * Add alias CoreClrTestBuildHost Intended to allow easy switching of test build host between OSX_x64 & Linux_x64 depending on availablility and reliability. * Add missing issues.targets from #36253 (#36351) * Add missing issues.targets from #36253 PR #36253 enabled building all tests on OSX. It was intended to include this list of tests to disable on unsupported platforms. This was dropped as part of rebasing for dev/infra branch. Lack of CI on dev/infra allowed this to be missed. * Revise issues.targets * Add src/coreclr/tests/src/baseservices/typeequivalence/simple * Remove #if Windows from DllImportPathTest * Build scripts while copying native binaries (#36482) * Build scripts while copying native binaries Stop building test scripts during the managed build phase Build after copying the native targets Fixes issues relative to building scripts on OSX * Fix superpmi script generation * Cleanup issues.targets * Keep 6 failing IJW tests * Fix IJW when managed tests are built on OSX (#36711) * Fix IJW when managed tests are built on OSX - Refactor Interop.setting.targets to Directory.Build.{targets,props} - Remove include Interop.setting.targets from interop projects rely on SDK including Directory.Build.* automatically - Add new CopyInteropNativeRuntimeDependencies target for copying IJW dependencies * Add <Copy/> task - Delete obsolete instruction in Interop/ReadMe.md * Cleanup Readme.md * Optional stress dependencies & don't populate CORE_ROOT twice (#36851) Bruce noticed that we're downloading and installing stress runtime dependencies several times during build. I have added a new option to suppress this behavior in build-test and I fixed the invocations of build-test based on Bruce's analysis. I have also patched the build scripts to avoid populating CORE_ROOT in "copynativeonly" mode. Thanks Tomas Fixes: #36797 * Build test scripts whenever CopyNativeProjectBinaries builds (#37028) * Fix pri0 test build. * Clean pri1 test build (#37266) * Clean pri1 test build * Undo testing comment * Make sure weakreferencetest does not run on unix * Fixup WinRT proj * Fix path Co-authored-by: Steve MacLean <Steve.MacLean@microsoft.com> Co-authored-by: Tomáš Rylek <trylek@microsoft.com> Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Andy Ayers <andya@microsoft.com> Co-authored-by: Adeel Mujahid <adeelbm@outlook.com> Co-authored-by: Ganbarukamo41 <ganbarukamo@gmail.com> Co-authored-by: monojenkins <jo.shields+jenkins@xamarin.com> Co-authored-by: thaystg <thaystg@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Aaron Robinson <arobins@microsoft.com> Co-authored-by: Kevin Jones <kevin@vcsjones.com> Co-authored-by: Jeremy Barton <jbarton@microsoft.com> Co-authored-by: Egor Bogatov <egorbo@gmail.com> Co-authored-by: Simon Nattress <nattress@gmail.com> Co-authored-by: Aleksey Kliger (λgeek) <alklig@microsoft.com> Co-authored-by: Krzysztof Wicher <kwicher@microsoft.com> Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com> Co-authored-by: Carol Eidt <carol.eidt@microsoft.com> Co-authored-by: Shimmy <2716316+weitzhandler@users.noreply.github.com> Co-authored-by: Stephen Toub <stoub@microsoft.com> Co-authored-by: Layomi Akinrinade <laakinri@microsoft.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> Co-authored-by: Jeremy Koritzinsky <jekoritz@microsoft.com> Co-authored-by: Jose Perez Rodriguez <joperezr@microsoft.com> Co-authored-by: Alexander Chermyanin <flamencist@mail.ru> Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> Co-authored-by: Sergey Andreenko <seandree@microsoft.com> Co-authored-by: Nikola Milosavljevic <nikolam@microsoft.com> Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: David Wrighton <davidwr@microsoft.com> Co-authored-by: buyaa-n <bunamnan@microsoft.com> Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com> Co-authored-by: Vladimir Sadov <vsadov@microsoft.com> Co-authored-by: Leandro Pereira <leandro.pereira@microsoft.com> Co-authored-by: Swaroop Sridhar <swaroop.sridhar@microsoft.com> Co-authored-by: Marie Píchová <11718369+ManickaP@users.noreply.github.com> Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com> Co-authored-by: Manish Godse <61718172+mangod9@users.noreply.github.com> Co-authored-by: Bruce Forstall <brucefo@microsoft.com> Co-authored-by: Alexis Christoforides <alexis@thenull.net> Co-authored-by: BrzVlad <BrzVlad@users.noreply.github.com> Co-authored-by: Alfred Myers <git@alfredmyers.com> Co-authored-by: Nick Craver <nrcraver@gmail.com> Co-authored-by: Nathan Ricci <naricc@microsoft.com> Co-authored-by: Thays Grazia <thaystg@gmail.com> Co-authored-by: Yoh Deadfall <yoh.deadfall@hotmail.com> Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com> Co-authored-by: Jo Shields <directhex@apebox.org> Co-authored-by: Jan Vorlicek <janvorli@microsoft.com> Co-authored-by: Anirudh Agnihotry <anirudhagnihotry098@gmail.com> Co-authored-by: Roman Marusyk <Marusyk@users.noreply.github.com> Co-authored-by: Zoltan Varga <vargaz@gmail.com> Co-authored-by: Youssef Victor <31348972+Youssef1313@users.noreply.github.com> Co-authored-by: Ben Adams <thundercat@illyriad.co.uk> Co-authored-by: Marek Safar <marek.safar@gmail.com> Co-authored-by: Anton Lapounov <antonl@microsoft.com> Co-authored-by: AraHaan <15173749+AraHaan@users.noreply.github.com> Co-authored-by: Tom Deseyn <tom.deseyn@gmail.com> Co-authored-by: Tomas Weinfurt <tweinfurt@yahoo.com> Co-authored-by: Tomas Weinfurt <furt@Shining.local> Co-authored-by: David Cantu <dacantu@microsoft.com> Co-authored-by: Sung Yoon Whang <suwhang@microsoft.com> Co-authored-by: Levi Broderick <GrabYourPitchforks@users.noreply.github.com> Co-authored-by: vargaz <vargaz@users.noreply.github.com> Co-authored-by: David Mason <davmason@microsoft.com> Co-authored-by: UnityAlex <UnityAlex@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: Tanner Gooding <tagoo@outlook.com> Co-authored-by: Egor Chesakov <Egor.Chesakov@microsoft.com> Co-authored-by: radical <radical@users.noreply.github.com> Co-authored-by: Ivan Diaz Sanchez <ivdiazsa@microsoft.com> Co-authored-by: Gleb Balykov <g.balykov@samsung.com> Co-authored-by: Honza Rameš <ramejan@gmail.com> Co-authored-by: Eric StJohn <ericstj@microsoft.com> Co-authored-by: Ivan <ivanbuha@outlook.com> Co-authored-by: Ryan Lucia <rylucia@microsoft.com> Co-authored-by: SRV <roman.sakno@gmail.com> Co-authored-by: Clinton Ingram <clinton.ingram@outlook.com> Co-authored-by: Mikhail Pilin <ww898@users.noreply.github.com> Co-authored-by: Jan Kotas <jkotas@microsoft.com> Co-authored-by: w-flo <w-flo@users.noreply.github.com> Co-authored-by: Peter Sollich <petersol@microsoft.com> Co-authored-by: grendello <grendello@users.noreply.github.com> Co-authored-by: Johan Lorensson <lateralusx.github@gmail.com> Co-authored-by: Erhan Atesoglu <47518605+eanova@users.noreply.github.com> Co-authored-by: Matt Mitchell <mmitche@microsoft.com> Co-authored-by: Josh Schreuder <joshschreuder@gmail.com> Co-authored-by: Josh Schreuder <josh.schreuder@gmail.com> Co-authored-by: Andrew Au <andrewau@microsoft.com> Co-authored-by: Eugene Rozenfeld <erozen@microsoft.com> Co-authored-by: Vlad Brezae <brezaevlad@gmail.com> Co-authored-by: Kenneth Pouncey <kjpou@pt.lu> Co-authored-by: Kirill Frolov <k.frolov@samsung.com> Co-authored-by: Tomas Weinfurt <furt@DESKTOP-0J9757E.corp.microsoft.com>

sdmaclea added the area-Infrastructure-coreclr label May 15, 2020

sdmaclea added this to the 5.0 milestone May 15, 2020

sdmaclea requested review from jashook, jkoritzinsky and trylek May 15, 2020 01:30

sdmaclea self-assigned this May 15, 2020

trylek approved these changes May 15, 2020

View reviewed changes

sdmaclea force-pushed the infra/buildSciptWithNative branch from 19e30d2 to 70256c9 Compare May 15, 2020 14:35

Build scripts while copying native binaries

57bfd7f

Stop building test scripts during the managed build phase Build after copying the native targets Remove issues relative to building scripts on OSX

sdmaclea force-pushed the infra/buildSciptWithNative branch from 70256c9 to 57bfd7f Compare May 15, 2020 15:19

jashook approved these changes May 15, 2020

View reviewed changes

Fix superpmi script genration

7c67cf7

swaroop-sridhar approved these changes May 15, 2020

View reviewed changes

jaredpar mentioned this pull request May 15, 2020

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

Closed

sdmaclea added 2 commits May 15, 2020 21:01

Fix Windows

e81f06e

Keep 6 failing IJW tests

2403dad

sdmaclea mentioned this pull request May 16, 2020

Triage tests regressing by building managed tests on OSX #36404

Closed

sdmaclea merged commit 1d9846a into dotnet:dev/infrastructure May 16, 2020

sdmaclea mentioned this pull request May 16, 2020

Refactor Build All OSX tests to reduce redundant data entry. #35865

Closed

sdmaclea deleted the infra/buildSciptWithNative branch September 26, 2020 16:53

ghost locked as resolved and limited conversation to collaborators Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build scripts while copying native binaries #36482

Build scripts while copying native binaries #36482

Uh oh!

sdmaclea commented May 15, 2020

Uh oh!

trylek left a comment

Uh oh!

jashook left a comment

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

sdmaclea commented May 15, 2020

Uh oh!

BruceForstall commented May 15, 2020

Uh oh!

sdmaclea commented May 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Build scripts while copying native binaries #36482

Build scripts while copying native binaries #36482

Uh oh!

Conversation

sdmaclea commented May 15, 2020

Uh oh!

trylek left a comment

Choose a reason for hiding this comment

Uh oh!

jashook left a comment

Choose a reason for hiding this comment

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

jashook commented May 15, 2020

Uh oh!

sdmaclea commented May 15, 2020

Uh oh!

BruceForstall commented May 15, 2020

Uh oh!

sdmaclea commented May 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants