Linux builds have increased in time beyond 1 hour average; needs investigation

Opened on behalf of @jcagme

Warnings: 2
Status Message: failed

Build : Master - 20180227.02 (Product Build)
Failing configurations:

• RedHat6
  • x64
• Linux
  • x64
    Mission Control Build Info

In the last 4 days at least 3 build have failed since Linux and/or RH6 legs have timed out

[joshfree]

cc: @ianhays @MattGal

[MattGal]

Nothing has changed about the VMs used for build here and there's no salient long-running task in the log; perhaps we should check in with @joperezr who made changes recently that slowed down builds (for instance we recently moved our OSX builds from 60 to 120 minute timeout, for this reason.)

[jcagme]

3 more hits:
https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build?_a=summary&buildId=1426668
https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build?_a=summary&buildId=1426669
https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build?_a=summary&buildId=1426079

[MattGal]

I added the change from 60-90 mins as was already done in the crossbuild linux as part of this PR:
https://github.com/dotnet/corefx/pull/27555/files

It seems silly to fail builds just because something caused a perf regression, I favor increase timeout then fix the problem.

[ianhays]

It seems silly to fail builds just because something caused a perf regression, I favor increase timeout then fix the problem.

I agree. Especially since the perf runner is deciding how long to run the tests and there isn't a process-wide max time that you can enforce (as far as I'm aware). I recently broke a bunch of builds because I added too many perf tests to a single test assembly and the perf harness decided to run those tests enough times that the total execution time was consistently above 60 minutes.

We can increase the timeout, but this is probably more of an issue with the perf harness than with the CI.

[MattGal]

@ianhays to clarify I meant more of a "build perf" regression, there's no product perf one I know of.

After further thought, I do note that something we did that correlates to the timeouts increasing was patching all our linux agents for Meltdown/Spectre, and if you read about those patches they're pretty much guaranteed to give a 20-30% perf reduction dependent on the type of code running; this can explain the issue.

Moving the builds to 90 mins for now is the simplest cleanest fix while folks can debate about why things are slower.

[ahsonkhan]

It is failing consistently now rather than random/intermittent - https://devdiv.visualstudio.com/DevDiv/_build/index?buildId=1427359&_a=summary

[MattGal]

@ahsonkhan I have a PR up to work around this: dotnet/corefx#27555

Given the average build product / build test part of this build is already over 40 minutes and disabling the "Cleanup agent" task which takes as long as it feels like is not presently an option, 60 minutes is just cutting it way too close. If I can get an LGTM on my PR we can try again...

[weshaggard]

@joperezr I still think the overall slow-down of the build is caused by the msbuild/toolset change you made so I'm assigning this to you to track that investigation.

[joperezr]

I agree, this must be related to my changes. I'm taking a look at this next.

[jcagme]

One more hit: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build?_a=summary&buildId=1445975