Enable Linux tests on VSTS

[natemcmaster]

There are issues with VSTS Linux test agents which cause test runs to hang indefinitely. When this is fixed, we should re-enable Linux tests in VSTS builds.

cref #2395
Internal bug: https://dotnet.visualstudio.com/internal/_workitems/edit/28

[mkArtakMSFT]

@natemcmaster, is this just a tracking bug?

[natemcmaster]

There is a small action item here: the yml files in this repo need to change to enable tests, but doing so is blocked by a bug in VSTS. I think the best course of action is to check with the VSTS team during our sync ups to make sure they are aware that this is a blocker for us getting off Jenkins.

[mkArtakMSFT]

Thanks @natemcmaster. Assigning this to you to track the blocker and make the necessary change then.

[natemcmaster]

@mkArtakMSFT I had a conversation with the VSTS guys. As far as we can tell, this is a symptom of tests which create orphan processes. I know this repo does some tests with rzc and msbuild which may keep alive some long-running background processes. Do we know if there are any background processes spawned by tests which are not cleaned up?

cc @ajaybhargavb

[natemcmaster]

I narrowed it down to one test project. Something in Microsoft.AspNetCore.Razor.Design.Test is causing the VSTS build to hang in Linux.

[natemcmaster]

I started digging through diagnostic dumps from VSTS to figure out which exact test case is getting stuck. The diagnostic dumps are not exactly easy to read, but it appears as if all test cases have completed, and yet the test process is still deadlocked. This could mean there is a bug in VSTest, or it could mean something in test cleanup, such as test fixture disposal, is hanging.

vstest-diagnostic-logs.zip

In the meantime, I discovered that the tests pass on agents from the "DotNetCore-Linux" pool, which are VMs managed by DDIT, and the tests deadlock on agents from the "Hosted Linux Preview" pool, which are VMs managed by VSTS with the agent running from inside a Docker container. From what I can tell, something about the "Hosted Linux Preview" pool is causing rzc.dll or the tests clients to deadlock (or get stuck in a long waiting loop). It could be something to do with how named pipes or networking runs inside a Docker container, something with Unix file permisssions, or something else altogether. I don't have memory dumps so I can say which it is, nor can I tell you the cause, but that's the next step in investigating this.

Or we can just switch agent pools and call it good. Given other bugs we've seen in razor build client/server, such as #2406, I think it's worth investigating a little more, but I'll leave that call up to you @mkArtakMSFT.

[ajaybhargavb]

Thanks for investigating it @natemcmaster.

Do we know if there are any background processes spawned by tests which are not cleaned up?

We do spin up other rzc processes as part of the test but they are always cleaned up unless the test process is killed abruptly.

This could mean there is a bug in VSTest, or it could mean something in test cleanup, such as test fixture disposal, is hanging.

We have a 1 minute timeout to prevent indefinite hangs from happening during test fixture disposal. Plus, the rzc processes that were not cleaned up by the test fixture has an idle timeout of 10 minutes so it shouldn't cause an indefinite hang. I would be surprised if the test fixture dispose is what is causing the hang.