Best practices for cross-version testing

[sharwell]

What is the recommended practice for testing an extension in Visual Studio 2010 or 2012, using Visual Studio 2013 or 2015 to launch the tests? I'm looking for guidance regarding automation of the following items:

1. Deploying the extension to the correct version of Visual Studio
2. Selecting the .testsettings file which matches that environment
3. Running tests outside the debugger
4. Running tests in the debugger

[zooba]

We haven't developed any best practices, so feel free to experiment to find what works well. Here are some suggestions for those items:

1. If you're using a VSIX, then you'll want to run a command like vsixinstaller.exe /s Pro /v 10.0 myext.vsix. MSIs will need to expose their own customization (or, like ours, be VS version specific).
2. For automation, we use mstest.exe /testSettings:vs2010.testsettings to launch tests. The script that does the launching has a configuration file that is manually filled out for a particular machine/image. mstest.exe is available in VS Community editions, so we install a recent version of that alongside the older version for the sole purpose of running tests. (Also for running tests in Express editions, which don't include mstest.exe.)
3. As above.
4. Tests in the debugger only make any sense when launching from VS, due to the magical trickery that actually does the attach. Generally we'd recommend launching tests individually under the debugger. There's no reason you can't launch multiple tests, but that's not how we use it.

[sharwell]

Deploy to Visual Studio 2010+ within a Visual Studio 2013+ build:

• Add VSSDK.IDE.Deploy packages for Visual Studio 2010+ tunnelvisionlabs/vsxdeps#23
• https://www.nuget.org/packages?q=vssdk.ide.deploy

[OmerRaviv]

Note that when incorporating these test in Continuous Integration, you may want to look into VsixUtil as a replacement for the vsixinstaller.exe tool @zooba mentioned above.

vsixinstaller.exe is asynchronous which makes it unsuitable for scripting.