Docs: SauceLabs Unreliable?

[geekdave]

From https://theintern.github.io/intern/#hosted-selenium :

As of February 2015, Sauce Labs VMs are slow and unreliable. We recommend using a different hosting provider until this issue is resolved.

We are currently using SauceLabs, and while we do notice that tests run more slowly and have occasional flakey behavior, we had chalked this up to "the price we pay" of using remote browser-as-a-service providers in general.

I was wondering if you had any apples-to-apples data comparing the performance and reliability of different providers? This would be a big help in understanding whether it would make sense for us to switch, or if we'd just run into the same issues elsewhere.

Thanks!

[csnover]

I used to have quantitative data but I can’t find it at the moment. Here’s what I can say, just anecdotally:

1. Since switching to BrowserStack for Intern’s self-tests we’ve had 0 false positives. When we were using Sauce Labs, earlier in the year, we would normally have to rerun CI 2–3 times before it would not have any false positives due to random service failures in the middle of testing.
2. We were never able to use Sauce Labs for CI with Dojo 1 because it was so slow everything would eventually time out. BrowserStack did not have this problem.
3. Team members I’ve polled all feel that BrowserStack is at least 2 times faster than Sauce Labs, and perhaps as much as 4–5x faster.

The good news is that outside of having to get an account with another provider, if you are using Intern switching to another provider is simply a matter of changing the tunnel configuration property and updating your access keys.