Rspec 2.14 + Ruby 2.1.0 + Simplecov 0.8.x doesn't pass along the correct exit status

[agrobbin]

Looks like Simplecov 0.8.x, Ruby 2.1.0, and Rspec 2.14 don't quite work together as you would expect. When running bundle exec rspec in that environment (with Rails 4.0.2), the exit code when tests fail is 0, when it should be 1. However, when I run them on Ruby 2.0.0(-p353) with Simplecov 0.8.x, it correctly exits 1. Even weirder, when running them on Ruby 2.1.0 and Simplecov 0.7.1, it also correctly exits 1.

[nanaya]

it seems to happen only when combined with rspec javascript test (capybara+poltergeist).

[nanaya]

I've created test case. Perhaps a bit too big though.

Travis report: https://travis-ci.org/edogawaconan/capytestruby21/builds/17829008

(pardon the repo name because initially I thought it's capybara bug)

[BanzaiMan]

Is this a dupe of #269?

[nanaya]

in my case, ruby 2.0 isn't affected though.

[BanzaiMan]

I dug a little deeper.

https://travis-ci.org/BanzaiMan/travis_production_test/jobs/18281155 tests this code base, running on Ruby 2.1.0 and using this Gemfile.lock. As you can see, the combination of Ruby 2.1.0, RSpec 2.14.1 and SimpleCov 0.8.2, is not sufficient to show the problem here.

I also added poltergeist for kicks (https://travis-ci.org/BanzaiMan/travis_production_test/jobs/18284015) but the problem does not manifest. Perhaps it is rspec-rails and/or Rails itself playing a part.

[BanzaiMan]

#269 (comment) implicates Rails 4.

[nanaya]

@BanzaiMan try enabling poltergeist by adding 'js: true' in the test case.

[BanzaiMan]

@edogawaconan I don't know how to exercise JavaScript, I guess. At any rate, it seems to me that developers from poltergeist should be involved to track this down.

[nanaya]

when I checked last time, same error also occured when using capybara-webkit driver.

[BanzaiMan]

@edogawaconan Could it be capybara, then? It's the common factor here, right?

[nanaya]

yeah, probably a combination of simplecov, capybara, and .

[colszowka]

So, I've finally also bumped into this at work. Mysteriously, it only manifests when I run the whole test suite, not for individual specs that fail (tried both with capybara acceptance spec and a model unit test). Digging in.

[colszowka]

Closed #269 in favor of this - I tried reverting the exit status changes from the 0.8 branch and also did a bit of binding.pry around the behaviour, I can not figure out what is going wrong, the behaviour is seemingly totally random :( This is happening with rspec 3 and rails 4 for me by the way. For now, I'm bailing and will downgrade to 0.7.1 myself...