CodeceptJS exits with status 0 when webdriverio target is not running

[attila]

Thanks for this project, I think it contains a lot of great ideas.

I used the WebdriverIO helper to run a few tests via both selenium standalone and phantomjs.
While working with it I noticed that codeceptjs does not return with a non-zero exit status when it cannot connect to selenium, i.e it skips executing steps and exits with status 0 as if everything was OK.

I think it should return with a non-zero exit status so for example CI environments can mark it as a failure.

codecept.json

{
  "tests": "./*.test.js",
  "timeout": 10000,
  "output": "./output",
  "helpers": {
    "WebDriverIO": {
      "url": "http://127.0.0.1:3011",
      "browser": "firefox",
      "host": "127.0.0.1",
      "port": 4444
    }
  },
  "bootstrap": false,
  "mocha": {},
  "name": "test"
}

example.test.js

Feature('Example');

Scenario('test', (I) => {
  I.amOnPage('/');
  I.see('Hello');
});

Output when selenium is running on localhost:4444:

$ codeceptjs run --steps
CodeceptJS v0.3.4
Test root is assumed to be /Users/attilab/tmp/example/test

Example --
 test
 • I am on page "/"
 • I see "Hello"
 ✓ OK in 2027ms


  OK  | 1 passed   // 2s

Exit status 0 here.

Output when selenium is NOT running anywhere:

$ codeceptjs run --steps
CodeceptJS v0.3.4
Test root is assumed to be /Users/attilab/tmp/example/test

Example --
 test

Exit status 0 here as well.

[attila]

It is also worth noting that this is not a duplicate of #98
When a step fails I do see a non-zero exit status.

[jgarber]

This has bitten us too and we haven't found a way around it. We can VNC into Circle CI and see the initial page load successfully in Firefox, but then it just bails out with exit code 0. Running locally, it finishes the test run and returns the correct exit code. Tried both selenium-standalone and phantomjs.

[byrnedo]

This is biting us too.

[aldarund]

+1

[joelgrimberg]

I am a codeceptjs noob; just starting writing codeceptjs.
I started the installation Guide; If I start the first script (I.amOnPage) I get no steps, and a green 'ok' line with 0 passed. Where /how do I check if codeceptjs has reached the targeted browser? I cannot find some sort of debugging function. (I don't even see the 'Feature('My First Test');' function being put on screen.

[ernoaapa]

Yeah same problem if you have something wrong in the bootstrap script.
Create boostrap.js:

require('this-is-invalid-require and should fail');

Now set the bootstrap configuration to point to above script.
Expected codeceptjs run --steps to fail with error code, but exits with 0 code.

[chasen]

+1 for returning non 0 status code for these failures.

[vkramskikh]

Also, if bootstrap's callback is called with an error, the exit code is also 0.

[APshenkin]

@vkramskikh @ernoaapa Is it still valid? I tested bootstrap file with mistake and get correct exit code?
If anybody has this error still, please create new issue with detailed information and code examples

[APshenkin]

Close due inactivity