Option to Run the Failed scripts only & to run selected scripts

[rjonam]

Motivation/Description of the PR

Currently Codecept don't have the option to rerun the failed scripts and to run the script with different pattern. This option of --rerun-tests will make it easier to run the failed tests alone and selected tests.

Applicable helpers:

• [ ✓ ] WebDriver

• Puppeteer

• Nightmare

• REST

• FileHelper

• Appium

• Protractor

• TestCafe

• Playwright

Applicable plugins:

• allure
• autoDelay
• autoLogin
• customLocator
• pauseOnFail
• puppeteerCoverage
• retryFailedStep
• screenshotOnFail
• selenoid
• stepByStepReport
• wdio

Type of change

• 🔥 Breaking changes
• [✓ ] 🚀 New functionality
• 🐛 Bug fix
• 📋 Documentation changes/updates
• ♨️ Hot fix
• 🔨 Markdown files fix - not related to source code
• 💅 Polish code

Checklist:

• Tests have been added
• Documentation has been added (Run npm run docs)
• [✓ ] Lint checking (Run npm run lint)
• [ ✓] Local tests are passed (Run npm test)

[rjonam]

Team, Can you please have look on this Pull request and advise. @DavertMik @Georgegriff

[DavertMik]

Thanks for the PR! Sorry for the delay in reviewing it.

Unfortunately, I can't merge it as it changes too many core classes for the sake of one feature.
Could you try to rewrite this feature as a plugin?

A plugin can do exactly what you need - hook up on events.
a plugin can also load tests from a file and put them into Mocha instance via MochaFactory

And the best part - plugin does not affect the overall behavior of codeceptjs so it can be merged safely. Could you try to rethink the implementation? After all you are on the right path and this seems like a good solution, but it needs to be transformed to not change the core classes of CodeceptJS

[DavertMik]

sorry, this looks like breaking the abstraction hierarchy.
this class represents only workers and should know nothing about failedTestRerun

[DavertMik]

this is not a correct place to put this logic
this listener only sets the exit code for failed tests

[rjonam]

Thanks a Lot for the review @DavertMik . I will check back and resubmit the code as requested.

[Georgegriff]

I'm not sure if this is what @DavertMik had in mind, but i've addressed this before with some hacky plugin code, which may not work in all cases, and may be dodgy with workers, might need a cross thread way of storing info about retries, potentially.

module.exports = function(config) {
  const event = codeceptjs.event;
  if (config.enabled) {
    const globalRetries =
      typeof config.retries === "undefined" || config.retries === ""
        ? 2
        : config.retries;
    // we dont retry on really slow tests
    const MINUTE = 1000 * 60;
    const maxTestDuration = parseInt(config.maxTestDuration) || 6 * MINUTE;

    event.dispatcher.on(event.test.started, function(test) {
      test.__retry_plugin_started = Date.now();
    });
    // to do make worker safe
    event.dispatcher.on(event.test.failed, function(test) {
      const duration = Date.now() - test.__retry_plugin_started;
      if (duration < maxTestDuration) {
        test.__retries = true
        test.retries(parseInt(globalRetries));
      }
    });
    event.dispatcher.on(event.test.finished, function(test) {
      if(test.__retries) {
        // we use allure  handy to tag  the retries accordingly
        const allure = codeceptjs.container.plugins('allure');
        if(allure) {
          allure.addLabel('tag', "@FLAKEY");
          allure.addLabel('package', "Flakey Tests");
        }

      }

    });
  }
};

[rjonam]

Requested changes are implemented in the below PR :

#2809.

Taking back this PR.