Do not allow Fallback Mode when the CI node was retried to avoid running the wrong set of tests

[ArturT]

Problem

You run tests in Queue Mode with flag KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true or Regular Mode with the flag KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT=true (default).

How to reproduce race condition:

• You run 1st CI build and one of parallel CI nodes tests failed.
• You retry failed CI node. We expect knapsack_pro will run the same set of tests on the CI node during retry as it was recorded for the first CI node run. Let's assume knapsack_pro could not connect to Knapsack Pro API. In such a case, the Fallback Mode is started and a different set of tests are run on the CI node. This can lead to false-positive because different tests can be green and the retried CI node did not run failing tests at all.

Solution

knapsack_pro should not allow running tests in Fallback Mode in the case when CI node was retried.

Some of CI providers expose ENV variable that knapsack_pro can read so it will know that Fallback Mode is not allowed.

This PR adds support for Buildkite CI env var BUILDKITE_RETRY_COUNT that has value 1 when CI node was retried.

How to fix your CI config

• If you use Buildkite CI then knapsack_pro won't allow for Fallback Mode when the parallel job was restarted. No action required on your side.
• If you use other CI provider that allows retrying failed CI node you should set KNAPSACK_PRO_CI_NODE_RETRY_COUNT=1 when failed CI node is restarted.

If you can't do it then please don't use retrying single failed CI node. Instead, you should retry the whole CI build to be sure that the whole test suite is passing green. In the case of Fallback Mode starting on any parallel CI nodes the knapsack_pro ensures each test file is executed at least once so you can be sure the whole test suite is green (passing).

If you really need to use retry failed CI node then you can disable fallback mode completely KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Thanks to that tests won't run when a connection with Knapsack Pro API is lost. Instead, exception will be raised.