Add option to ignore wait_cond() result

[ligurio]

Imagine we have open tickets that blocks passing wait_cond() statement in a test.
In such cases we can add some option to wait_cond() that inverts old wait_cond()behavior: when condition in wait_cond() is False it sets overall test status XFail with appropriate reason of XFail (say number of ticket 'GH-3301') and when condition in wait_cond() is True it sets overall test status to XPass and test failed. When test has status XPass we should remove option in wait_cond().

NOTE: Behavior described above will work only for non-flaky tests. Otherwise it will be a pain.

[Totktonada]

I would generalize it to ability to set an xfail or xpass status from a test and support of those statuses in test-run.

The test_run:wait_cond() function is just to spin and check a function result. It is NOT something like test:ok() in the tap build-in module, t.assert() in the luatest module, self.assertTrue() in unitest (Python) or assert in pytest. Sometimes it is used to wait a particular state and the 'real' 'assertion' is afterwards.

Anyway, you always can use a wait_cond() result to set a test status using a separate function(s):

res = wait_cond(function() <...> end)
test_run:xfail_if(not res)
test_run:xpass_if(res)

TBD: The Lua API (naming, in particular).

TBD: How to combine this result with an exit code / a result file (mis)compare / a TAP13 result.

In fact, for a 'core = tarantool' test, where each expression is compared against an expected result, the test code above is equivalent to:

res = wait_cond(function() <...> end)
not not res -- xfail (expected false for a while)

The only different thing is the status in the statistics after the testing. So we can just statically list such tests in suite.ini to show amount of xfails after each testing. Or track them in the issue tracker using a special label. Or just grep by xfail.

So the question is what would be simple, intuitive and convenient.

[ligurio]

I would agree that it is better too keep wait_cond() as is, but add feature to set overall status from tests with additional status like xfail(). You don't need to add a function xpass() because that status used when you "expect fail" (XFAIL), but got a PASS. IN that case you should remove logic with XFAIL and test will PASS.

To continue to generalize I would be replace condition-like function:

test_run:xfail_if(not res)

to a simple one:

test_run:xfail(reason)

One can combine test_run:xfail(reason) with if ... then ... to get a behaviour of xfail_if().