unit tests with expected execution time

[wyli]

Is your feature request related to a problem? Please describe.
in many cases it is necessary to check whether a unit test could finish within some expected execution time, in addition to the correctness checks of the outputs.

Describe the solution you'd like
this ticket looks for a generic timer wrapper for the unit testing module, so that it automatically fails a test (early) if the test takes longer than the expected execution time when running it on the monai CI/CD servers.

would also be great to have a global flag to switch this feature on/off so that the developers can run the tests locally without an execution time constraint

[Nic-Ma]

Hi @IsaacYangSLA , as we discussed yesterday, this can be a next step optimization for our CI/CD.
Thanks.

[bhashemian]

@Nic-Ma, to clarify the requirements for this feature, do you want to set timeout for

• each individual test function, or
• each test case, or
• the whole test suite?

[Nic-Ma]

Hi @behxyz ,

Thanks for your help here. This is related to #927.
Maybe we can implement by steps?

1. Add testing time for integration tests.
2. Add testing time for the overall time of every test suite.
3. Add testing time for each test case.

@wyli can also help share some comments.

Thanks.