-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Describe the bug
Failures of network dependent iotools tests due to failures of the API server receiving the request affect consistency and confidence in our CI test results.
To Reproduce
- This is difficult to reproduce since API failures are sporadic.
- An example of our CI failure can be found here.
Expected behavior
There are two important functionalities that are desirable:
-
Ideally, a failure due to API server failure would be repeated with the hope that communication can be established.
-
In the event that communication cannot be established during the test, it should be conditionally skipped with a warning that the test wasn't properly run.
Additional context
This was originally identified by @wholmgren in #897 (comment)_
Proposed solution
Implement @pytest.mark.flaky()for all remote API dependent tests to reattempt if API is temporarily unreachable. This is already done for tests in test_psm3.py decorated with @pytest.mark.remote_data.
I'm going to look for a pytest functionality to enable a conditional skip of the test in the event the API server is completely unavailable during the test duration. If you have a suggestion on how to accomplish this please let me know.
@pytest.mark.skipif() should work well for this application but the trick will be the implementation for each network dependent API.