Skip to content

iotools tests fail when unable to connect to remote API #917

@CameronTStark

Description

@CameronTStark

Describe the bug
Failures of network dependent iotools tests due to failures of the API server receiving the request affect consistency and confidence in our CI test results.

To Reproduce

  • This is difficult to reproduce since API failures are sporadic.
  • An example of our CI failure can be found here.

Expected behavior
There are two important functionalities that are desirable:

  1. Ideally, a failure due to API server failure would be repeated with the hope that communication can be established.

  2. In the event that communication cannot be established during the test, it should be conditionally skipped with a warning that the test wasn't properly run.

Additional context

This was originally identified by @wholmgren in #897 (comment)_

Proposed solution
Implement @pytest.mark.flaky()for all remote API dependent tests to reattempt if API is temporarily unreachable. This is already done for tests in test_psm3.py decorated with @pytest.mark.remote_data.

I'm going to look for a pytest functionality to enable a conditional skip of the test in the event the API server is completely unavailable during the test duration. If you have a suggestion on how to accomplish this please let me know.
@pytest.mark.skipif() should work well for this application but the trick will be the implementation for each network dependent API.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions