Skip to content

Update pytest framework#1493

Merged
wpbonelli merged 1 commit into
modflowpy:developfrom
wpbonelli:tests
Aug 10, 2022
Merged

Update pytest framework#1493
wpbonelli merged 1 commit into
modflowpy:developfrom
wpbonelli:tests

Conversation

@wpbonelli
Copy link
Copy Markdown
Member

@wpbonelli wpbonelli commented Aug 10, 2022

Pytest framework changes include:

  • use pytest-benchmark's builtin profiling capability instead of manual implementation
  • update DEVELOPER.md to reflect above
  • remove requires_exe("mf6") from test_mf6.py tests that don't run models/simulations
  • add @requires_spatial_reference marker to conftest.py (for tests depending on spatialreference.org)
  • split test_export.py::test_polygon_from_ij into network-bound and non-network-bound cases (and use marker above)
  • try both importlib.import_module and pkg_resources.get_distribution in @requires_pkg marker
  • mark test_lgr.py::test_simple_lgr_model_from_scratch as flaky (occasional forrtl error (65): floating invalid)
  • add comments to flaky tests with links to potentially similar issues
  • remove unneeded markers from pytest.ini (profile, benchmark)
  • match profiling/benchmarking test files in pytest.ini
    • profile_*.py
    • benchmark_*.py
    • *_profile*.py
    • *_benchmark*.py
  • mark get-modflow script tests as flaky again due to internal github API egress issues
  • various tidying/cleanup

CI changes include:

  • add timeouts to CI jobs (10min for build, lint, & smoke, 45min for test, 90min for daily jobs)
  • fix benchmark artifact names (previously Windows benchmark artifacts were saved as benchmark--<python version> because matrix.os was incorrectly used with no job matrix, now fixed with runner.os so that all benchmark artifacts are named as benchmark-<platform>-<python version>)
  • cache benchmark results in daily CI and compare with prior runs (a performance regression failure condition is introduced such that if any benchmark sees an avg increase > 25%, the benchmark job is failed)

@codecov

This comment was marked as spam.

* use pytest-benchmark's builtin profiling capability instead of manual implementation
* remove requires_exe(mf6) from test_mf6.py tests that don't run models/simulations
* add @requires_spatial_reference marker to conftest.py (for tests depending on spatialreference.org)
* try both importlib.import_module and pkg_resources.get_distribution in @requires_pkg marker
* mark test_lgr.py::test_simple_lgr_model_from_scratch as flaky (occasional forrtl error (65): floating invalid)
* split test_export.py::test_polygon_from_ij into network-bound and non-network-bound cases
* add comments to flaky tests with links to potentially similar issues
* add timeouts to CI jobs (10min for build, lint, & smoke, 45min for test, 90min for daily jobs)
* remove unneeded markers from pytest.ini
* match profiling/benchmarking test files in pytest.ini
* mark get-modflow tests as flaky (modflowpy#1489 (comment))
* cache benchmark results in daily CI and compare with prior runs
* various tidying/cleanup
@wpbonelli wpbonelli marked this pull request as ready for review August 10, 2022 22:52
@wpbonelli wpbonelli merged commit 3b8eedb into modflowpy:develop Aug 10, 2022
wpbonelli added a commit to wpbonelli/flopy that referenced this pull request Aug 10, 2022
* use pytest-benchmark's builtin profiling capability instead of manual implementation
* remove requires_exe(mf6) from test_mf6.py tests that don't run models/simulations
* add @requires_spatial_reference marker to conftest.py (for tests depending on spatialreference.org)
* try both importlib.import_module and pkg_resources.get_distribution in @requires_pkg marker
* mark test_lgr.py::test_simple_lgr_model_from_scratch as flaky (occasional forrtl error (65): floating invalid)
* split test_export.py::test_polygon_from_ij into network-bound and non-network-bound cases
* add comments to flaky tests with links to potentially similar issues
* add timeouts to CI jobs (10min for build, lint, & smoke, 45min for test, 90min for daily jobs)
* remove unneeded markers from pytest.ini
* match profiling/benchmarking test files in pytest.ini
* mark get-modflow tests as flaky (modflowpy#1489 (comment))
* cache benchmark results in daily CI and compare with prior runs
* various tidying/cleanup
@wpbonelli wpbonelli deleted the tests branch August 11, 2022 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant