CI updates/fixes, test updates

[wpbonelli]

This PR:

• fixes Codecov configuration
• fixes PathlineFile.get_destination_pathline_data() performance #1479
• fixes Intermittent pytest-xdist failures on Mac CI runners #1491
• fixes/updates CI workflows
• corrects ci.yml to commit.yml
• adds/updates some tests
• groups scripts in scripts

Codecov configuration

Uploads coverage reports on pull_request as well as push events to fix inaccurate codecov bot comments on PRs.

Also:

• don't measure & upload coverage for smoke tests, regression & example tests, and benchmarks
  • smoke tests and benchmarks are subsets of the standard test suite
  • example scripts/notebooks are not included in coverage measurements (only the flopy module is)
  • excluding regression tests does not seem to change coverage appreciably (as seems appropriate — ideally the standard test suite should exercise everything?) and I can plan to add a few cases to make up any omissions
• decrease coverage precision from 3 to 1 decimal place to avoid small/spurious deltas
• update to codecov-action@v3

PathlineFile performance

Fix #1479 by sorting once in the ctor instead of each call to parent's _ModpathSeries method get_data() (occurs in a loop in _MoadpathSeries.get_destination_data()).

Previous performance:

Name (time in ms)	Min	Max	Mean	StdDev	Median	IQR	Outliers	OPS	Rounds	Iterations
test_get_destination_pathline_data[well-backward]	26,589.8808 (>1000.0)	26,652.4284 (>1000.0)	26,623.4600 (>1000.0)	25.6051 (831.50)	26,616.2695 (>1000.0)	40.1569 (>1000.0)	2;0	0.0376 (0.00)	5	1
test_get_destination_pathline_data[well-forward]	40,028.3296 (>1000.0)	40,195.3408 (>1000.0)	40,104.5977 (>1000.0)	70.1838 (>1000.0)	40,077.0743 (>1000.0)	115.3085 (>1000.0)	2;0	0.0249 (0.00)	5	1
test_get_destination_pathline_data[river-forward]	61,112.0500 (>1000.0)	61,440.9841 (>1000.0)	61,247.5391 (>1000.0)	132.9419 (>1000.0)	61,200.5654 (>1000.0)	201.5238 (>1000.0)	2;0	0.0163 (0.00)	5	1
test_get_destination_pathline_data[river-backward]	568,730.0336 (>1000.0)	575,857.0278 (>1000.0)	571,281.7784 (>1000.0)	2,803.3254 (>1000.0)	571,121.0215 (>1000.0)	3,365.4625 (>1000.0)	1;0	0.0018 (0.00)	5	1

New benchmarks:

Name (time in ms)	Min	Max	Mean	StdDev	Median	IQR	Outliers	OPS	Rounds	Iterations
test_get_destination_pathline_data[well-backward]	128.9273 (120.76)	207.2077 (111.50)	153.9030 (136.77)	28.1448	(463.77)	145.9356 (130.04)	21.2064 (545.76)	1;1	6.4976 (0.01)	6
test_get_destination_pathline_data[well-forward]	171.6738 (160.80)	252.1039 (135.66)	204.7025 (181.91)	39.9782	(658.77)	181.4749 (161.71)	72.8909 (>1000.0)	1;0	4.8851 (0.01)	5
test_get_destination_pathline_data[river-forward]	238.6348 (223.52)	356.1929 (191.67)	293.0763 (260.45)	51.4346	(847.55)	269.4483 (240.10)	87.1385 (>1000.0)	2;0	3.4121 (0.00)	5
test_get_destination_pathline_data[river-backward]	2,282.6307 (>1000.0)	2,325.8112 (>1000.0)	2,303.5692 (>1000.0)	17.5329	(288.91)	2,298.4413 (>1000.0)	27.9918 (720.38)	2;0	0.4341 (0.00)	5

The operation completes in a fraction of the previous time. I don't think moving the sort() will cause problems as _ModpathSeries and children don't expose methods to update internal data after it's set in the constructor.

CI updates

Moves benchmarking, example scripts & notebook tests, and regression tests into their own workflows to allow for independent testing and scheduling. (However all three still run on the same daily schedule as before).

• updates/fixes benchmarking workflow
• addresses macos CI flakiness in Intermittent pytest-xdist failures on Mac CI runners #1491
• refactor modflow executable caching

Benchmarks

Previously benchmark comparisons (with pytest --benchmark-compare) were failing due to an issue in the CI caching configuration. Now benchmarks are no longer saved in the cache, only as artifacts, and artifacts are postprocessed after benchmark jobs complete. The script scripts/process_benchmarks.py accepts 2 arguments, the input and output directory, and:

• finds JSON files created by pytest with --benchmark-autosave or --benchmark-json=<file>, searching recursively in the input directory
• collects results in a dataframe and saves it to CSV
• creates and saves plots for each benchmarked test case

For instance, to look in the default benchmark save location autotest/.benchmarks and save results in the same place:

python scripts/process_benchmarks.py autotest/.benchmarks autotest/.benchmarks

Individual artifacts are still saved for each combination of system and python version. An aggregate artifact is created by process_benchmarks.py in a job that runs after all benchmark jobs finish (example here).

Macos/matplotlib errors

Tentatively fixes #1491 by using the non-interactive agg matplotlib backend on macos CI. Previously the macos backend was selected by default. This involves a FigureManager, which wraps a Cocoa NSWindow. Maybe something in Cocoa doesn't like being invoked from multiple processes.

Modflow exe caching

Cache modflow executables in commit-triggered and daily workflows with new composite action .github/actions/cache_exes. This installs & caches modflow executables, checking the exe repo's latest release for a code.json asset and hashing its contents if found to determine whether to invalidate the cache. Required inputs are:

• path: the path to restore the cache to
• github_token: the GitHub access token to authenticate API requests

Linux/MacOS

To cache Modflow executables in .local/bin in the runner's home directory, add a step like the following:

      - name: Install Modflow executables
        uses: ./.github/actions/cache_exes
        with:
          path: ~/.local/bin
          github_token: ${{ secrets.GITHUB_TOKEN }}

Windows

An equivalent step for Windows:

      - name: Install Modflow executables
        uses: ./.github/actions/cache_exes
        with:
          path: C:\Users\runneradmin\.local\bin
          github_token: ${{ secrets.GITHUB_TOKEN }}

The home directory path is explicitly provided here — although tilde expansion works in recent versions of Powershell, it does not seem to work on Windows runners.

Tests

• skip ex-gwtgwt-mt3dms-p10 mf6 example test (fix(gwtgwt): allow exchange to work without XT3D active for dispersion MODFLOW-ORG/modflow6#1008)
• give test_mt3d.py::test_mfnwt_CrnkNic more retries due to fortran errors
• remove plt.show() usage in test_sfr.py (plot tests can check attributes of ax)
• tidy up and add some explanatory comments to conftest.py and regression/conftest.py
• add test case that PathlineFile sorts on initialization

Scripts

Rename release folder to scripts, move pull_request_prepare.py from autotest to scripts, and add new scripts for postprocessing benchmarks.

[codecov]

Codecov Report

Merging #1495 (02bec83) into develop (c9e6f61) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 02bec83 differs from pull request most recent head 2b08442. Consider uploading reports for the commit 2b08442 to get more accurate results

@@           Coverage Diff            @@
##           develop    #1495   +/-   ##
========================================
  Coverage    72.47%   72.48%           
========================================
  Files          250      250           
  Lines        54039    54039           
========================================
+ Hits         39166    39170    +4     
+ Misses       14873    14869    -4     

Impacted Files	Coverage Δ
flopy/utils/modpathfile.py	85.96% <100.00%> (ø)
flopy/plot/crosssection.py	62.42% <0.00%> (-1.56%)	⬇️
flopy/mf6/modflow/mfsimulation.py	60.46% <0.00%> (+1.70%)	⬆️

[jdhughes-dev]

What do the benchmark plot look like?

[wpbonelli]

@jdhughes-usgs it makes one like this for each benchmarked test

full example here

[christianlangevin]

This is looking really nice, Wes. Is this ready to merge in?

[wpbonelli]

@langevin-usgs yes, I think it's ready.

[jdhughes-dev]

How will the cache be updated when there is a new release on the modflow-executables repo?

Here and elsewhere.

code.json in the zip files has the version numbers. We could add the code.json to the release and maybe that could be hashed so that the latest will always be used.

[wpbonelli]

Hashing code.json should work if it lives in the executables repo. Flopy CI would just need to check it out. The code.json file is currently only in the release archives though.

Another option is invalidating the cache via API per MODFLOW-ORG/executables#7.

The former would require changes to this PR, latter some changes to the executables repo PR.

[jdhughes-dev]

I was thinking of adding it as a release asset but could add it as a file to the repo. If it is a release asset, it could be pulled using the GitHub API before the MODFLOW executable step in the workflows. Adding it as a file in the repo would allow us to tag the commit and tie the release to the tag. I think we would have to make a new release after adding code.json to the repo to test this option.

[wpbonelli]

The release asset option might be simplest. I'll update this PR to look for code.json in the executables repo release assets & use its hash in the cache key if it finds one, otherwise just using modflow-exes-<system> as above. That should allow caching to work now as well as if/when we do include code.json as a release asset.

Marked as draft again, will re-mark for review when done.

[wpbonelli]

Added cache_exes composite action to .github/actions. This action packages the modflow exe install so workflows can all reuse it. The action checks the latest executables release for a code.json file, and if found, hashes it to determine if there's been a new release. (This file is not yet a release asset, and the cache won't be invalidated until it becomes one.)

The cache_exes action works on ubuntu and mac runners & there's a functionally identical Windows one
The cache_exes action is cross-platform and works on Windows, Linux and Mac runners.

A further option is to move the action to its own repository so modflow6 and other repos could use them.