Skip to content

Conversation

@rburghol
Copy link
Contributor

@rburghol rburghol commented Jan 30, 2026

Ready to merge @PaulDudaRESPEC -- All fixes with respect to the pandas 3.0 transition are now passing with full coverage from python 3.9-12. There was one lingering issue with the transform() function that just needed me to understand it a little more fully to insure that the calculations would be correct. Thus, SOLRAD was the only timeseries that was not identical (in test10), but that resulted in many changes to the PWATER simulation, which then cascaded into differences in HYDR, and other NPS constituents. This is also preliminarily valid for pandas 3.0.0, that is, code the will execute without error under pandas 3.0.0 and and this same code produces identical results to hspf in pandas < 3.0.0. So this is a good step forward IMO.

Under pandas 3.0.0 the test coverage fails -- an error related to pandas, I suspect that there may be deprecated pandas code in RegressTest, and if so, likely it can be resolved, and then we can see if the test coverage is actually valid. That will be tracked in the formal pandas 3.0.0 issue #209

  • Adds a close() method to the custom local HDF5 class
  • Invokes this method a the end of the run() function to insure that the hdf5 is handled cleanly -- will at last eliminate a warning.
  • Closing the hdf5 at the end of the run eliminates the pytest error: ValueError: The file '/opt/model/HSPsquared/tests/test10/HSPFresults/test10.h5' is already opened, but not in read-only mode (as requested). in pandas >= 3.0.0
  • Integrates the changes to utilities.py and main.py from a brief comment to trigger git #207
  • Enhances the changes from a brief comment to trigger git #207 to do all timeseries transformations correctly.
  • Fixes the value comparison problems in python < 3.11
  • Adds a script cmd_regression.py that creates a command prompt friendly tester, as well as leveraging the intermediate products of the RegressTest class for simulation debugging.
  • Updates the class RegressTest in the file tests/convert/regression_base.py to have a silent mode to be a bit more screen friendly (@austinorr this is False by default so no change in previous behavior is made).
  • The model has run successfully under pandas 3.0.0, and I have manually verified the model flow quartiles are identical with hspf. However, this should just be considered a first step in pandas 3.0.0 update.
    • This does NOT fix the pandas-3.0.0 error pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match)

THIS IS NOT YET Pandas 3.0 safe. The change to pandas3 are substantial. Most specifically, there have been some things removed from classes, such as the to_timedelta() method of the .delta property of a pandas.timerseries.freq, but most importantly, pandas no longer allows Timedelta to have things like Y, or M me valid intervals, since they are non-ambiguous. Thus, I am not super optimistic about the IF-THEN constructs in the utilities.py transform() function continuing to work since it looks at things like the string value of a frequency to see if they have the characters Y and M. The results look OK with test10, but I think we should consider that with more complicated datasets we could definitely encounter timeseries resampling issues. Or maybe not. We can track that in #209

@rburghol rburghol changed the title Saved2 Pandas 3.0, create new HDF5 close() method Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant