Use graphs for commodity flow validation

[tsmbland]

Description

This replaces the code for validating commodity flows with a new and (hopefully) more robust method using the graph library that we're already using for commodity ordering.

The point of this is to make sure that each commodity abides by the rules for its type (SVD vs SED vs OTH), and ultimately to ensure that there's a viable pathway to meet commodity demands.

• Service demand commodities (SVD): if these are demanded, there must also be a process that can produce them
• Supply equals demand commodities (SED): if there are processes that can consume these commodities, there must also be processes that can produce them
• Other commodities (OTH): these can only be consumed or produced but not both (these are usually side-products such as CO2, renewables such as solar energy, or raw materials that we don't model the production of)

The process is as follows:

1. Create initial commodities graph for each region and timeslice.
2. Perform topological sorts on these graphs. This will also make sure there are no loops in the graph
3. Extra validation to check that each commodity abides by the rules for its type. This also takes into account process availabilities and commodity demands, which are added to the graph. Since commodities can have different balance levels, and therefore different timescales over which the rules must apply, we do several rounds of validation at different levels of granularity:
   • Add annual availabilities and annual demands to the graph, and perform validation for Annual-level commodities
   • Add seasonal availabilities and seasonal demands to the graph, and perform validation for Season-level commodities
   • Add timeslice availabilities and timeslice demands to the graph, and perform validation for DayNight-level commodities

If it passes all of this then we're good to go!

I've tried to be faithful to the original code by keeping roughly the same checks that were originally in place. If we're being smart, then we probably don't need to do validation checks for every commodity but just need to ensure that each _DEMAND node is reachable from at least one _SOURCE or OTH node via a directed path through the graph, which is ultimately what we care about. (See has_connecting_path or all_simple_paths)

(Ultimately, I wonder if we actually need the "commodity type" concept at all, but that's a conversation for another day)

I think as the model gets more complicated the benefits of this approach will become stronger. For example, commodity trade where a commodity may be produced in one region and consumed in another (in this case we'll probably have to combine all regions into a single graph)

Fixes #737 I'm not directly addressing this issue, but I've deleted the part of the code where the panic occurred, and checked that it no longer panics in the new version (it gives a nice error message instead)

Re #743 I can't actually reproduce the panicking (I remember it being a bit random as to whether it panics or gives an error message), but the part of the code where it supposedly did panic is now gone and I've been careful not to include the same mistake in the new code. The error message persists (unrelated to this code) which is still a problem so I'll leave the issue up and change the description

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

❌ Patch coverage is 86.69951% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.53%. Comparing base (47f02bc) to head (d56a542).
⚠️ Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
src/graph.rs	86.73%	18 Missing and 8 partials ⚠️
src/input.rs	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #771      +/-   ##
==========================================
- Coverage   86.77%   86.53%   -0.24%     
==========================================
  Files          46       46              
  Lines        4787     4828      +41     
  Branches     4787     4828      +41     
==========================================
+ Hits         4154     4178      +24     
- Misses        423      437      +14     
- Partials      210      213       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[alexdewar]

This is great! I've made some small suggestions.

Requesting changes mostly because I think you should use enums for edges/nodes rather than IDs with sentinel values.

[alexdewar]

Is this a limitation? Will things break for commodities with a different time_slice_level?

[tsmbland]

Nope. We only validate SVD commodities at their appropriate time_slice_level, and other commodities won't be affected by the presence/absence of demand nodes

[tsmbland]

@alexdewar Thanks for the comments, I think this is ready for another look

[alexdewar]

I've suggested a few small tweaks, but otherwise all good!