Dispatch with flexible capacity (and partial implementation for cycles)

[tsmbland]

Description

This PR adds to ability to have flexible capacity assets in the dispatch optimisation, and adds the beginnings of an investment algorithm for cycles (select_assets_for_cycle)

This follows on from #1004, where markets were put in an order that minimises potential conflicts (scenarios where a market may have its demand profile altered by a market further down the order). With this in place, we can almost treat the cycle like any other ordered set of markets, except that some conflict is inevitable so we need an approach to deal with this when it arises.

The approach I've come up with here is to perform dispatch optimisation after each market with a limited degree of freedom on the capacities of any new assets selected for markets in the cycle. The point of this is that markets that are part of a cycle may see their demand profiles change after their investments were initially performed, and this is a way of allowing capacities to change a small amount across the system to "balance-out" these demand changes, without having to trigger complete re-investment.

Capacity variables are added to the dispatch (for the subset of assets that we want to have flexible capacity), and activity constraints are reformulated in a way that takes these variables into account (similar to what we already do in the appraisal optimisation).

Capacity variables are bound to the original capacity of the asset within some degree of tolerance (e.g. 20%), which is specified by a user parameter. Note: we only allow this for newly selected assets (Selected type). Existing assets, or any assets provided in assets.csv are not allowed to change their capacity. I'm using capital/fixed costs as the coefficients for these variables (suggested by Adam).

The overall point of this (and for limiting the tolerance to a small value), is that small changes in demand are unlikely to have a significant qualitative impact on the choice of assets in a market. If the system is still infeasible, even allowing for this tolerance, then at this point we'll probably want to trigger re-investment for certain markets, although I haven't done this here. Users currently have the option of increasing capacity_margin, which may be sufficient (but not ideal), and I also anticipate that other yet-to-implement features such as capacity growth limits and capacity share constraints may help to balance the cycle.

Main functions to look at:

• select_assets_for_cycle
• add_capacity_variables
• add_activity_constraints
• calculate_capacity_coefficient

Other changes:

• Caried over some of the tidy-ups from Ability to specify markets with unmet demand vars #985 which I ultimately didn't merge as I didn't need it in the end
• Doing price filtering within DispatchRun rather than outside. So now we're passing it all prices (from the previous year), and it will decide which ones it needs to keep based on which markets its applying balance constraints to (if there's a balance constraint, prices will be generated internally, so we throw away the previous year prices). I think it's cleaner like this (see select_input_prices, a replacement for check_input_prices)
• Removing some log messages: we don't need to log the objective value because we're saving it to debug_solver.csv. I've also globally turned off logging for highs because it was giving loads of really annoying messages like Adding a problem with 64 variables and 88 constraints to HiGHS which was obscuring the log messages that I actually wanted to see
• Added a new example model (originally from Adam). I haven't added a regression test for it (partially because it doesn't run to completion!). If it's a bad idea to include this in the repo I can remove it, although it's really useful for development and I've added a warning to the readme

Discussion/remaining work

This doesn't work in all scenarios:

• if a market relies entirely on existing assets, then there's no flexibility added here so this will not help
• if a market selects no assets (i.e. because it has no demand when it's first met, but may do later), this will also not help
• if all new assets selected by markets in the cycle draw on the cycle itself, then no amount of cycle balancing will get that to work (e.g. in the electricity/hydrogen cycle, if the electricity agent decides to get all electricity from hydrogen, and the hydrogen agent decides to get all hydrogen from electricity. This currently happens in the example model added here, although there are a few rounds before this where it does work, and it's possible to get something that runs to completion by fiddling with parameters).
• if none of these apply, it can still fail if capacity_margin isn't big enough. Sure, you can increase this, but eventually you might expect large imbalances to have a qualitative impact on investment decisions (i.e. different technologies selected), so a better approach might be to redo investments and iterate in the case of large imbalances

Were going to need to come up with an approach that addresses these scenarios, but also avoids the potential for unrealistically spiraling loops and unnecessary capacity expansion. We also want to, as best as we can, flag from the outset any flow cycles that are likely to be problematic (e.g. probably any cycle needs to have at least one input from outside the cycle).

I think the solution will probably involve redoing investments with a new demand profile for any markets that can't keep up with the imbalances. I worry about this potentially leading to an endless loop, particularly in the case of point 3 above, so we may need something else as well.

Anyway, if we're happy that this is a partial solution, hidden behind the please_allow_broken_results option, then hopefully that's good enough for now.

Fixes #794

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

❌ Patch coverage is 36.48649% with 188 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.59%. Comparing base (178647b) to head (4ba2927).
⚠️ Report is 34 commits behind head on main.

Files with missing lines	Patch %	Lines
src/simulation/investment.rs	7.92%	92 Missing and 1 partial ⚠️
src/simulation/optimisation.rs	48.25%	70 Missing and 4 partials ⚠️
src/simulation/optimisation/constraints.rs	55.00%	16 Missing and 2 partials ⚠️
src/asset.rs	33.33%	2 Missing ⚠️
src/model/parameters.rs	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #999      +/-   ##
==========================================
- Coverage   84.28%   81.59%   -2.69%     
==========================================
  Files          52       52              
  Lines        5974     6210     +236     
  Branches     5974     6210     +236     
==========================================
+ Hits         5035     5067      +32     
- Misses        694      889     +195     
- Partials      245      254       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dalonsoa]

I've a couple of questions, but otherwise it looks good. Actually, I think it is quite neat.

About some of your scenarios where this does not work:

1. If a market relies only on existing assets, then there is no solution to the problem, cycle or not cycle, as you cannot install anything new to fulfil the demand, right?
2. Should't the graph select those if there is the potential to be needed? In other words, if there is an asset that could be invested in, but we don't know if it will, that requires commodity A, shouldn't assets that produce such commodity be included, just in case?
3. I don't see how that scenario can work at all, to start with, even from a physics perspective, if there's no external input at all, to be honest... unless we are talking about nuclear fusion.
4. OK!

[dalonsoa]

In the select_assets function above, we say * seen_markets – Markets whose demand-balancing has already been fixed, however, here we are balancing them again. Either this is an error - probably not - or the explanation above is misleading.

[tsmbland]

These are markets that have previously been solved and their asset capacities are now locked in place (i.e. other investment sets earlier in the investment order that are independent of the cycle currently being solved).

By "demand-balancing has already been fixed" I mean that we have already ensured that there's enough capacity in these markets to meet the demands of the market, and these capacities are now fixed. Because of the way the investment order is defined and solved sequentially, we can guarantee that these markets won't see any new demands, so there won't be any need for further investments in these markets.

When you say that we are "balancing them again" - this is also sort of true. We're giving the system the end use demands and allowing the system to utilise these confirmed assets in an appropriate way to meet these demands. But assets in these markets can only change activity and not capacity. (It's actually a bit of an API limitation that we need to do this, because we know that activity won't need to change either, because there can't be any new demands on these markets, but for now we just have to include them in markets_to_balance)

I think the confusion is because I use the term "balancing" to mean slightly different things in different contexts. In all cases this is ensuring that supply meets demand, but in two different ways:

• balancing only via activity variables: this is allowed for previously seen markets
• balancing via activity and capacity variables: this is only allowed for markets in the cycle currently being solved

Agree that that doc comment could be clearer

[tsmbland]

I've a couple of questions, but otherwise it looks good. Actually, I think it is quite neat.

About some of your scenarios where this does not work:

1. If a market relies only on existing assets, then there is no solution to the problem, cycle or not cycle, as you cannot install anything new to fulfil the demand, right?

This would happen if, when the market is first solved, it sees no extra demand compared to the previous year (and none of the existing assets have reached the end of their life). Not a problem in models without a cycle. Quite common actually. However, if there's a cycle, there's the possibility that new demands could be encountered after we've initially decided just to keep the existing assets, at which point the existing assets might no longer be sufficient. The problem is that the solution proposed here (allow newly selected assets to change capacity a bit to meet the new demands) doesn't apply here because there are no newly selected assets.

2. Should't the graph select those if there is the potential to be needed? In other words, if there is an asset that could be invested in, but we don't know if it will, that requires commodity A, shouldn't assets that produce such commodity be included, just in case?

We include these when we run asset selection on the market. But if no assets are selected (because there's no demand at that point, even though there may be demand later because of the circularity problem) then these aren't carried forward to the "cycle balancing". We could select the best one as a "dummy asset" with zero capacity, but would have to reformulate the capacity limits because a percentage limit on zero capacity is still zero capacity.

3. I don't see how that scenario can work at all, to start with, even from a physics perspective, if there's no external input at all, to be honest... unless we are talking about nuclear fusion.

I think in a lot of cases it would be impossible (and, indeed, impossible to get a stable solution). That said, it's not mandatory to model energy inputs as actual commodities. E.g. you could model a wind turbine producing electricity without modelling "wind" as a commodity; in this case the process is producing "something from nothing", although we know that the energy isn't coming from nowhere. So, following similar logic, I could imagine a realistic scenario where a cycle is modelled without any external inputs.

I guess it will depends on the coefficients. In this case we've got:
1.3 ELC -> 1 H2
1.5 H2 -> 1 ELC
So 1 unit of electricity supplied by the loop would require 2 units of electricity, so this can only work if there's an external source of electricity

That said, the agents in each market are "selfish and short sighted". The electricity agent doesn't know that investing entirely in electricity production from hydrogen would ultimately lead to ever more electricity demand, so it could very well decide to do so. Then, when we get to the full system "balancing dispatch", there's no way to get this to balance without shutting off the loop entirely

4. OK!

[tsmbland]

I've experimented a bit with the NPV objective. It seems that H2YPRD always gets priced in such a way that the H2YGEN has a profitability index of zero, so never gets chosen. But, if the agent was able to look at the loop as a whole (ELCTRI->H2YPRD->ELCTRI), the price of H2YPRD is irrelevant as it balances out, you're only interested in the price of electricity in vs electricity out, which could be very profitable if you're able to buy electricity when it's cheap, store as hydrogen, and sell when it's expensive.

Perhaps we can fix with pricing methods that give a more realistic price for H2YPRD?

Otherwise, I think our approach of making agents select assets for one commodity at a time, even with the subsequent global "balancing" in this PR, can lead to some overly-myopic behaviour which we should be careful about

[tsmbland]

@ahawkes a few scattered observations/discussion points above about the circularities problem.

Happy to get your take on anything raised above, and any/all of the following:

• Are you happy with the general approach in this PR, at least as a partial solution?
• Prices for commodities in cycles - are current prices reasonable? Do we need a new approach?
• How to deal with the situation where multiple commodities in the cycle are owned by the same agent?
• How to prevent endless feedback-loops where markets in the cycle draw entirely on the cycle itself?
• Upfront validation - how to identify upfront "impossible" circuits?

[ahawkes]

@ahawkes a few scattered observations/discussion points above about the circularities problem.

Happy to get your take on anything raised above, and any/all of the following:

• Are you happy with the general approach in this PR, at least as a partial solution?

Yes I think this is a good starting point. Thanks for the effort!

• Prices for commodities in cycles - are current prices reasonable? Do we need a new approach?

Not sure about this. I think it's fair to say prices are consistent with the philosophy we're adopting to estimate them. Which is a good thing. However further thought would be needed on how realistic they are.

Another priority might be a pricing method that incorporates capital cost. I guess there's already a separate issue for this? (and probably won't help re cycles anyway)

• How to deal with the situation where multiple commodities in the cycle are owned by the same agent?

I see your point here I think. I don't want to complicate things by agents considering multiple commodities simultaneously yet! In some senses this is similar to the two-outputs problem noted previously, where both commodities should probably be considered for investment at the same time, lest you end up with over-investment, or poor investment choices.

• How to prevent endless feedback-loops where markets in the cycle draw entirely on the cycle itself?

I think as a first port of call we should implement capacity addition constraints. We may end up needing to require users to enter these, which can prevent endless investment in cycles.

• Upfront validation - how to identify upfront "impossible" circuits?

Or rather, how to find an algorithm where this can never happen. Or constrain user input data so that it can never happen (e.g. capacity addition constraints required). And warnings for excessive capacity addition?

[tsmbland]

Thanks @ahawkes

I agree that addition limits and pricing methods should be priorities. We've got issues #1009 and #124, with a few questions on both of them