Allow year ranges in input files

[tsmbland]

Is your feature request related to a problem? Please describe

Many input files have a "years" column which allows users to specify a list of years to apply the parameter(s) to. This is neat, however if you want to apply the parameter to lots of years (but not all years, for which you could use "all"), then this list could get very long. For example this

Describe the solution you'd like

It would be nice if we could accept a range of years so users don't have to write out each and every year. Not sure what syntax this should take. We could use rust syntax, for example:

• "2000 up to and including 2020" would be 2000..=2020
• "any year up to and including 2020" would be ..=2020

Describe alternatives you've considered

The rust syntax may be a bit obscure to some users so we could consider coming up with custom syntax of our own

[alexdewar]

I think we should write these as 2000..2020 and ..2020 instead. I know that means the range isn't inclusive in Rust, but, as you say, most of the users won't be familiar with Rust anyway. We should just put a note in the docs pointing out that it's different to the Rust syntax, in case that confuses anyone.

[tsmbland]

Maybe something like:

• 2020..2024: 2020 to 2024 inclusive
• 2020..: 2020 until (but not including) the next milestone year
• 2020...: indefinitely from 2020 onwards
• ..2020: 2020 and all years after (but not including) the preceding milestone year
• ...2020: 2020 and all preceding years
• ...2020..: 2020, all preceding years, and all years before the next milestone year

Related to #770 (comment)

[alexdewar]

I fear this might be a bit too subtle. We could do ..< or ..^ or something to mean an exclusive range (though I don't know if we need it), but anything beyond that is maybe too complex.

If we decide to allow processes to be commissioned in non-milestone years within the time horizon, that will possibly make all this a bit easier. In that case, I'd be inclined to just have the basic 2000..2020 syntax, with the first and last number being optional (meaning no lower or upper limit, respectively). I think it's a bit subtle to have a special syntax that means "up until the next milestone".

We could also use this range syntax to express availability bounds (see #363).

[tsmbland]

I fear this might be a bit too subtle. We could do ..< or ..^ or something to mean an exclusive range (though I don't know if we need it), but anything beyond that is maybe too complex.

If we decide to allow processes to be commissioned in non-milestone years within the time horizon, that will possibly make all this a bit easier. In that case, I'd be inclined to just have the basic 2000..2020 syntax, with the first and last number being optional (meaning no lower or upper limit, respectively). I think it's a bit subtle to have a special syntax that means "up until the next milestone".

We could also use this range syntax to express availability bounds (see #363).

True. Actually I think "up until the next milestone" might be a bad idea, but "up until the next year which data is provided for" could be an option (although probably quite fiddly).

The raw data is probably going to be for a certain set of years, which may not match the milestone years of the simulation, and may be different between files. For example, they may have process parameters for [2023, 2028, 2033], existing assets from [2018, 2024, 2025, 2031 etc.], demand data for [2020, 2025, 2030, 2035], and want to run the simulation with milestone years [2020, 2022, 2024, 2026, 2028, 2030, 2032]. We want users to be able to develop each file independently (e.g. it's annoying if users have to look at the existing assets while building the process parameters file), and it should be seamless to change the milestone years for greater/less granularity without having to change every input file.

For most parameters, if there's no data available for the year in which the parameter is required, I think most users would either want to interpolate (in which case they should do this externally so that data is provided for every year), use the parameter from the most recently available year (we wouldn't want to hardcode this, but the ["2023..", "2028..", "2033.."] syntax would be an easy way for users to tell the program to do this), or use the parameter from the next available year (["..2023", "..2028", "..2033"]).

The unbound ".." could mean "forwards/backwards extension until the next year that data is provided for"

I imagine that would get really fiddly though. So maybe we just stick to the simple range syntax you suggested (so users would have to write ["2023..2027", "2028..2032", "2033.."] or ["..2023", "2024..2028", "2029..2033"])

[alexdewar]

I imagine that would get really fiddly though. So maybe we just stick to the simple range syntax you suggested (so users would have to write ["2023..2027", "2028..2032", "2033.."] or ["..2023", "2024..2028", "2029..2033"])

I think this is probably the right place to start. If down the line users are complaining that they want to be able to write it more tersely, then we can revisit, but it's hard to take out a feature like this once you've put it in and if we can avoid having to maintain it forever then it'll make our lives easier 😄

So I think we're agreed on how to do this then? Shall we add an issue about allowing process params for non-milestone years to start with, then once that's done, we can look at some of these related ones?

[tsmbland]

I imagine that would get really fiddly though. So maybe we just stick to the simple range syntax you suggested (so users would have to write ["2023..2027", "2028..2032", "2033.."] or ["..2023", "2024..2028", "2029..2033"])

I think this is probably the right place to start. If down the line users are complaining that they want to be able to write it more tersely, then we can revisit, but it's hard to take out a feature like this once you've put it in and if we can avoid having to maintain it forever then it'll make our lives easier 😄

So I think we're agreed on how to do this then? Shall we add an issue about allowing process params for non-milestone years to start with, then once that's done, we can look at some of these related ones?

Sure. I think that's a good place to start, then maybe we can chat on Friday to firm up the other plans

[alexdewar]

Sure. I think that's a good place to start, then maybe we can chat on Friday to firm up the other plans

Sounds good. I'm WFH on Fri, but we can chat on Teams