Release version v1 of the file format

[cschwan]

Here's what I'd like to change in a newer version of the file format, v1, replacing the current (v0) one:

• merge the types BinInfo and BinRemapper into BinLimits. The reason for having them separate is historical only (see PineAPPL file format and backwards compatibility issues #83 (comment)) and merging them will make parts of the code much easier
• merge Mmv3 into Grid, which means that metadata will always be present making metadata-related code much shorter
• change the Order's member from u32 to u8 and add another member to support Add support for additional couplings #98
• remove most of the Subgrid types and keep only the ones we use (see PineAPPL file format and backwards compatibility issues #83 (comment))
• merge all Subgrid types into a single module
• export most important types into the main module
• use bincode version 2 with varint encoding, see also item here: Add offline file-size optimisation #45 (comment)
• make some of the metadata proper members of Grid: initial_state_1, initial_state_2 and possibly lumi_id_types, which are important in some places; Clarify initial states and PDFs used #135 is a related Issue; done in Implement v1 file format #299
• replace set_convolutions with convolutions_mut; done in Implement v1 file format #299
• Unify SubgridParams with ExtraSubgridParams and rename them to InterpolationParams; make their parameters pub instead of having many setters and getters; this has now been merged into single Interp type
• Generalize convolutions to support fragmentation functions (see Generalize convolutions for both initial and final states #172)
• Rename LumiCache to ConvolutionCache, which better reflects its purpose
• generalize Channel to allow more or less than 2 PIDs per linear combination; done in Implement v1 file format #299
• Remove Convolutions::None
• Add machine-readable MC results and uncertainties to calculate interpolation error #270

Originally posted by @cschwan in #83 (comment)

[cschwan]

Technically we can have different file formats at the same time by versioning also the modules:

• pineappl::v0:: would contain the old code
• pineappl::v1:: would contain the new file format
• and so on should we need even more changes.
• in the module pineappl we can insert pub use ... statements to import the most recent file format.

Grid::read would be the only method that has access to the old v0 module, which it uses if the file header indicates a non-current version. It then elects the appropriate Grid, either from v0 or v1. Grids loaded from v0 would automatically be converted into objects from v1, so everything should go smoothly without the user even noticing.

In this way we can also easily get rid of old file format - if we decide to do that - by simply throwing away the v0 module and changing Grid::read to return an error. In that case the user would have to manually upgrade the file format, for instance by using an older version of the CLI.

[felixhekhorn]

Since you want to guarantee backwards compatibility, you need to have both the old and the new code inside and also the converter (which could go in v0 since it can be thrown away together) ... (just a style comment: you maybe want to scope them a bit more then just pineappl, because this sounds like the whole library changes - e.g. pineappl::io or similar?)

[cschwan]

Yet another way (no idea how that would work) is to let the pineappl crate depend on an older version of itself, or separate that part into a different crate, for instance pineappl-upgrade.

[alecandido]

Personally, I don't care so much about splitting crates. But if the other is not obviously the good choice, I'd stick with one for simplicity.

The dependency on the older version is potentially a mess.

I've read in the Go ecosystem an advise for new major releases, that was to create a completely new module (in this case a new crate), and depend on the old one.
Unfortunately, in Go is simpler, because of the way modules are addressed. In Rust is not so common, so it would appear messy.
But given that we are talking about the file format, I would imitate inside the crate itself (essentially with ::io::v0 and ::io::v1).

[cschwan]

I think I'll first go with v0 and v1 modules, and after some time we might decide split off the v0 part into a separate crate.

[cschwan]

Yet another way (no idea how that would work) is to let the pineappl crate depend on an older version of itself, or separate that part into a different crate, for instance pineappl-upgrade.

This stackoverflow answer describes how to do it: https://stackoverflow.com/a/58739076/812178. That seems pretty straightforward.

[alecandido]

Consider that, going this way, you might have to maintain also the older version, in case of bugs preventing some kind of applications.
It is possible, but not the easiest, for a project with a restricted number of maintainers.

In my opinion, I would try to keep the version management straight, and just increase the version number in new releases. Thus, I'm more in favor of pineappl::v0, pineappl::v1... (or pineappl::io::v0 or whatever...).
When you have a new one, you deprecate the old at some point, and eventually drop.

[cschwan]

@alecandido those two ideas aren't exclusive. You'd write

use pineappl_v0 as v0;

in pineappl/src/lib.rs and in pineappl/Cargo.toml:

[dependencies]
pineappl_v0 = { package = "pineappl", version = "0.5.8" }

In this way we'd preserve an older version in pineappl::v0, while the newer version would be in pineappl. The main difference to explicitly keeping the old copy in pineappl/src/v0/... is that we can't modifiy the old source, but I think this is an advantage. With the method described above we'd rely on crates.io to get 0.5.8, which no one needs to maintain (you can't change published versions).

The only functionality that we need from pineappl::v0 is to load a Grid and update the structures to those in v1.

[alecandido]

For as long as it is working, that's fine.

Problem is if Grid becomes incompatible, then you have to write a method to convert v0::Grid into Grid.
However, if the only operation allowed with the old crate in the new crate is loading, that's just the worst case: you need a converter. Then, all the operations and everything will be done by the new crate with the new logic, even if incompatible.

The only room remained is if you are also going to dump an old version, or just the new one (I'd recommend the later).
E.g., if you optimize an old grid, will it still be old? Or if you merge old grids?
This is a potential mess, since then you also have to deal with merging new and old grids and so on, so I'd say all the operations should be performed as if the grids were all new, and the old ones are converted.
Notice that this is potentially breaking: if your code depends on PineAPPL v0, and you optimize your grid with PineAPPL CLI v1, then you won't be able to use it any longer. However, I would say that this should be solved by the user either using CLI v0, or upgrading the dependency.