Add option to use the same tr/vl split for different replicas

[scarlehoff]

Even though we are using a tr/vl split per replica it would be useful to have an option such that the tr/vl split is informed just by the seed in the runcard (and not the combination of the seed in the runcard + the replica).

In practice this will be useful for running n3fit in GPU (where it is necessary to have the data in the same position for all replicas).

I'm wondering, @wilsonmr, whether this option exists already somewhere (since you explored this)?

[wilsonmr]

There was a branch but the implementation was not very nice. It's fairly easy to do though in the trvl seed action!

[siranipour]

Sorry if I'm being slow, but why do we want the same tr/val split in the case of GPU running?

[wilsonmr]

Well at the moment the multireplica fits on GPUs all fit the same data, that would include the data having the same training validation split.

In the future the idea would be to fit pseudodata replicas with different tr/vl splits but I think there is the question of how to support that in the GPU memory

[wilsonmr]

see #1153: genrep has to be set to False

[siranipour]

Ah I see so it's a question of memory not being big enough.

[scarlehoff]

Sorry if I'm being slow, but why do we want the same tr/val split in the case of GPU running?

The way the convolution is implemented, the tr/vl mask is applied at the level of the fktable, so by using the same tr/vl split the same convolution is performed for all replicas.

If instead one would like to have different tr/vl mask per replica either you have to load a fktable per replica (impossible) or load the whole fktable and apply the mask to the output. This was much worse in the CPU, I haven't tried here.

see #1153: genrep has to be set to False

Now that the tr/vl split can be forced to be the same for all replicas I should be able to do replicas as well (I'm testing it, but I want it to behave like replica_range and keep both options)

Ah I see so it's a question of memory not being big enough.

In summary, yes.