ENH add fista restart solver

[HippolyteLBRRR]

No description provided.

[mathurinm]

Thanks Hippolyte, a few comments on variable names.
Do you have results on simple datasets to share ?

[mathurinm]

can you try to have more explicit names like outer_iteration, inner_iteration etc?

[HippolyteLBRRR]

Thank you for your comments I will correct that quickly!
I am still doing some tests but I will share my results when I have some!

[tomMoral]

A few comments but overall this is nice.

I am wondering if it would make sense to only compute the loss only after the restart, in the outer while loop? this would simplify the use of the callback.

[tomMoral]

This looks good! :) A few more comments to make the code clearer.

Did you run a few benchmarks? Can you share one or 2 convergence curves (maybe with celer and modopt-fista['greedy']?

[HippolyteLBRRR]

"I am wondering if it would make sense to only compute the loss only after the restart, in the outer while loop? this would simplify the use of the callback."
I didn't see your remark sorry! I think that it is the way it is coded now but I may not understand your point.
I am still doing some tests but I will share some convergence curves as soon as possible!

[mathurinm]

I get a few division by zero errors, do you know where they come from @HippolyteLBRRR ?

/home/mathurin/workspace/benchs/lasso/solvers/fista_automatic_restart.py:66: RuntimeWarning: divide by zero encountered in divide                                                                
  mu = np.min(4 * L / (ite_per_restart[:n_restarts-1]+1) ** 2
/home/mathurin/workspace/benchs/lasso/solvers/fista_automatic_restart.py:66: RuntimeWarning: invalid value encountered in divide
  mu = np.min(4 * L / (ite_per_restart[:n_restarts-1]+1) ** 2

[HippolyteLBRRR]

I get a few division by zero errors, do you know where they come from @HippolyteLBRRR ?

/home/mathurin/workspace/benchs/lasso/solvers/fista_automatic_restart.py:66: RuntimeWarning: divide by zero encountered in divide                                                                
  mu = np.min(4 * L / (ite_per_restart[:n_restarts-1]+1) ** 2
/home/mathurin/workspace/benchs/lasso/solvers/fista_automatic_restart.py:66: RuntimeWarning: invalid value encountered in divide
  mu = np.min(4 * L / (ite_per_restart[:n_restarts-1]+1) ** 2

I think that it comes from the fact that for a sufficient number of iterations, the value of the loss function does not vary much between two restarts. Consequently objectives[1:-1]-objectives[-1] (or F_tab[1:n_restarts] - last_F in the old version) may have some small coefficient which are interpreted as zeros. It does not have negative consequences on the algorithm as far as I know.

[HippolyteLBRRR]

Here are two convergence curves for the dataset leukemia. Our restart strategy is not particularly efficient in that context compared to the other one and far behind Celer.

[tomMoral]

Nice!! :)
It is cool that you are faster than Modopt greedy strategy.

Could you run the algorithm on larger datasets and with small reg (which tend to be more favorable to FISTA)?

[HippolyteLBRRR]

I have put two times the same graph, here it is for reg=0.05! Yes I will test the methods on other datasets.