New argument: max_inner_iter

[Malte Londschien (mlondschien)]

contributes to #850

[Marc-Antoine Schmidt (MarcAntoineSchmidtQC)]

This is great, and was probably originally meant to be exposed. Thanks.

I'm wondering if we should change the default value to 1000 (as mentioned in the linked issue). Can you provide a simulation to show the difference in runtime between the two?

I'm happy to merge this without the change and switch the discussion to another issue.

[Malte Londschien (mlondschien)]

Can you provide a simulation to show the difference in runtime between the two?

I have not managed to reproduce the convergence issues I encountered on a simulated dataset. The dataset I am using is an aggregation of ICU benchmarks. A training and approval by the dataset owner is required to access. So while not proprietary, it's not easy to share.

The defining feature of this dataset is that there are many strongly correlated features and many mean-imputed missing values.

[Luca Bittarello (lbittarello)]

Martin Stancsics (@stanmart), ping

[Martin Stancsics (stanmart)]

LGTM, but maybe don't close #850 until we test if a lower default works better.

[Malte Londschien (mlondschien)]

LGTM, but maybe don't close #850 until we test if a lower default works better.

My understanding is the the max_iter_inner is only needed if there is some problem with the inner solver, e.g., the problem is not strictly convex. Exposing max_iter_inner to the user to avoid the solver trying to find a unique minimum that does not exist for many, many iterations is a hotfix. It would be good to solve the underlying problem: Why does the inner solver not converge in say 10 iterations? Quantco/tabmat#408 is a first step. Still, glum does not behave as expected for problems such as the one in #872. Why? Probably something with the hessian.