Implement LBFGS optimizer

[orionarcher]

LBFGS would be an excellent additional optimizer alongside FIRE and gradient descent

[corinwagen]

agreed! FIRE is quick but doesn't seem to converge things as tightly as LBFGS

[abhijeetgangan]

Do you have a comparison that it does generally better than FIRE?

Here are some references comparing different optimizers:

• https://www.sciencedirect.com/science/article/pii/S0927025620300756#s0010
• https://docs.matlantis.com/atomistic-simulation-tutorial/en/2_3_opt-algorithm.html
• https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.97.170201

FIRE seems to be more robust but can struggle with very small fmax. In any case adding LBFGS with linesearch would a good addition.

[corinwagen]

Agreed re: FIRE vs BFGS, it does seem generally like small fmax can be tough but that FIRE's often more robust. Both seem good to have in the toolbox IMO.

Here's a concrete case where BFGS performs much better than FIRE—optimization of valinomycin w/ MACE-MP-0 (or MACE-MPA-0, they keep stealth-changing which model is used and I can't keep track since there's no publication haha). Here's valinomycin:

Script for BFGS, FIRE is the same if you replace BFGS with FIRE:

from ase.optimize import BFGS, FIRE, QuasiNewton
from mace.calculators import mace_mp
from ase.io import read

mol = read("valinomycin.xyz")
mol.calc = mace_mp()

BFGS(mol).run(fmax=0.01)

BFGS converges to a final energy of -993.677917 in 405 steps. FIRE converges to a final energy of -993.098267 in 497 steps, so it doesn't get the right minimum.

Here's the input file for valinomycin:

168
valinomycin
N        2.98    2.984    1.224
C        2.288    3.512    2.4
C        0.808    3.268    2.176
C        2.768    2.832    3.664
C        4.272    3.148    3.956
C        1.92    3.248    4.876
O        0.312    2.208    1.86
O        0.088    4.372    2.44
C        -1.372    4.204    2.396
C        -1.828    4.144    0.92
C        -1.944    5.468    3.04
O        -3.012    3.704    0.768
N        -1.068    4.456    -0.064
C        -1.528    4.34    -1.448
C        -1.924    2.9    -1.676
C        -0.488    4.76    -2.5
C        -0.236    6.276    -2.284
C        -0.916    4.44    -3.876
O        -1.26    1.944    -1.36
O        -3.136    2.784    -2.244
C        -3.612    1.476    -2.624
C        -4.144    0.692    -1.48
C        -4.644    1.652    -3.728
C        -5.836    2.448    -3.208
C        -4.052    2.304    -4.948
O        -4.584    -0.444    -1.732
N        -4.22    1.212    -0.272
C        -4.74    0.408    0.828
C        -3.892    -0.84    0.936
C        -4.764    1.18    2.168
C        -5.716    2.352    2.052
C        -5.176    0.304    3.316
O        -2.664    -0.864    1.02
O        -4.636    -1.964    0.96
C        -3.96    -3.248    1.132
C        -3.076    -3.564    -0.056
C        -5.036    -4.32    1.256
O        -2.132    -4.356    0.092
N        -3.34    -3.036    -1.252
C        -2.516    -3.388    -2.408
C        -1.092    -2.984    -2.112
C        -3.032    -2.732    -3.672
C        -4.476    -3.164    -3.984
C        -2.164    -3.116    -4.872
O        -0.752    -1.904    -1.672
O        -0.224    -3.968    -2.408
C        1.184    -3.656    -2.236
C        1.668    -3.888    -0.86
C        1.96    -4.548    -3.224
C        1.74    -6.008    -3.052
C        1.728    -4.1    -4.652
O        2.86    -3.672    -0.58
N        0.84    -4.248    0.144
C        1.316    -4.26    1.532
C        1.816    -2.844    1.856
C        0.26    -4.728    2.512
C        -0.124    -6.196    2.236
C        0.736    -4.552    3.956
O        1.2    -1.84    1.644
O        3.028    -2.88    2.424
C        3.576    -1.604    2.856
C        4.032    -0.736    1.672
C        4.728    -1.932    3.82
O        4.344    0.436    1.888
N        4.184    -1.34    0.48
C        4.596    -0.492    -0.656
C        3.568    0.616    -0.8
C        4.704    -1.34    -1.928
C        5.744    -2.428    -1.756
C        5.1    -0.452    -3.092
O        2.404    0.468    -0.824
O        4.188    1.796    -0.936
C        3.36    2.956    -1.252
C        2.652    3.492    0.016
C        4.256    3.992    -1.9
C        5.264    4.512    -0.948
C        3.444    5.092    -2.568
O        1.828    4.392    -0.108
H        3.96    3.14    1.356
H        2.476    4.488    2.516
H        2.668    1.848    3.516
H        4.552    2.684    4.796
H        4.388    4.136    4.064
H        4.832    2.828    3.192
H        2.256    2.784    5.696
H        0.964    2.992    4.72
H        1.984    4.24    5.004
H        -1.672    3.372    2.864
H        -2.944    5.416    3.044
H        -1.652    6.268    2.52
H        -1.612    5.544    3.98
H        -0.816    5.416    0.068
H        -2.296    4.968    -1.568
H        0.364    4.244    -2.388
H        0.44    6.6    -2.948
H        0.108    6.428    -1.36
H        -1.092    6.776    -2.408
H        -0.212    4.728    -4.524
H        -1.772    4.92    -4.08
H        -1.06    3.452    -3.96
H        -2.84    0.936    -2.96
H        -4.96    0.744    -4.004
H        -6.508    2.56    -3.936
H        -5.528    3.348    -2.9
H        -6.256    1.96    -2.44
H        -4.756    2.404    -5.652
H        -3.308    1.732    -5.304
H        -3.692    3.204    -4.704
H        -3.296    1.488    -0.016
H        -5.692    0.172    0.632
H        -3.836    1.5    2.352
H        -5.732    2.852    2.92
H        -6.632    2.016    1.844
H        -5.408    2.96    1.324
H        -5.18    0.844    4.16
H        -4.532    -0.452    3.408
H        -6.092    -0.056    3.148
H        -3.38    -3.216    1.948
H        -4.604    -5.216    1.376
H        -5.596    -4.332    0.428
H        -5.616    -4.124    2.044
H        -4.28    -3.288    -1.48
H        -2.56    -4.376    -2.564
H        -3    -1.748    -3.512
H        -4.784    -2.712    -4.824
H        -5.076    -2.9    -3.228
H        -4.512    -4.156    -4.108
H        -2.52    -2.672    -5.696
H        -2.18    -4.108    -4.996
H        -1.224    -2.816    -4.716
H        1.328    -2.68    -2.408
H        2.924    -4.416    -3.004
H        2.28    -6.508    -3.732
H        0.772    -6.216    -3.176
H        2.028    -6.28    -2.132
H        2.24    -4.692    -5.276
H        2.036    -3.156    -4.76
H        0.752    -4.16    -4.864
H        0.064    -3.62    0.112
H        2.056    -4.924    1.628
H        -0.556    -4.16    2.388
H        -0.82    -6.492    2.892
H        0.688    -6.776    2.332
H        -0.484    -6.28    1.308
H        0.024    -4.868    4.584
H        0.928    -3.588    4.128
H        1.568    -5.088    4.1
H        2.876    -1.056    3.316
H        5.144    -1.084    4.148
H        5.42    -2.476    3.34
H        4.376    -2.452    4.596
H        4.876    -2.052    0.592
H        5.5    -0.088    -0.496
H        3.816    -1.764    -2.108
H        5.8    -2.972    -2.592
H        5.484    -3.02    -0.992
H        6.636    -2.012    -1.568
H        5.168    -1.004    -3.924
H        5.984    -0.024    -2.904
H        4.408    0.26    -3.224
H        2.628    2.708    -1.888
H        4.768    3.544    -2.632
H        5.836    5.192    -1.404
H        5.836    3.76    -0.624
H        4.804    4.94    -0.168
H        4.064    5.76    -2.984
H        2.876    5.548    -1.884
H        2.86    4.692    -3.276

Here's another case, for this crystal structure of pyrazole from the X23 benchmark set:

Using the same script as above, BFGS gets -458.832733 after 103 steps, while FIRE gets -458.827972 after 454 steps. If I apply a FrechetCellFilter, FIRE doesn't seem close to convergence after 750 steps, while BFGS converges after 413 steps.

Input file for pyrazole crystal structure:

72
Lattice="8.08736833 0.01590956 0.04985313 -0.05378646 12.67199266 -0.11418380 -0.03286503 0.05385492 6.83082997" Properties=species:S:1:pos:R:3
c        6.79929760       1.00965814       3.92257364
c        0.19526255       6.82990012       3.14053012
c        0.31571540       9.80122407       1.00862592
c        7.50691535       0.33953661       2.94030800
c        4.28166139       0.49491727       1.17933815
c        3.42857907      11.77040367       4.80480711
c        4.35566645       2.31672141       6.68800293
c        7.68444256       1.98847825       4.40071625
c        3.61183327       8.33989266       6.75783761
c        2.74590548       7.32710466       0.42047118
c        3.43690439       6.67936194       1.42924394
c        6.80585279       4.74100344       0.47361264
c        7.70362627       3.78338460       0.97209735
c        3.63156629      10.14147771       3.32526130
c        2.73581567      11.10983007       3.80548307
c        1.08167795      10.84342589       0.51430903
c        0.17610136      11.62293617       6.51721329
c        5.15825654       1.22794792       0.36193050
c        4.37027972       3.48480811       3.25860568
c        1.06768333       7.57756090       3.94902300
c        0.29808932       8.62590959       4.42501615
c        4.24056933       5.28803382       4.52958648
c        5.12717019       4.54058185       3.73715656
c        7.47028591       5.44802709       6.30371463
h        5.78632806       0.82037092       4.24063561
h        5.42771951      11.37543001       5.53012051
h        1.42268639       0.72408136       2.18010525
h        4.41980622      12.27698218       1.65370997
h        1.73581299       7.13744244       0.09441442
h        4.60674714       3.13568064       6.02806905
h        3.46728461       9.05783316       5.96082174
h        3.48595485       9.41313948       2.53789731
h        2.10976560       7.38611829       4.15205828
h        0.54701734       9.45792124       5.06895198
h        6.32868460       9.11386244       4.04161032
h        6.20332387       1.03596393       0.17219139
h        1.38829641       5.04013222       5.54753499
h        7.18889303       6.25870211       5.64702175
h        5.79150259       4.92572203       0.79067301
h        7.55701902       3.06434724       1.76765059
h        1.72619052      11.29819070       3.47691554
h        3.14767075      12.59053891       5.44940716
h        3.15144371       5.87126506       2.08747026
h        7.18194830      12.18154978       2.18947051
h        0.38996049       5.92668933       2.57388955
h        5.43541284       7.07971411       2.15110722
h        2.32580286       2.73567132       0.26115719
h        2.30975600       3.00568281       3.66505903
h        6.17062098       4.73783203       3.54530883
h        0.36795768      12.51468406       5.93195243
h        4.63369529       2.65019631       2.62360890
h        2.12162376      11.04556405       0.71868932
h        0.56624229       8.98498125       1.67163977
h        6.35094711       9.32507624       0.62707400
h        4.42338791       6.19712442       5.09117401
h        7.52540280       2.72605391       5.17712295
n        7.04550966      11.08070268       6.55146890
n        7.06158052       7.39704954       3.16854235
n        3.12424997       3.63963173       3.75548800
n        3.02212583       4.73030926       4.53975541
n        7.16602106       9.95825411       0.53595310
n        7.14702651       8.48268392       3.96117435
n        3.14615493       2.11035642       0.36394126
n        4.79571061      10.19968907       3.98442533
n        4.65806570       7.25387862       1.49613605
n        4.80907334       8.23441186       0.58708817
n        0.60637834       4.86091821       6.19507615
n        3.05808621       1.04135739       1.17905469
n        4.64803685      11.19274188       4.87976163
n        0.78516258       3.82458163       0.27174364
n        0.63709547       0.90626554       2.82236346
n        0.76900480       1.91212429       3.70622134

Hopefully these examples are demonstrative enough...

[abhijeetgangan]

@corinwagen Good examples! Personally I haven't seen much comparison with molecules / molecular crystals so I think there is a compelling case to add BFGS as you demonstrated above.

[CompRhys]

c.f. https://github.com/erwallace/neural-optimiser

[ysanspeur]

Hi @orionarcher! Wanted to ask what the status/priority is for this feature? We are seeing something similar when we try to relax really large periodic slabs where is takes ~ 3 min with a traditional ASE LBFGS optimizer but takes 20 min with FIRE in torchsim and if you do not set the max steps to a very large number, sometimes it will not find a minimum comparable to what ASE LBFGS does. Thanks!

[Andrew-S-Rosen]

We (@sihoonchoi and I) are also interested in a BFGS-type implementation --- FIRE is horribly slow for many of the systems that we study here. BFGS does much better.

[orionarcher]

Thanks for the bump @Andrew-S-Rosen and @ysanspeur, I am not the right person to implement this but will poke around to see who has bandwidth to take this on.

[abhijeetgangan]

I have a implementation for BFGS and L-BFGS but I might need help with testing it with ASE. Will open a PR this week. Would appreciate if someone could benchmark it with ASE similar to FIRE.

[Andrew-S-Rosen]

I have a implementation for BFGS and L-BFGS but I might need help with testing it with ASE. Will open a PR this week. Would appreciate if someone could benchmark it with ASE similar to FIRE.

@sihoonchoi, perhaps you can benchmark on a few representative systems?