Bias terms are also regularized.

[zgrkpnr]

match r with
        | L1Reg l -> fun (w:DV) -> l * (DV.l1norm w)
        | L2Reg l -> fun w -> l * (DV.l2normSq w)
        | NoReg -> fun _ -> D 0.f

This regularization methods penalizes bias terms, as well. Bias terms should be excluded from being penalized.

[smoothdeveloper]

@zgrkpnr the library doesn't have tests yet, could you write a bit of code along with expected and actual results showing the problem with regularization involving bias terms and it will be put under unit test.

[zgrkpnr]

@smoothdeveloper Are you asking for a code for fix or do you want me to show that regularizing bias terms causes weak performance? If latter, it is not necessary. Perceptron theory suggests that bias terms don't depend on input values and they have nothing to do with overfitting. Therefore, they are excluded from regularization. Moreover, regularization is done in order to obtain a "low bias" model.

[smoothdeveloper]

@zgrkpnr I was looking for a complete code sample exercising regularization and including bias terms to look at.

From what you describe, the outcome of the test should be identical results but obtained faster.

When a bug is found in a library we tend to write "non-regression test" for it, tests are run any time a change is made to the library to make sure we didn't introduce known issues back.

With such sample I could maybe write a test which shows that the implementation is failing, and when the implementation gets fixed the test would be passing and adding some security for later releases.