ggml : fix quants nans when all the group weights are very close to zero

[slaren]

When the group abs max value is very close to zero but not zero, it may still result in a division by zero when computing the scale, which ends with a nan scale. To avoid this, we check the max value against an epsilon instead of zero. With the IQ quants, this could also result in a Oops: found point %u not on grid error.

While doing this, I noticed that there was already a similar check with 1e-30 epsilon in make_qx_quants, however values this small can still result in nan, so I bumped it to 1e-20 and extended it to all the cases that I could find. I used the commented code in test-backend-ops to find these cases. It is possible that an even higher epsilon may be necessary.

I don't expect this to result in lower precision in the quants since the epsilon is so small, but it may be worth checking.

Fixes #7311.

[JohannesGaessler]

There is no way the change in threshold has any significant effects on the results. Even a threshold of $10^{-10}$ should still be fine.

[slaren]

I tried progressively higher values and found that some quants still fail with 1e-8, so I increased the eps to 1e-7.

[JohannesGaessler]

While increasing GROUP_MAX_EPS by factors of 10, I could go as high as $10^{-4}$ before the results started to change for LLaMA 3 8b q6_K.

[JohannesGaessler]

To clarify, I'm not sure how generalizable my results are to other models; I think the model for which the fix is needed at least should also be checked since that particular model seems to have some blocks with only very small values.

[slaren]

I have tried to find the lowest possible eps for the quants that require lower than 1e-15, so that only the quants that actually require it use the higher eps, to reduce the risk of introducing errors. Mostly that's the IQ quants.

I don't really like this solution, I think the best way to handle this would be to check for zero before doing the division, but that would require deeper changes, the code is not very easy to follow, and don't want to risk introducing bugs that may cause models with bad quants to be distributed.