ISSUE-60: Fix for division by zero caused by float precision#61
Conversation
|
Have you checked what happens with numbers between 0.0 and 1.0 ? goes everything OK? |
5fe96b9 to
86cc13c
Compare
|
@castarco Yeah I did check, but I've updated the test to include a sample to verify that a number between 0.0 and 1.0 still behaves the same as prior to the change. The guard I've added only guards against negative numbers after the log approx. is calculated which would otherwise cause the division by zero (number pow negative = 0). This guard doesn't necessarily correct the behaviour (perhaps the approx log10 behaviour needs to take into account the natural precision vs just the values string length?) however does guard against the div by zero. I've manually verified several cases that we've encountered. There is some inaccuracies with precision (the sample I've provided performs a Is there some other specific sample cases you'd like my to verify? |
Aims to resolve #60
Note: This fix is based on my assumption that for numbers where the
$value_length<$in_scale + 1the$value_log10_approxshould be clamped to zero. This matches current test expectations (and log10 of the sample provided), but may be the incorrect place to address this (perhaps the natural scale for floats is the true source of this issue?)Proof/Testing
New regression test covering example case and asserting that log10 precision for affected case returns expected value.
Ping @castarco