You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In good old days Druid Double Sum/Min/Max/First/Last Aggregators would aggregate double values during ingestion then this double aggregated value will be stored as a truncated float. Effectively you send to Druid a double number x and druid will store it as (float) x thus this can save 32 bits at the expense of more error accumulations/confusions.
PR #4491 has made the following changes to address the problem.
Added Float Aggregator factory that can be used if the user is okay with 32 bits floating numbers.
Made Double Aggregator factory use actual 64bits floating point numbers.
PRs 4491/4944 were released as part of 0.11 thus any cluster running 0.11 is capable of reading segments where Doubles are stored as 64 bits also all the indexing nodes understand float aggregators.
As discussed in #4605 the plan was to make Double aggregator use 64bits by default in 0.12.
As part of the last dev meeting, I proposed to release double as 64bits by default. FYI this is not a blocker for 0.12 but i will be nice to get this done one day.
In Addition to that @leventov had some concerns about keeping the same names while changing the implementation. He suggested that we add new kind of aggregator and deprecated old ones, @roman feel free to elaborate more on your point of view.
IMO it will be confusing more than helping the user if we add new aggregators and deprecate the old one.
In addition, we are not saving the user anything because at the end of the day you have to either move out from the deprecated ones or use the float aggregators instead of double.
In good old days Druid Double Sum/Min/Max/First/Last Aggregators would aggregate double values during ingestion then this double aggregated value will be stored as a truncated float. Effectively you send to Druid a double number
xand druid will store it as(float) xthus this can save 32 bits at the expense of more error accumulations/confusions.PR #4491 has made the following changes to address the problem.
PRs 4491/4944 were released as part of 0.11 thus any cluster running 0.11 is capable of reading segments where Doubles are stored as 64 bits also all the indexing nodes understand float aggregators.
As discussed in #4605 the plan was to make Double aggregator use 64bits by default in 0.12.
As part of the last dev meeting, I proposed to release double as 64bits by default.
FYI this is not a blocker for 0.12 but i will be nice to get this done one day.
In Addition to that @leventov had some concerns about keeping the same names while changing the implementation. He suggested that we add new kind of aggregator and deprecated old ones, @roman feel free to elaborate more on your point of view.
IMO it will be confusing more than helping the user if we add new aggregators and deprecate the old one.
In addition, we are not saving the user anything because at the end of the day you have to either move out from the deprecated ones or use the float aggregators instead of double.
https://groups.google.com/d/msg/druid-development/VEYkW_5dWd8/AKL6_uk5AwAJ