[Proposal] Adding double presion mertrics.

[b-slim]

Proposal Adding Double precision metrics.

Currently Druid stores double aggregated numbers as floats which leads to lose of precision and not expected results set for the case of Min/Max where you sort of expect to see the same precision of the actual indexed data.
The goal of this work is to add support of actual double indexed columns.
There is several ways on how to get there would like to poll the community about which way to go.

Option 1

Make the double aggregator store numbers as double period !

• This might be radical and surprise druid user since the storage footprint of double columns will double.
• Less code duplication (will need to keep float ser/deser code to ensure backward compatible).

Option 1+

If storage footprint is a deal breaker, do the same as option 1 and add new suite of FloatsXXXAggregators.

• Make it more flexible if storage is an issue.
• Lot of code duplication.

Option 2

Instead of modifying existing Double aggregator, create new set of aggregator called DoubleSum64...etc.

• lot of code duplication
• some how confusing and ugly to have DoubleSum64 and DoubleSum etc
• probably the most transparent for current druid users.

Option 3

Add a flag to indexSpec ie "StoreDoubleAsFloat" and set it to true/false

• less code duplication that 1.1 and 1.2
• need to have a new precision flag within the column to find out how to deserialize the column (or Maybe not haven't fleshed out this yet but it is doable).
• same as option 2 don't see why index double numbers then store it as floats.

[b-slim]

related to #2252 and #3294

[gianm]

I prefer option 1+ or 3.

I'd avoid option 1 since storing floats as 32-bit works for a lot of people, and doubling their width would make segments bigger for no good reason. And I'd avoid option 2 since it's too weird in terms of API (DoubleSum and DoubleSum64 would be super confusing to new users, who would rightly think: why is there a difference? Isn't a double 64 bits?).

It'd be great to add real support for doubles throughout the query runtime. The current state of things led to the following (pretty sad) docs for Druid SQL:

Druid natively supports four main column types: "long" (64 bit signed int), "float" (32 bit float), "string" (UTF-8 encoded strings), and "complex" (catch-all for more exotic data types like hyperUnique and approxHistogram columns). [...] At runtime, Druid will widen floats to "double" (64 bit float) for certain features, like SUM aggregators. But this widening is not universal; some floating point operations retain 32 bit precision.

Right now it's difficult for users to understand when floating point operations are 32 bit and when they're 64 bit.

[b-slim]

@gianm thanks, i have started working on 1+, i preferred this one since it has less config to worry about form the user perspective.

[himanshug]

+1 4 1+

[drcrallen]

We will need to NOT modify druid segment size footprint uncontrollably. We already have considerations for page cache thrashing and shaking that up is very problematic from our side. So whatever the solution, as long as we have the ability to retain the current on disk sizing and precision, and a smooth upgrade path, and no performance degradation for the prior behavior (float-on-disk --> double-agg), it should be good.

[leventov]

is it fixed?

[b-slim]

fixed by #4491