More accurate dictionary size estimation in RowBasedKeySerde

[jihoonson]

The original discussion is #4704 (comment).

RowBasedKeySerde currently builds a dictionary at query time. To avoid using too many memory, the dictionary size is limited by user configuration. However, the current dictionary size estimation is based on a rough calculation.

JOL might be an alternative for more accurate dictionary size estimation if its overhead is not so large.

[leventov]

What do you mean by "overhead" here? I thought the only concern with it might be that it only supports HotSpot JVM (though I'm not sure). But it could be fine, if Druid doesn't really aim to support other JVMs. Apart from that, it should be perfectly fine to use JOL.

Note e. g. that Presto, a high-profile project similar to Druid, uses it.

[leventov]

Ok, and another issue could be the license, because JOL is GPL.

[jihoonson]

I meant the overhead to get the size of each string in dictionary because it will be a very frequent operation.

Ok, and another issue could be the license, because JOL is GPL.

This is a really sad news..

[leventov]

@jihoonson see #4771

[leventov]

Or, if still using this approach, byte-buddy-agent library could be used to obtain Instrumentation instance, and then calling Instrumentation.getObjectSize().

[jihoonson]

#5583 might be related as well.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[jihoonson]

Still relevant.

[stale]

This issue is no longer marked as stale.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.