refactor numeric primitive aggregators in sql compatible mode#10666
refactor numeric primitive aggregators in sql compatible mode#10666clintropolis wants to merge 4 commits intoapache:masterfrom
Conversation
|
This PR is going to fail the coverage check in CI on the 'processing' unit tests, because most of the code here is only used in SQL compatible mode. |
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions. |
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
This pull request/issue has been closed due to lack of activity. If you think that |
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
This pull request/issue has been closed due to lack of activity. If you think that |
Description
#10219 added
hasNullstoColumnCapabilities, allowing things to know if a column has any null values in SQL compatible mode (druid.generic.useDefaultValueForNull=false). With this information available,NullableNumericAggregatorFactorycan be modified to use a version of the null-aware wrapper aggregators which do not need to checkisNullon each aggregate call, which should reduce the overhead of enabling this mode for columns which do not have null values (I haven't actually measured this yet so I'm unsure of the difference).To achieve this, I have pulled out most of the logic of
NullableNumericAggregator,NullableNumericBufferAggregator, andNullableNumericVectorAggregatorinto a new set of abstract classes,NullAwareNumericAggregator,NullAwareNumericVectorAggregator, andNullAwareNumericVectorAggregatorrespectively which the former now extend. For processing columns which do not have null values, a new set of 'non-null' aggregator wrappers have been introduced,NonnullNumericAggregator,NonnullNumericBufferAggregator, andNonnullNumericVectorAggregator, which also extend the 'null-aware' base classes, so that they initialize to a null value and are compatible with the expectations of aggregator behavior with filtering (which reasonably expect an aggregator 'get' to produce the correct result, even if no values were aggregated), but can skip the 'is null check'.NullableNumericAggregatorFactory(which should probably more correctly be renamedNullAwareNumericAggregatorFactorybut its marked@ExtensionPointso I tried not to mess with it too much), has also been expanded to include a new method to check if the aggregator input has null values, with a default implementation of:with implementations that override in the 'simple' aggregator factories. Since there was a lot of duplicated code between the 'simple' aggregator factories (
SimpleLongAggregatorFactory,SimpleFloatAggregatorFactory,SimpleDoubleAggregatorFactory), I have introduced yet another base class,SimpleNumericAggregatorFactoryto consolidate all of the 'fieldName'/'expression' handling stuff.which also handles the 'hasNulls' check override of
NullableNumericAggregatorFactory. This class could probably be consolidated withNullableNumericAggregatorFactory, but since that class is@ExtensionPoint, I avoided making this change at this time.I'll try to find some time to measure the difference, but naively it seems obvious that using the 'non-null' family of aggregators should be better due to having fewer method calls and branching opportunities.
There is a potential further optimization that callers which "know" that there are values to aggregate (e.g. no filter) could avoid the extra byte of overhead used for null tracking for columns which don't have any null values by modifying the 'factorize' methods to let callers communicate this information (aggregators don't know such things), but I haven't made this modification at this time since this PR was already starting to get a bit big, so can be done as a follow-up.
This PR has:
Key changed/added classes in this PR
NullableNumericAggregatorFactoryNullAwareNumericAggregatorNullAwareNumericBufferAggregatorNullAwareNumericVectorAggregatorNullableNumericAggregatorNullableNumericBufferAggregatorNullableNumericVectorAggregatorNonnullNumericAggregatorNonnullNumericBufferAggregatorNonnullNumericVectorAggregatorSimpleNumericAggregatorFactorySimpleLongAggregatorFactorySimpleFloatAggregatorFactorySimpleDoubleAggregatorFactory