Skip to content

fix array_agg to work with complex types and bugs with expression aggregator complex array handling#13781

Merged
clintropolis merged 8 commits intoapache:masterfrom
clintropolis:array-agg-and-complex-types
Feb 13, 2023
Merged

fix array_agg to work with complex types and bugs with expression aggregator complex array handling#13781
clintropolis merged 8 commits intoapache:masterfrom
clintropolis:array-agg-and-complex-types

Conversation

@clintropolis
Copy link
Copy Markdown
Member

@clintropolis clintropolis commented Feb 9, 2023

Description

This PR removes an overly restrictive validation check on ARRAY_AGG which was preventing use with druid COMPLEX types, such as COMPLEX<json>. This check is a relic from when this function was added, which was before the native expression system supported COMPLEX types.

After removing this check, I ran into a couple of rough edges, and a bug in ObjectStrategyComplexTypeStrategy, which was missing a check to ensure that the positional read method actually moved the buffer position forward by the number of bytes read (useful for array_agg which is serializing arrays) instead of relying on the underlying ObjectStrategy to move the buffer.

After the bugs, the result seems pretty useful, particularly for COMPLEX<json> (though certainly not the speediest thing around since its implemented as an expression aggregator):
Screenshot 2023-02-09 at 3 06 39 AM
even distinct works:

Screenshot 2023-02-09 at 3 06 05 AM

While here, I also fixed a bug with the output type inference of ARRAY_OFFSET and ARRAY_ORDINAL functions, as well as made native ARRAY typed expressions more consistently honor druid.generic.useDefaultValueForNull.

This PR has:

  • been self-reviewed.
  • a release note entry in the PR description.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@clintropolis clintropolis merged commit f09f836 into apache:master Feb 13, 2023
@clintropolis clintropolis deleted the array-agg-and-complex-types branch February 13, 2023 06:01
@clintropolis clintropolis added this to the 26.0 milestone Apr 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants