Skip to content

GroupedHashAggregateStream should create smaller spill batches #8003

@milenkovicm

Description

@milenkovicm

Is your feature request related to a problem or challenge?

At the moment GroupedHashAggregateStream will spill state as a single batch, which is not optimal when merging as it loads whole file in memory as a single batch.

Describe the solution you'd like

I'd like to spit spill batch into smaller chunks with default batch size same as default batch size set in configuration property.

Describe alternatives you've considered

I have considered setting batch size to a fixed size or read from configuration property, but at the moment I did not do it as it would be bigger change.

Additional context

Relates to #7858

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions