Skip to content

Conversation

@liukun4515
Copy link
Contributor

@liukun4515 liukun4515 commented Nov 27, 2021

Which issue does this PR close?

WIP, this pr is not ready for review.

related #122
Closes #.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

@liukun4515 liukun4515 marked this pull request as ready for review November 30, 2021 07:37
@liukun4515 liukun4515 marked this pull request as draft November 30, 2021 10:38
@liukun4515
Copy link
Contributor Author

This pull request has been split into several small pull requests.

@liukun4515 liukun4515 closed this Dec 17, 2021
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
## Which issue does this PR close?

## Rationale for this change

Current Comet unified memory pool is a greedy pool. One thread (consumer) can take a large amount of memory that causes OOM for other threads, especially for aggregation.

## What changes are included in this PR?

Added a fair version of unified memory pool similar to DataFusion `FairSpilPool` that caps the memory usage at `pool_size/num`
The fair unified memory pool is the default for off-heap mode with this PR

## How are these changes tested?

Exisiting tests
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
## Which issue does this PR close?

Closes apache#1388 

## Rationale for this change

Following up on apache#1369 and apache#1386 

## What changes are included in this PR?

Updated the doc

## How are these changes tested?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant