Skip to content

Add a readOnly() method for PartitionedOutputChannel#13755

Merged
cryptoe merged 16 commits intoapache:masterfrom
LakshSingla:supersorter-oom
Mar 10, 2023
Merged

Add a readOnly() method for PartitionedOutputChannel#13755
cryptoe merged 16 commits intoapache:masterfrom
LakshSingla:supersorter-oom

Conversation

@LakshSingla
Copy link
Copy Markdown
Contributor

Description

With SuperSorter using the PartitionedOutputChannels for sorting, it might OOM on inputs of reasonable size because the channel consists of both the writable frame channel and the frame allocator, both of which are not required once the output channel has been written to.
This change adds a readOnly to the output channel which contains only the readable channel, due to which unnecessary memory references to the writable channel and the memory allocator are lost once the output channel has been written to, preventing the OOM.


Key changed/added classes in this PR
  • PartitionedOutputChannel
  • SuperSorter

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@cryptoe cryptoe added the Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 label Feb 6, 2023
})::get;
writableFrameChannelsBuilder.add(() -> channel.get().getWritableChannel());
readableFrameChannelSuppliersBuilder.add(() -> channel.get().getReadableChannelSupplier().get());
readableFrameChannelSuppliersBuilder.add(() -> channel.get().readOnly().getReadableChannelSupplier().get());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will open a readOnly channel for all ComposingOutputChannel's. That shouldn't be the case no ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please add a comment saying that you should only request a readable channel if the outputchannel is completely written too.

@LakshSingla
Copy link
Copy Markdown
Contributor Author

After debugging, it was found that there can be the following places of memory leaks (before this patch):

  1. The SuperSorter holds the PartitionedOutputChannels' map which contains the writableChannel and the memory allocator, which are not useful once the channel has been written to. Therefore while storing the PartitionedOutputChannels, we convert them to readOnly() which throws away the reference to the writableChannel and the memory allocator allowing them to be GCed and thereby reducing the footprint of the SuperSorter. Attached below is a heap dump before the PR

Screenshot 2023-03-03 at 11 52 00 AM

  1. The output channels created by ComposingOutputChannelFactory contain a collection of writableChannels and the readableChannelSuppliers (from which it is composed off). Even if we throw away the reference to the frame memory allocators of the original channel, the readableChannelSuppliers still hold the reference to the memory allocators of the individual output channels. To alleviate this, while building the readableChannelSuppliers in the ComposingOutputChannelFactory, we only get the readOnly() version of the output channel.

  2. Another potential memory hog is when the output channels created by the ComposingOutputChannelFactory contain more than one outputChannel. Once an output channel is exhausted in the composition, we move on to the next output channel and never write to the older one again. However, we still hold the reference to the memory allocator of the older channels because the code assumes that we can write on it again (till the composition itself is marked as readOnly()). Therefore multiple memory allocators can be held up for a composition, even though we require one at a time. While the future ones are created lazily, the older ones are not closed, and this PR addresses that as well.

Copy link
Copy Markdown
Contributor

@cryptoe cryptoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add some UT's to this change.

composingWritableFrameChannel.write(new FrameWithPartition(Mockito.mock(Frame.class), 2));
composingWritableFrameChannel.write(new FrameWithPartition(Mockito.mock(Frame.class), 3));

partitionToChannelMap.get(0);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be empty no ?

@cryptoe
Copy link
Copy Markdown
Contributor

cryptoe commented Mar 7, 2023

@LakshSingla The failures look legit.

@cryptoe cryptoe merged commit 5b0b3a9 into apache:master Mar 10, 2023
@clintropolis clintropolis added this to the 26.0 milestone Apr 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants