Add a readOnly() method for PartitionedOutputChannel#13755
Add a readOnly() method for PartitionedOutputChannel#13755cryptoe merged 16 commits intoapache:masterfrom
Conversation
| })::get; | ||
| writableFrameChannelsBuilder.add(() -> channel.get().getWritableChannel()); | ||
| readableFrameChannelSuppliersBuilder.add(() -> channel.get().getReadableChannelSupplier().get()); | ||
| readableFrameChannelSuppliersBuilder.add(() -> channel.get().readOnly().getReadableChannelSupplier().get()); |
There was a problem hiding this comment.
This will open a readOnly channel for all ComposingOutputChannel's. That shouldn't be the case no ?
There was a problem hiding this comment.
Can we please add a comment saying that you should only request a readable channel if the outputchannel is completely written too.
|
After debugging, it was found that there can be the following places of memory leaks (before this patch):
|
cryptoe
left a comment
There was a problem hiding this comment.
Lets add some UT's to this change.
| composingWritableFrameChannel.write(new FrameWithPartition(Mockito.mock(Frame.class), 2)); | ||
| composingWritableFrameChannel.write(new FrameWithPartition(Mockito.mock(Frame.class), 3)); | ||
|
|
||
| partitionToChannelMap.get(0); |
|
@LakshSingla The failures look legit. |

Description
With SuperSorter using the PartitionedOutputChannels for sorting, it might OOM on inputs of reasonable size because the channel consists of both the writable frame channel and the frame allocator, both of which are not required once the output channel has been written to.
This change adds a readOnly to the output channel which contains only the readable channel, due to which unnecessary memory references to the writable channel and the memory allocator are lost once the output channel has been written to, preventing the OOM.
Key changed/added classes in this PR
PartitionedOutputChannelSuperSorterThis PR has: