update default value of allocatorPoolingConcurrency by sunshujie1990 · Pull Request #3001 · apache/bookkeeper

sunshujie1990 · 2022-01-19T12:56:23Z

Set a more reasonable default value to avoid OutOfDirectMemoryError.

Descriptions of the changes in this PR:

Motivation

The default value of allocatorPoolingConcurrency is 2 * Runtime.getRuntime().availableProcessors(). It's used to specify the num of Arena in PooledByteBufAllocator.

Assume:
40 processors
80 arena (processors2)
2 chunk per arena
16MiB per chunk
JVM's total direct mem should be larger then 802*16=2560MiB, otherwise OutOfDirectMemoryError occured.

OutOfDirectMemoryError much more likely to occur in Docker, cause JDK versions earlier than Java SE 8U131 does not support Docker CPU limits.

Changes

Use Netty default arena num PooledByteBufAllocator.defaultNumDirectArena() as default allocatorPoolingConcurrency.

Set a more reasonable default value to avoid OutOfDirectMemoryError.

sunshujie1990 · 2022-03-09T11:36:33Z

@eolivelli Please help to review it.

hangc0276 · 2022-06-24T16:18:44Z

     */
    public int getAllocatorPoolingConcurrency() {
-        return this.getInteger(ALLOCATOR_POOLING_CONCURRENCY, 2 * Runtime.getRuntime().availableProcessors());
+        return this.getInteger(ALLOCATOR_POOLING_CONCURRENCY, PooledByteBufAllocator.defaultNumDirectArena());


The default value of defaultNumDirectArena is 2 * cpus
https://github.com/netty/netty/blob/4e439264df0523fb6efce5f8f6c7c7fa74addd07/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L413-L418

The default value of defaultNumDirectArena is 2 * cpus https://github.com/netty/netty/blob/4e439264df0523fb6efce5f8f6c7c7fa74addd07/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L413-L418

Thanks for your reply. Netty default value use the min value between 2*cpus and max mem defaultChunkSize / 2 / 3 to avoid oom. And it helps me resolved bookeeper oom problem in my product environment. Can you help me to merge it?

DEFAULT_NUM_DIRECT_ARENA = Math.max(0, SystemPropertyUtil.getInt( "io.netty.allocator.numDirectArenas", (int) Math.min( defaultMinNumArena, PlatformDependent.maxDirectMemory() / defaultChunkSize / 2 / 3)));

hangc0276 · 2022-07-26T09:49:56Z

ping @merlimat @eolivelli @dlg99, Would you please help take a look? thanks.

horizonzy

Nice catch! It happend at low maxDirectMemory.

horizonzy · 2022-07-26T15:27:29Z

And there is a more suitable value for it.

In we use DbLedgerStorage, it will occupy 1/2 unpooled direct memory for write cache and read cache, it's unpooled but occupy direct memory, we shuold consider it.

When we calculate the concurrency, we should subtract the memory which writeCache and readCache total occupy.

See:

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java

Lines 153 to 156 in 39a9c28

    
           long writeCacheMaxSize = getLongVariableOrDefault(conf, WRITE_CACHE_MAX_SIZE_MB, 
        
                   DEFAULT_WRITE_CACHE_MAX_SIZE_MB) * MB; 
        
           long readCacheMaxSize = getLongVariableOrDefault(conf, READ_AHEAD_CACHE_MAX_SIZE_MB, 
        
                   DEFAULT_READ_CACHE_MAX_SIZE_MB) * MB;

.

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/WriteCache.java

Line 116 in 39a9c28

    
           cacheSegments[segmentsCount - 1] = Unpooled.directBuffer(lastSegmentSize, lastSegmentSize);

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/ReadCache.java

Line 79 in 39a9c28

cacheSegments.add(Unpooled.directBuffer(segmentSize, segmentSize));

hangc0276 · 2022-07-27T01:57:23Z

ping @sunshujie1990, would you please take a look at the last comment?

sunshujie1990 · 2022-07-27T02:34:57Z

ping @sunshujie1990, would you please take a look at the last comment?

sure

eolivelli

LGTM

StevenLuMT

lgtm

StevenLuMT · 2022-07-27T15:54:53Z

if no problem, please merge it,thanks @eolivelli @dlg99 @hangc0276

Set a more reasonable default value to avoid OutOfDirectMemoryError. Descriptions of the changes in this PR: ### Motivation The default value of allocatorPoolingConcurrency is 2 * Runtime.getRuntime().availableProcessors(). It's used to specify the num of Arena in PooledByteBufAllocator. Assume: 40 processors 80 arena (processors*2) 2 chunk per arena 16MiB per chunk JVM's total direct mem should be larger then 80*2*16=2560MiB, otherwise OutOfDirectMemoryError occured. OutOfDirectMemoryError much more likely to occur in Docker, cause JDK versions earlier than Java SE 8U131 does not support Docker CPU limits. ### Changes Use Netty default arena num `PooledByteBufAllocator.defaultNumDirectArena()` as default allocatorPoolingConcurrency. (cherry picked from commit 01c8824)

Set a more reasonable default value to avoid OutOfDirectMemoryError. Descriptions of the changes in this PR: ### Motivation The default value of allocatorPoolingConcurrency is 2 * Runtime.getRuntime().availableProcessors(). It's used to specify the num of Arena in PooledByteBufAllocator. Assume: 40 processors 80 arena (processors*2) 2 chunk per arena 16MiB per chunk JVM's total direct mem should be larger then 80*2*16=2560MiB, otherwise OutOfDirectMemoryError occured. OutOfDirectMemoryError much more likely to occur in Docker, cause JDK versions earlier than Java SE 8U131 does not support Docker CPU limits. ### Changes Use Netty default arena num `PooledByteBufAllocator.defaultNumDirectArena()` as default allocatorPoolingConcurrency. (cherry picked from commit 01c8824) (cherry picked from commit 35d0ecf)

Set a more reasonable default value to avoid OutOfDirectMemoryError. Descriptions of the changes in this PR: ### Motivation The default value of allocatorPoolingConcurrency is 2 * Runtime.getRuntime().availableProcessors(). It's used to specify the num of Arena in PooledByteBufAllocator. Assume: 40 processors 80 arena (processors*2) 2 chunk per arena 16MiB per chunk JVM's total direct mem should be larger then 80*2*16=2560MiB, otherwise OutOfDirectMemoryError occured. OutOfDirectMemoryError much more likely to occur in Docker, cause JDK versions earlier than Java SE 8U131 does not support Docker CPU limits. ### Changes Use Netty default arena num `PooledByteBufAllocator.defaultNumDirectArena()` as default allocatorPoolingConcurrency.

update ALLOCATOR_POOLING_CONCURRENCY default value

8c28144

Set a more reasonable default value to avoid OutOfDirectMemoryError.

hangc0276 reviewed Jun 24, 2022

View reviewed changes

hangc0276 requested review from dlg99, eolivelli, hezhangjian, nicoloboschi and zymap July 25, 2022 01:47

hangc0276 assigned sunshujie1990 Jul 25, 2022

hangc0276 added area/bookie area/config labels Jul 25, 2022

hangc0276 added this to the 4.16.0 milestone Jul 25, 2022

hangc0276 approved these changes Jul 26, 2022

View reviewed changes

horizonzy approved these changes Jul 26, 2022

View reviewed changes

eolivelli approved these changes Jul 27, 2022

View reviewed changes

dlg99 approved these changes Jul 27, 2022

View reviewed changes

StevenLuMT approved these changes Jul 27, 2022

View reviewed changes

hangc0276 merged commit 01c8824 into apache:master Jul 27, 2022

horizonzy mentioned this pull request Jul 28, 2022

Tuning pool concurrency #3432

Closed

hangc0276 added release/4.14.6 release/4.15.3 type/improvement labels Oct 9, 2022

zymap added the cherry-picked/branch-4.15 label Oct 26, 2022

hangc0276 added the cherry-picked/branch-4.14 label Nov 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update default value of allocatorPoolingConcurrency#3001

update default value of allocatorPoolingConcurrency#3001
hangc0276 merged 1 commit intoapache:masterfrom
sunshujie1990:master

sunshujie1990 commented Jan 19, 2022

Uh oh!

sunshujie1990 commented Mar 9, 2022

Uh oh!

hangc0276 Jun 24, 2022

Uh oh!

sunshujie1990 Jun 25, 2022 •

edited

Loading

Uh oh!

hangc0276 commented Jul 26, 2022

Uh oh!

horizonzy left a comment

Uh oh!

horizonzy commented Jul 26, 2022

Uh oh!

hangc0276 commented Jul 27, 2022

Uh oh!

sunshujie1990 commented Jul 27, 2022

Uh oh!

eolivelli left a comment

Uh oh!

StevenLuMT left a comment

Uh oh!

StevenLuMT commented Jul 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

sunshujie1990 commented Jan 19, 2022

Motivation

Changes

Uh oh!

sunshujie1990 commented Mar 9, 2022

Uh oh!

hangc0276 Jun 24, 2022

Choose a reason for hiding this comment

Uh oh!

sunshujie1990 Jun 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hangc0276 commented Jul 26, 2022

Uh oh!

horizonzy left a comment

Choose a reason for hiding this comment

Uh oh!

horizonzy commented Jul 26, 2022

Uh oh!

hangc0276 commented Jul 27, 2022

Uh oh!

sunshujie1990 commented Jul 27, 2022

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

StevenLuMT left a comment

Choose a reason for hiding this comment

Uh oh!

StevenLuMT commented Jul 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sunshujie1990 Jun 25, 2022 •

edited

Loading