Unable to upgrade bookie to 2.5.0

[gperinazzo]

Describe the bug
When upgrading an existing 2.4.1 bookie cluster to 2.5.0 on kubernetes, the bookie fails to start with the following exception during initialization: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2147483648, max: 2147483648)

The node where the bookie was deployed has more than enough free memory. Rolling the bookie back to versions 2.4.1 or 2.4.2 allows the bookie to initialize successfully.

To Reproduce
Steps to reproduce the behavior:

1. Have an existing 2.4.1 bookie cluster on kuberentes
2. Update the image on the stateful set definition and set it to start in read only mode.
3. Restart one of the bookies to update the image version
4. See error on the bookie logs

Expected behavior
Upgrading a bookie should work

Additional context
bookie.log

[gperinazzo]

Seems that reducing dbStorage_writeCacheMaxSizeMb and dbStorage_readAheadCacheMaxSizeMb to 512 each allows the bookie to initialize, while setting it to 2048 (after adjusting the MaxDirectMemorySize to 14g) gives me the following error: https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java#L114

Maybe netty is not picking the memory setting correctly?

[merlimat]

Seems like the max memory size is set to 2GBs, are you setting directly the dbStorage_writeCacheMaxSizeMb or leaving them empty?

[gperinazzo]

Seems like the max memory size is set to 2GBs, are you setting directly the dbStorage_writeCacheMaxSizeMb or leaving them empty?

I had them set at first, but then tested leaving it empty. It defaults to 512Mb, no matter what I set XX:MaxDirectMemorySize to (tried up to 14Gb)

[merlimat]

It defaults to 512Mb

It should get set to 1/4th of overall direct mem size:

https://github.com/apache/bookkeeper/blob/163db4496d50cae198b80c9d88c14579f6d2e9f3/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java#L83

[gperinazzo]

Yes, it's seeing a max direct memory of 2gb regardless of what I set it. The same settings worked properly for pulsar 2.4.x

[merlimat]

Got it.

It works for standalone:

-XX:MaxDirectMemorySize=4g

11:48:26.691 [main] INFO  org.apache.bookkeeper.bookie.Bookie - Using ledger storage: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
11:48:26.693 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Started Db Ledger Storage
11:48:26.694 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Number of directories: 1
11:48:26.694 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Write cache size: 1024 MB
11:48:26.694 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Read Cache: 1024 MB

I think this got badly broken in #5675 because it's applying the mem setting from conf/bkenv.sh.

[gperinazzo]

I just tested it and I can confirm that #5675 is the reason: I added BOOKIE_MEM to my deployment and bin/apply-config-from-env.py conf/bkenv.sh to the container command (don't think this is necessary, but it's done for pulsar_env.sh), and the default has changed to the expected value given my MaxDirectMemorySize, as expected.

This should be changed in the example kubernetes yaml and helm chart, as they currently don't have this environment variable set. Maybe also add a note about this in the migration guide.