[improve][misc] Set format_version=5, checksum=kxxHash in Bookkeeper RocksDB configs

[lhotari]

Motivation

RocksDB format_version 5 has been supported since RocksDB 6.6 . It's required for certain performance optimizations.
Currently the default is format_version=2 which is really old. It's better to default to a more modern version.

Changes

• specify format_version=5 and checksum=kxxHash in Bookkeeper RocksDB config files
  • Bookkeeper RocksDB config is specified in separate files since change rocksdb init: use OptionsUtil bookkeeper#3056
• fix invalid CFOptions config (fix on BK side in fix[rocksdb]: fix error rocksdb default config for CFOptions bookkeeper#4466)
• make max_background_jobs match pre-4.15 configuration for entry location db.
  • In BK 4.15, the parallelism was changed from 32 to 2 which is a significant change and regression.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[nodece]

Could you explain the format_version compatible issue?

1. There are RocksDB data with format_version=2 and then migrate format_version to 5 from 2
2. There are RocksDB data with format_version=5 and then migrate format_version to 2 from 5

[lhotari]

Could you explain the format_version compatible issue?

1. There are RocksDB data with format_version=2 and then migrate format_version to 5 from 2
2. There are RocksDB data with format_version=5 and then migrate format_version to 2 from 5

@nodece The option is used for new database files. It won't prevent reading existing database files created with a previous format_version such as 2. The format_version setting is ignored for existing database files (new files created in compaction will get written with configured format_version).
format_version=2 is really old and there shouldn't be any reason to use it.
format_version=5 is the default between RocksDB version >= 6.6 and < 9.0.

There's a problem in the current config files that format_version isn't set for conf/ledger_metadata_rocksdb.conf. It will use whatever is the default. This is a problem after upgrading to RocksDB 9.0+ since it would use format_version=6 and it wouldn't be possible to downgrade to older RocksDB versions because of that.
That's why it's necessary to specify the format_version.