Skip to content

KAFKA-13599: Upgrade RocksDB to 6.27.3#11690

Merged
cadonna merged 3 commits intoapache:trunkfrom
linux-on-ibm-z:rocksdbjni-6.27.3
Feb 2, 2022
Merged

KAFKA-13599: Upgrade RocksDB to 6.27.3#11690
cadonna merged 3 commits intoapache:trunkfrom
linux-on-ibm-z:rocksdbjni-6.27.3

Conversation

@jonathan-albrecht-ibm
Copy link
Copy Markdown
Contributor

RocksDB v6.27.3 has been released and it is the first release to support s390x. RocksDB is currently the only dependency in gradle/dependencies.gradle without s390x support.

RocksDB v6.27.3 has added some new options that require an update to streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java but no other changes are needed to upgrade.

I have run the unit/integration tests locally on s390x and also the :streams tests on x86_64 and they pass.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@cadonna
Copy link
Copy Markdown
Member

cadonna commented Jan 19, 2022

@jonathan-albrecht-ibm Thank you for the PR! I will review it this or next week.

@jonathan-albrecht-ibm
Copy link
Copy Markdown
Contributor Author

Thanks @cadonna!

@jonathan-albrecht-ibm
Copy link
Copy Markdown
Contributor Author

Hi @cadonna, just wondering if you are still able to review this week or if it will be pushed out to next? Not trying to rush anything, I appreciate the time you can spend on it.

@cadonna
Copy link
Copy Markdown
Member

cadonna commented Jan 28, 2022

Sorry, I got sick and need to move the review to next week.

@jonathan-albrecht-ibm
Copy link
Copy Markdown
Contributor Author

Thanks @cadonna. Get well soon

Copy link
Copy Markdown
Member

@showuon showuon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-albrecht-ibm , thanks for the PR. Overall lgtm. Just a minor comment left.

@cadonna , get well soon!

}

@Override
public Options setEnableBlobFiles(final boolean enableBlobFiles) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Started here, they are options for blobsDB. In the upstream Options.java, there are comments at the beginning and the end of the options. Could we add something similar here?

 //
  // BEGIN options for blobs (integrated BlobDB)
  //
 //
  // END options for blobs (integrated BlobDB)
  //

ref: https://github.com/facebook/rocksdb/blob/main/java/src/main/java/org/rocksdb/Options.java#L1996-L1998

Thanks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR @showuon! I've added in the comments, let me know how it looks.

Copy link
Copy Markdown
Member

@cadonna cadonna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-albrecht-ibm Thanks for the PR!

LGTM!

@cadonna
Copy link
Copy Markdown
Member

cadonna commented Feb 1, 2022

I re-triggered the builds. Once the builds are acceptable, I will merge.

@cadonna
Copy link
Copy Markdown
Member

cadonna commented Feb 2, 2022

Failing test are unrelated:

Build / JDK 11 and Scala 2.13 / kafka.admin.LeaderElectionCommandTest.[1] Type=Raft, Name=testTopicPartition, Security=PLAINTEXT
Build / JDK 11 and Scala 2.13 / kafka.api.TransactionsExpirationTest.[2] quorum=kraft
Build  / JDK 17 and Scala 2.13 / kafka.admin.LeaderElectionCommandTest.[1]  Type=Raft, Name=testAllTopicPartition, Security=PLAINTEXT
Build / JDK 17 and Scala 2.13 / kafka.admin.LeaderElectionCommandTest.[1] Type=Raft, Name=testTopicPartition, Security=PLAINTEXT
Build / JDK 17 and Scala 2.13 / kafka.api.TransactionsTest.testAbortTransactionTimeout()
Build / JDK 17 and Scala 2.13 / kafka.server.KRaftClusterTest.testCreateClusterAndCreateListDeleteTopic()
Build / JDK 17 and Scala 2.13 / kafka.network.SocketServerTest.remoteCloseWithIncompleteBufferedReceive()
Build / JDK 8 and Scala 2.12 / kafka.network.SocketServerTest.remoteCloseWithIncompleteBufferedReceive()

@cadonna cadonna merged commit ec05f90 into apache:trunk Feb 2, 2022
@jonathan-albrecht-ibm
Copy link
Copy Markdown
Contributor Author

Thanks @cadonna!

cadonna pushed a commit that referenced this pull request Nov 16, 2022
RocksDB v6.27.3 has been released and it is the first release to support s390x. RocksDB is currently the only dependency in gradle/dependencies.gradle without s390x support.

RocksDB v6.27.3 has added some new options that require an update to streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java but no other changes are needed to upgrade.

I have run the unit/integration tests locally on s390x and also the :streams tests on x86_64 and they pass.

Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
cadonna pushed a commit that referenced this pull request Nov 16, 2022
RocksDB v6.27.3 has been released and it is the first release to support s390x. RocksDB is currently the only dependency in gradle/dependencies.gradle without s390x support.

RocksDB v6.27.3 has added some new options that require an update to streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java but no other changes are needed to upgrade.

I have run the unit/integration tests locally on s390x and also the :streams tests on x86_64 and they pass.

Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants