KAFKA-10221: Backport fix for KAFKA-9603 to 2.5#8987
Conversation
KAFKA-9603 reports that the number of open files keeps increasing in RocksDB. The reason is that bulk loading is turned on but never turned off in segmented state stores for standby tasks. This bug was fixed in 2.6 by using code that is not present in 2.5. This PR backports the fix to 2.5.
|
Call for review: @vvcephei @ableegoldman @guozhangwang @mjsax @abbccdda |
|
Test this please |
| // have the local RocksDB instance for the segment. In this case, toggleDBForBulkLoading | ||
| // will only close the database and open it again with bulk loading enabled. | ||
| if (!bulkLoadSegments.contains(segment)) { | ||
| if (!bulkLoadSegments.contains(segment) && context instanceof ProcessorContextImpl) { |
There was a problem hiding this comment.
Woah, this is subtle. IIUC, the fix works by asserting that we should only enable bulk loading if the provided context is a ProcessorContextImpl, which is the kind of context that is only provided when adding the store to an active task.
This seems correct to me, and although it's very subtle, it also seems ok as a patch for an older codebase that won't need to be maintained much. But maybe we can have a comment, or an internal method for the check, like
| if (!bulkLoadSegments.contains(segment) && context instanceof ProcessorContextImpl) { | |
| if (!bulkLoadSegments.contains(segment) && isStoreForActiveTask(context)) { |
so that it'll be more obvious what's going on here?
|
Retest this please |
|
Test this please |
1 similar comment
|
Test this please |
|
Retest this please |
|
The test failure was unrelated. Thanks for the backport, @cadonna ! I'll go ahead and merge now. |
3 participants
KAFKA-9603 reports that the number of open files keeps increasing
in RocksDB. The reason is that bulk loading is turned on but
never turned off in segmented state stores for standby tasks.
This bug was fixed in 2.6 through PR #8661 by using code that is not present in 2.5.
So cherry-picking was not possible.
This PR backports the fix to 2.5.
Committer Checklist (excluded from commit message)