Make maxConcurrentZkRequests for gc configurable #2797
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nicoloboschi
requested changes
Sep 17, 2021
Contributor
nicoloboschi
left a comment
nicoloboschi
left a comment
There was a problem hiding this comment.
Good job, I think we should rename the property avoiding to mention Zookkeeper
| protected static final String GC_WAIT_TIME = "gcWaitTime"; | ||
| protected static final String IS_FORCE_GC_ALLOW_WHEN_NO_SPACE = "isForceGCAllowWhenNoSpace"; | ||
| protected static final String GC_OVERREPLICATED_LEDGER_WAIT_TIME = "gcOverreplicatedLedgerWaitTime"; | ||
| protected static final String GC_OVERREPLICATED_LEDGER_MAX_CONCURRENT_ZK_REQUESTS = |
Contributor
There was a problem hiding this comment.
since this property is not tightly related to ZK but it's more a logical option, IMHO we should not mention ZK in the name of the property
GC_OVERREPLICATED_LEDGER_MAX_CONCURRENT_REQUESTS ?
Contributor
Author
|
@nicoloboschi |
eolivelli
approved these changes
Sep 18, 2021
Contributor
eolivelli
left a comment
eolivelli
left a comment
There was a problem hiding this comment.
Lgtm
I left a comment about a small typo
| this.zkLedgersRootPath = ZKMetadataDriverBase.resolveZkLedgersRootPath(conf); | ||
| LOG.info("Over Replicated Ledger Deletion : enabled=" + enableGcOverReplicatedLedger + ", interval=" | ||
| + gcOverReplicatedLedgerIntervalMillis); | ||
| LOG.info("Over Replicated Ledger Deletion : enabled={}, interval={}, maxConcurrentRequest={}", |
Contributor
There was a problem hiding this comment.
Typo: maxConcurrentRequest > maxConcurrentRequests
Contributor
Author
There was a problem hiding this comment.
@eolivelli
can you please review again and merge this PR?
Contributor
Author
|
rerun failure checks |
1 similar comment
Contributor
Author
|
rerun failure checks |
dlg99
approved these changes
Sep 30, 2021
zymap
pushed a commit
that referenced
this pull request
Oct 22, 2021
### Motivation - Issue is as described in [PR#2797](#2797). > In one day, zookeepers became high cpu usage and disk full. > The cause of this is bookie's gc of overreplicated ledgers. > Gc created/deleted zk nodes under /ledgers/underreplication/locks very frequently and some bookies ran gc at same time. > As a result, zookeepers created a lot of snapshots and became disk full. - I want to reduce the number of lock node creations and deletions in ZK. ### Changes - Add an ensemble check before creating the lock node. This is to reduce the number of lock node creations and deletions in ZK. - ~~If [PR#2797](#2797) was merged, this PR needs to be fixed.~~
zymap
pushed a commit
that referenced
this pull request
Oct 26, 2021
### Motivation - Issue is as described in [PR#2797](#2797). > In one day, zookeepers became high cpu usage and disk full. > The cause of this is bookie's gc of overreplicated ledgers. > Gc created/deleted zk nodes under /ledgers/underreplication/locks very frequently and some bookies ran gc at same time. > As a result, zookeepers created a lot of snapshots and became disk full. - I want to reduce the number of lock node creations and deletions in ZK. ### Changes - Add an ensemble check before creating the lock node. This is to reduce the number of lock node creations and deletions in ZK. - ~~If [PR#2797](#2797) was merged, this PR needs to be fixed.~~ (cherry picked from commit 53954ca)
hrsakai
force-pushed
the
make_max_zk_requests_configurable
branch
from
6fd01be to
24c1dd1
Compare
hrsakai
force-pushed
the
make_max_zk_requests_configurable
branch
from
24c1dd1 to
0118a27
Compare
Contributor
Author
|
PTAL Rebased to newer version of master and resolved conflicts. |
Contributor
Author
|
rerun failure checks |
Ghatage
pushed a commit
to sijie/bookkeeper
that referenced
this pull request
Jul 12, 2024
### Motivation - Issue is as described in [PR#2797](apache#2797). > In one day, zookeepers became high cpu usage and disk full. > The cause of this is bookie's gc of overreplicated ledgers. > Gc created/deleted zk nodes under /ledgers/underreplication/locks very frequently and some bookies ran gc at same time. > As a result, zookeepers created a lot of snapshots and became disk full. - I want to reduce the number of lock node creations and deletions in ZK. ### Changes - Add an ensemble check before creating the lock node. This is to reduce the number of lock node creations and deletions in ZK. - ~~If [PR#2797](apache#2797) was merged, this PR needs to be fixed.~~
Ghatage
pushed a commit
to sijie/bookkeeper
that referenced
this pull request
Jul 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
In one day, zookeepers became high cpu usage and disk full.
The cause of this is bookie's gc of overreplicated ledgers.
Gc created/deleted zk nodes under
/ledgers/underreplication/locksvery frequently and some bookies ran gc at same time.As a result, zookeepers created a lot of snapshots and became disk full.
I want to configure max zk concurrent requests lower than 1000(default) to avoid heavy traffic at a specific time.
Changes