Broker change dynamic configuration based on ZK latency to reduce zk-load

[rdhabalia]

Existing behavior

Sometimes broker sees high zk-latency due to gc pauses or high load on zookeeper, and it may cause brokers to lose zk-session which can create cold restart situation and that subsequently creates high pressure on zk as cold restart loads high number of concurrent topics and lookup-requests.

Broker has capability to throttle concurrent lookup-request and topic-loading which helps to reduce back-pressure on zookeeper. with PR #320 Broker will also have capability to monitor zk-latency at real time as well.

Change

So, broker should dynamically control throttling if it sees zk-latency is going higher than configured threshold (eg. threshold=0.75*zkSessionTimeOut) which can reduce zk-load and helps broker to keep zk-session alive and avoid cold-restart. So, we can implement controller which monitors zk-stats and takes appropriate actions (can be enhanced to consider additional variables).

@merlimat @saandrews any thought?

[saandrews]

1. Can we list down what actions(e.g: topic load/unload, new subscription, ledger rollover) will be impacted when we implement this change?
2. If all of them are impacted, can we prioritize among them as to who can access zk? If we do this, would it indirectly address some of the benefits we would get if we had 2 zk clusters?
3. Should we have different thresholds for read vs. write latency?

[rdhabalia]

Can we list down what actions(e.g: topic load/unload, new subscription, ledger rollover) will be impacted when we implement this change?

for now, we have mechanism to control concurrent topic-loading and lookup-request dynamically. So, for now, action could be changing throttling limits for topic-loading and lookup-request.

If all of them are impacted, can we prioritize among them as to who can access zk? If we do this, would it indirectly address some of the benefits we would get if we had 2 zk clusters?

actually, we don't have any gate at broker which can be enabled dynamically to prioritize zk access. So, not sure what exactly we can do here.

Should we have different thresholds for read vs. write latency?

yes, but considering write latency could be enough.?

[sijie]

@rdhabalia are you guys actively working on this?

[jiazhai]

ping @rdhabalia to confirm

[rdhabalia]

@jiazhai I had some changes checked in local branch and wanted to get thoughts before creating the PR. Will try to look into it again soon. let me know if you guys have any thoughts on it.