Fix the queue size in writeThreadPool exceeds the configured size

[lordcheng10]

Motivation

Fix the queue size in writeThreadPool exceeds the configured size.

I print out the queue size every time I submit a task, and then set the queue size to 1:
maxPendingAddRequestsPerThread=1

The queue size of the log output is greater than 1：

It means that the queue size set by maxPendingAddRequestsPerThread does not limit the queue size.

[lordcheng10]

/pulsarbot run-failure-checks

[lordcheng10]

/pulsarbot run-failure-checks

[lordcheng10]

/pulsarbot run-failure-checks

[lordcheng10]

rerun failure checks

[lordcheng10]

rerun failure checks

[lordcheng10]

rerun failure checks

[lordcheng10]

rerun failure checks

[lordcheng10]

rerun failure checks

[lordcheng10]

rerun failure checks

[lordcheng10]

@hangc0276 @eolivelli PTAL,thanks!

[lordcheng10]

@shoothzj PTAL,thanks!

[merlimat]

This change will cause an incredible amount of mutex contention and big performance regression.

[lordcheng10]

This change will cause an incredible amount of mutex contention and big performance regression.

Do you have any good idea about the queue size exceeding the configured size? At present, the queue size we configured cannot control the queue size in high concurrency scenarios. @merlimat

[lordcheng10]

incredible

@merlimat OK, I will try some other solutions.

[eolivelli]

a style note..... you can push synchronized to the method signature

[dlg99]

@lordcheng10 +1 to Matteo's concern about performance.
Original implementation #1309 concentrated on "lightweight" over "precise size" hence it is optimistic and I think this is ok.
checkQueue() and methods using it are not synchronized, it is possible to call it from multiple threads, pass the check and end up with more tasks in queue than the soft limit. Performance benefits eher outweigh the precision.

[hangc0276]

+1 to Matteo's concern about performance.