KAFKA-4222: Adjust timeout in low resource environments

[enothereska]

No description provided.

[enothereska]

This other PR is also needed of course: #2982

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3649/
Test PASSed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3643/
Test PASSed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3639/
Test FAILed (JDK 8 and Scala 2.12).

[mjsax]

@enothereska Can you elaborate in this PR. With #2982 I am wondering why we need this?

[enothereska]

@mjsax yes, I tested all the streams test in an ultra-low resource environment (1 wimpy core, 1GB of RAM) and even with the RocksDB fix the queryOnRebalance test would fail. Sometimes the network would appear flaky, requiring more retries (I've changed to 10). Sometimes the rebalance would take forever, requiring new timeouts.

[enothereska]

cc @guozhangwang

[guozhangwang]

@eno I'm still wondering why the rebalance could take forever, since we only have a single thread that will own all the tasks, and it has graceful shutdown / re-start so there should be no restoration at all?

[enothereska]

In a low resource env, everything competes for a bit of resources, the producer thread, the main streams thread and the rebalancing. It all crawls.

[guozhangwang]

In a low resource env, everything competes for a bit of resources, the producer thread, the main streams thread and the rebalancing. It all crawls.

@enothereska Okay, I think I'm a bit over-concerned with your description that rebalance take forever since it is only a single thread, so if it really take very long to create tasks / restore it would be a problem.

[guozhangwang]

Merged to trunk.