KAFKA-4484: Set more conservative default values on RocksDB for memory usage

[dguy]

Lowered the default RocksDB settings for the block cache and write buffers

[dguy]

@guozhangwang @mjsax @enothereska
I've been doing some perf testing today with 3 different configurations for this. And have opted to halve the current settings, so 50MB block cache and 16MB write buffers.
This results in about a 10% drop in throughput on the SimpleBenchmark for the standard state store test. And the difference is negligible for the join test. Numbers below:

Block Cache 100MB. Write buffers 32MB
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/128282/233859.77767730466/24.079761774839803
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/105610/284064.0090900483/29.24912413597197
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/106104/282741.4612078715/29.11294578903717

Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000030/112141/133760.5157792422/13.768380877645107
Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/117062/128137.22642702158/13.189591840221421
Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/122080/122870.249017038/12.647444298820446


Block Cache 50MB. Write buffers 16MB
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/119193/251692.63295663337/25.91595144010135
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/114822/261273.97188692063/26.902509971956594
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/119311/251443.70594496734/25.890320255466804

Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/109789/136625.70931514085/14.0633396788385
Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/113989/131591.64480783232/13.545166638886208
Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/116520/128733.26467559217/13.250944043940954
	


Block Cache 25MB. Write buffers 8MB
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/134337/223318.96647982314/22.994409581872457
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/126444/237259.1819303407/24.429787099427415	
Streams Performance [records/latency/rec-sec/MB-sec source+store]: 30000000/129629/231429.69551566395/23.82954431492953

Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/113363/132318.30491430184/13.61996418584547
Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000000/115488/129883.62427265171/13.369354391798282
Streams KTableKTable LeftJoin Performance [records/latency/rec-sec/MB-sec joined]: 15000006/116558/128691.3553767223/13.246623998352751

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1595/
Test PASSed (JDK 8 and Scala 2.12).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/1598/
Test PASSed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/1595/
Test PASSed (JDK 7 and Scala 2.10).

[guozhangwang]

This is GREAT! Thanks @dguy , will take a look soon.

[mjsax]

Just discussed with @guozhangwang why it is "great" that we get a "throughput drop" -- I was confused about this comment. But that's not what he meant. The "great" is for the general investigation done :)

[enothereska]

@dguy can we try the "count" test that does aggregates? (albeit each key happens only once, so kind of worst case perf).

[enothereska]

How are these changes comparable to what Henry has done to fine tune performance (see description on the JIRA)? Have we understood the implication of changing the cache size on a variety of workloads? We are in the middle of doing performance tests to understand this behaviour, I wouldn't recommend lowering these values just yet. Thanks.

[dguy]

@enothereska this task has nothing to do with performance tuning. It is about lowering some settings so that by default we are more conservative on memory.

[enothereska]

@dguy I tried Henry's configuration, the only thing that made a difference for me was the compaction set to LEVEL (not UNIVERSAL) and compression set to SNAPPY. Both of them improved performance quite a bit. Do you want to make those two changes too in this JIRA so we have 1 JIRA with the changes? So in RocksDbStore:

private static final CompressionType COMPRESSION_TYPE = CompressionType.SNAPPY;
private static final CompactionStyle COMPACTION_STYLE = CompactionStyle.LEVEL;

[dguy]

@enothereska I've done further testing of this with the settings you have suggested etc.
Settings as they currently are in this PR:

Writes/sec = 101679, Write MB/S 97, Reads/sec = 56980, Read MB/S = 54
Writes/sec = 104482, Write MB/S 100, Reads/sec = 49922, Read MB/S = 47
Writes/sec = 104652, Write MB/S 100, Reads/sec = 58885, Read MB/S = 56
Writes/sec = 105116, Write MB/S 100, Reads/sec = 55788, Read MB/S = 53

Snappy Compression

Writes/sec = 85989, Write MB/S 82, Reads/sec = 73267, Read MB/S = 69
Writes/sec = 88758, Write MB/S 84, Reads/sec = 42505, Read MB/S = 40
Writes/sec = 87545, Write MB/S 83, Reads/sec = 60225, Read MB/S = 57
Writes/sec = 86298, Write MB/S 82, Reads/sec = 45477, Read MB/S = 43

Level Compaction

Writes/sec = 63873, Write MB/S 61, Reads/sec = 84412, Read MB/S = 80
Writes/sec = 69998, Write MB/S 67, Reads/sec = 84650, Read MB/S = 80
Writes/sec = 72436, Write MB/S 69, Reads/sec = 68100, Read MB/S = 64
Writes/sec = 65296, Write MB/S 62, Reads/sec = 83869, Read MB/S = 79

[enothereska]

I'd go with the first option as they are currently on PR. Thanks. I guess this is ready to merge? LGTM.

[enothereska]

Btw, I made the array bytes random here: ed32d49

[guozhangwang]

Merged to trunk.