*java.lang.NullPointerException* at org.xerial.snappy.BufferRecycler.releaseInputBuffer(BufferRecycler.java:153)

[bmistry13]

Hi Snappy Dev Team,

I have unique issue about the code snappy-java lib which is used by Kafka. Please see the Kafka Issue at https://issues.apache.org/jira/browse/KAFKA-1721 about the problem of NP. Stack trace is;
2014-10-20 18:55:21.841 [kafka-producer-network-thread] ERROR
org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
kafka producer I/O thread:
java.lang.NullPointerException
at
org.xerial.snappy.BufferRecycler.releaseInputBuffer(BufferRecycler.java:153)
at org.xerial.snappy.SnappyOutputStream.close(SnappyOutputStream.java:317)
at java.io.FilterOutputStream.close(FilterOutputStream.java:160)
at org.apache.kafka.common.record.Compressor.close(Compressor.java:94)
at
org.apache.kafka.common.record.MemoryRecords.close(MemoryRecords.java:119)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.drain(RecordAccumulator.java:285)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:744)

Can you guys please help me resolve this issue ? This only happen in multi-threaded environment.

Thanks,
Bhavesh

[xerial]

Thanks for reporting, i'll check this issue. Which version of snappy-java are you using?

[xerial]

This problem happens because BufferRecycler is not thread-safe. The current BufferRecycler expects SnappyOutputStream is created by multiple-threads, but if several instances of SnappyOutputStream are generated in a single thread, then used by multiple-thread, the same BufferRecycler instance is used and causes this type of error.

I'll fix this problem by introducing a thread-safe buffer recycler (or make the buffer allocation policy configurable)

[xerial]

I created a new version that probably fixes your problem. Please test this version, and if this works for you, I'll release this version as 1.1.1.4:
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.1.1.4-SNAPSHOT/

[bmistry13]

Hi Taro,

Thanks for quick patch. I will test this out today.

Thanks,

Bhavesh

On Tue, Oct 21, 2014 at 8:06 AM, Taro L. Saito notifications@github.com
wrote:

I created a new version that probably fix your problem. Please test this
version, and if this works for you, I'll release this version as 1.1.1.4:

https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.1.1.4-SNAPSHOT/

—
Reply to this email directly or view it on GitHub
#88 (comment).

[ewencp]

Thanks for addressing this quickly. I couldn't build against your jar, got this complaint:

Class org.xerial.snappy.buffer.BufferAllocatorFactory not found - continuing with a stub.

Not sure if something was just stale when you built it, but building it myself worked.

However, I think the SoftReference handling isn't right in CachedBufferAllocator. getAllocator doesn't actually check the value it gets back, so it can return null values. In fact it's even possible that when that method allocates a new BufferAllocator that it gets destroyed before it has a chance to return it. This was triggering exceptions like this:

java.lang.NullPointerException
    at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:95)
    at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:83)
    at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:74)
    at kafka.message.CompressionFactory$.apply(CompressionFactory.scala:35)

This minor patch should fix the issue: https://gist.github.com/ewencp/ef7343d21aa84b84526b. With that, I haven't been able to reproduce the original issue.

[bmistry13]

HI @ewencp ,

Thanks for patch. Do you have generated jar or any maven repo ? Otherwise, I will have to apply your patch and try again.

Thanks,

Bhavesh

[ewencp]

I didn't save it but it was really easy to build -- just clone the repo and run make.

[xerial]

Thanks for the fix @ewencp. I applied the patch and updated the snapshot jar: https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.1.1.4-SNAPSHOT/

[bmistry13]

HI Guys,

Thanks for all help ! I did not get chance to verify today. I will do this
tomorrow. Thank you all for help !!

Thanks,

Bhavesh

On Tue, Oct 21, 2014 at 5:55 PM, Taro L. Saito notifications@github.com
wrote:

Thanks for the fix @ewencp https://github.com/ewencp. I applied the
patch and updated the snapshot jar:
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.1.1.4-SNAPSHOT/

—
Reply to this email directly or view it on GitHub
#88 (comment).

[bmistry13]

Following jar has issue. I had to build my own and work. Also, I have to run 100 times etc to make sure this issue is fixed. Please bare with me. You may generate another jar new classes are missing.

https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.1.1.4-SNAPSHOT/

Thanks,

Bhavesh

[xerial]

Sorry for the broken jar. I rebuilt it and uploaded again. Thanks.

[bmistry13]

HI Taro,

Thanks for your help !! I will close the issue in couple of days.

Thanks,

Bhavesh

On Wed, Oct 22, 2014 at 5:16 PM, Taro L. Saito notifications@github.com
wrote:

Sorry for the broken jar. I rebuilt it and uploaded again. Thanks.

—
Reply to this email directly or view it on GitHub
#88 (comment).