Skip to content

Upgrade lz4-java to 1.5.0#6478

Merged
b-slim merged 1 commit intoapache:masterfrom
patelh:upgrade-lz4
Oct 18, 2018
Merged

Upgrade lz4-java to 1.5.0#6478
b-slim merged 1 commit intoapache:masterfrom
patelh:upgrade-lz4

Conversation

@patelh
Copy link
Copy Markdown
Contributor

@patelh patelh commented Oct 16, 2018

The current version of lz4-java is using lz4 r123 release and the latest version of 1.5.0 is using 1.8.3 which claims to have speed improvements. lz4 r123 is from 09/2014. lz4 release notes show speed up for high compression of up to 10% and decompression speed up of 10%.

From change log:

Upgraded LZ4 to 1.8.3. Updated JNI bindings for Linux/amd64,
   Linux/ppc64le, Linux/s390x, Linux/aarch64, and Mac OS Darwin/x86_64.
   A speed-up is expected on these platforms.
   Note that the JNI bindings for Linux/i386 and Win32 still work
   but are based on old LZ4 r123. Contributions of the JNI
   bindings for these and other platforms are more than welcome.

@drcrallen
Copy link
Copy Markdown
Contributor

screen shot 2018-10-17 at 2 28 07 pm

Here's a partial flame graph from some of our hot nodes. I'll be excited if the lz4 decrypt is faster. Even 10% would be good

@b-slim b-slim merged commit f0b977e into apache:master Oct 18, 2018
@b-slim
Copy link
Copy Markdown
Contributor

b-slim commented Oct 18, 2018

@drcrallen keep us posted !

@patelh patelh deleted the upgrade-lz4 branch October 30, 2018 01:55
@drcrallen
Copy link
Copy Markdown
Contributor

screen shot 2018-11-30 at 9 06 26 am

Looking at just the mean topN segment scan times for non-zero reported scan times, this change seems to indicate an approximately 13% improvement in performance for 1.5.0 vs 1.3.0 in a real cluster.

Pretty solid and easy upgrade.

clintropolis pushed a commit to implydata/druid-public that referenced this pull request Nov 30, 2018
@patelh
Copy link
Copy Markdown
Contributor Author

patelh commented Nov 30, 2018

@drcrallen Nice! Did you notice any improvement in indexing/merging time ?

@patelh
Copy link
Copy Markdown
Contributor Author

patelh commented Jan 23, 2019

@drcrallen @b-slim @clintropolis @gianm any idea which release this change will be part of?

@clintropolis
Copy link
Copy Markdown
Member

It will be part of the upcoming 0.14 release, which I believe we plan to start the process at the end of the month.

clintropolis pushed a commit to implydata/druid-public that referenced this pull request Feb 5, 2019
@jon-wei jon-wei added this to the 0.14.0 milestone Feb 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants