CompressionUtils performance improvement ask

[crodier]

FileUtils, needs (on or around line 234), a BufferedOutputStream instead of a FilterOutputStream (we think.) Apologies for not submitting a proper patch.

Affected Version

0.14.2-incubating

Description

When we compress a 20mb smoosh folder today into a zip. It takes about 15 seconds, whereas the previous implementation of Compression Utils takes about a second. This makes the hand-off compression much slower.

The software is amazing, thank you.

[himanshug]

what was the version where same zipping operation was taking 1 second ?

If you use "local" deep storage then I "guess" it could be due to explicit "fsync" that was enabled sometime last year and could be really responsible for this time. but that was done to fix corruption issues.

[crodier]

Yes, local deep storage, to a disk drive. The "atomic" update yes, which was enabled last year, made it 20x slower. You can still do the atomic update, but need to use the Buffered stream, not FilterOutput stream. It is not the fsync, it is the FilterOutputStream. This doesn't use buffers, is pushing a byte at a time? The change (almost certainly) is merely to use, a buffered output stream, need not remove the Atomic fixes for corruption issues. Thx again!

[himanshug]

I think #6677 is the PR in question. that PR didn't change anything wrt use[or no use] of buffered output steam, only change introduced is to write to temp file instead of final file and do an fsync . so not sure why that specific change caused perf to go 20x slower (if fsync() is not the problem)
can you point to the change that removed usage of Buffered OutputStream that is causing the slowness ?

[gianm]

It's possible the change from FileOutputStream to FileChannel on https://github.com/apache/incubator-druid/pull/6677/files#diff-8cbeaa1874f0c7db8634387441433351L81 had some effect. Maybe FileOutputStream does buffering differently from the OutputStream you get by doing Channels.newOutputStream(fileChannel)?

[crodier]

#5344 seems to have it? Should be using something other than, FilterOutputStream?

#5344 (review)

https://stackoverflow.com/questions/47721200/javas-filteroutputstream-adds-a-performance-penalty

Hope this helps, and we meet one day at an event, terrific software, ty!

[himanshug]

@crodier thanks . ok, I see now. default FilterOutputStream.write(byte b[], int off, int len) impl is dumber than it needs to be.
In this case, Performance degradation is not related to BufferedOutputStream which we never had in that context but due to the use of FilterOutputStream . I think that is what you were trying to say as well. I did some tests and #8024 fixes the regression.

[gianm]

Thanks for the investigation @himanshug & @crodier!!