8x speed-up by buffering of InputStream during reading of uncompressed files

[sbittrich]

I ran some benchmarks for the BinaryCIF project/format and in comparison the Java implementation of the MMTF codec was surprisingly slow. Especially when uncompressed (non-gzipped) files were processed. Find benchmark details in the RCSB internal ciftools-performance repo.

By employing a BufferedInputStream with 65536 buffer size the performance can be improved drastically, resulting in a traversal of the currently 154k structures in 70 s (10 minutes with the current code).

For comparison, read times for BinaryCIF and mmCIF parsing are given (which should be slower due to higher overhead). A performance increase for gzipped files can be expected by using a GZIPInputStream with an equally sized buffer of 65536 (in contrast to the default buffer of 512 bytes).

[coveralls]

Coverage increased (+0.02%) to 81.926% when pulling 7960934 on JonStargaryen:master into 59287c5 on rcsb:master.

[josemduarte]

Fantastic, thank you!

[pwrose]

Excellent, Sebastian!

…

On Mon, Jul 29, 2019, 3:21 PM Jose Manuel Duarte ***@***.***> wrote: ***@***.**** approved this pull request. Fantastic, thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AA53AEGKDPKI3PPPY6IQQQTQB5UNZA5CNFSM4IHWTA2KYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB75HBDQ#pullrequestreview-268071054>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA53AEANYFJNJCT7TLNHGUDQB5UNZANCNFSM4IHWTA2A> .

[josemduarte]

@JonStargaryen do you know if this is this still relevant when running under java 11+ JRE?

If so I'd like to merge and release a new bugfix as soon as possible.

[sbittrich]

@josemduarte Yeah, it's still an issue on Java 11. I didn't run it with warm-up iterations or redundancy though but the trend is clear.

Here the times to read the archive:

Benchmark	Mode
MMTF explicitly buffered	71.851 s/op
MMTF current impl	547.395 s/op

Probably a good idea to release a new version on Maven before releasing BioJava.

[josemduarte]

Thanks, @JonStargaryen !

I'll go ahead and make a new release today