Microbenchmarks based on Caliper are broken

[turu]

All microbenchmarks using Google Caliper library are broken. They cannot execute properly due to dependency version mismatch. The root cause is that Druid declares dependency on com.google.guava:guava:16.0.1 and com.google.caliper:caliper:0.5-rc1. Caliper in turn, needs Guava in version < 16.0, because it uses http://docs.guava-libraries.googlecode.com/git-history/v15.0/javadoc/com/google/common/io/Closeables.html#closeQuietly(java.io.Closeable) internally.

Affected benchmarks:

1. JavaScriptAggregatorBenchmark
2. CardinalityAggregatorBenchmark
3. HyperLogLogCollectorBenchmark
4. TopNBinaryFnBenchmark
5. MemcachedCacheBenchmark

Steps to reproduce:

1. Checkout current master
2. Attempt to run any of the affected benchmarks.

Expected result:

1. Benchmark completes successfully

Actual result:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.io.Closeables.closeQuietly(Ljava/io/Closeable;)V
    at com.google.caliper.Runner.measure(Runner.java:301)
    at com.google.caliper.Runner.runScenario(Runner.java:229)
    at com.google.caliper.Runner.runOutOfProcess(Runner.java:378)
    at com.google.caliper.Runner.run(Runner.java:97)
    at com.google.caliper.Runner.main(Runner.java:423)
    at com.google.caliper.Runner.main(Runner.java:436)
    at io.druid.query.aggregation.hyperloglog.HyperLogLogCollectorBenchmark.main(HyperLogLogCollectorBenchmark.java:167)

How to fix:
There are two ways this can be fixed.

1. Lower version of Guava dependency to 15.0 globally.
2. Bump up Caliper dependency to the newest 1.0-beta-2 and migrate affected benchmarks to the new API (there have been some API changes between 0.5-rc1 and 1.0-beta-1).

I will try to open a pull request fixing this today.

[turu]

I'm working on a pr to fix this and so far it seems that it will be a bit of a challenge to get those benchmarks going again. Bumping caliper to 1.0-beta-2 requires bumping Guava to 18.0. This in turn breaks quite a bit of Druid code, which uses deprecated features that have been removed since 16.0.1.

[drcrallen]

We cannot bump druid guava above 16.0.1 due to hadoop incompatibilities.

The stuff under the benchmark directory worked last time I checked (and we were using guava 16.0.1)

Can you elaborate more on the error you are seeing?

[turu]

The benchmarks I'm talking about are not inside benchmark directory.

The one I'm specifically interested in is io.druid.query.aggregation.hyperloglog.HyperLogLogCollectorBenchmark from processing module. All the other failing benchmarks are located in processing and server (the last one) modules.

The error is as described above. The com.google.caliper.Runner attempts to call the com.google.common.io.Closeables.closeQuietly method which is always going fail under Guava 16.0.1, since this method has been removed from Guava API between versions 15.0 and 16.0 (see: http://docs.guava-libraries.googlecode.com/git-history/v16.0/jdiff/changes.html).

Benchmarks I'm talking about are probably broken since Guava was bumped to 16.0.1, over a year and a half ago.

[turu]

If Guava can't be bumped above 16.0.1, then the only way to fix those benchmarks is to go back to 15.0, or rewrite them completely from scratch, without Caliper.

[xvrl]

Thank you @turu for offering to fix the benchmarks. I don't think we want or even can downgrade Guava. I faced this problem a while back. Caliper completely changed their Api and didn't seem to provide a way to dump results on the console last time I looked at upgrading to 1.0. I basically gave up on caliper at that point and decided it wasn't worth my time.

I looked a little bit at using junit-benchmark but it appears that has also been deprecated in favor of JMH since I last looked.

Going forward I believe we should be using JMH for any serious microbenchmark. We already have a couple on JMH, and if you'd like to help porting the remaining ones that would be awesome!

Let us know if you need help

[turu]

Ok then. Thanks for your response. Perhaps I will give it a shot. Having those benchmarks running may prove useful. :)

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.