move integration tests from ZooKeeper 3.4.x to 3.5.x

[xvrl]

as we deprecate ZooKeeper 3.4.x, the first step is to move our integration tests from 3.4.x to 3.5.x
see #10780 for context

• Removes the need to build separate docker-base image:
  • use multi-stage build for the base image
  • use openjdk base image instead of building our own JDK base
  • workaround Debian not including MySQL by using MariaDB and
    downloading mysql connector directly
• Run integration tests with ZK 3.5 by default
• Run a subset of our integration tests with ZK 3.4 for backwards compatibility testing.

[gianm]

I think it'd be good for now to test both ZK 3.4 and 3.5 since we're meant to be in a migratory period where both servers are supported. Do you think you could add a smoke test that uses ZK 3.4? (I don't think we need to run every single test, but it'd be good to run some subset of the ITs.)

@clintropolis @gianm what's the process to build and update the base image?

I think someone on the Imply team would need to push a new base image. That's probably the easiest way.

We could also move the base image to the "apache" Docker account? We started using a base image from Imply in #5060 for performance reasons, and IIRC that pre-dates when Druid joined ASF, so using the ASF account wouldn't have been an option at the time. Maybe it is now.

[clintropolis]

Hmm, since the integration tests are using docker-compose now, I wonder if we should just pull stock zookeeper (and kafka, metadata store) images and use those in the compose files instead...

[xvrl]

@gianm agree we should ideally test both. I'll see if it's easy to add separate 3.4.x integration test. All our "unit" tests that rely on curator's TestingCluster and TestingServer classes would continue to test against embedded 3.4.x nodes until we move to 3.5.x

[jihoonson]

Hi @xvrl, I'm planning to finish the 0.21.0 release early next week. Do you think we can get this PR merged before that? Let me know if there is anything I can help.

[xvrl]

@jihoonson I've updated the integration tests to remove the need for an external image build and added some ZK 3.4 integration tests.

[jihoonson]

@xvrl great, thanks! I will take a look this weekend.

[jihoonson]

@xvrl, can you add another test against ZooKeeper 3.4 that tests ingestion and segment loading? The "Kafka index integration test with various formats" test looks good enough to test both.

[jihoonson]

nit: better to bump the mysql connector version in the pom to match to this. If you are going to change the version, can you please add a comment in the pom file that says these versions should be in sync? It would be nice if we can sync all these library versions automatically, but it could be done in a different PR in the future.

[xvrl]

I updated the docker build to pull in the mysql connector version from the pom property

[jihoonson]

This change makes every Travis job to download Kafka and ZooKeeper binaries every time. This can be a problem because integration tests will become flaky if the HTTP connection is unstable. I tested the flakiness by restarting all integration tests of this PR and it seems OK. So, I'm OK with this change in this PR, but we need to address this potential flakiness as soon as possible.

[xvrl]

true, on the flipside we now ensure the integration tests always match the base image definition. Hopefully adding the ASF mirror will not be make things too flaky. My only concern would be the ZK 3.4.x binaries, which are no longer available from the main mirror and have to be downloaded from the ASF archive.

We can do a follow-up PR to save the intermediate docker cache to a file in the travis cache, so we only have to rebuild the base image if it changes.

[jihoonson]

LGTM. +1 after CI.