Integrations tests 🚀

[PierreZ]

This PR is adding:

• integrations tests for KoP with testcontainers,
• librdkafka support by allowing ListOffsets in v0

Produce and Consume support

Golang

• https://github.com/Shopify/sarama
• https://github.com/confluentinc/confluent-kafka-go

Rust

• https://github.com/fede1024/rust-rdkafka

NodeJS

• https://github.com/Blizzard/node-rdkafka

Partial support

kafka-node

Producing is working, but consuming is failing as the library is sending FETCH v0 regardless of API_VERSIONS responses:

DEBUG io.streamnative.kop.KafkaCommandDecoder - Write kafka cmd response back to client. request: RequestHeader(apiKey=FETCH, apiVersion=0, clientId=kafka-node-client, correlationId=4)
INFO  io.streamnative.kop.KafkaIntegrationTest - STDOUT: Error: Not a message set. Magic byte is 2

[jiazhai]

[�[1;34mINFO�[m] Running io.streamnative.kop.�[1mKafkaIntegrationTest�[m
[�[1;31mERROR�[m] �[1;31mTests �[0;1mrun: �[0;1m7�[m, �[1;31mFailures: �[0;1;31m1�[m, Errors: 0, �[1;33mSkipped: �[0;1;33m6�[m, Time elapsed: 40.458 s�[1;31m <<< FAILURE!�[m - in io.streamnative.kop.�[1mKafkaIntegrationTest�[m
[�[1;31mERROR�[m] setup(io.streamnative.kop.KafkaIntegrationTest)  Time elapsed: 40.354 s  <<< FAILURE!
org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:322)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
	at org.testcontainers.containers.PortForwardingContainer.createSSHSession(PortForwardingContainer.java:41)
	at org.testcontainers.containers.PortForwardingContainer.getSshConnection(PortForwardingContainer.java:26)
	at org.testcontainers.containers.PortForwardingContainer.exposeHostPort(PortForwardingContainer.java:67)
	at org.testcontainers.Testcontainers.exposeHostPorts(Testcontainers.java:21)
	at io.streamnative.kop.KafkaIntegrationTest.setup(KafkaIntegrationTest.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:168)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageNameFuture=java.util.concurrent.CompletableFuture@4679554d[Completed normally], imagePullPolicy=DefaultPullPolicy(), dockerClient=LazyDockerClient.INSTANCE)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1265)
	at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:600)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:311)
	... 37 more
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:150)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:135)
	at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
	at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
	at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
	at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
	at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:62)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:25)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:27)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1263)
	... 39 more

[�[1;31mERROR�[m] �[1;31mFailures: �[m
[�[1;31mERROR�[m] �[1;31m  KafkaIntegrationTest.setup:146 ? ContainerLaunch Container startup failed�[m
[�[1;34mINFO�[m] 
[�[1;31mERROR�[m] �[1;31mTests run: 198, Failures: 1, Errors: 0, Skipped: 1�[m
[�[1;34mINFO�[m] 

[PierreZ]

Mmmmh, does docker is enabled on the pipeline?

[sijie]

@PierreZ @jiazhai it might be worth moving to using Github Actions

[sijie]

The change has been merged in #67