Skip to content

Upgrade AWS SDK#10386

Closed
mitchlloyd wants to merge 1 commit intoapache:masterfrom
mitchlloyd:aws-sdk-upgrade
Closed

Upgrade AWS SDK#10386
mitchlloyd wants to merge 1 commit intoapache:masterfrom
mitchlloyd:aws-sdk-upgrade

Conversation

@mitchlloyd
Copy link
Copy Markdown
Contributor

@mitchlloyd mitchlloyd commented Sep 11, 2020 •
edited
Loading

Fixes #10375.

Description

Upgrades the AWS SDK to get new SDK features including STS regional endpoint config via environment variable.

This PR has:

  • been self-reviewed.
  • been tested in a test Druid cluster.

@mitchlloyd mitchlloyd mentioned this pull request Sep 11, 2020
Closed
@suneet-s
Copy link
Copy Markdown
Contributor

suneet-s commented Sep 14, 2020

@mitchlloyd it looks like the license check job is complaining about the licenses file not being updated

Error: found 3 missing licenses. These licenses are reported, but missing in the registry
druid_module: kinesis-indexing-service, groupId: com.amazonaws, artifactId: jmespath-java, version: 1.11.854, license: Apache License version 2.0
druid_module: kinesis-indexing-service, groupId: com.amazonaws, artifactId: aws-java-sdk-sts, version: 1.11.854, license: Apache License version 2.0
druid_module: kinesis-indexing-service, groupId: com.amazonaws, artifactId: aws-java-sdk-kinesis, version: 1.11.854, license: Apache License version 2.0

Have you been able to run the AWS integration tests with the new sdk version?

@mitchlloyd
Copy link
Copy Markdown
Contributor Author

mitchlloyd commented Sep 14, 2020

@suneet-s I'm running the tests following the documentation:

However I'm running into some issues. First the documented command to run seems to run all tests which fail at different points on different runs:

mvn verify -P integration-tests \
  -Dgroups=s3-deep-storage \
  -Doverride.config.path=s3-creds-override \
  -Ddruid.test.config.streamEndpoint=kinesis.us-west-2.amazonaws.com \
  -Ddruid.test.config.cloudBucket=<MY-BUCKET-NAME> \
  -Ddruid.test.config.cloudPath=s3-tests/

I added the -pl integration-tests option and this seems to start up Docker:

mvn verify -pl integration-tests \
   -P integration-tests \
  -Dgroups=s3-deep-storage \
  -Doverride.config.path=s3-creds-override \
  -Ddruid.test.config.streamEndpoint=kinesis.us-west-2.amazonaws.com \
  -Ddruid.test.config.cloudBucket=<MY-BUCKET-NAME> \
  -Ddruid.test.config.cloudPath=s3-tests/

I notice that during start up there is a message that says:

Could not set docker host IP - integration tests can not run

I also see this error:

Error 
Step 8/36 : RUN find /var/lib/mysql -type f -exec touch {} \; && /etc/init.d/mysql start       && echo "CREATE USER 'druid'@'%' IDENTIFIED BY 'diurd'; GRANT ALL ON druid.* TO 'druid'@'%'; CREATE database druid DEFAULT CHARACTER SET utf8mb4;" | mysql -u root       && /etc/init.d/mysql stop
 ---> Running in d25c117522f4
 * /etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full!
The command '/bin/sh -c find /var/lib/mysql -type f -exec touch {} \; && /etc/init.d/mysql start       && echo "CREATE USER 'druid'@'%' IDENTIFIED BY 'diurd'; GRANT ALL ON druid.* TO 'druid'@'%'; CREATE database druid DEFAULT CHARACTER SET utf8mb4;" | mysql -u root       && /etc/init.d/mysql stop' returned a non-zero code: 1
Error response from daemon: No such container: druid-historical
Error: No such container: druid-historical
Error response from daemon: No such container: druid-historical-for-query-retry-test
Error: No such container: druid-historical-for-query-retry-test
Error response from daemon: No such container: druid-coordinator
Error: No such container: druid-coordinator
Error response from daemon: No such container: druid-overlord
Error: No such container: druid-overlord
Error response from daemon: No such container: druid-router
Error: No such container: druid-router
Error response from daemon: No such container: druid-router-permissive-tls
Error: No such container: druid-router-permissive-tls
Error response from daemon: No such container: druid-router-no-client-auth-tls
Error: No such container: druid-router-no-client-auth-tls
Error response from daemon: No such container: druid-router-custom-check-tls
Error: No such container: druid-router-custom-check-tls
Error response from daemon: No such container: druid-broker
Error: No such container: druid-broker
Error response from daemon: No such container: druid-middlemanager
Error: No such container: druid-middlemanager
Error response from daemon: No such container: druid-zookeeper-kafka
Error: No such container: druid-zookeeper-kafka
Error response from daemon: No such container: druid-metadata-storage
Error: No such container: druid-metadata-storage
Error response from daemon: No such container: druid-it-hadoop
Error: No such container: druid-it-hadoop
druid-it-net

Possibly of note, after that the test stops with what I know is a user prompt:

Pulling druid-zookeeper-kafka (druid/cluster:)...
The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

I can get past that by pressing y and enter to bypass the prompt.

Then I get

Pulling druid-zookeeper-kafka (druid/cluster:)...
pull access denied for druid/cluster, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

After that a test section runs but runs no tests

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

Then I get errors from org.apache.maven.plugins:maven-failsafe-plugin:2.22.0:verify

Error 
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:16 min
[INFO] Finished at: 2020-09-14T12:27:30-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.22.0:verify (integration-tests) on project druid-integration-tests: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/mitlloyd/projects/apache/druid/integration-tests/target/failsafe-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] Unable to provision, see the following errors:
[ERROR]
[ERROR] 1) Error in custom provider, java.lang.RuntimeException: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR]   while locating org.apache.druid.https.SSLContextProvider
[ERROR]   while locating javax.net.ssl.SSLContext annotated with interface org.apache.druid.guice.annotations.EscalatedClient
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.guice.annotations.EscalatedClient()
[ERROR]     for the 3rd parameter of org.apache.druid.testing.guice.DruidTestModule.getHttpClient(DruidTestModule.java:72)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.testing.guice.TestClient()
[ERROR]     for the 2nd parameter of org.apache.druid.testing.clients.CoordinatorResourceTestClient.(CoordinatorResourceTestClient.java:63)
[ERROR]   while locating org.apache.druid.testing.clients.CoordinatorResourceTestClient
[ERROR]     for field at org.apache.druid.tests.indexer.AbstractIndexerTest.coordinator(AbstractIndexerTest.java:47)
[ERROR]   while locating org.apache.druid.tests.coordinator.duty.ITAutoCompactionTest
[ERROR] Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR] 	at org.apache.druid.server.security.TLSUtils.createSSLContext(TLSUtils.java:249)
[ERROR] 	at org.apache.druid.server.security.TLSUtils$ClientSSLContextBuilder.build(TLSUtils.java:148)
[ERROR] 	at org.apache.druid.https.SSLContextProvider.get(SSLContextProvider.java:66)
[ERROR] 	at org.apache.druid.https.SSLContextProvider.get(SSLContextProvider.java:30)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
[ERROR] 	at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
[ERROR] 	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
[ERROR] 	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
[ERROR] 	at org.apache.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:118)
[ERROR] 	at org.apache.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:83)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
[ERROR] 	at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
[ERROR] 	at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
[ERROR] 	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
[ERROR] 	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
[ERROR] 	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
[ERROR] 	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
[ERROR] 	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
[ERROR] 	at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:402)
[ERROR] 	at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
[ERROR] 	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
[ERROR] 	at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
[ERROR] 	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:120)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
[ERROR] 	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
[ERROR] 	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
[ERROR] 	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
[ERROR] 	at org.testng.internal.ClassImpl.getInstanceFromGuice(ClassImpl.java:141)
[ERROR] 	at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:119)
[ERROR] 	at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:190)
[ERROR] 	at org.testng.TestClass.getInstances(TestClass.java:95)
[ERROR] 	at org.testng.TestClass.initTestClassesAndInstances(TestClass.java:81)
[ERROR] 	at org.testng.TestClass.init(TestClass.java:73)
[ERROR] 	at org.testng.TestClass.(TestClass.java:38)
[ERROR] 	at org.testng.TestRunner.initMethods(TestRunner.java:389)
[ERROR] 	at org.testng.TestRunner.init(TestRunner.java:271)
[ERROR] 	at org.testng.TestRunner.init(TestRunner.java:241)
[ERROR] 	at org.testng.TestRunner.(TestRunner.java:167)
[ERROR] 	at org.testng.DruidTestRunnerFactory$DruidTestRunner.(DruidTestRunnerFactory.java:76)
[ERROR] 	at org.testng.DruidTestRunnerFactory.newTestRunner(DruidTestRunnerFactory.java:57)
[ERROR] 	at org.testng.SuiteRunner$ProxyTestRunnerFactory.newTestRunner(SuiteRunner.java:713)
[ERROR] 	at org.testng.SuiteRunner.init(SuiteRunner.java:260)
[ERROR] 	at org.testng.SuiteRunner.(SuiteRunner.java:198)
[ERROR] 	at org.testng.TestNG.createSuiteRunner(TestNG.java:1295)
[ERROR] 	at org.testng.TestNG.createSuiteRunners(TestNG.java:1273)
[ERROR] 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
[ERROR] 	at org.testng.TestNG.runSuites(TestNG.java:1049)
[ERROR] 	at org.testng.TestNG.run(TestNG.java:1017)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
[ERROR] Caused by: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR] 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
[ERROR] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[ERROR] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
[ERROR] 	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
[ERROR] 	at java.nio.file.Files.newByteChannel(Files.java:361)
[ERROR] 	at java.nio.file.Files.newByteChannel(Files.java:407)
[ERROR] 	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
[ERROR] 	at java.nio.file.Files.newInputStream(Files.java:152)
[ERROR] 	at org.apache.druid.server.security.TLSUtils.createSSLContext(TLSUtils.java:188)
[ERROR] 	... 67 more
[ERROR]
[ERROR] 2) Error in custom provider, java.lang.RuntimeException: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR]   while locating org.apache.druid.https.SSLContextProvider
[ERROR]   while locating javax.net.ssl.SSLContext annotated with interface org.apache.druid.guice.annotations.EscalatedClient
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.guice.annotations.EscalatedClient()
[ERROR]     for the 3rd parameter of org.apache.druid.testing.guice.DruidTestModule.getHttpClient(DruidTestModule.java:72)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.testing.guice.TestClient()
[ERROR]     for the 2nd parameter of org.apache.druid.testing.clients.OverlordResourceTestClient.(OverlordResourceTestClient.java:64)
[ERROR]   while locating org.apache.druid.testing.clients.OverlordResourceTestClient
[ERROR]     for field at org.apache.druid.tests.indexer.AbstractIndexerTest.indexer(AbstractIndexerTest.java:47)
[ERROR]   while locating org.apache.druid.tests.coordinator.duty.ITAutoCompactionTest
[ERROR] Caused by: java.lang.RuntimeException (same stack trace as error #1)
[ERROR] 3) Error in custom provider, java.lang.RuntimeException: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR]   while locating org.apache.druid.https.SSLContextProvider
[ERROR]   while locating javax.net.ssl.SSLContext annotated with interface org.apache.druid.guice.annotations.EscalatedClient
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.guice.annotations.EscalatedClient()
[ERROR]     for the 3rd parameter of org.apache.druid.testing.guice.DruidTestModule.getHttpClient(DruidTestModule.java:72)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.testing.guice.TestClient()
[ERROR]     for the 2nd parameter of org.apache.druid.testing.clients.QueryResourceTestClient.(QueryResourceTestClient.java:41)
[ERROR]   while locating org.apache.druid.testing.clients.QueryResourceTestClient
[ERROR]     for the 2nd parameter of org.apache.druid.testing.utils.TestQueryHelper.(TestQueryHelper.java:38)
[ERROR]   while locating org.apache.druid.testing.utils.TestQueryHelper
[ERROR]     for field at org.apache.druid.tests.indexer.AbstractIndexerTest.queryHelper(AbstractIndexerTest.java:47)
[ERROR]   while locating org.apache.druid.tests.coordinator.duty.ITAutoCompactionTest
[ERROR] Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR] 	at org.apache.druid.server.security.TLSUtils.createSSLContext(TLSUtils.java:249)
[ERROR] 	at org.apache.druid.server.security.TLSUtils$ClientSSLContextBuilder.build(TLSUtils.java:148)
[ERROR] 	at org.apache.druid.https.SSLContextProvider.get(SSLContextProvider.java:66)
[ERROR] 	at org.apache.druid.https.SSLContextProvider.get(SSLContextProvider.java:30)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
[ERROR] 	at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
[ERROR] 	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
[ERROR] 	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
[ERROR] 	at org.apache.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:118)
[ERROR] 	at org.apache.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:83)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
[ERROR] 	at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
[ERROR] 	at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
[ERROR] 	at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
[ERROR] 	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
[ERROR] 	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
[ERROR] 	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
[ERROR] 	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
[ERROR] 	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
[ERROR] 	at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:402)
[ERROR] 	at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
[ERROR] 	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
[ERROR] 	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
[ERROR] 	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
[ERROR] 	at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:54)
[ERROR] 	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:120)
[ERROR] 	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
[ERROR] 	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
[ERROR] 	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
[ERROR] 	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
[ERROR] 	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
[ERROR] 	at org.testng.internal.ClassImpl.getInstanceFromGuice(ClassImpl.java:141)
[ERROR] 	at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:119)
[ERROR] 	at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:190)
[ERROR] 	at org.testng.TestClass.getInstances(TestClass.java:95)
[ERROR] 	at org.testng.TestClass.initTestClassesAndInstances(TestClass.java:81)
[ERROR] 	at org.testng.TestClass.init(TestClass.java:73)
[ERROR] 	at org.testng.TestClass.(TestClass.java:38)
[ERROR] 	at org.testng.TestRunner.initMethods(TestRunner.java:389)
[ERROR] 	at org.testng.TestRunner.init(TestRunner.java:271)
[ERROR] 	at org.testng.TestRunner.init(TestRunner.java:241)
[ERROR] 	at org.testng.TestRunner.(TestRunner.java:167)
[ERROR] 	at org.testng.DruidTestRunnerFactory$DruidTestRunner.(DruidTestRunnerFactory.java:76)
[ERROR] 	at org.testng.DruidTestRunnerFactory.newTestRunner(DruidTestRunnerFactory.java:57)
[ERROR] 	at org.testng.SuiteRunner$ProxyTestRunnerFactory.newTestRunner(SuiteRunner.java:713)
[ERROR] 	at org.testng.SuiteRunner.init(SuiteRunner.java:260)
[ERROR] 	at org.testng.SuiteRunner.(SuiteRunner.java:198)
[ERROR] 	at org.testng.TestNG.createSuiteRunner(TestNG.java:1295)
[ERROR] 	at org.testng.TestNG.createSuiteRunners(TestNG.java:1273)
[ERROR] 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
[ERROR] 	at org.testng.TestNG.runSuites(TestNG.java:1049)
[ERROR] 	at org.testng.TestNG.run(TestNG.java:1017)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
[ERROR] 	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
[ERROR] 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
[ERROR] Caused by: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR] 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
[ERROR] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[ERROR] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
[ERROR] 	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
[ERROR] 	at java.nio.file.Files.newByteChannel(Files.java:361)
[ERROR] 	at java.nio.file.Files.newByteChannel(Files.java:407)
[ERROR] 	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
[ERROR] 	at java.nio.file.Files.newInputStream(Files.java:152)
[ERROR] 	at org.apache.druid.server.security.TLSUtils.createSSLContext(TLSUtils.java:188)
[ERROR] 	... 72 more
[ERROR]
[ERROR] 4) Error in custom provider, java.lang.RuntimeException: java.nio.file.NoSuchFileException: client_tls/truststore.jks
[ERROR]   while locating org.apache.druid.https.SSLContextProvider
[ERROR]   while locating javax.net.ssl.SSLContext annotated with interface org.apache.druid.guice.annotations.EscalatedClient
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   at org.apache.druid.guice.http.HttpClientModule.configure(HttpClientModule.java:77) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.http.HttpClientModule)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.guice.annotations.EscalatedClient()
[ERROR]     for the 3rd parameter of org.apache.druid.testing.guice.DruidTestModule.getHttpClient(DruidTestModule.java:72)
[ERROR]   while locating org.apache.druid.java.util.http.client.HttpClient annotated with @org.apache.druid.testing.guice.TestClient()
[ERROR]     for the 2nd parameter of org.apache.druid.testing.clients.CompactionResourceTestClient.(CompactionResourceTestClient.java:54)
[ERROR]   while locating org.apache.druid.testing.clients.CompactionResourceTestClient
[ERROR]     for field at org.apache.druid.tests.coordinator.duty.ITAutoCompactionTest.compactionResource(ITAutoCompactionTest.java:54)
[ERROR]   while locating org.apache.druid.tests.coordinator.duty.ITAutoCompactionTest
[ERROR] Caused by: java.lang.RuntimeException (same stack trace as error #1)
[ERROR] 4 errors
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:658)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:278)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:244)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1194)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1022)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:868)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

I decided to try debugging the first error I saw about the docker host. I tried following the section of the docs to start the docker container manually. I get:

docker-compose -f integration-tests/docker/docker-compose.yml up
WARNING: The DRUID_INTEGRATION_TEST_GROUP variable is not set. Defaulting to a blank string.
Pulling druid-zookeeper-kafka (druid/cluster:)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y
Pulling druid-zookeeper-kafka (druid/cluster:)...
ERROR: pull access denied for druid/cluster, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I'm not sure how to best direct my debugging time here. Anything stand out to you?

@jihoonson
Copy link
Copy Markdown
Contributor

jihoonson commented Sep 17, 2020

Could not set docker host IP - integration tests can not run

@mitchlloyd it seems like you need to export DOCKER_IP. Please check out our doc for more details.

Pulling druid-zookeeper-kafka (druid/cluster:)...
pull access denied for druid/cluster, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

This error seems like you need to run docker login. Have you installed docker desktop? Then you can just run docker login in your terminal.

@mitchlloyd
Copy link
Copy Markdown
Contributor Author

mitchlloyd commented Sep 23, 2020

I probably should have specifically called out that I have the DOCKER_IP environment variable set to 127.0.0.1 per the instructions since that seems like a likely culprit.

I took another shot at this today. I ran docker login to make sure I was logged into Dockerhub. I also disabled some maven configuration that I have that pulls packages through an Artifactory proxy repository in case that mattered.

As a demonstration I also put this env var in the command and still saw Could not set docker host IP - integration tests can not run:

DOCKER_IP=127.0.0.1 mvn verify -pl integration-tests \
   -P integration-tests \
  -Dgroups=s3-deep-storage \
  -Doverride.config.path=s3-creds-override \
  -Ddruid.test.config.streamEndpoint=kinesis.us-west-2.amazonaws.com \
  -Ddruid.test.config.cloudBucket=<MY-BUCKET-NAME> \
  -Ddruid.test.config.cloudPath=s3-tests/

So then I honed in on this command which seemed to be not working and giving me the error.

dscacheutil -q host -a name "$(HOSTNAME)" | perl -nle '/ip_address: (.*)/ && print $1' | tail -n1

I don't have a HOSTNAME env var so I tried setting that to localhost. Then I needed to remove some parenthesis around HOSTNAME to get that command to return a value:

> dscacheutil -q host -a name "$HOSTNAME" | perl -nle '/ip_address: (.*)/ && print $1' | tail -n1
> 127.0.0.1

With that change I see a lot more RSA keys being generated:

Generating RSA private key, 1024 bit long modulus

And now I'm able to run the tests!

From here it looks like to make sure all of my config is right but at least I have the tests running 👍

@mitchlloyd
Copy link
Copy Markdown
Contributor Author

mitchlloyd commented Sep 23, 2020 •
edited
Loading

So my tests appear to be timing out after a very long time.

2020-09-23T23:32:28,960 INFO [main] org.apache.druid.testing.clients.OverlordResourceTestClient - Index status response{"task":"index_parallel_wikipedia_index_test_b5c3e438-9b0b-42d2-8bd2-666897539e07 Россия 한국 中国!?_mhepmemn_2020-09-23T23:29:35.975Z","status":{"id":"index_parallel_wikipedia_index_test_b5c3e438-9b0b-42d2-8bd2-666897539e07 Россия 한국 中国!?_mhepmemn_2020-09-23T23:29:35.975Z","groupId":"index_parallel_wikipedia_index_test_b5c3e438-9b0b-42d2-8bd2-666897539e07 Россия 한국 中国!?_mhepmemn_2020-09-23T23:29:35.975Z","type":"index_parallel","createdTime":"2020-09-23T23:29:35.996Z","queueInsertionTime":"1970-01-01T00:00:00.000Z","statusCode":"RUNNING","status":"RUNNING","runnerStatusCode":"RUNNING","duration":-1,"location":{"host":"172.172.172.7","port":8100,"tlsPort":8101},"dataSource":"wikipedia_index_test_b5c3e438-9b0b-42d2-8bd2-666897539e07 Россия 한국 中国!?","errorMsg":null}}
2020-09-23T23:32:28,961 INFO [main] org.apache.druid.testing.utils.ITRetryUtil - Attempt[17]: Task index_parallel_wikipedia_index_test_b5c3e438-9b0b-42d2-8bd2-666897539e07 Россия 한국 中国!?_mhepmemn_2020-09-23T23:29:35.975Z still not complete. Next retry in 10000 ms

I was able to connect to the UI during the test once I found out that the basic http auth password was "priest".

Screen Shot 2020-09-23 at 4 29 58 PM

I wanted to see what the submitted job was but I was only getting 400 errors when looking at the details.
Screen Shot 2020-09-23 at 4 30 22 PM

The build output from Docker doesn't seem entirely happy:

---> 23a6faa7d678
Successfully built 23a6faa7d678
Successfully tagged druid/cluster:latest
druid-historical
druid-historical
Error response from daemon: No such container: druid-historical-for-query-retry-test
Error: No such container: druid-historical-for-query-retry-test
druid-coordinator
druid-coordinator
druid-overlord
druid-overlord
druid-router
druid-router
Error response from daemon: No such container: druid-router-permissive-tls
Error: No such container: druid-router-permissive-tls
Error response from daemon: No such container: druid-router-no-client-auth-tls
Error: No such container: druid-router-no-client-auth-tls
Error response from daemon: No such container: druid-router-custom-check-tls
Error: No such container: druid-router-custom-check-tls
druid-broker
druid-broker
druid-middlemanager
druid-middlemanager
druid-zookeeper-kafka
druid-zookeeper-kafka
druid-metadata-storage
druid-metadata-storage
Error response from daemon: No such container: druid-it-hadoop
Error: No such container: druid-it-hadoop
druid-it-net

I logged the spec output for AbstractBatchITIndexTest many seem to be referencing hdfs paths which I'm not sure is going to work without more setup. Do I need to resolve these "No such container" errors or is that expected?

@maytasm
Copy link
Copy Markdown
Contributor

maytasm commented Sep 24, 2020

@mitchlloyd

  1. You can view the logs (druid logs and task logs) from the shared mount path on your machine. By default this should be something like /Users/<LocalMachineUser>/shared/log and /Users/<LocalMachineUser>/shared/tasklogs. The UI have some problem getting the task logs since the task name contains non-ASCII (I think that's the reason why you are getting 400 errors but not 100% sure on that).

  2. Can you check which tests are timeout? Is it all of the tests? Are any test passing?

  3. Is the build output from Docker from the beginning of the run? Which step / command resulted in that output? There is a teardown step which make sure any docker you may have running from previous running IT is destroyed. If none were running then those may be normal.

@mitchlloyd
Copy link
Copy Markdown
Contributor Author

mitchlloyd commented Oct 20, 2020

Those logs are very helpful. I took another crack at this and most of the errors seems related to Hadoop versions. There appeared to be just 2 basic s3 tests that didn't use Hadoop.

I'm probably not the best person to slog through figuring out how to setup some local Hadoop clusters for testing; I've just never used Hadoop before.

I'll close this PR but keep the issue open in case someone else wants to follow this trail. Maybe they'll find some useful info for running the tests on OSX at least.

@mitchlloyd mitchlloyd closed this Oct 20, 2020
@mitchlloyd
Copy link
Copy Markdown
Contributor Author

mitchlloyd commented Apr 30, 2021

Luckily this slipped into #10541 so this issue is now resolved!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Area - Dependencies

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Upgrade AWS SDK

4 participants

@mitchlloyd @suneet-s @jihoonson @maytasm