The instruction for running integration tests from integration-tests/README.md doesn't work

[leventov]

[ERROR] 1) Error in custom provider, java.lang.RuntimeException: java.io.FileNotFoundException: client_tls/truststore.jks (No such file or directory)

FYI @jon-wei

[jon-wei]

@leventov Could you provide the full logs after you start running the tests?

[leventov]

db9a714abe45acc2011799fe4dfd6b89c82653f321a4e92dab5903cd89b800d0
Sending build context to Docker daemon  136.6MB
Step 1/31 : FROM imply/druiditbase:0.2
 ---> 765db104f306
Step 2/31 : RUN echo "[mysqld]\ncharacter-set-server=utf8\ncollation-server=utf8_bin\n" >> /etc/mysql/my.cnf
 ---> Using cache
 ---> ad36ebd753ea
Step 3/31 : RUN find /var/lib/mysql -type f -exec touch {} \; && /etc/init.d/mysql start       && echo "CREATE USER 'druid'@'%' IDENTIFIED BY 'diurd'; GRANT ALL ON druid.* TO 'druid'@'%'; CREATE database druid DEFAULT CHARACTER SET utf8mb4;" | mysql -u root       && /etc/init.d/mysql stop
 ---> Using cache
 ---> 159e843c63c9
Step 4/31 : ADD lib/* /usr/local/druid/lib/
 ---> 030dd8ed2167
Step 5/31 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Running in a9dda1aac6c0
Removing intermediate container a9dda1aac6c0
 ---> 02a3a02e16e6
Step 6/31 : RUN apt-get update && apt-get install -y --no-install-recommends apt-utils libmysql-java
 ---> Running in 75c78a31155a
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [855 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [557 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6113 B]
Get:10 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1249 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [968 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [19.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7942 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8532 B]
Fetched 15.8 MB in 58s (271 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  apt libapt-inst2.0 libapt-pkg5.0
Suggested packages:
  aptitude | synaptic | wajig dpkg-dev apt-doc python-apt libslf4j-java
The following NEW packages will be installed:
  apt-utils libapt-inst2.0 libmysql-java
The following packages will be upgraded:
  apt libapt-pkg5.0
2 upgraded, 3 newly installed, 0 to remove and 41 not upgraded.
Need to get 3006 kB of archives.
After this operation, 2473 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libapt-pkg5.0 amd64 1.2.32 [713 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt amd64 1.2.32 [1086 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libapt-inst2.0 amd64 1.2.32 [55.8 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt-utils amd64 1.2.32 [196 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/universe amd64 libmysql-java all 5.1.38-1 [955 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 3006 kB in 9s (316 kB/s)
(Reading database ... 13797 files and directories currently installed.)
Preparing to unpack .../libapt-pkg5.0_1.2.32_amd64.deb ...
Unpacking libapt-pkg5.0:amd64 (1.2.32) over (1.2.29ubuntu0.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libapt-pkg5.0:amd64 (1.2.32) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
(Reading database ... 13797 files and directories currently installed.)
Preparing to unpack .../archives/apt_1.2.32_amd64.deb ...
Unpacking apt (1.2.32) over (1.2.29ubuntu0.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up apt (1.2.32) ...
Installing new version of config file /etc/apt/apt.conf.d/01autoremove ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Selecting previously unselected package libapt-inst2.0:amd64.
(Reading database ... 13806 files and directories currently installed.)
Preparing to unpack .../libapt-inst2.0_1.2.32_amd64.deb ...
Unpacking libapt-inst2.0:amd64 (1.2.32) ...
Selecting previously unselected package apt-utils.
Preparing to unpack .../apt-utils_1.2.32_amd64.deb ...
Unpacking apt-utils (1.2.32) ...
Selecting previously unselected package libmysql-java.
Preparing to unpack .../libmysql-java_5.1.38-1_all.deb ...
Unpacking libmysql-java (5.1.38-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Setting up libapt-inst2.0:amd64 (1.2.32) ...
Setting up apt-utils (1.2.32) ...
Setting up libmysql-java (5.1.38-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Removing intermediate container 75c78a31155a
 ---> c36a0f77fc0d
Step 7/31 : RUN ln -sf /usr/share/java/mysql-connector-java.jar /usr/local/druid/lib/mysql-connector-java.jar
 ---> Running in 8163029f2fc3
Removing intermediate container 8163029f2fc3
 ---> fcee6844fed7
Step 8/31 : RUN find /var/lib/mysql -type f -exec touch {} \; && service mysql start       && java -cp "/usr/local/druid/lib/*" -Ddruid.metadata.storage.type=mysql org.apache.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://localhost:3306/druid" --user=druid --password=diurd       && /etc/init.d/mysql stop
 ---> Running in 3fa4e3e5aaf0
 * Starting MySQL database server mysqld
   ...done.
Jun 06, 2019 1:42:21 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
2019-06-06T13:42:21,935 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.common.config.NullValueHandlingConfig] from props[druid.generic.] as [org.apache.druid.common.config.NullValueHandlingConfig@5fe1ce85]
2019-06-06T13:42:21,971 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', useExtensionClassloaderFirst=false, hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=null}]
2019-06-06T13:42:22,068 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}]
2019-06-06T13:42:22,200 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.storage.s3.S3StorageDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,201 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.firehose.s3.S3FirehoseDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,204 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.gcp.GcpModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,205 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,211 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.testing.guice.ITTLSCertificateCheckerModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,216 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.aws.AWSModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,219 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.histogram.ApproximateHistogramDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,221 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.security.basic.BasicSecurityDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,224 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.https.SSLContextModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,227 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,231 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.server.lookup.namespace.NamespaceExtractionModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,241 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,243 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,245 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,288 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:22,291 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:42:23,300 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.server.security.Escalator] from props[druid.escalator.] as [NoopEscalator{}]
2019-06-06T13:42:23,313 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.CuratorConfig] from props[druid.zk.service.] as [org.apache.druid.curator.CuratorConfig@23940f86]
2019-06-06T13:42:23,321 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [org.apache.druid.curator.ExhibitorConfig@4a7a965d]
2019-06-06T13:42:23,399 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
2019-06-06T13:42:23,401 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2019-06-06T13:42:23,461 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [org.apache.druid.server.initialization.ZkPathsConfig@58d3f4be]
2019-06-06T13:42:23,468 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain=null, authorizers=null, unsecuredPaths=[], allowUnauthenticatedHttpOptions=false}]
2019-06-06T13:42:23,527 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [org.apache.druid.server.metrics.DruidMonitorSchedulerConfig@41b1f51e]
2019-06-06T13:42:23,531 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
2019-06-06T13:42:23,532 INFO [main] org.apache.druid.server.emitter.EmitterModule - Underlying emitter for ServiceEmitter: org.apache.druid.java.util.emitter.core.NoopEmitter@64f1fd08
2019-06-06T13:42:23,533 INFO [main] org.apache.druid.server.emitter.EmitterModule - Extra service dimensions: {version=0.16.0-incubating-SNAPSHOT}
2019-06-06T13:42:23,535 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.query.ExecutorServiceMonitor@2e807c54]
2019-06-06T13:42:23,536 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@4c18621b]
2019-06-06T13:42:23,550 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.metadata.storage.mysql.MySQLConnectorConfig] from props[druid.metadata.mysql.ssl.] as [MySQLConnectorConfig{useSSL='false', clientCertificateKeyStoreUrl='null', clientCertificateKeyStoreType='null', verifyServerCertificate='false', trustCertificateKeyStoreUrl='null', trustCertificateKeyStoreType='null', enabledSSLCipherSuites=null, enabledTLSProtocols=null}]
2019-06-06T13:42:23,632 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Configured MySQL as metadata storage
2019-06-06T13:42:24,005 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_dataSource]
2019-06-06T13:42:24,020 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_pendingSegments]
2019-06-06T13:42:24,067 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_segments]
2019-06-06T13:42:24,098 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_rules]
2019-06-06T13:42:24,122 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_config]
2019-06-06T13:42:24,135 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_tasks]
2019-06-06T13:42:24,159 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_tasklogs]
2019-06-06T13:42:24,180 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_tasklocks]
2019-06-06T13:42:24,209 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_audit]
2019-06-06T13:42:24,261 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_supervisors]
 * Stopping MySQL database server mysqld
   ...done.
Removing intermediate container 3fa4e3e5aaf0
 ---> 0d54a842d7ee
Step 9/31 : ADD sample-data.sql sample-data.sql
 ---> 1d52173b3afc
Step 10/31 : RUN find /var/lib/mysql -type f -exec touch {} \; && service mysql start       && cat sample-data.sql | mysql -u root druid       && /etc/init.d/mysql stop
 ---> Running in 6990b5641d87
 * Starting MySQL database server mysqld
   ...done.
 * Stopping MySQL database server mysqld
   ...done.
Removing intermediate container 6990b5641d87
 ---> 5b3aee086457
Step 11/31 : ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf
 ---> 64f26b1d64f5
Step 12/31 : ADD run-mysql.sh /run-mysql.sh
 ---> f1fd8daeae2d
Step 13/31 : ADD docker_ip docker_ip
 ---> 1c7176a01685
Step 14/31 : RUN perl -pi -e "s/#listeners=.*/listeners=INTERNAL:\/\/172.172.172.2:9092,EXTERNAL:\/\/172.172.172.2:9093/" /usr/local/kafka/config/server.properties
 ---> Running in 9291c8613f7e
Removing intermediate container 9291c8613f7e
 ---> 1f1048baf413
Step 15/31 : RUN perl -pi -e "s/#advertised.listeners=.*/advertised.listeners=INTERNAL:\/\/172.172.172.2:9092,EXTERNAL:\/\/$(cat docker_ip):9093/" /usr/local/kafka/config/server.properties
 ---> Running in a1e30c7d1a67
Removing intermediate container a1e30c7d1a67
 ---> e2227d34c928
Step 16/31 : RUN perl -pi -e "s/#listener.security.protocol.map=.*/listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT\ninter.broker.listener.name=INTERNAL/" /usr/local/kafka/config/server.properties
 ---> Running in 4e5eaa8875d3
Removing intermediate container 4e5eaa8875d3
 ---> e7d3b1581709
Step 17/31 : RUN perl
 ---> Running in 65b54cd5cf40
Removing intermediate container 65b54cd5cf40
 ---> 1bcb3f61a3cd
Step 18/31 : ADD tls tls
 ---> 7e713a39eb71
Step 19/31 : ADD client_tls client_tls
 ---> 5664ca7b6a9c
Step 20/31 : EXPOSE 8081 8281
 ---> Running in d22fa4da4a8c
Removing intermediate container d22fa4da4a8c
 ---> afcd54890a98
Step 21/31 : EXPOSE 8082 8282
 ---> Running in fe3336b40391
Removing intermediate container fe3336b40391
 ---> 07b06b16190b
Step 22/31 : EXPOSE 8083 8283
 ---> Running in 67aa05561e45
Removing intermediate container 67aa05561e45
 ---> 2cffe76dec56
Step 23/31 : EXPOSE 8090 8290
 ---> Running in af6ccb8489d6
Removing intermediate container af6ccb8489d6
 ---> b2385a0c3227
Step 24/31 : EXPOSE 8091 8291
 ---> Running in ba7f4b0ba113
Removing intermediate container ba7f4b0ba113
 ---> b6e5706a61e1
Step 25/31 : EXPOSE 3306
 ---> Running in b32669a4e48f
Removing intermediate container b32669a4e48f
 ---> 49d60ebebe6e
Step 26/31 : EXPOSE 2181 2888 3888
 ---> Running in 9c962d7880ec
Removing intermediate container 9c962d7880ec
 ---> 755625e6c02f
Step 27/31 : EXPOSE 8100 8101 8102 8103 8104 8105
 ---> Running in eceb5521dcbb
Removing intermediate container eceb5521dcbb
 ---> 2af9955649e3
Step 28/31 : EXPOSE 8300 8301 8302 8303 8304 8305
 ---> Running in 7495ce6836a6
Removing intermediate container 7495ce6836a6
 ---> e911f6b42218
Step 29/31 : EXPOSE 9092 9093
 ---> Running in dc131e16a2af
Removing intermediate container dc131e16a2af
 ---> 572abadb4dff
Step 30/31 : WORKDIR /var/lib/druid
 ---> Running in 3fe9f944bb74
Removing intermediate container 3fe9f944bb74
 ---> c1e03f9d06aa
Step 31/31 : ENTRYPOINT export HOST_IP="$(resolveip -s $HOSTNAME)" && /tls/generate-server-certs-and-keystores.sh && exec /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
 ---> Running in aba3405c1933
Removing intermediate container aba3405c1933
 ---> be3328500aeb
Successfully built be3328500aeb
Successfully tagged druid/cluster:latest
bf2fc0f5bead287b10fbc2d124b2b5d3bdd1a188f0e12e2a14ebcdf1386898da
6d3a0eb4a76e8eb9f8c5eecac44082013378423aff7671a25b16da7f91d33ecf
feb2523a9591f3f5b902af894ac6d714b21515818022164726e1a712b24fb4f8
bea098ac043aa55be5deeb6fc7017e906a577a8951adf39fe3f6b610e02c3a80
d958ef13675dcade612c374e1b59122f89b27f323658def73cdb495378d13978
ecae9635c9ff6e4d64136579b88fa36c3c0c971e9566f7adf087ff2f4676cf13
be71b2eb1dca0f30ede32d570e0037ed35b5b3ac29ed44a86a9784b9a9be933d
17a4628e2d2bdf8803aa7464a8d008c52b708aab9ae59598cde6961642087d26
133d3aabf3742ba272fb405649c454c5d3203850fdf2b7dbdc81789cf24a68d6
60a551811617b0e5be7baa3c0743daa00938c3e704fb527020ab32af6489559b
966f61175d1d31baaf95a830aabe7dce1a3782e15486014ab16ad41dec07e7d1
[INFO]
[INFO] --- maven-failsafe-plugin:2.22.0:integration-test (integration-tests) @ druid-integration-tests ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.druid.tests.security.ITTLSTest
Jun 06, 2019 1:43:09 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
2019-06-06T13:43:09,402 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.common.config.NullValueHandlingConfig] from props[druid.generic.] as [org.apache.druid.common.config.NullValueHandlingConfig@59532566]
2019-06-06T13:43:09,415 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}]
2019-06-06T13:43:09,533 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', useExtensionClassloaderFirst=false, hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=null}]
2019-06-06T13:43:09,537 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.testing.guice.ITTLSCertificateCheckerModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,540 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.storage.s3.S3StorageDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,541 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.firehose.s3.S3FirehoseDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,543 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,544 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,546 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,548 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,550 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,553 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.query.aggregation.histogram.ApproximateHistogramDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,554 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,556 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,558 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.security.basic.BasicSecurityDruidModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,560 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.server.lookup.namespace.NamespaceExtractionModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,561 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.https.SSLContextModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,565 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.aws.AWSModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,568 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.gcp.GcpModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension
2019-06-06T13:43:09,611 WARN [main] org.apache.druid.initialization.Log4jShutterDownerModule - Shutdown callback registry expected class [org.apache.druid.common.config.Log4jShutdown] found [org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry]. Skipping shutdown registry
2019-06-06T13:43:10,444 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.server.security.Escalator] from props[druid.escalator.] as [NoopEscalator{}]
2019-06-06T13:43:10,463 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.testing.IntegrationTestingConfigProvider] from props[druid.test.config.] as [org.apache.druid.testing.DockerConfigProvider@4ed5a1b0]
2019-06-06T13:43:10,471 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [org.apache.druid.curator.ExhibitorConfig@2d0b05]
2019-06-06T13:43:10,545 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode
2019-06-06T13:43:10,547 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2019-06-06T13:43:10,588 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [org.apache.druid.server.initialization.ZkPathsConfig@58d3f4be]
2019-06-06T13:43:10,596 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain=null, authorizers=null, unsecuredPaths=[], allowUnauthenticatedHttpOptions=false}]
2019-06-06T13:43:10,649 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [org.apache.druid.server.metrics.DruidMonitorSchedulerConfig@2c26ba07]
2019-06-06T13:43:10,654 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
2019-06-06T13:43:10,658 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.java.util.emitter.core.LoggingEmitterConfig] from props[druid.emitter.logging.] as [LoggingEmitterConfig{loggerClass='org.apache.druid.java.util.emitter.core.LoggingEmitter', logLevel='info'}]
2019-06-06T13:43:10,660 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@59c500f7]
2019-06-06T13:43:10,679 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.https.SSLClientConfig] from props[druid.client.https.] as [SSLClientConfig{protocol='null', trustStoreType='null', trustStorePath='client_tls/truststore.jks', trustStoreAlgorithm='null', keyStorePath='client_tls/client.jks', keyStoreType='null', certAlias='druid', keyManagerFactoryAlgorithm='null', validateHostnames='null'}]
2019-06-06T13:43:10,687 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.http.DruidHttpClientConfig] from props[druid.broker.http.] as [org.apache.druid.guice.http.DruidHttpClientConfig@13803a94]
2019-06-06T13:43:10,690 INFO [main] org.apache.druid.https.SSLContextProvider - Creating SslContext for https client using config [SSLClientConfig{protocol='null', trustStoreType='null', trustStorePath='client_tls/truststore.jks', trustStoreAlgorithm='null', keyStorePath='client_tls/client.jks', keyStoreType='null', certAlias='druid', keyManagerFactoryAlgorithm='null', validateHostnames='null'}]
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-failsafe-plugin:2.22.0:verify (integration-tests) @ druid-integration-tests ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] druid-integration-tests 0.16.0-incubating-SNAPSHOT . FAILURE [03:17 min]
[INFO] druid-benchmarks ................................... SKIPPED
[INFO] druid-avro-extensions .............................. SKIPPED
[INFO] druid-bloom-filter ................................. SKIPPED
[INFO] druid-kerberos ..................................... SKIPPED
[INFO] druid-hdfs-storage ................................. SKIPPED
[INFO] druid-stats ........................................ SKIPPED
[INFO] druid-kafka-eight .................................. SKIPPED
[INFO] druid-kafka-extraction-namespace ................... SKIPPED
[INFO] druid-kinesis-indexing-service ..................... SKIPPED
[INFO] druid-orc-extensions ............................... SKIPPED
[INFO] druid-parquet-extensions ........................... SKIPPED
[INFO] postgresql-metadata-storage ........................ SKIPPED
[INFO] druid-protobuf-extensions .......................... SKIPPED
[INFO] druid-lookups-cached-single ........................ SKIPPED
[INFO] druid-ec2-extensions ............................... SKIPPED
[INFO] druid-google-extensions ............................ SKIPPED
[INFO] druid-influx-extensions ............................ SKIPPED
[INFO] druid-azure-extensions ............................. SKIPPED
[INFO] druid-cassandra-storage ............................ SKIPPED
[INFO] druid-rocketmq ..................................... SKIPPED
[INFO] druid-cloudfiles-extensions ........................ SKIPPED
[INFO] graphite-emitter ................................... SKIPPED
[INFO] druid-kafka-eight-simple-consumer .................. SKIPPED
[INFO] druid-rabbitmq ..................................... SKIPPED
[INFO] druid-distinctcount ................................ SKIPPED
[INFO] statsd-emitter ..................................... SKIPPED
[INFO] druid-time-min-max ................................. SKIPPED
[INFO] druid-virtual-columns .............................. SKIPPED
[INFO] druid-thrift-extensions ............................ SKIPPED
[INFO] ambari-metrics-emitter ............................. SKIPPED
[INFO] sqlserver-metadata-storage ......................... SKIPPED
[INFO] kafka-emitter ...................................... SKIPPED
[INFO] druid-redis-cache .................................. SKIPPED
[INFO] druid-opentsdb-emitter ............................. SKIPPED
[INFO] materialized-view-maintenance ...................... SKIPPED
[INFO] materialized-view-selection ........................ SKIPPED
[INFO] druid-momentsketch ................................. SKIPPED
[INFO] druid-moving-average-query ......................... SKIPPED
[INFO] tdigestsketch ...................................... SKIPPED
[INFO] druid-influxdb-emitter ............................. SKIPPED
[INFO] distribution 0.16.0-incubating-SNAPSHOT ............ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

[jon-wei]

Do you have logs from before that?

The cert generation should be happening before the section you pasted, e.g.:

[INFO] --- exec-maven-plugin:1.2.1:exec (build-and-start-druid-cluster) @ druid-integration-tests ---
Error response from daemon: No such container: druid-historical
Error response from daemon: No such container: druid-historical
Error response from daemon: No such container: druid-coordinator
Error response from daemon: No such container: druid-coordinator
Error response from daemon: No such container: druid-overlord
Error response from daemon: No such container: druid-overlord
Error response from daemon: No such container: druid-router
Error response from daemon: No such container: druid-router
Error response from daemon: No such container: druid-router-permissive-tls
Error response from daemon: No such container: druid-router-permissive-tls
Error response from daemon: No such container: druid-router-no-client-auth-tls
Error response from daemon: No such container: druid-router-no-client-auth-tls
Error response from daemon: No such container: druid-router-custom-check-tls
Error response from daemon: No such container: druid-router-custom-check-tls
Error response from daemon: No such container: druid-broker
Error response from daemon: No such container: druid-broker
Error response from daemon: No such container: druid-middlemanager
Error response from daemon: No such container: druid-middlemanager
Error response from daemon: No such container: druid-zookeeper-kafka
Error response from daemon: No such container: druid-zookeeper-kafka
Error response from daemon: No such container: druid-metadata-storage
Error response from daemon: No such container: druid-metadata-storage
Error response from daemon: network druid-it-net not found
Generating RSA private key, 4096 bit long modulus
..........................................................................................................................................................................................................++
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
Generating RSA private key, 4096 bit long modulus
.........................................................................................++
...........++
e is 65537 (0x10001)
Generating RSA private key, 1024 bit long modulus
...............++++++
......++++++
e is 65537 (0x10001)
Using configuration from root_for_expired_client.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 286331157 (0x11111115)
        Validity

From run_cluster.sh:

# so docker IP addr will be known during docker build
echo ${DOCKER_IP:=127.0.0.1} > $DOCKERDIR/docker_ip

# setup client keystore
./docker/tls/generate-client-certs-and-keystores.sh
rm -rf docker/client_tls
cp -r client_tls docker/client_tls

[leventov]

@jon-wei thank you, yes, the cert generation step failed because of the following problems:

../docker/tls/generate-expired-client-cert.sh: line 18: resolveip: command not found

and

Error Loading request extension section req_ext
4681430636:error:22FFF076:X509 V3 routines:func(4095):bad ip address:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/x509v3/v3_alt.c:529:value=
4681430636:error:22FFF080:X509 V3 routines:func(4095):error in extension:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/crypto/x509v3/v3_conf.c:96:name=subjectAltName, value=@alt_names

I'm going to look for the root cause and a solution later.

[freyayunfu]

@leventov Any updates? I'm getting the same error.

Thanks.

[freyayunfu]

@leventov I created a PR to resolve the issue, but still seeing connection refused issue since coordinator is not starting up correctly in time for some reason, which requires more research to fix.

[leventov]

I "reproduced" this problem in Travis: https://travis-ci.org/apache/incubator-druid/jobs/615425842 in this PR: #6702. This must be due to these changes to integration-tests' Dockerfile and some following .sh scripts.