MariaDB as metadata storage: facing ClassNotFoundException: com.mysql.jdbc.exceptions.MySQLTransientException

[kazuhirokomoda]

Affected Version

• Druid: 0.22.1
• MariaDB: (On-Prem) MariaDB-Galera 10.5.12 cluster
• Java: OpenJDK 11.0.12

Description

Trying to use MariaDB as metadata storage, but facing MySQLTransientException when trying to run metadata-init tool, during druid_pendingSegments table creation.

The issue does not happen when I try the same steps with MariaDB Docker container on a server.

Thank you for your support in advance.

Steps to reproduce

1. Download the 0.22.1 release.
2. In your terminal, extract Druid and change directories to the distribution directory (as shown in official doc)
3. Download mariadb-java-client-2.7.3.jar and move it under ./extensions/mysql-metadata-storage
4. (Prepare MariaDB cluster)
5. Run this command, following official doc, note that -Ddruid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver is added

java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[\"mysql-metadata-storage\"] -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver org.apache.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://<host>:<port>/<database>" --user <user> --password <password> --base druid

6. Facing the following MySQLTransientException exception

log output including MySQLTransientException exception

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/druid/apache-druid-0.22.1/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-08-30T04:28:34,518 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final
2023-08-30T04:28:34,794 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mariadb-java-client-2.7.3.jar
2023-08-30T04:28:34,892 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mariadb-java-client-2.7.3.jar
2023-08-30T04:28:35,373 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions [{version=0.22.1}].
2023-08-30T04:28:35,375 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 2 monitors: org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2023-08-30T04:28:35,382 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Loading "MySQL" metadata connector driver org.mariadb.jdbc.Driver
2023-08-30T04:28:35,428 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Configured MySQL as metadata storage
2023-08-30T04:28:35,519 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Table[druid_dataSource] already exists
2023-08-30T04:28:35,525 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_pendingSegments]
Exception in thread "main" java.lang.NoClassDefFoundError: com/mysql/jdbc/exceptions/MySQLTransientException
	at org.apache.druid.metadata.storage.mysql.MySQLConnector.connectorIsTransientException(MySQLConnector.java:208)
	at org.apache.druid.metadata.SQLMetadataConnector.isTransientException(SQLMetadataConnector.java:169)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:135)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153)
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:138)
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:148)
	at org.apache.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:182)
	at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:210)
	at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:544)
	at org.apache.druid.cli.CreateTables.run(CreateTables.java:121)
	at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.exceptions.MySQLTransientException
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 13 more

Using MariaDB Docker container instead of On-Prem cluster

This produces no issue.

docker pull mariadb:10.5.12
docker run --name mariadbtest -e MYSQL_ROOT_PASSWORD=mypass -p 3306:3306 -d mariadb:10.5.12
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mariadbtest
docker run --rm -ti mariadb:10.5 mysql -u root -p -h 240.0.0.2
 
# https://druid.apache.org/docs/0.22.1/development/extensions-core/mysql.html#setting-up-mysql
-----
CREATE DATABASE druid102 DEFAULT CHARACTER SET utf8mb4;
CREATE USER 'druid102'@'%' IDENTIFIED BY 'diurd';
GRANT ALL PRIVILEGES ON druid102.* TO 'druid102'@'%';
-----
 
java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[\"mysql-metadata-storage\"] -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver org.apache.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://240.0.0.2:3306/druid102" --user druid102 --password diurd --base druid

docker stop mariadbtest
docker rm -v mariadbtest

Alternative attempts and results

• Put mysql-connector-java-5.1.48.jar instead of mariadb-java-client-2.7.3.jar
  • java.sql.SQLException: Unknown system variable 'transaction_isolation'
• Change to -Ddruid.metadata.storage.type=mariadb
  • Unknown provider[mariadb] of Key[type=org.apache.druid.metadata.MetadataStorageConnector, annotation=[none]], known options[[mysql, derby]]
• Change jdbc:mysql to jdbc:mariadb
  • The same java.lang.ClassNotFoundException: com.mysql.jdbc.exceptions.MySQLTransientException
• Put both mysql-connector-java-5.1.48.jar and mariadb-java-client-2.7.3.jar under ./extensions/mysql-metadata-storage
  • java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)

log output including EOFException exception

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/druid/apache-druid-0.22.1/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-08-30T05:15:27,521 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final
2023-08-30T05:15:27,806 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mysql-connector-java-5.1.48.jar, mariadb-java-client-2.7.3.jar
2023-08-30T05:15:27,914 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mysql-connector-java-5.1.48.jar, mariadb-java-client-2.7.3.jar
2023-08-30T05:15:28,393 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions [{version=0.22.1}].
2023-08-30T05:15:28,394 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 2 monitors: org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2023-08-30T05:15:28,402 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Loading "MySQL" metadata connector driver org.mariadb.jdbc.Driver
2023-08-30T05:15:28,451 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Configured MySQL as metadata storage
2023-08-30T05:15:28,546 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Table[druid_dataSource] already exists
2023-08-30T05:15:28,553 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_pendingSegments]
2023-08-30T05:15:28,562 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 1,136ms.
org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server) [statement:"null", located:"null", rewritten:"null", arguments:null]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:138) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) [druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) [druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) [druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) [druid-core-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:138) [druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:148) [druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:182) [druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:210) [druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:544) [druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.cli.CreateTables.run(CreateTables.java:121) [druid-services-0.22.1.jar:0.22.1]
	at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.22.1.jar:0.22.1]
Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server) [statement:"null", located:"null", rewritten:"null", arguments:null]
	at org.skife.jdbi.v2.Batch.execute(Batch.java:131) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:194) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:184) ~[druid-server-0.22.1.jar:0.22.1]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	... 12 more
Caused by: java.sql.BatchUpdateException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
	at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:324) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.MariaDbStatement.executeBatch(MariaDbStatement.java:1398) ~[mariadb-java-client-2.7.3.jar:?]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.skife.jdbi.v2.Batch.execute(Batch.java:121) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:194) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:184) ~[druid-server-0.22.1.jar:0.22.1]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	... 12 more
Caused by: java.sql.SQLNonTransientConnectionException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:155) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:320) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.MariaDbStatement.executeBatch(MariaDbStatement.java:1398) ~[mariadb-java-client-2.7.3.jar:?]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.skife.jdbi.v2.Batch.execute(Batch.java:121) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:194) ~[druid-server-0.22.1.jar:0.22.1]
	at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:184) ~[druid-server-0.22.1.jar:0.22.1]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	... 12 more
Caused by: java.sql.SQLNonTransientConnectionException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2089) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1540) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1519) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:132) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:67) ~[mariadb-java-client-2.7.3.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
	at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:249) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:218) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1538) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1519) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:132) ~[mariadb-java-client-2.7.3.jar:?]
	at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:67) ~[mariadb-java-client-2.7.3.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

[LakshSingla]

This is fixed by #12205. Can you please upgrade to a more recent version and check if the error persists?

On a related note, the version of Druid in use is pretty old, and there have been significant improvements made to Druid since then. To make use of those, and to get rid of the error, it is recommended that you upgrade to a more recent version.

[kazuhirokomoda]

@LakshSingla Thank you for sharing. I tried newer versions (26.0.0, 25.0.0, 24.0.2, 0.23.0), they didn't throw MySQLTransientException but instead they froze until I pressed Ctrl+C.

With 0.22.1, along with the MySQLTransientException, I observed a particular table (druid_pendingSegments) was created on 2 nodes, but not on 1 node (out of 3 nodes).

Also I highlight that manual execution of CREATE TABLE statements (which I obtained from our another old MySQL 5.7.25 cluster) worked fine.

I talked with DB Admin of the on-prem cluster, and he addressed some questions to ask:

• Is the metadata-init tool compatible with MariaDB 10.6.15 (upgraded from 10.5.12), 3-node Galera cluster? Especially when it uses Total Order Isolation for DDL?
• Could you provide entire/exact SQL it uses to create tables?
• Is there any retry mechanism inside the tool?
• Does it open transaction, or set any variable before execution?