Unbounded insert into MySQL fails

[benvogan]

My environment is totally horked from being down for so long and having a bunch of stuck/failed tasks. I don't know precisely what is leading to having such a large number of values in the payload, but obviously there needs to be some checking of this value.

Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: com.mysql.jdbc.PacketTooBigException: Packet for query is too large (1201867 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable. [statement:"INSERT INTO druid_tasklogs (task_id, log_payload) VALUES (:entryId, :payload)", located:"INSERT INTO druid_tasklogs (task_id, log_payload) VALUES (:entryId, :payload)", rewritten:"INSERT INTO druid_tasklogs (task_id, log_payload) VALUES (?, ?)", arguments:{ positional:{}, named:{payload:[123, 34, 116, 121, 112, 101, 34, 58, 34, 115, 101, 103, 109, 101, 110, 116, 84, 114, 97, 110, 115, 97, 99, 116, 105, 111, 110, 97, 108, 73, 110, 115, 101, 114, 116, 34, 44, 34, 115, 101, 103, 109, 101, 110, 116, 115, 34, 58, 91, 123, 34, 100, 97, 116, 97, 83, 111, 117, 114, 99, 101, 34, 58, 34, 115, 111, 114, 95, 98, 117, 115, 105, 110, 101, 115, 115, 95, 101, 118, 101, 110, 116, 95, 115, 117, 99, 99, 101, 115, 115, 95, 118, 49, 34, 44, 34, 105, 110, 116, 101, 114, 118, 97, 108, 34, 58, 34, 50, 48, 49, 54, 45, 48, 55, 45, 49, 53, 84, 50, 50, 58, 48, 48, 58, 48, 48, 46, 48, 48, 48, 90, 47, 50, 48, 49, 54, 45, 48, 55, 45, 49, 53, 84, 50, 51, 58, 48, 48, 58, 48, 48, 46, 48, 48, 48, 90, 34, 44, 34, 118, 101, 114, 115, 105, 111, 110, 34, 58, 34, 50, 48, 49, 54, 45, 48, 55, 45, 50, 49, 84, 48, 51, 58, 52, 50, 58, 48, 48, 46, 51, 52, 57, 90, 34, 44, 34, 108, 111, 97, 100, 83, 112, 101, 99, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 104, 100, 102, 115, 34, 44, 34, 112, 97, 116, 104, 34, 58, 34, 104, 100, 102, 115, 58, 47, 47, 104, 100, 102, 115, 48, 48, 49, 58, 56, 48, 50, 48, 47, 100, 114, 117, 105, 100, 50, 47, 115, 101, 103, 109, 101, 110, 116, 115, 47, 115, 111, 114, 95, 98, 117, 115, 105, 110, 101, 115, 115, 95, 101, 118, 101, 110, 116, 95, 115, 117, 99, 99, 101, 115, 115, 95, 118, 49, 47, 50, 48, 49, 54, 48, 55, 49, 53, 84, 50, 50, 48, 48, 48, 48, 46, 48, 48, 48, 90, 95, 50, 48, 49, 54, 48, 55, 49, 53, 84, 50, 51, 48, 48, 48, 48, 46, 48, 48, 48, 90, 47, 50, 48, 49, 54, 45, 48, 55, 45, 50, 49, 84, 48, 51, 95, 52, 50, 95, 48, 48, 46, 51, 52, 57, 90, 47, 57, 47, 105, 110, 100, 101, 120, 46, 122, 105, 112, 34, 125, 44, 34, 100, 105, 109, 101, 110, 115, 105, 111, 110, 115, 34, 58, 34, 117, 115, 101, 114, 95, 103, 101, 110, 100, 101, 114, 44, 117, 115, 101, 114, 95, 116, 111, 100, 97, 121, 95, 105, 115, 95, 98, 97, 100, 103, 101, 100, 44, 117, 115, 101, 114, 95, 115, 111, 99, 105, 97, 108, 95, 102, 97, 99, 101, 98, 111, 111, 107, 44, 117, 115, 101, 114, 95, 97, 103, 101, 44, 117, 115, 101, 114, 95, 97, 103, 101, 95, 98, 114, 97, 99, 107, 101, 116, 44, 117, 115, 101, 114, 95, 114, 101, 103, 105, 115, 116, 114, 97, 116, 105, 111, 110, 95, 97, 103, 101, 95, 98, 114, 97, 99, 107, 101, 116, 44, 117, 115, 101, 114, 95, 114, 101, 103, 105, 115, 116, 114, 97, 116, 105, 111, 110, 95, 97, 103, 101, 95, 100, 97, 121, 44, 117, 115, 101, 114, 95, 114, 101, 103, 105, 115, 116, 114, 97, 116, 105, 111, 110, 95, 97, 103, 101, 95, 119, 101, 101, 107, 44, 101, 118, 101, 110, 116, 95, 116, 121, 112, 101, 44, 101, 118, 101, 110, 116, 95, 115, 116, 97, 116, 117, 115, 44, 101, 118, 101, 110, 116, 95, 105, 115, 95, 111, 102, 102, 108, 105, 110, 101, 44, 101, 118, 101, 110, 116, 95, 99, 105, 116, 121, 44, 101, 118, 101, 110, 116, 95, 114, 101, 103, 105, 111, 110, 44, 101, 118, 101, 110, 116, 95, 99, 111, 117, 110, 116, 114, 121, 44, 117, 115, 101, 114, 95, 99, 111, 117, 110, 116, 114, 121, 44, 99, 104, 97, 105, 110, 95, 110, 97, 109, 101, 44, 99, 108, 105, 101, 110, 116, 95, 97, 112, 112, 95, 118, 101, 114, 115, 105, 111, 110, 44, 99, 108, 105, 101, 110, 116, 95, 105, 112, 95, 97, 100, 100, 114, 101, 115, 115, 44, 99, 108, 105, 101, 110, 116, 95, 108, 111, 99, 97, 108, 101, 44, 99, 108, 105, 101, 110, 116, 95, 111, 115, 95, 110, 97, 109, 101, 44, 99, 108, 105, 101, 110, 116, 95, 117, 115, 101, 114, 95, 97, 103, 101, 110, 116, 44, 100, 101, 118, 105, 99, 101, 95, 112, 108, 97, 116, 102, 111, 114, 109, 95, 116, 121, 112, 101, 44, 100, 101, 118, 105, 99, 101, 95, 110, 97, 109, 101, 44, 108, 111, 99, 97, 116, 105, 111, 110, 95, 110, 97, 109, 101, 44, 108, 111, 99, 97, 116, 105, 111, 110, 95, 112, 114, 111, 118, 105, 100, 101, 114, 95, 105, 100, 44, 108, 111, 99, 97, 116, 105, 111, 110, 95, 112, 114, 111, 118, 105, 100, 101, 114, 95, 110, 97, 109, 101, 44, 117, 115, 101, 114, 95, 107, 105, 99, 107, 115, 95, 98, 97, 108, 97, 110, 99, 101, 44, 97, 112, 112, 95, 105, 100, 44, 112, 97, 114, 116, 110, 101, 114, 95, 105, 100, 44, 108, 111, 99, 97, 116, 105, 111, 110, 95, 105, 100, 44, 99, 104, 97, 105, 110, 95, 105, 100, 44, 117, 115, 101, 114, 95, 105, 100, 44, 100, 101, 118, 105, 99, 101, 95, 105, 100, 44, 99, 108, 105, 101, 110, 116, 95, 101, 118, 101, 110, 116, 95, 116, 105, 109, 101, 44, 99, 108, 105, 101, 110, 116, 95, 114, 101, 113, 117, 101, 115, 116, 95, 116, 105, 109, 101, 44, 115, 101, 114, 118, 101, 114, 95, 114, 101, 113, 117, 101, 115, 116, 95, 116, 105, 109, 101, 44, 117, 115, 101, 114, 95, 114, 101, 103, 105, 115, 116, 114, 97, 116, 105, 111, 110, 95, 116, 105, 109, 101, 44, 101, 118, 101, 110, 116, 95, 99, 111, 111, 114, 100, 105, 110, 97, 116, 101, 115, 44, 108, 111, 99, 97, 116, 105, 111, 110, 95, 99, 111, 111, 114, 100, 105, 110, 97, 116, 101, 115, 34, 44, 34, 109, 101, 116, 114, 105, 99, 115, 34, 58, 34, 99, 111, 117, 110, 116, 44, 116, 114, 97, 110, 115, 97, 99, 116, 105, 111, 110, 95, 107, 105, 99, 107, 115, 44, 116, 114, 97, 110, 115, 97, 99, 116, 105, 111, 110, 95, 99, 111, 115, 116, 44, 116, 114, 97, 110, 115, 97, 99, 116, 105, 111, 110, 95, 114, 101, 118, 101, 110, 117, 101, 44, 117, 110, 105, 113, 117, 101, 95, 117, 115, 101, 114, 95, 105, 100, 44, 117, 110, 105, 113, 117, 101, 95, 117, 115, 101, 114, 95, 100, 101, 118, 105, 99, 101, 95, 105, 100, 44, 117, 110, 105, 113, 117, 101, 95, 108, 111, 99, 97, 116, 105, 111, 110, 95, 105, 100, 34, 44, 34, 115, 104, 97, 114, 100, 83, 112, 101, 99, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 110, 117, 109, 98, 101, 114, 101, 100, 34, 44, 34, 112, 97, 114, 116, 105, 116, 105, 111, 110, 78, 117, 109, 34, 58, 57, 44, 34, 112, 97, 114, 116, 105, 116, 105, 111, 110, 115, 34, 58, 48, 125, 44, 34, 98, 105, 110, 97, 114, 121, 86, 101, 114, 115, 105, 111, 110, 34, 58, 57, 44, 34, 115, 105, 122, 101, 34, 58, 52, 51, 53, 55, 56, 50, 56, 44, 34, 105, 100, 101, 110, 116, 105, 102, 105, 101, 114, 34, 58, 34, 115, 111, 114, 95, 98, 117, 115, 105, 110, 101, 115, 115, 95, 101, 118, 101, 110, 116, 95, 115, 117, 99, 99, 101, 115, 115, 95, 118, 49, 95, 50, 48, 49, 54, 45, 48, 55, 45, 49, 53, 84, 50, 50, 58, 48, 48, 58, 48, 48, 46, 48, 48, 48, 90, 95, 50, 48, 49, 54, 45, 48, 55, 45, 49, 53, 84, 50, 51, 58, 48, 48, 58, 48, 48, 46, 48, 48, 48, 90, 95, 50, 48, 49, 54, 45, 48, 55, 45, 50, 49, 84, 48, 51, 58, 52, 50, 58, 48, 48, 46, 51, 52, 57, 90, 95, 57, 34, 125, 44, 123, 34, 100, 97, 116, 97, 83, 111, 117, 114, 99, 101, 34, 58, 34, 115, 111, 114, 95, 98, 117, 115, 105, 110, 101, 115, 115, 95, 101, 118, 101, 110, 116, 95, 115, 117, 99, 99, 101, 115, 115, 95, 118, 49, 34, 44, 34, 105, 110, 116, 101, 114, 118, 97, 108, 34, 58, 34, 50, 48, 49, 54, 45, 48, 55, 45, 49, 53, 84, 50, 50, 58, 48, 48, 58, 48, 48, 46, 48, 48, 48, 90, 47, 50, 48, 49, 54, 45, 48, 55, 45, 49, 53, 84, 50, 51, 58, 48, 48, 58, 48, 48, 46, 48, 48, 48, 90, 34, 44, 34, 118, 101, 114, 115, 105, 111, 110, 34, 58, 34, 50, 48, 49, 54, 45, 48, 55, 45, 50, 49, 84, 48, 51, 58, 52, 50, 58, 48, 48, 46, 51, 52, 57, 90, 34, 44,...

    at org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1334) ~[jdbi-2.63.1.jar:2.63.1]
    at org.skife.jdbi.v2.Update.execute(Update.java:56) ~[jdbi-2.63.1.jar:2.63.1]
    at io.druid.metadata.SQLMetadataStorageActionHandler$10.withHandle(SQLMetadataStorageActionHandler.java:359) ~[druid-server-0.9.1.jar:0.9.1]
    at io.druid.metadata.SQLMetadataStorageActionHandler$10.withHandle(SQLMetadataStorageActionHandler.java:347) ~[druid-server-0.9.1.jar:0.9.1]
    at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
    ... 64 more

Caused by: com.mysql.jdbc.PacketTooBigException: Packet for query is too large (1201867 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable.
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3583) ~[?:?]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2460) ~[?:?]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2625) ~[?:?]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2551) ~[?:?]
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[?:?]
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[?:?]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1328) ~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.Update.execute(Update.java:56) ~[jdbi-2.63.1.jar:2.63.1]
at io.druid.metadata.SQLMetadataStorageActionHandler$10.withHandle(SQLMetadataStorageActionHandler.java:359) ~[druid-server-0.9.1.jar:0.9.1]
at io.druid.metadata.SQLMetadataStorageActionHandler$10.withHandle(SQLMetadataStorageActionHandler.java:347) ~[druid-server-0.9.1.jar:0.9.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
... 64 more

[dclim]

Task log is currently a single entry per task containing segment metadata for all segments generated by that task. With the Kafka indexing service being able to create new segments on demand to hold events with arbitrary timestamps + the Kafka indexing service creating a segment per Kafka partition:

long running task * large range of historical data * small segment granularities * lots of Kafka partitions * big segment metadata (for example lots of dimensions) == massively sized payload

Maybe we should consider splitting the task log table to have separate entries for each segment in a task.

[gianm]

I think we can close this due to the incremental handoff ability in indexing tasks (#4815). We shouldn't be generating these huge publish actions anymore.