InlineFirehose does not work with index_parallel ingestion

[vogievetsky]

In Druid 0.16.0

As the title says, if you try to submit a task like:

{
  "type": "index_parallel",
  "spec": {
    "type": "index_parallel",
    "ioConfig": {
      "type": "index_parallel",
      "firehose": {
        "type": "inline",
        "data": "{\"name\":\"Vadim\"}"
      }
    },
    "tuningConfig": {
      "type": "index_parallel"
    },
    "dataSchema": {
      "dataSource": "some_data",
      "granularitySpec": {
        "type": "uniform",
        "queryGranularity": "HOUR",
        "rollup": true,
        "segmentGranularity": "DAY"
      },
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "!!!_no_such_column_!!!",
            "missingValue": "2010-01-01T00:00:00Z"
          },
          "dimensionsSpec": {
            "dimensions": [
              "name"
            ]
          }
        }
      },
      "metricsSpec": [
        {
          "name": "count",
          "type": "count"
        }
      ]
    }
  }
}

You get an error of:

{"error":"Instantiation of [simple type, class org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask] value failed: [InlineFirehoseFactory] should implement FiniteFirehoseFactory"}

However changing the ingestion to just index:

{
  "type": "index",
  "spec": {
    "type": "index",
    "ioConfig": {
      "type": "index",
      "firehose": {
        "type": "inline",
        "data": "{\"name\":\"Vadim\"}"
      }
    },
    "tuningConfig": {
      "type": "index"
    },
    "dataSchema": {
      "dataSource": "some_data",
      "granularitySpec": {
        "type": "uniform",
        "queryGranularity": "HOUR",
        "rollup": true,
        "segmentGranularity": "DAY"
      },
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "!!!_no_such_column_!!!",
            "missingValue": "2010-01-01T00:00:00Z"
          },
          "dimensionsSpec": {
            "dimensions": [
              "name"
            ]
          }
        }
      },
      "metricsSpec": [
        {
          "name": "count",
          "type": "count"
        }
      ]
    }
  }
}

Makes it all work

[ccaominh]

To work with parallel indexing, one possibility is to modifyInlineFirehose to implement FiniteFirehoseFactory so that it is splittable and always returns 1 split (that contains the entire contents).

[jihoonson]

I was writing about the same comment with @ccaominh. InlineFirehose could implement FiniteFirehoseFactory but it should always run in the sequential mode.

[jnaous]

If this gets fixed, is it possible to fix these cryptic messages that are sent out? I assume it's because of Jackson deserialization exceptions being regurgitated to the user?

…

On Mon, Oct 14, 2019 at 8:53 PM Jihoon Son ***@***.***> wrote: I was writing about the same comment with @ccaominh <https://github.com/ccaominh>. InlineFirehose could implement FiniteFirehoseFactory but it should always run in the sequential mode. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8673>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPSYCV6EA3XXBLRKGW5PCTQOU5DRANCNFSM4JAWTATQ> .

-- Jad Naous Imply | VP R&D 650-521-3425 jad.naous@imply.io

[gianm]

If this gets fixed, is it possible to fix these cryptic messages that are sent out? I assume it's because of Jackson deserialization exceptions being regurgitated to the user?

Yeah, those are Jackson exceptions, and it would be nice to make them more pleasant in a systematic way. I'm not sure if there's an easy way to do that without destroying the information in the error message or needing to parse their text. Maybe Jackson provides a way to get errors in a more programmatic way and we can write our own messages.