Doc question: Can protobuf extension be used with "index_parallel"?

[cswarth]

The Protobuf extension documentation demonstrates use of the extension to decode Kafka events.
Can the protobuf extension also be used to decode files, or it is only suitable for streaming input?

I tried make an example "index_parallel" task definition that uses protobuf but it gets rejected,

{"error":"Cannot construct instance of `org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexIngestionSpec`,
  problem: Cannot use parser and inputSource together. Try using inputFormat instead of parser.
 at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 77, column: 1] 
(through reference chain: org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask[\"spec\"])"
}

Task definition:

curl -v http://localhost:8888/druid/indexer/v1/task -H 'Content-Type: application/json' -d '
{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "local",
        "filter": "metrics.bin",
        "baseDir": "./"
      }
    },
    "tuningConfig": {
      "type": "index_parallel",
      "partitionsSpec": {
        "type": "dynamic"
      }
    },
    "dataSchema": {
      "dataSource": "metrics",
      "parser": {
        "type": "protobuf",
        "descriptor": "file:///tmp/metrics.desc",
        "protoMessageType": "Metrics",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "timestamp",
            "format": "auto"
          },
          "dimensionsSpec": {
            "dimensions": [
              "unit",
              "http_method",
              "http_code",
              "page",
              "metricType",
              "server"
            ],
            "dimensionExclusions": [
              "timestamp",
              "value"
            ]
          }
        }
      },
      "metricsSpec": [
        {
          "name": "count",
          "type": "count"
        },
        {
          "name": "value_sum",
          "fieldName": "value",
          "type": "doubleSum"
        },
        {
          "name": "value_min",
          "fieldName": "value",
          "type": "doubleMin"
        },
        {
          "name": "value_max",
          "fieldName": "value",
          "type": "doubleMax"
        }
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": "NONE"
      }
  }
}
'

[cswarth]

I suspect the answer is "No, protobuf cannot be used to parse file contents", based on this comment,
#10839 (comment)

If you are interested in doing a follow-up PR or two, the protobuf extension is one of the few ingestion formats that has not been ported to the newer InputFormat interface which has replaced parsers. There are a handful of implementations that could be used as reference (JSON, Avro though it is missing an InputFormat for streaming support and using schema registry itself, and many others). I think this should be relatively straight-forward with the protobuf extension since most of the work is being done in the decoder implementations, which could also be re-used by the InputFormat implementation.

[clintropolis]

I am not familiar enough with protobuf encoded files to know if this will work, but the error you are seeing is related to trying to use inputSource with a parser. You need to use the older 'parser' based ingestion spec to not see this error, see https://druid.apache.org/docs/latest/ingestion/index.html#parser-deprecated. (no inputSource or inputFormat on parser based specs, instead "firehoses" are used in place of input source iirc, https://druid.apache.org/docs/latest/ingestion/native-batch.html#firehoses-deprecated)

The protobuf parser depends on getting byte chunks of encoded proto messages, so any file reader would need to read out individual message binary blobs from the underlying file to feed to the parser, which is the part that makes me unsure that protobuf files with batch would work correctly. For example the CSV parser is fed single lines from an underlying text file, where the line is expected to be a CSV row. A protobuf file parser would need something to do something similar with the binary message blobs from the file, and i'm not sure if just having the message schema is enough for that to work.

I think if the inputFormat did exist, it might still have this issue, a file based protobuf decoder might be a specialized InputFormat implementation that is separate from a streaming individual message processor format (again I'm not familiar with the file side of things much).

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.