druid-orc-extensions hadoop-common dependency is broken

[egor-ryashin]

druid-orc-extensions hadoop-common dependency is broken or maybe the extension isn't properly documented

Affected Version

0.13.0-incubating

Description

Using this modules:
druid.extensions.loadList=["mysql-metadata-storage", "druid-kafka-indexing-service", "druid-orc-extensions", "druid-hdfs-storage"]

du extensions/*
20032	extensions/druid-cassandra-storage
46928	extensions/druid-hdfs-storage
4240	extensions/druid-kafka-indexing-service
168	extensions/druid-lookups-cached-global
56640	extensions/druid-orc-extensions
136	extensions/druid-s3-extensions
1968	extensions/mysql-metadata-storage

Posting this task:

{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "my_orc_test",
      "metricsSpec": [
        {
          "type": "count",
              "name": "count"
            }
      ],
      "granularitySpec": {
          "segmentGranularity": "DAY",
          "queryGranularity": "second",
          "intervals" : [ "2018-07-10/2018-07-11" ]
       },
        "parser": {
          "type": "orc",
          "parseSpec": {
            "format": "timeAndDims",
            "timestampSpec": {
              "column": "time",
              "format": "auto"
            },
            "dimensionsSpec": {
              "dimensions": [
                "tag"
              ],
              "dimensionExclusions": [],
              "spatialDimensions": []
            }
          },
          "typeString": "struct<time:string,tag:string>",
          "mapFieldNameFormat": "<PARENT>_<CHILD>"
        }
            
    },
    "ioConfig": {
        "type": "index_parallel",
        "firehose": {
          "type": "local",
          "baseDir": "./",
          "filter": "*.orc"
        }
    }
  }
}

Got error from the spawned subtask:

2019-04-10T22:16:10,413 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Uncaught Throwable while running task[AbstractTask{id='index_sub_my_orc_test_2019-04-10T22:16:02.661Z', groupId='index_parallel_my_orc_test_2019-04-10T22:15:55.541Z', taskResource=TaskResource{availabilityGroup='index_sub_my_orc_test_2019-04-10T22:16:02.661Z', requiredCapacity=1}, dataSource='my_orc_test', context={}}]
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable

The log also says the needed dependency jar is loaded beforehand:

2019-04-10T22:16:04,921 INFO [main] org.apache.druid.initialization.Initialization - added URL[file:/Users/egorryashin/a-druid-0.13-i/extensions/druid-hdfs-storage/hadoop-common-2.8.3.jar] for extension[druid-hdfs-storage]

The task doesn't work neither with druid-hdfs-storage loaded nor without it.

I spotted that while I was investigating #6925

[jihoonson]

Hi @egor-ryashin, index_parallel task supports only text formats and IngestSegmentFirehose currently. Probably the error message should be improved.

[jon-wei]

I think this is hitting the same extension classloading issue discussed here: #6430 (review)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.