S3 InputSource issue when using prefix mode if a directory contains _SUCCESS files

[sascha-coenen]

Affected Version

v 0.17.0

Description

We set up Druid Indexer nodes to test the new native parallel ingestion.
Then we used the following InputSource section within an index_parallel spec to point to a "directory" in S3 that would contain a _SUCCESS file along with a bunch of data files.

      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://smt-druid-ingestion-stage/SI-835/year=2020/month=01/day=20/hour=00/1580297687716/auction"]
      }

The index_parallel task fails and we observed in the logs that the above section got rewritten to the following

      "inputSource": {
        "type": "s3",
        "uris": null,
        "prefixes": null,
        "objects": [
          {
            "bucket": "smt-druid-ingestion-stage",
            "path": "SI-835/year=2020/month=01/day=20/hour=00/1580297687716/auction/_SUCCESS"
          }
        ]
      }

This looks to me like an attempt was made to support filtering out _SUCCESS files from the file list and that inadvertently the filter condition is doing the opposite.

[jihoonson]

Hi @sascha-coenen, I think the second inputSource with the _SUCCESS file is created for a subtask. In 0.17, the parallel task lists all objects starting with the given prefixes and creates subtasks to process each object. Unfortunately, there is no way to filter out unwanted objects for s3 input source now.

[gianm]

Are you saying that the data files did not get ingested?

At first glance I don't see code to do anything one way or the other with _SUCCESS files. They should be treated the same as any other file. There's some code here to filter out directory placeholders, but it doesn't seem related: https://github.com/apache/druid/blob/druid-0.17.0/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryIterator.java#L154

I could be missing something, but could you double-check that there are actually objects under this prefix and they are nonzero size etc.

[sascha-coenen]

Are you saying that the data files did not get ingested?

yes. the ingestion fails as long as we have the _SUCCESS file in the folder. Then I deleted the _SUCCESS file and the same ingestion spec resubmitted a second time was now working. At least the first phase was running successfully.

WIth Druid 0.16.0 we had some successful attempts at using index_parallel. With 0.17.0 the second phase immediately fails. Currently we are clueless why, but as far as this report scope is concerned, it is sufficient that the first phase completes successfully. I can see in the logs that the files got picked up and processed and that there were as many slots used as there are files in the s3 location.

When the _SUCCESS file is present, only one slot will be used, so it doesn't get to the point that the other files would also be scheduled. Perhaps I'm wrong. I can double check tomorrow to be sure. But now I know that there is no dedicated code to deal with shadowed files so I know to remove them for now.

[vikramsinghchandel]

Hi @gianm any update on this?

the _SUCCESS file is actually of zero size(0 B) and is created as a result of a Spark ETL job via EMR. This is the error that pops while it tries to sample the dataset

[jihoonson]

Hi @vikramsinghchandel, #9450 can work around this issue.

[gianm]

@jihoonson, @vikramsinghchandel, does #9450 fix this issue? Could we close it?

[jihoonson]

#9450 could be a workaround for this issue, but probably #9813 would be the real issue. I think we can track it in #9813 and close this.