Files left on disk when batch task with IngestSegmentFirehose is done

[zhihuij]

We use batch task(normal batch task, not hadoop), and found that after the batch task completed, there were many files left on disk, and after serval days, we have to clean the files. Can Druid clean these files after the batch task is done?

Example directory list:
drwxr-xr-x 3 user group 4096 Nov 4 00:05 2016-11-03T00:00:00.000Z_2016-11-03T01:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 00:05 2016-11-03T01:00:00.000Z_2016-11-03T02:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 00:05 2016-11-03T02:00:00.000Z_2016-11-03T03:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 00:05 2016-11-03T03:00:00.000Z_2016-11-03T04:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 00:05 2016-11-03T04:00:00.000Z_2016-11-03T05:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 00:05 2016-11-03T05:00:00.000Z_2016-11-03T06:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 06:05 2016-11-03T06:00:00.000Z_2016-11-03T07:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 06:05 2016-11-03T07:00:00.000Z_2016-11-03T08:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 06:05 2016-11-03T08:00:00.000Z_2016-11-03T09:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 06:05 2016-11-03T09:00:00.000Z_2016-11-03T10:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 06:05 2016-11-03T10:00:00.000Z_2016-11-03T11:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 06:05 2016-11-03T11:00:00.000Z_2016-11-03T12:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 12:05 2016-11-03T12:00:00.000Z_2016-11-03T13:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 12:05 2016-11-03T13:00:00.000Z_2016-11-03T14:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 12:05 2016-11-03T14:00:00.000Z_2016-11-03T15:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 12:05 2016-11-03T15:00:00.000Z_2016-11-03T16:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 12:05 2016-11-03T16:00:00.000Z_2016-11-03T17:00:00.000Z
drwxr-xr-x 3 user group 4096 Nov 4 12:05 2016-11-03T17:00:00.000Z_2016-11-03T18:00:00.000Z

[dclim]

@onlychoice index task shouldn't leave intermediate files behind. What directory is that, and what is contained in those directories? They look like segments to me. Is that your deep storage directory?

[zhihuij]

1. The directory is "/srv/nbs/0/druid/task/reingest/work/online_request", "online_request" is the datasource name, "/srv/nbs/0/druid" is the local directory for druid files.

2. The content of the directory:
   user@classa-test1:/srv/nbs/0/druid/task/reingest/work/online_request/2016-11-03T00:00:00.000Z_2016-11-03T06:00:00.000Z/2016-11-03T16:05:00.173Z$ ll
   total 20
   drwxr-xr-x 2 user group 4096 Nov 4 18:10 0
   drwxr-xr-x 2 user group 4096 Nov 4 18:10 1
   drwxr-xr-x 2 user group 4096 Nov 4 18:10 2
   drwxr-xr-x 2 user group 4096 Nov 4 18:10 3
   drwxr-xr-x 2 user group 4096 Nov 4 18:10 4
   user@classa-test1:/srv/nbs/0/druid/task/reingest/work/online_request/2016-11-03T00:00:00.000Z_2016-11-03T06:00:00.000Z/2016-11-03T16:05:00.173Z$ cd 0/
   user@classa-test1:/srv/nbs/0/druid/task/reingest/work/online_request/2016-11-03T00:00:00.000Z_2016-11-03T06:00:00.000Z/2016-11-03T16:05:00.173Z/0$ ll
   total 363068
   -rw-r--r-- 1 user group 371770584 Nov 4 18:10 00000.smoosh
   -rw-r--r-- 1 user group 499 Nov 4 18:10 meta.smoosh
   -rw-r--r-- 1 user group 4 Nov 4 18:10 version.bin

   The batch task is submitted every six hours.

3. No, segment files are stored on HDFS.

[dclim]

@onlychoice I was able to repro this, looks like the IngestSegmentFirehoseFactory doesn't cleanup the files pulled by SegmentLoader. Thanks for filing the issue.

[gianm]

This is related to #3608, a fix to that should fix this one too.

[zhihuij]

#4069 fix this, the issue should be closed, @dclim @gianm