This problem was first reported by @clintropolis when verifying the 0.21.1-RC1.
On Linux platform, tasks fail due to failure of creating task log dir. However, on macOS, tasks works well.
middlemanager | 2021-05-18T21:20:39,239 INFO [forking-task-runner-0]
org.apache.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
middlemanager | org.apache.druid.java.util.common.IOE: Unable to create task log dir[/opt/data/indexing-logs]
middlemanager | at org.apache.druid.indexing.common.tasklogs.FileTaskLogs.pushTaskLog(FileTaskLogs.java:59)
~[druid-indexing-service-0.21.1.jar:0.21.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:386)
[druid-indexing-service-0.21.1.jar:0.21.1]
middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:137)
[druid-indexing-service-0.21.1.jar:0.21.1]
middlemanager | at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
middlemanager | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[?:1.8.0_275]
middlemanager | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[?:1.8.0_275]
middlemanager | at java.lang.Thread.run(Thread.java:748)[?:1.8.0_275]
middlemanager | 2021-05-18T21:20:39,247 INFO [forking-task-runner-0]
org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task
directory: var/druid/task/index_parallel_wikipedia_gmkopeph_2021-05-18T21:20:10.899Z
In the example docker-compose.yml, /opt/data is mounted to a host directory distribution\docker\storage, if the host directory is belong to root, the permission of /opt/data inside container is also root. While druid processes inside contains are running as a normal user, they have no permission to create sub directories under /opt/data.
There's no way for us to control the permission of the directory on host OS in docker-compose, so either we leave this problem for users to setup the permission of the host directories or uses a named volume to avoid the problem.
- If we choose the first, we have to update the docker.md to tell users how to do that.
- If we choose the latter, docker will create a dir automatically on the host OS and setup permission correctly, but that directory is located at
/var/lib/docker/volumes/ by default. Since this dir usually has smaller disk size, putting druid data and task logs under that directory might consume that directory quickly. I don't know how many people use this example docker-compose for long term use, but we might need to state this in the doc to let users know how to change the default directory to a directory they want to keep data.
BTW, @clintropolis What's your initial thinking about using host directory in the docker-compose.yml since I see you added these volumes according to the code history ?
cc @jihoonson @clintropolis @xvrl
This problem was first reported by @clintropolis when verifying the 0.21.1-RC1.
On Linux platform, tasks fail due to failure of creating task log dir. However, on macOS, tasks works well.
In the example docker-compose.yml,
/opt/datais mounted to a host directorydistribution\docker\storage, if the host directory is belong to root, the permission of/opt/datainside container is also root. While druid processes inside contains are running as a normal user, they have no permission to create sub directories under/opt/data.There's no way for us to control the permission of the directory on host OS in docker-compose, so either we leave this problem for users to setup the permission of the host directories or uses a named volume to avoid the problem.
/var/lib/docker/volumes/by default. Since this dir usually has smaller disk size, putting druid data and task logs under that directory might consume that directory quickly. I don't know how many people use this example docker-compose for long term use, but we might need to state this in the doc to let users know how to change the default directory to a directory they want to keep data.BTW, @clintropolis What's your initial thinking about using host directory in the docker-compose.yml since I see you added these volumes according to the code history ?
cc @jihoonson @clintropolis @xvrl