Unable to use AWS Fargate bind mounts with Dockerfile

[mattmassicotte]

Affected Version

0.21.1-rc1

Description

So far, I've never been able to use AWS Fargate bind mounts (ie Docker volumes) with Druid. I have a workaround of using EFS mounts, but it is a real pain to use. I was really excited to see that 0.21.1-rc1 included a change to the Dockefile, so I decided to check it out. Unfortunately, it still does not work.

I want to stress that this is not a regression.

The druid process fails to start with the following logging:

mkdir: can't create directory 'var/tmp': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied
mkdir: can't create directory 'var/druid/': Permission denied

I'm neither a Fargate, Druid, or Docker expert. But, from what I can gather, Fargate is very sensitive to the nature of the path in the VOLUME command in the Dockerfile.

Some details here:

aws/containers-roadmap#938 (comment)

An AWS employee actually looked the the Druid Dockefile and identified the symlinking as the issue. I figured I'd open this bug since it is indeed still not quite compatible with Fargate and this area was just touched.

If I can help in some way, I'd really love to, as this is currently affecting my ability to easily scale Druid. Each new instance requires a special EFS mount point, which it turn requires a special ECS task definition. Regular docker volume support would be awesome.

[jihoonson]

Hi @mattmassicotte, thank you for the report! Do you have a fix in your mind that could work? I'm wondering if the fix would be just like renaming the directory instead of creating a symlink, but am not an expert of Fargate.

[FrankChen021]

This problem is very interesting. I agree that we eliminate the symlink to make a try.

@mattmassicotte Could you do a test by changing the line

druid/distribution/docker/Dockerfile

Line 44 in fcb4eaa

&& ln -s /opt/apache-druid-${VERSION} /opt/druid

to

&& mv /opt/apache-druid-${VERSION} /opt/druid 

and build docker images by your own to see if it works ?

[mattmassicotte]

Thanks so much for looking into this, and for the suggestions.I'm attempting to do this.

Unfortunately, I'm on an arm Mac, and I'm having a very hard time building the image. I also kinda don't know what I'm doing, but I'm fairly sure the issues I'm hitting are platform-related.

I'm going to try to get set up on an amd64 machine and give it another go. Will report back.

[mattmassicotte]

Ok, so that indeed seems to have resolved the issue! I was able to make and test the change from the druid-0.21.1-rc1 tag, and successfully start up a coordinator. I have not tried a full system. But, this is major progress. Thank you all so much!