Symlink file from a symlink directory not being copied across stages compared with docker build

[tanguydelignieresaccenture]

Actual behavior
A clear and concise description of what the bug is.
For docker build, copy symlink file from a symlink directory copy the actual file across stages.
For kaniko build, copy symlink file from a symlink directory throws the following error across stages:

error building image: error building stage: failed to execute command: failed to get fileinfo for /kaniko/0/test/link: lstat /kaniko/0/test/link: no such file or directory

Expected behavior
A clear and concise description of what you expected to happen.
For kaniko build, copy symlink file from a symlink directory would copy the actual file across stages, as in docker build.

To Reproduce
Steps to reproduce the behavior:

1. Use the provided Dockerfile

2. Run kaniko build

Additional Information
Please provide either the Dockerfile you're trying to build or one that can reproduce this error.

• Dockerfile (derived from Fix Symlinks not being copied across stages #971 Dockerfile_test_copy_symlink test)

FROM busybox as t
RUN echo "hello" > /tmp/target
RUN ln -s /tmp /test && \
	ln -s /test/target /test/link

FROM busybox
COPY --from=t /test/link /anotherplace/link

• Build Context

Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
Nothing more than the provided Dockerfile.

• Kaniko Image (fully qualified with digest)

gcr.io/kaniko-project/executor:debug-v0.17.1

• Kaniko build command line and result

$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --no-push
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Retrieving image manifest busybox
INFO[0001] Retrieving image manifest busybox
INFO[0003] Retrieving image manifest busybox
INFO[0004] Retrieving image manifest busybox
INFO[0005] Built cross stage deps: map[0:[/test/link]]
INFO[0005] Retrieving image manifest busybox
INFO[0006] Retrieving image manifest busybox
INFO[0007] Unpacking rootfs as cmd RUN echo "hello" > /tmp/target requires it.
INFO[0008] Taking snapshot of full filesystem...
INFO[0008] RUN echo "hello" > /tmp/target
INFO[0008] cmd: /bin/sh
INFO[0008] args: [-c echo "hello" > /tmp/target]
INFO[0008] Taking snapshot of full filesystem...
INFO[0008] RUN ln -s /tmp /test &&      ln -s /test/target /test/link
INFO[0008] cmd: /bin/sh
INFO[0008] args: [-c ln -s /tmp /test &&        ln -s /test/target /test/link]
INFO[0008] Taking snapshot of full filesystem...
INFO[0008] Saving file /tmp/target for later use
INFO[0008] Saving file /test/link for later use
INFO[0008] Deleting filesystem...
INFO[0008] Retrieving image manifest busybox
INFO[0009] Retrieving image manifest busybox
INFO[0010] Unpacking rootfs as cmd COPY --from=t /test/link /anotherplace/link requires it.
INFO[0011] Taking snapshot of full filesystem...
INFO[0011] COPY --from=t /test/link /anotherplace/link
error building image: error building stage: failed to execute command: failed to get fileinfo for /kaniko/0/test/link: lstat /kaniko/0/test/link: no such file or directory

• Docker build command line and result

$ docker build -t docker-test .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as t
 ---> 6d5fcfe5ff17
Step 2/5 : RUN echo "hello" > /tmp/target
 ---> Using cache
 ---> 9735661d6942
Step 3/5 : RUN ln -s /tmp /test &&      ln -s /test/target /test/link
 ---> Using cache
 ---> 90bea1a8c973
Step 4/5 : FROM busybox
 ---> 6d5fcfe5ff17
Step 5/5 : COPY --from=t /test/link /anotherplace/link
 ---> 58cf0c1cbb36
Successfully built 58cf0c1cbb36
Successfully tagged docker-test:latest

$ docker run --rm -ti docker-test sh
/ # ls -lh /
total 40K
drwxr-xr-x    2 root     root        4.0K Feb  5 10:32 anotherplace
drwxr-xr-x    2 root     root       12.0K Dec 23 19:21 bin
drwxr-xr-x    5 root     root         360 Feb  5 10:34 dev
drwxr-xr-x    1 root     root        4.0K Feb  5 10:34 etc
drwxr-xr-x    2 nobody   nogroup     4.0K Dec 23 19:21 home
dr-xr-xr-x  191 root     root           0 Feb  5 10:34 proc
drwx------    1 root     root        4.0K Feb  5 10:34 root
dr-xr-xr-x   13 root     root           0 Feb  5 10:34 sys
drwxrwxrwt    2 root     root        4.0K Dec 23 19:21 tmp
drwxr-xr-x    3 root     root        4.0K Dec 23 19:21 usr
drwxr-xr-x    4 root     root        4.0K Dec 23 19:21 var
/ # ls -lh /anotherplace/
total 4K
-rw-r--r--    1 root     root           6 Feb  5 10:30 link
/ # cat /anotherplace/link
hello
/ #

The use case is having a symlinked directory with a symlinked file in the stage 0 image, and needing to extract the actual target to the base image of the multistage build.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[raijinsetsu]

I ran into this issue as well but only one of my symlinks was causing an issue but not the other 30+ symlinks. I haven't dug deeper into it yet.

[raijinsetsu]

Minor addendum to my prior comment: sometimes the symlink causes an issue during unpacking and other times the symlink is just... missing.

[tejal29]

Thanks @tanguydelignieresaccenture. I just merged #1030 which should fix this issue.
I will verify this.

[tanguydelignieresaccenture]

Thanks @tejal29 , let me know.

[tanguydelignieresaccenture]

@tejal29, FYI, I've tested with the debug-e0b913997f07dab0cc5751a7b2d26f62d9c9830e release, supposing it includes the merge of #1030 and have confirm I did not reproduce the problem anymore in the context of this issue (nor in my original more complex case).

$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/1036:/workspace gcr.io/kaniko-project/executor:debug-e0b913997f07dab0cc5751a7b2d26f62d9c9830e /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --no-push
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Resolved base name busybox to busybox
INFO[0000] Retrieving image manifest busybox
INFO[0002] Retrieving image manifest busybox
INFO[0004] Retrieving image manifest busybox
INFO[0005] Retrieving image manifest busybox
INFO[0007] Built cross stage deps: map[0:[/test/link]]
INFO[0007] Retrieving image manifest busybox
INFO[0009] Retrieving image manifest busybox
INFO[0010] Unpacking rootfs as cmd RUN echo "hello" > /tmp/target requires it.
INFO[0012] Taking snapshot of full filesystem...
INFO[0012] RUN echo "hello" > /tmp/target
INFO[0012] cmd: /bin/sh
INFO[0012] args: [-c echo "hello" > /tmp/target]
INFO[0012] Taking snapshot of full filesystem...
INFO[0012] RUN ln -s /tmp /test &&      ln -s /test/target /test/link
INFO[0012] cmd: /bin/sh
INFO[0012] args: [-c ln -s /tmp /test &&        ln -s /test/target /test/link]
INFO[0012] Taking snapshot of full filesystem...
INFO[0012] Saving file /tmp/target for later use
INFO[0012] Saving file /test/link for later use
INFO[0012] Deleting filesystem...
INFO[0012] Retrieving image manifest busybox
INFO[0014] Retrieving image manifest busybox
INFO[0016] Unpacking rootfs as cmd COPY --from=t /test/link /anotherplace/link requires it.
INFO[0017] Taking snapshot of full filesystem...
INFO[0017] COPY --from=t /test/link /anotherplace/link
INFO[0017] Taking snapshot of files...
INFO[0017] RUN ls -lh / &&      ls -lh /anotherplace/ &&        cat /anotherplace/link
INFO[0017] cmd: /bin/sh
INFO[0017] args: [-c ls -lh / &&        ls -lh /anotherplace/ &&        cat /anotherplace/link]
total 56K
drwxr-xr-x    2 root     root        4.0K Feb  9 22:43 anotherplace
drwxr-xr-x    2 root     root       12.0K Feb  9 22:43 bin
drwxr-xr-x    2 root     root       12.0K Feb  9 22:43 busybox
drwxr-xr-x    5 root     root         340 Feb  9 22:43 dev
drwxr-xr-x    1 root     root        4.0K Feb  9 22:43 etc
drwxr-xr-x    2 nobody   nogroup     4.0K Feb  9 22:43 home
drwxr-xr-x    1 root     root        4.0K Feb  9 22:43 kaniko
dr-xr-xr-x  127 root     root           0 Feb  9 22:43 proc
drwx------    2 root     root        4.0K Feb  9 22:43 root
dr-xr-xr-x   13 root     root           0 Feb  9 22:43 sys
drwxrwxrwt    2 root     root        4.0K Feb  9 22:43 tmp
drwxr-xr-x    3 root     root        4.0K Feb  9 22:43 usr
drwxr-xr-x    4 root     root        4.0K Feb  9 22:43 var
drwxrwxrwx    1 root     root           0 Feb  7 13:53 workspace
total 4K
-rw-r--r--    1 root     root           6 Feb  9 22:43 link
hello
INFO[0017] Taking snapshot of full filesystem...
INFO[0018] No files were changed, appending empty layer to config. No layer added to image.
INFO[0018] Skipping push to container registry due to --no-push flag

Dockerfile used:

FROM busybox as t
RUN echo "hello" > /tmp/target
RUN ln -s /tmp /test && \
	ln -s /test/target /test/link

FROM busybox
COPY --from=t /test/link /anotherplace/link
RUN ls -lh / && \
	ls -lh /anotherplace/ && \
	cat /anotherplace/link

[tejal29]

I just verified your dockerfile in latest master gcr.io/tejal-test/executor

docker run -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test_1036
INFO[0000] Resolved base name busybox to busybox        
INFO[0000] Resolved base name busybox to busybox        
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name busybox to busybox        
INFO[0000] Resolved base name busybox to busybox        
INFO[0000] Retrieving image manifest busybox            
INFO[0001] Retrieving image manifest busybox            
INFO[0002] Retrieving image manifest busybox            
INFO[0003] Retrieving image manifest busybox            
INFO[0004] Built cross stage deps: map[0:[/test/link]]  
INFO[0004] Retrieving image manifest busybox            
INFO[0004] Retrieving image manifest busybox            
INFO[0005] Unpacking rootfs as cmd RUN echo "hello" > /tmp/target requires it. 
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving paths                              
INFO[0005] RUN echo "hello" > /tmp/target               
INFO[0005] cmd: /bin/sh                                 
INFO[0005] args: [-c echo "hello" > /tmp/target]        
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving paths                              
INFO[0005] RUN ln -s /tmp /test && 	ln -s /test/target /test/link 
INFO[0005] cmd: /bin/sh                                 
INFO[0005] args: [-c ln -s /tmp /test && 	ln -s /test/target /test/link] 
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving paths                              
INFO[0006] Saving file /tmp/target for later use        
INFO[0006] Saving file /test/link for later use         
INFO[0006] Deleting filesystem...                       
INFO[0006] Retrieving image manifest busybox            
INFO[0006] Retrieving image manifest busybox            
INFO[0007] Unpacking rootfs as cmd COPY --from=t /test/link /anotherplace/link requires it. 
INFO[0007] Taking snapshot of full filesystem...        
INFO[0007] Resolving paths                              
INFO[0007] COPY --from=t /test/link /anotherplace/link  
INFO[0007] Resolving paths                              
INFO[0007] Taking snapshot of files... 

I checked the built image:
The results are consistent

tejaldesai@@kaniko (r-v0.18.0)$ docker run --rm -ti gcr.io/tejal-test/test_1036  sh
Unable to find image 'gcr.io/tejal-test/test_1036:latest' locally
latest: Pulling from tejal-test/test_1036
bdbbaa22dec6: Pull complete 
3cde9973cb26: Pull complete 
Digest: sha256:9fb30b08121d1d76a48a02de71479d05a9beb217390eb22b30eb882b69545cdd
Status: Downloaded newer image for gcr.io/tejal-test/test_1036:latest
/ # ls
anotherplace  bin           dev           etc           home          proc          root          sys           tmp           usr           var
/ # ls -lh /
total 40K    
drwxr-xr-x    2 root     root        4.0K Mar  6 04:32 anotherplace
drwxr-xr-x    2 root     root       12.0K Dec 23 19:21 bin
drwxr-xr-x    5 root     root         360 Mar  6 04:34 dev
drwxr-xr-x    1 root     root        4.0K Mar  6 04:34 etc
drwxr-xr-x    2 nobody   nogroup     4.0K Dec 23 19:21 home
dr-xr-xr-x  483 root     root           0 Mar  6 04:34 proc
drwx------    1 root     root        4.0K Mar  6 04:34 root
dr-xr-xr-x   13 root     root           0 Mar  6 04:34 sys
drwxrwxrwt    2 root     root        4.0K Dec 23 19:21 tmp
drwxr-xr-x    3 root     root        4.0K Dec 23 19:21 usr
drwxr-xr-x    4 root     root        4.0K Dec 23 19:21 var
/ # ls -lh /anotherplace/
total 4K     
-rw-r--r--    1 root     root           6 Mar  6 04:32 link
/ # cat /anotherplace/link
hello
/ #