COPY resolves symlinks it shouldn't try to resolve

[gilbsgilbs]

Actual behavior

Since v0.18.0, COPY tries to resolve some symlinks it shouldn't try to resolve, which is a regression.

Expected behavior

I would expect kaniko to let those symlinks remain symlinks when copied in the image.

To Reproduce

For instance, we can create a symlink that points to nothing to make kaniko crash:

mkdir example
cd example/

echo "FROM alpine\nCOPY . /example" > Dockerfile
ln -s ./missing symlink

docker run --rm -it -v $(pwd):/workspace:ro gcr.io/kaniko-project/executor:debug --no-push

👇 👇 👇

[…]                           
error building image: error building stage: failed to execute command: lstat /workspace/missing: no such file or directory

If we create the symlink target, it builds successfully but the symlink is replaced with a plain file within the image:

touch missing
docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'

👇 👇 👇

[…]
total 4
-rw-r--r--    1 1000     1000            29 Mar  6 14:28 Dockerfile
-rw-r--r--    1 1000     1000             0 Mar  6 14:28 missing
-rw-r--r--    1 1000     1000             0 Mar  6 14:28 symlink

Whereas in v0.17.1 (or using docker build) the symlink is not resolved:

$ docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug-v0.17.1 /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
[…]
total 8
-rw-r--r--    1 1000     1000            29 Mar  6 14:29 Dockerfile
-rw-r--r--    1 1000     1000             0 Mar  6 14:29 missing
lrwxrwxrwx    1 root     root             9 Mar  6 14:29 symlink -> ./missing

Additional Information

This is a regression from v0.18.0. It worked properly in v0.17.1.

Note that Kaniko v0.18.0 changed the way symlinks work to better match actual Docker's behavior. That probably introduced this bug as a side-effect.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[gilbsgilbs]

FWIW, this was probably introduced by #943 .

[tejal29]

@gilbsgilbs Thanks for the issue.

1. Copying a dead symlink throws an error.

We changed copy to be as close to docker build.
I verified your case on docker version 1.9.03

Dockerfile

tejaldesai@@tmep$ cat Dockerfile 
FROM scratch

COPY deadlink workdir/

My workdir

tejaldesai@@tmep$ ls -al 
total 24
drwxr-xr-x  5 tejaldesai primarygroup 4096 Mar  6 15:49 .
drwxr-x--- 11 tejaldesai primarygroup 4096 Mar  6 11:57 ..
lrwxrwxrwx  1 tejaldesai primarygroup    4 Mar  6 15:48 deadlink -> test

Dockerbuild log

tejaldesai@@tmep$ docker build . 
Sending build context to Docker daemon   7.68kB
Step 1/2 : FROM scratch
 ---> 
Step 2/2 : COPY deadlink workdir/
COPY failed: stat /usr/local/google/docker/tmp/docker-builder885367293/test: no such file or directory

2. Copying a single symlink creates a regular file.

We also made changes to make COPY behavior is mimic to docker build
Dockerfile

FROM scratch
COPY sym.link workdir/

My dir

tejaldesai@@tmep$ ls -al 
drwxr-xr-x  5 tejaldesai primarygroup 4096 Mar  6 15:56 .
drwxr-x--- 11 tejaldesai primarygroup 4096 Mar  6 11:57 ..
-rw-r--r--  1 tejaldesai primarygroup   37 Mar  6 15:56 Dockerfile
-rw-r--r--  1 tejaldesai primarygroup    6 Mar  6 15:56 hello
lrwxrwxrwx  1 tejaldesai primarygroup    5 Mar  6 15:56 sym.link -> hello

Run docker build

context: path ",." not found
tejaldesai@@tmep$ docker build . 
Sending build context to Docker daemon  8.704kB
Step 1/2 : FROM scratch
 ---> 
Step 2/2 : COPY sym.link workdir/
 ---> 3725f395af68
Successfully built 3725f395af68

Then, i exploded the built image.

tejaldesai@@tar$ tar -xvf 48233d291de8fc27f595bf188b1ed62de4ca3d3a562add526f302e8220e8385c/layer.tar 
workdir/
workdir/sym.link
tejaldesai@@tar$ ls -al workdir/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar  6 15:57 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar  6 15:59 ..
-rw-r--r-- 1 tejaldesai primarygroup    6 Mar  6 15:56 sym.link
tejaldesai@@tar$ 

If you see the sym.link is a regular file.

3. Copying a directory with symlinks preserves symlinks

However, if you copy a directory, then symlinks are preserved.
Dockerfile

FROM scratch

COPY src/ workdir/

My dir

tejaldesai@@tmep$ ls -al src/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar  6 15:19 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar  6 16:01 ..
lrwxrwxrwx 1 tejaldesai primarygroup    5 Mar  6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup   10 Mar  6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup   19 Mar  6 15:19 zSym.link -> ../anotherSrc/hello
tejaldesai@@tmep$ ls -al anotherSrc/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar  6 15:18 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar  6 16:01 ..
-rw-r--r-- 1 tejaldesai primarygroup   14 Mar  6 15:18 hello
tejaldesai@@tmep$ 

Build Docker image

Sending build context to Docker daemon   7.68kB
Step 1/2 : FROM scratch
 ---> 
Step 2/2 : COPY src/ workdir/
 ---> Using cache
 ---> 0ab1ea7e4e3f
Successfully built 0ab1ea7e4e3f

Exploded image tar

tejaldesai@@tar$ ls -al workdir/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar  6 15:19 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar  6 16:03 ..
lrwxrwxrwx 1 tejaldesai primarygroup    5 Mar  6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup   10 Mar  6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup   19 Mar  6 15:19 zSym.link -> ../anotherSrc/hello
tejaldesai@@tar$

In this case, zSymlink is a deadlink since anotherSrc wasn't copied over.

4. Copying a dir with symlink and dead symlinks
   Dockerfile

FROM scratch

COPY src/ workdir/

My workdir

tejaldesai@@src$ ls -al 
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar  6 16:21 .
drwxr-xr-x 5 tejaldesai primarygroup 4096 Mar  6 16:20 ..
lrwxrwxrwx 1 tejaldesai primarygroup    1 Mar  6 16:21 deadlink -> t
lrwxrwxrwx 1 tejaldesai primarygroup    5 Mar  6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup   10 Mar  6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup   19 Mar  6 15:19 zSym.link -> ../anotherSrc/hello
tejaldesai@@src$ 

Docker build

Step 1/2 : FROM scratch
 ---> 
Step 2/2 : COPY src/ workdir/
 ---> 81a9999565db
Successfully built 81a9999565db

Exploded tar

tejaldesai@@tard$ ls -al workdir/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar  6 16:23 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar  6 16:24 ..
lrwxrwxrwx 1 tejaldesai primarygroup    1 Mar  6 16:21 deadline -> t
lrwxrwxrwx 1 tejaldesai primarygroup    5 Mar  6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup   10 Mar  6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup   19 Mar  6 15:19 zSym.link -> ../anotherSrc/hello

[WestonThayer]

I'm also seeing build failures that are likely caused by this when using npm link during image build. All fine in v0.17.1.

[tejal29]

@WestonThayer can you please provide your dockerfile?

I am not very familiar with npm link

[gilbsgilbs]

@tejal29 I agree that Kaniko is closer to Docker's behavior in the cases you described and that the title of this issue may be slightly incorrect (I'll fix it). But I insist, the case I gave in the original post still is a regression regarding docker 19.03 and kaniko 0.17.1 behavior. Kaniko v0.18.0 tries to resolve some symlinks it shouldn't resolve. The regression label should be kept. Are you unable to reproduce the two repro I gave in the original post?

Case 0: dead symlink

Setup

[nils@buks example]$ lsb_release -a
LSB Version:	1.4
Distributor ID:	Arch
Description:	Arch Linux
Release:	rolling
Codename:	n/a

[nils@buks example]$ uname -a
Linux buks 5.5.6-arch1-1 #1 SMP PREEMPT Mon, 24 Feb 2020 12:20:16 +0000 x86_64 GNU/Linux

[nils@buks example]$ docker --version
Docker version 19.03.6-ce, build 369ce74a3c

[nils@buks example]$ cat Dockerfile 
FROM alpine
COPY . /example

[nils@buks example]$ ls -la
total 4
drwxr-xr-x  2 nils nils  80 Mar  7 06:13 .
drwxrwxrwt 16 root root 360 Mar  7 06:23 ..
-rw-r--r--  1 nils nils  28 Mar  7 06:13 Dockerfile
lrwxrwxrwx  1 nils nils   9 Mar  7 06:13 symlink -> ./missing

docker build

works 👇

[nils@buks example]$ sudo docker build --no-cache -t testimage .
Sending build context to Docker daemon   2.56kB
Step 1/2 : FROM alpine
 ---> e7d92cdc71fe
Step 2/2 : COPY . /example
 ---> 881fa1ebd1ec
Successfully built 881fa1ebd1ec
Successfully tagged testimage:latest

[nils@buks example]$ sudo docker run --rm -it testimage ls -la /example
total 8
drwxr-xr-x    1 root     root            34 Mar  7 05:23 .
drwxr-xr-x    1 root     root           148 Mar  7 05:23 ..
-rw-r--r--    1 root     root            28 Mar  7 05:13 Dockerfile
lrwxrwxrwx    1 root     root             9 Mar  7 05:13 symlink -> ./missing

Kaniko v0.18.0

Crashes 👇

[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro gcr.io/kaniko-project/executor:debug-v0.18.0 --no-push
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
INFO[0001] Retrieving image manifest alpine             
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine             
INFO[0002] Retrieving image manifest alpine             
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it. 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Resolving paths                              
INFO[0004] COPY . /example                              
error building image: error building stage: failed to execute command: lstat /workspace/missing: no such file or directory

Kaniko v0.17.1

works 👇

[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro gcr.io/kaniko-project/executor:debug-v0.17.1 --no-push
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
INFO[0001] Retrieving image manifest alpine             
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine             
INFO[0002] Retrieving image manifest alpine             
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0004] COPY . /example                              
INFO[0004] Taking snapshot of files...                  
INFO[0004] Skipping push to container registry due to --no-push flag

Case 1: good symlink

Environment

[nils@buks example]$ ls -la
total 4
drwxr-xr-x  2 nils nils 100 Mar  7 06:36 .
drwxrwxrwt 16 root root 360 Mar  7 06:36 ..
-rw-r--r--  1 nils nils  28 Mar  7 06:13 Dockerfile
-rw-r--r--  1 nils nils   0 Mar  7 06:36 missing
lrwxrwxrwx  1 nils nils   9 Mar  7 06:13 symlink -> ./missing

[nils@buks example]$ cat Dockerfile 
FROM alpine
COPY . /example

docker build

Doesn't crash and doesn't resolve the symlink

[nils@buks example]$ sudo docker build --no-cache -t testimage .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM alpine
 ---> e7d92cdc71fe
Step 2/2 : COPY . /example
 ---> 89d33df5422b
Successfully built 89d33df5422b
Successfully tagged testimage:latest

[nils@buks example]$ sudo docker run --rm -it testimage ls -la /example
total 8
drwxr-xr-x    1 root     root            48 Mar  7 05:36 .
drwxr-xr-x    1 root     root           148 Mar  7 05:36 ..
-rw-r--r--    1 root     root            28 Mar  7 05:13 Dockerfile
-rw-r--r--    1 root     root             0 Mar  7 05:36 missing
lrwxrwxrwx    1 root     root             9 Mar  7 05:13 symlink -> ./missing

kaniko v0.18.0

Doesn't crash but resolves the symlink:

[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug-v0.18.0 /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
INFO[0001] Retrieving image manifest alpine             
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine             
INFO[0002] Retrieving image manifest alpine             
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] COPY . /example                              
INFO[0003] Resolving paths                              
INFO[0003] Taking snapshot of files...                  
INFO[0003] Skipping push to container registry due to --no-push flag 
total 4
-rw-r--r--    1 1000     1000            28 Mar  7 05:39 Dockerfile
-rw-r--r--    1 1000     1000             0 Mar  7 05:39 missing
-rw-r--r--    1 1000     1000             0 Mar  7 05:39 symlink

kaniko v0.17.1

Doesn't crash and doesn't resolve the symlink:

[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug-v0.17.1 /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
INFO[0001] Retrieving image manifest alpine             
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine             
INFO[0002] Retrieving image manifest alpine             
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] COPY . /example                              
INFO[0003] Taking snapshot of files...                  
INFO[0003] Skipping push to container registry due to --no-push flag 
total 8
-rw-r--r--    1 1000     1000            28 Mar  7 05:39 Dockerfile
-rw-r--r--    1 1000     1000             0 Mar  7 05:39 missing
lrwxrwxrwx    1 root     root             9 Mar  7 05:39 symlink -> ./missing

Let me know if I can provide any further information that may help you.

[tejal29]

Are you unable to reproduce the two repro I gave in the original post?

yes i am. The reason why we made that change was to mimic what docker does and picked up the regression. I will work on fixing it and fixing the tests.

[gilbsgilbs]

Thanks. Sorry for the misunderstanding. Given that you removed the regression label and since your response sounded like “not a bug” to me, I wasn't sure you were actually able to reproduce 😄 .

[tejal29]

No worries at all. I fixed up my PR to revert back to old behavior.
Can you try on your end to see if that fixes your issue?

gcr.io/kaniko-project/executor:debug
gcr.io/kaniko-project/executor:debug-edge

[gilbsgilbs]

Thank you. I can confirm this issue is fixed on debug-edge.