Kaniko saves a corrupted /bin/ directory in image

[golpa]

Actual behavior
v0.17.0 corrupts /bin:

$ docker run --rm -ti --entrypoint "" logstash:7.6.0 ls -l /bin/
ls: cannot access /bin/rpm: No such file or directory
total 0
c????????? ? ? ? ?            ? rpm

works correctly with v0.16.0 (and docker):

$ docker run --rm -ti --entrypoint "" logstash:7.6.0 ls -l /bin/
total 40856
-rwxr-xr-x 1 root root   41488 Aug 20 06:25 [
-rwxr-xr-x 1 root root    1661 Nov 20  2015 abs2rel
-rwxr-xr-x 1 root root   29104 Feb  4 16:35 addr2line
-rwxr-xr-x 1 root root      29 Aug  8  2019 alias
... snip ...

Expected behavior
A functioning /bin just like with v0.16.0

To Reproduce
Steps to reproduce the behavior:

1. /kaniko/executor --context /path/to/Dockerfile/ --destination logstash:7.6.0
2. docker run --rm -ti --entrypoint "" logstash:7.6.0 ls -l /bin/

Additional Information

• Dockerfile:

  FROM logstash:7.6.0
  RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-relp
  

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:debug
  025bd79d3e06

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[tanguydelignieresaccenture]

Repro with #1039 context too.

$ docker run --rm -ti tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1 bash
bash: /usr/libexec/grepconf.sh: /bin/sh: bad interpreter: No such file or directory
[root@6e9bfc7e1f67 /]# ls -lh /bin
ls: cannot access /bin/rpm: No such file or directory
total 0
c????????? ? ? ? ?            ? rpm
[root@6e9bfc7e1f67 /]#

But not fixed by @cvgw try to fix #1039 with tag debug-1039-fix-test

$ docker run --rm -ti tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test bash
bash: /usr/libexec/grepconf.sh: /bin/sh: bad interpreter: No such file or directory
[root@4236156e5144 /]# ls -lh /bin
ls: cannot access /bin/rpm: No such file or directory
total 0
c????????? ? ? ? ?            ? rpm
[root@4236156e5144 /]#

[tanguydelignieresaccenture]

Of course, no problem with debug-v0.16.0 tag.
Note that this issue is related to symlink once again.

$ docker run --rm -ti tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0 bash
[root@90a11173e978 /]# ls -lh /bin
lrwxrwxrwx 1 root root 7 Nov 19 12:53 /bin -> usr/bin
[root@90a11173e978 /]#

[tejal29]

@tanguydelignieresaccenture Thank you for opening this issue. Sorry about this. While the team is working on this, please use gcr.io/kaniko-project/debug-v0.16.0

[tanguydelignieresaccenture]

@tejal29 , I think you wanted to mention @golpa as the opener !

Anyway, I'm in a situation where I face the #1036, #1039 then #1058 with debug-v0.17.1 but needs #971 in it because of #915 in <= debug-v0.16.0.
So until overall symlink/multistages issues known actually are fixed, I'm afraid to be in a dead end.

[cvgw]

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

[tanguydelignieresaccenture]

I've re-run the context of the #1039 with the debug-a1af057f997316bfb1c4d2d82719d78481a02a79 tag, build correctly the image, pushed it, pulled it and confirm it fixes the context of this issue too.

$ docker run --rm -ti index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d7
8481a02a79 bash
Unable to find image 'tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79' locally
debug-a1af057f997316bfb1c4d2d82719d78481a02a79: Pulling from tanguydelignieres/kaniko_bugs_20200207-001_build1
d327c1598329: Already exists
48ed3bfd8226: Already exists
601e0925f3b7: Pull complete
Digest: sha256:8f6c13ff4a8013fa21c6e9595a4438e5abc8ea60ffce4c4b871acac1fa974edb
Status: Downloaded newer image for tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
[root@23af0af0004a /]# ls -lh /bin
lrwxrwxrwx 1 root root 7 Nov 19 12:53 /bin -> usr/bin
[root@23af0af0004a /]#

[michaelhixson]

This is most likely the same issues as #1039, can you try tags a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79

Both of these fix the issue for me. I'm not doing exactly the same test as the OP, but I was seeing the same symptoms (same output for ls /bin) in my own builds with v0.17.1.

[tejal29]

I verified this on the latest master gcr.io/tejal-test/executor:debug

tejaldesai@@kaniko (master)$ docker run -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test --tarPath=image.tar
INFO[0000] Resolved base name logstash:7.6.0 to logstash:7.6.0 
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name logstash:7.6.0 to logstash:7.6.0 
INFO[0000] Retrieving image manifest logstash:7.6.0     
INFO[0001] Retrieving image manifest logstash:7.6.0     
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest logstash:7.6.0     
INFO[0002] Retrieving image manifest logstash:7.6.0     
INFO[0003] Unpacking rootfs as cmd RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-relp requires it. 
INFO[0016] Taking snapshot of full filesystem...        
INFO[0016] Resolving paths                              
INFO[0022] RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-relp 
INFO[0022] cmd: /bin/sh                                 
INFO[0022] args: [-c /usr/share/logstash/bin/logstash-plugin install logstash-input-relp] 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Validating logstash-input-relp
Installing logstash-input-relp
Installation successful
INFO[0040] Taking snapshot of full filesystem...        
INFO[0040] Resolving paths                              

The destination image inow lists the /bin/

ejaldesai@@kaniko (master)$ docker run --rm -ti --entrypoint "" gcr.io/tejal-test/test_1078  ls -l /bin/
Unable to find image 'gcr.io/tejal-test/test_1078:latest' locally
latest: Pulling from tejal-test/test_1078
ab5ef0e58194: Pull complete 
..
Digest: sha256:f7ae07fb184453374983eecf2d8cdf0cdcf6525070000ba0f2bc5d9dce24c7c0
Status: Downloaded newer image for gcr.io/tejal-test/test_1078:latest
total 40856
-rwxr-xr-x 1 root root    1661 Nov 20  2015 abs2rel
-rwxr-xr-x 1 root root   29104 Feb  4 16:35 addr2line
-rwxr-xr-x 1 root root      29 Aug  8  2019 alias
-rwxr-xr-x 1 root root   62688 Feb  4 16:35 ar
-rwxr-xr-x 1 root root   33080 Aug 20  2019 arch
-rwxr-xr-x 1 root root  386416 Feb  4 16:35 as
-rwxr-xr-x 1 root root   28888 Aug  8  2019 aserver
lrwxrwxrwx 1 root root       4 Oct  1 01:15 awk -> gawk
-rwxr-xr-x 1 root root   37360 Aug 20  2019 base64
-rwxr-xr-x 1 root root   29032 Aug 20  2019 basename
-rwxr-xr-x 1 root root  964600 Aug  8  2019 bash
lrwxrwxrwx 1 root root      10 Oct  1 01:15 bashbug -> bashbug-64
-rwxr-xr-x 1 root root    6964 Aug  8  2019 bashbug-64
-rwxr-xr-x 1 root root   45248 Oct 30  2018 bdftopcf

...