error building image: error building stage GCC

[dllozhkin]

I try build gcc image with kaniko
and get error

INFO[0003] Unpacking rootfs as cmd RUN cmake -DCMAKE_BUILD_TYPE=Release requires it. 
error building image: error building stage: removing whiteout usr/bin/cmake/.wh..wh..opq: lstat /usr/bin/cmake/.wh..opq: operation not permitted

kaniko image : gcr.io/kaniko-project/executor:debug

stack:

k8s, gitlab, gitlab-runner(into k8s)

[donmccasland]

Are you using cacheing? Might be related to #742

[tejal29]

@denitol Can you provide more information on your docker file?

[z0mb1ek]

same problem

INFO[0030] RUN apt-get update && apt-get -y install wget build-essential git libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev
INFO[0030] Found cached layer, extracting to filesystem
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

[z0mb1ek]

my docker image

FROM nginx:1.17.8

COPY devops/nginx/ssl/certbundle.pem /etc/ssl/certbundle.pem

COPY devops/nginx/ssl/1.key /etc/ssl/1.key

COPY devops/nginx/nginx.conf /etc/nginx/nginx.conf

COPY devops/nginx/default.conf /etc/nginx/conf.d/default.conf

RUN apt-get update && apt-get -y install wget build-essential git libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev

RUN git clone https://github.com/kaltura/nginx-vod-module.git

RUN wget 'https://nginx.org/download/nginx-1.17.8.tar.gz' && tar xvzf nginx-1.17.8.tar.gz

WORKDIR nginx-1.17.8

RUN ./configure --add-dynamic-module=/nginx-vod-module \
	--with-file-aio \
	--with-threads \
	--with-cc-opt="-O3"

RUN ls -la /nginx-vod-module

[ribx]

We are also on gitlab/gitlab-runner + k8s.

I can reproduce this sometimes always (if cache is present) with this simple Dockerfile:

FROM nginx:1.17

RUN echo "server {listen 80; root /html/; \
          location /api/ {return 500;} \
          location /static/ {} \
          location / {try_files \$uri /index.html;}}" > /etc/nginx/conf.d/default.conf

COPY ./build /html

with this command:

/kaniko/executor
      --context "./"
      --dockerfile Dockerfile
      --destination "$REGISTRY/$GROUP/$PROJECT/frontend:$TAG" # maybe this is important?
      --cache=true

inside the image "gcr.io/kaniko-project/executor:debug" (sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3)
and also "gcr.io/kaniko-project/executor:debug-v0.16.0"

When the cache is cleared, the build works fine.

[Flauschbaellchen]

We're running into the same bug which is reproducable on every run as long as the cache exists.
Currently, we disabled the cache to be able to build our images again.

Can I help, e.g. do you need another Dockerfile example or debug output?
We're using gitlab-runner 12.7.1 on Debian 9 with Docker version 17.05.0-ce (build 89658be) and gcr.io/kaniko-project/executor:debug (025bd79d3e06)

[cvgw]

We're running into the same bug which is reproducable on every run as long as the cache exists.
Currently, we disabled the cache to be able to build our images again.

Can I help, e.g. do you need another Dockerfile example or debug output?
We're using gitlab-runner 12.7.1 on Debian 9 with Docker version 17.05.0-ce (build 89658be) and gcr.io/kaniko-project/executor:debug (025bd79d3e06)

You probably need to clear your cache so that it doesn't keep re-using the bad cached layer.

[Flauschbaellchen]

@cvgw Well, that's not a solution as within the next run a newly created cache would be saved which is also faulty.

[cvgw]

@cvgw Well, that's not a solution as within the next run a newly created cache would be saved which is also faulty.

Apologies if I'm misunderstanding you; my suggestion was to clear the cache when going from a known bad version of kaniko to a good version

E.G

image was previously built with kaniko v0.17.0 and cached
switch to kaniko v0.17.1
clear cache
build image with v0.17.1
at that point the cache is good to use for v0.17.1