unexpected status: 401 Unauthorized

[blabla1337]

Hi All,

So I saw this was posted before but without a proper solution or breakdown why this happens. #139 (comment)
I'm also having this issue from time to time and its really unclear what it triggers.

First I do the docker login and that goes well, after that I start the buildx for the 2 images but I still get the 401 error in the end as you can see. I'm using it on Mac Docker Desktop 2.3.2.0 Edge. if you need additional information please let me know.

docker login
Authenticating with existing credentials...
Login Succeeded

docker buildx build -f Docker/alpine-cloud/api/Dockerfile  --platform linux/amd64,linux/arm/v7 -t blabla1337/skf-api:dev --push .
[+] Building 1483.8s (38/38) FINISHED                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                    0.1s
 => => transferring dockerfile: 32B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.1s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/python:3.7                                                                                                               9.2s
 => [linux/arm/v7 internal] load metadata for docker.io/library/python:3.7                                                                                                             11.5s
 => [internal] load build context                                                                                                                                                      89.0s
 => => transferring context: 967.06MB                                                                                                                                                  81.4s
 => [linux/arm/v7 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                              0.0s
 => => resolve docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                                                     0.0s
 => [linux/amd64 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                               0.0s
 => CACHED [linux/amd64 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython          0.0s
 => CACHED [linux/amd64 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                    0.0s
 => [linux/amd64 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                                 27.8s
 => CACHED [linux/arm/v7 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython         0.0s
 => CACHED [linux/arm/v7 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                   0.0s
 => CACHED [linux/arm/v7 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                          0.0s
 => CACHED [linux/arm/v7 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                      0.0s
 => CACHED [linux/arm/v7 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                       0.0s
 => CACHED [linux/arm/v7 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                    0.0s
 => CACHED [linux/arm/v7 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                         0.0s
 => CACHED [linux/arm/v7 stage-1  9/16] RUN mkdir .kube                                                                                                                                 0.0s
 => CACHED [linux/arm/v7 stage-1 10/16] RUN mkdir .config                                                                                                                               0.0s
 => CACHED [linux/arm/v7 stage-1 11/16] RUN mkdir .config/pip                                                                                                                           0.0s
 => CACHED [linux/arm/v7 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                     0.0s
 => CACHED [linux/arm/v7 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                              0.0s
 => CACHED [linux/arm/v7 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                  0.0s
 => CACHED [linux/arm/v7 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi           0.0s
 => [linux/arm/v7 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                         325.9s
 => [linux/amd64 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                              1.2s
 => [linux/amd64 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                               0.2s
 => [linux/amd64 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                            0.2s
 => [linux/amd64 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                                 0.0s
 => [linux/amd64 stage-1  9/16] RUN mkdir .kube                                                                                                                                         0.2s
 => [linux/amd64 stage-1 10/16] RUN mkdir .config                                                                                                                                       0.2s
 => [linux/amd64 stage-1 11/16] RUN mkdir .config/pip                                                                                                                                   0.2s
 => [linux/amd64 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                             0.2s
 => [linux/amd64 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                                      0.2s
 => [linux/amd64 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                         22.0s
 => [linux/amd64 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi                  16.0s
 => [linux/amd64 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                           65.1s
 => ERROR exporting to image                                                                                                                                                         1056.5s
 => => exporting layers                                                                                                                                                               433.2s
 => => exporting manifest sha256:651a8c2a92c21b35939f84379ac3ce6a59a102ad12c1ffd77b87ae5943c10440                                                                                       0.0s
 => => exporting config sha256:75bc89b31a6e8e5bcb67ef00ed3afbec60af799da0412d17c13852eb956d4e5f                                                                                         0.0s
 => => exporting manifest sha256:92aad5879eb60c8e4a776ecca27906068d0ae6264b69afefad152e8b566756ad                                                                                       0.0s
 => => exporting config sha256:24273c4da9572c159210060f87c5889f2f1508feeab36708a27d55fae989945e                                                                                         0.0s
 => => exporting manifest list sha256:cbbd8a5daf155203ad74613fafddeb7dd1187b7d35da33682c150247c6871c27                                                                                  0.0s
 => => pushing layers                                                                                                                                                                 621.9s
 => => pushing manifest for docker.io/blabla1337/skf-api:dev                                                                                                                            1.2s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:651a8c2a92c21b35939f84379ac3ce6a59a102ad12c1ffd77b87ae5943c10440": unexpected status: 401 Unauthorized

[blabla1337]

So I tried it a couple more times again and now its working:

docker buildx build -f Docker/alpine-cloud/api/Dockerfile  --platform linux/amd64,linux/arm/v7 -t blabla1337/skf-api:dev --push .
[+] Building 1148.9s (38/38) FINISHED                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                    0.0s
 => => transferring dockerfile: 32B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.1s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/python:3.7                                                                                                               3.9s
 => [linux/arm/v7 internal] load metadata for docker.io/library/python:3.7                                                                                                              6.2s
 => [internal] load build context                                                                                                                                                     132.2s
 => => transferring context: 967.06MB                                                                                                                                                 122.9s
 => [linux/amd64 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                               0.0s
 => => resolve docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                                                     0.0s
 => [linux/arm/v7 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                              0.0s
 => => resolve docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                                                     0.0s
 => CACHED [linux/amd64 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython          0.0s
 => CACHED [linux/amd64 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                    0.0s
 => CACHED [linux/amd64 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                           0.0s
 => CACHED [linux/amd64 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                       0.0s
 => CACHED [linux/amd64 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                        0.0s
 => CACHED [linux/amd64 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                     0.0s
 => CACHED [linux/amd64 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                          0.0s
 => CACHED [linux/amd64 stage-1  9/16] RUN mkdir .kube                                                                                                                                  0.0s
 => CACHED [linux/amd64 stage-1 10/16] RUN mkdir .config                                                                                                                                0.0s
 => CACHED [linux/amd64 stage-1 11/16] RUN mkdir .config/pip                                                                                                                            0.0s
 => CACHED [linux/amd64 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                      0.0s
 => CACHED [linux/arm/v7 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython         0.0s
 => CACHED [linux/arm/v7 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                   0.0s
 => CACHED [linux/arm/v7 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                          0.0s
 => CACHED [linux/arm/v7 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                      0.0s
 => CACHED [linux/arm/v7 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                       0.0s
 => CACHED [linux/arm/v7 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                    0.0s
 => CACHED [linux/arm/v7 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                         0.0s
 => CACHED [linux/arm/v7 stage-1  9/16] RUN mkdir .kube                                                                                                                                 0.0s
 => CACHED [linux/arm/v7 stage-1 10/16] RUN mkdir .config                                                                                                                               0.0s
 => CACHED [linux/arm/v7 stage-1 11/16] RUN mkdir .config/pip                                                                                                                           0.0s
 => CACHED [linux/arm/v7 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                     0.0s
 => CACHED [linux/arm/v7 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                              0.0s
 => CACHED [linux/arm/v7 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                  0.0s
 => CACHED [linux/arm/v7 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi           0.0s
 => [linux/arm/v7 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                         371.1s
 => [linux/amd64 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                                      3.7s
 => [linux/amd64 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                         23.3s
 => [linux/amd64 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi                  12.3s
 => [linux/amd64 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                           65.4s
 => exporting to image                                                                                                                                                                638.5s
 => => exporting layers                                                                                                                                                               302.5s
 => => exporting manifest sha256:dd1ff1af15e0d11d75c276aa648a56a7debc16bb6242489cb95a9172f4524e7a                                                                                       0.0s
 => => exporting config sha256:50adb73b1fe9b5e04bf4dd764188f12fabe1a6f693d793a2b06d906edd7dc016                                                                                         0.0s
 => => exporting manifest sha256:0395b10a6c7045398acb39c4e94e9dc5c3870bf96f001772d0e72b403a523fdb                                                                                       0.0s
 => => exporting config sha256:15bad4ce8ffeda2ded7e88b7009209fad96589adbde0111a01c35f324601c3f8                                                                                         0.0s
 => => exporting manifest list sha256:8061b5764cfeeeebf0fc5acb8f0a3bad1475a018a06ca8c0412fbffe56f77ca1                                                                                  0.0s
 => => pushing layers                                                                                                                                                                 332.2s
 => => pushing manifest for docker.io/blabla1337/skf-api:dev                                                                                                                            3.6s

No idea what the issue can be...

[jc21]

I'm experiencing this in my CI builds, after docker login then a docker buildx and it fails with unauthorized only 1 hour and 1 minute later. How long is this authentication token meant to last? I was sure it would be longer than this.

Here is a full build history if anyone is interested:
https://ci.nginxproxymanager.com/blue/organizations/jenkins/nginx-proxy-manager/detail/master/36/pipeline

EDIT: I should note that retriggering this build works as it uses the cached docker images and pushes faster.

[emwjacobson]

I ran into this problem today creating large images. What I did was use the --cache-to flag when building it, then when I go to push it to dockerhub I log in and use --cache-from.
An example of the workflow would be something like this:

# Build images and cache them to a local cache/ folder, this part takes the longest
docker buildx build --cache-to=type=local,dest=cache,mode=max --platform linux/amd64,linux/arm64 --tag myimagename:latest .
# Login
docker login -u username -p password
# Build again, but use cache from local source
docker buildx build --push --cache-from=type=local,src=cache --platform linux/amd64,linux/arm64 --tag myimagename:latest .

As long as it uploads the image somewhat quickly, it hopefully shouldn't time you out.

Note that this does take a lot more disk space. I made a VM and gave it 64gb of disk space, and with a 7gb docker image crosscompiled to x86 and arm it ran out of disk space while caching all of the layers.

Hope that helps a bit.

EDIT: You could also play around with caching to/from a registry. I mentioned local caching because my internet would be a bottleneck, but it might reduce disk usage if that is a concern, but I havent personally tested it.

[tonistiigi]

Please test with moby/buildkit#1636 docker buildx create --driver-opt image=moby/buildkit:master

[andrewufrank]

congratulation! now I can push. each push is a success, when before it mostly failed (after running for 10 minutes).
thank you very much!
andrew

[tyranron]

I'm hitting this at the moment with Quay.io. GitHub Container Registry works OK.

I'm able to reproduce this locally via CLI. The interesting thing that a regular docker push works OK.

Another interesting thing that the first push to Quay.io (when images not yet existed) worked OK.

[shepmaster]

@tyranron same deal here: pushing to Quay.io from a GitHub Actions workflow on the subsequent run.

[tonistiigi]

I think you guys are hitting containerd/containerd#4622 (comment)