[ECR] [request]: support cache manifest

[lifeofguenter]

Would be great if ECR could support cache-manifest (see: https://medium.com/titansoft-engineering/docker-build-cache-sharing-on-multi-hosts-with-buildkit-and-buildx-eb8f7005918e)

NOTE FROM AWS: We shipped this on BuildKit 0.12, see here for details - https://aws.amazon.com/blogs/containers/announcing-remote-cache-support-in-amazon-ecr-for-buildkit-clients/. We are keeping this issue open for the time being to allow the community to discuss and gather further feedback

[TBBle]

BuildKit 0.8 will default to using an OCI media type for its caches (see moby/buildkit#1746) which I assume should make this work, but I haven't tested it myself.

[aleks-fofanov]

It still doesn't work with recently released buildkit 0.8.0
It can write the layers and config, but it is unable to upload manifest to ECR:

=> ERROR exporting cache                                                                                                     5.4s
 => => preparing build cache for export                                                                                       0.2s
 => => writing layer sha256:0d48cc65d93fe2ee9877959ff98ebc98b95fe4b2fc467ff50f27103c1c5d6973                                  0.3s
 => => writing layer sha256:2ade286d53f2e045413601ca0e3790de3792ea34abd3d025cd2cd9c3cb5231de                                  0.3s
 => => writing layer sha256:64befcf53942ba04c144cde468548885d497e238001e965e983e39eb947860c2                                  0.3s
 => => writing layer sha256:7415f0cbea8739c1bf353568b16ac74a9cfbc0b36327602e3a025abf919a38a6                                  0.3s
 => => writing layer sha256:76a1f73c618c30eb1b1d90cf043fe3f855a1cce922d1fb47458defd3dbe1c783                                  0.3s
 => => writing layer sha256:8674739c0ada3e834b816667d26dd185aa5ea089f33701f11a05b7be03f43026                                  0.3s
 => => writing layer sha256:9dc80bcd2805b2a441bd69bc9468df2e81994239e34879567bed7bdef6cb605d                                  0.3s
 => => writing layer sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08                                  0.3s
 => => writing layer sha256:ce4e6de84945ab498f65d16920c9b801dfea3792871e44f89e6438e232a690b3                                  0.3s
 => => writing layer sha256:d46583c5d4c69b34cb46866838d68f53a38686dc7f2d1347ae0f252e8eb0ed4c                                  0.2s
 => => writing config sha256:33c76a0f8a74a06e461926d8a8d1845371c0cf9e86753db2483a4873aede8889                                 2.0s
 => => writing manifest sha256:0f69a7e6626f6a24a0a95ed915613ebdf9459280d4986879480d87e34849aea8                               0.6s
------
 > importing cache manifest from XXXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/test-repo:buildcache:
------
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:0f69a7e6626f6a24a0a95ed915613ebdf9459280d4986879480d87e34849aea8": unexpected status: 400 Bad Request

[errm]

I am seeing the same error on buildkit 0.8.0

even when setting oci-mediatypes explicitly to true: --export-cache type=registry,ref=${REPO}:buildcache,oci-mediatypes=true

 => ERROR exporting cache                                                                                                                                                                                                                                                                                                                                                                                                                                                                1.4s
 => => preparing build cache for export                                                                                                                                                                                                                                                                                                                                                                                                                                                  0.0s
 => => writing layer sha256:757d39990544d20fbebf7a88e29a5dd2bb6a4fdb116d67df9fe8056843da794d                                                                                                                                                                                                                                                                                                                                                                                             0.1s
 => => writing layer sha256:7597eaba0060104f2bd4f3c46f0050fcf6df83066870767af41c2d7696bb33b2                                                                                                                                                                                                                                                                                                                                                                                             0.1s
 => => writing config sha256:0e308fd4eee4cae672eee133cbd77ef7c197fa5d587110b59350a99b289f7000                                                                                                                                                                                                                                                                                                                                                                                            0.8s
 => => writing manifest sha256:8eb142b16e0ec25db4517f2aecff795cca2b1adbe07c32f5c571efc5c808cbcd                                                                                                                                                                                                                                                                                                                                                                                          0.3s
------
 > importing cache manifest from xxx.dkr.ecr.us-east-1.amazonaws.com/errm/test:buildcache:
------
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:8eb142b16e0ec25db4517f2aecff795cca2b1adbe07c32f5c571efc5c808cbcd": unexpected status: 400 Bad Request

Deamon logs:

time="2020-12-09T13:42:48Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2020-12-09T13:44:09Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
time="2020-12-09T13:44:10Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:8eb142b16e0ec25db4517f2aecff795cca2b1adbe07c32f5c571efc5c808cbcd\": unexpected status: 400 Bad Request\n"

[AlexLast]

Also seeing this for private repos, although it doesn't seem to be an issue with public ECR repos..

[n1ru4l]

Is there a timeframe for this feature request available? This could help to tremendously speed up CI builds.

[jellevanhees]

We have been experimenting with this buildkit feature for some time now and it works wonders.
currently, we are still dependant upon dockehub so having this functionality in private ecr would greatly benefit our ci/cd workflow

[davidfm]

Any indication as to if/when this will ever be available? Using buildkit would really improve our CI build times

[devopsmash]

One year passed and still nothing 😔

[pieterza]

We'd really like to see support of this with ECR private repos 🙏

As of today, it still does not work:

error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:75f32e1bb4df7c6333dc352ea3ea9d04d1e04e4a14ba79b59daa019074166519": unexpected status: 400 Bad Request

[hf]

Yes please!