containerd worker: use diff service instead of client-side implementation

[ktock]

Fixes: #2365

This commit fixes containerd worker to use containerd's diff service instead of client-side implementation to avoid the performance regression. Overlayfs differ will be enabled only for OCI worker.

As discussed in #2365, to allow containerd worker using overlayfs differ, we'll need to port the differ implementation to containerd diff serivce.

[ktock]

Containerd's diff service doesn't support custom compresor (containerd/containerd#5735) so zstd support is limited to OCI worker.

https://github.com/moby/buildkit/pull/2366/checks?check_run_id=3623798830#step:7:1046

=== CONT  TestIntegration/TestBuildExportZstd/worker=containerd
    client_test.go:2203: 
        	Error Trace:	client_test.go:2203
        	            				run.go:176
        	Error:      	Received unexpected error:
        	            	unsupported diff media type: application/vnd.oci.image.layer.v1.tar+zstd: not implemented
        	            	github.com/moby/buildkit/util/stack.Enable
        	            		/src/util/stack/stack.go:77
        	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
        	            		/src/util/grpcerrors/grpcerrors.go:188
        	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
        	            		/src/util/grpcerrors/intercept.go:41
        	            	google.golang.org/grpc.(*ClientConn).Invoke
        	            		/src/vendor/google.golang.org/grpc/call.go:35
        	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        	            		/src/api/services/control/control.pb.go:1320
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		/src/client/solve.go:201
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1581
        	            	failed to solve
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		/src/client/solve.go:214
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1581
        	Test:       	TestIntegration/TestBuildExportZstd/worker=containerd

[ktock]

@tonistiigi WDYT about #2366 (comment) ?
Client-side differ is needed to support non-gzip compressions (zstd,esgz,etc) that aren't supported by containerd.
Should we limit the support of such compressions to the OCI worker, or have overlayfs-differ (10-20% slower than containerd differ as discussed in #2365) to enable them for containerd worker as well?

[AkihiroSuda]

Does “fuse-overlayfs” really support overlay diff?

[ktock]

Thank you for pointing this. No, it doesn't because fuse-overlayfs snapshotter doesn't provide overlayfs mounts. Disabled it for fuse-overlayfs snapshotter.

[tonistiigi]

I'm not really sure if these hacks are worth it. If we can fix the buffering issue and performance compared to oci worker is in +10-20% range I'd be ok with just using the client-side implementation always. Some cleanup in this area might be good though.

Currently, we have exceptions for overlay-native, windows and zstd. All of them could be implemented in containerd as well if the performance parity is needed but I don't think it is very important to have it instantly.

In #buildkit while discussing mergeop with @sipsma we've come up with another exception where we want certain refs to store their changes list for upper layer directly in ref metadata. This is so that we can guarantee the diff remains persistent when it is copied to different snapshot chains. If that list exists for a ref it would be used as a source for the differ, so again something that containerd API itself does not support.

FYI @aaronlehmann

[ktock]

Thank you for the comment. At least fixing the buffering issue seems to be needed instantly. I'll open a PR for the fix.

[ktock]

Closing in favor of #2388 and #2390.