no active session for <id>: context deadline exceeded

[errordeveloper]

I’m seeing buildx error like this:

#25 exporting to image
#25 exporting layers
#25 exporting layers 2.4s done
#25 exporting manifest sha256:bee6a0ade00b7bc2bcaae86deefc8e88e6a274dee097c8ade078e4b3a88de520 done
#25 exporting config sha256:ac3b1fe36ac072dafadc6f2aed84ff305d810a92a3e62f71a820928c832d9ac9
#25 exporting config sha256:ac3b1fe36ac072dafadc6f2aed84ff305d810a92a3e62f71a820928c832d9ac9 0.0s done
#25 exporting manifest sha256:b7812799f180bbded0a8de865efba29dcf351c858f958e506cc13e612657e4b8 0.0s done
#25 exporting config sha256:b7efbce252b10357c495758cfb1299a4c6908153eec71fc8e3097bcca5d3eb8f done
#25 exporting manifest list sha256:cbf719abf8d8c678350283152e91f9f49c1138e0087758eeaa67f6697ae4dfda done
#25 ERROR: no active session for os44n22472jqaz3bk2t3jktq2: context deadline exceeded

Things to note:

• buildx v0.4.2
• build takes 4h50m
• it's meant to push to quay.io
• full build log

[errordeveloper]

My first hunch is that some context is initialised with a short deadline, and that deadline expires soon since the build is taking so long.

The error is very ambiguous, however no active session for is quite unique and is returned from here:

buildx/vendor/github.com/moby/buildkit/session/manager.go

Lines 176 to 178 in fb7b670

	case <-ctx.Done():
	sm.mu.Unlock()
	return nil, errors.Wrapf(ctx.Err(), "no active session for %s", id)

Also, ERROR: is a unique prefix too, perhaps somewhat surprisingly!

buildx/vendor/github.com/moby/buildkit/util/progress/progressui/printer.go

Lines 151 to 158 in fb7b670

	if v.Error != "" {
	if v.logsPartial {
	fmt.Fprintln(p.w, "")
	}
	if strings.HasSuffix(v.Error, context.Canceled.Error()) {
	fmt.Fprintf(p.w, "#%d CANCELED\n", v.index)
	} else {
	fmt.Fprintf(p.w, "#%d ERROR: %s\n", v.index, v.Error)

All I can tell is that this came from here:

buildx/util/progress/printer.go

Lines 47 to 53 in fb7b670

	var c console.Console
	if cons, err := console.ConsoleFromFile(out); err == nil && (mode == "auto" || mode == "tty") {
	c = cons
	}
	// not using shared context to not disrupt display but let is finish reporting errors
	pw.err = progressui.DisplaySolveStatus(ctx, "", c, out, statusCh)
	close(doneCh)

I am not entirely sure where to go from here, this is seems like some kind of generic processing queue thing and this can be coming from just about anywhere in either buildkit or buildx code...

[SphtKr]

Not much to add, but seeing same behavior here today, similar setup, building multiarch across three nodes, amd64 + aarch64 + armv7l ... the arm7l one took close to eight hours and then hit this. Using plain TCP socket context to the two ARM nodes issuing the build on the amd64 node (UNIX socket).

[tonistiigi]

Was that export supposed to end with a push? What version of buildkit?

[errordeveloper]

Was that export supposed to end with a push? What version of buildkit?

@tonistiigi yes, in my case it was.

[errordeveloper]

Using plain TCP socket context to the two ARM nodes issuing the build on the amd64 node (UNIX socket).

In my case everything was built in the same docker instance, with qemu for some of the Arm stages.

[errordeveloper]

@tonistiigi ping!

[tonistiigi]

It looks like maybe the session connection just dropped because your build took almost 5h. You should see that from the daemon logs. If that is the case then maybe we could add logic to redial. Although not really different from build request connection itself dropping or if this happens at the same time session is being used it would still fail.

[errordeveloper]

Would you mind pointing me to where the redial logic would have to be implemented? Also, how can we improve the error message? I'm happy to have a go at fixing this one, just need some pointers. I'm pretty sure it's some sort of a drop.

[tonistiigi]

Session request happens in https://github.com/moby/buildkit/blob/master/client/solve.go#L165 but first you should figure out if it is dropping and if there is any error message of condition that causes it.

[carlonluca]

Same happens to me, very frequently. In some cases it is almost impossible to push images to docker hub. In my case I'm crossbuilding on arm to other arm versions or amd64.

#13 exporting to image
#13 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#13 exporting layers
#13 exporting layers 103.9s done
#13 exporting manifest sha256:eb3e1510baff02d53aa0afd867f1e4d297935fc99ee88d8152bc7cd977ba8108 0.1s done
#13 exporting config sha256:5c767564ca11f26821ce80e1fe80a713b1b0bffae5fff3e47c03cd29f295e013
#13 exporting config sha256:5c767564ca11f26821ce80e1fe80a713b1b0bffae5fff3e47c03cd29f295e013 0.1s done
#13 pushing layers
#13 pushing layers 765.1s done
#13 ERROR: no active session for mxx75rit9yyhguolxzs23e4oq: context deadline exceeded
------
 > exporting to image:
------
error: failed to solve: rpc error: code = DeadlineExceeded desc = no active session for mxx75rit9yyhguolxzs23e4oq: context deadline exceeded

[pgehring]

We are getting the same error while pushing to GitHub Container Registry. We though this is related to the 5GB limit of a single image layer, but we were able to push images with layersizes >5GB.
As we observed, the layers of this example is pushed succesfully according to the logfile.

#9 exporting to image
#9 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#9 exporting layers
#9 exporting layers 989.6s done
#9 exporting manifest sha256:78e6645f9a005842a4243947bd2e6e928f9a45b071637e2a1ba42ad9bf1c1136
#9 exporting manifest sha256:78e6645f9a005842a4243947bd2e6e928f9a45b071637e2a1ba42ad9bf1c1136 0.4s done
#9 exporting config sha256:4782358c0f8cd40619f1244a283965c470e3977ef112276f93a84eb5e5111a27
#9 exporting config sha256:4782358c0f8cd40619f1244a283965c470e3977ef112276f93a84eb5e5111a27 0.4s done
#9 pushing layers
#9 pushing layers 85.5s done
#9 ERROR: no active session for d2mw2899j00mfampt13ooz376: context deadline exceeded
------
 > exporting to image:
------
error: failed to solve: rpc error: code = DeadlineExceeded desc = no active session for d2mw2899j00mfampt13ooz376: context deadline exceeded
Error: buildx call failed with: error: failed to solve: rpc error: code = DeadlineExceeded desc = no active session for d2mw2899j00mfampt13ooz376: context deadline exceeded

The following log is copied from an image with layersizes above 5GB size. As of this log, the pushing of the manifest data fails for the example above.

#9 exporting to image
#9 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#9 pushing layers 461.4s done
#9 pushing manifest for ghcr.io/...
#9 pushing manifest for ghcr.io/... 1.2s done
#9 DONE 805.3s
🛒 Extracting digest...
sha256:56e2cf1b9b5d78b293851ef6e68e23494885974db6e945141677240c458d380c

[errordeveloper]

moby/buildkit#2019 is potentially related with regard to improving how the underlying error handling machinery is implemented.

[deepio]

For anyone else running into this issue as part of their CI pipeline, we've decided to downgrade to legacy builds instead because reliability is more important than speed.

[hfedcba]

This also happens with short build times, in this case less than 10 minues. Often the same build passes with a build time of about 12 minutes. I'd say it fails in about 50 % of the jobs.

#9 DONE 528.6s
#10 exporting to image
#10 exporting layers
#10 exporting layers 10.6s done
#10 exporting manifest sha256:e341e2c0bc3510d62cc7650f3ee0e5006bb679499977832fc8a6b493d5dc8491
#10 exporting manifest sha256:e341e2c0bc3510d62cc7650f3ee0e5006bb679499977832fc8a6b493d5dc8491 0.4s done
#10 exporting config sha256:082f6382fb3047e7182df5bf0360c2cf5b9927b4b4fd8664ad058c75663bc485
#10 exporting config sha256:082f6382fb3047e7182df5bf0360c2cf5b9927b4b4fd8664ad058c75663bc485 0.4s done
#10 ERROR: no active session for 7c6uxna8fa27v8oanfq8zcj3l: context deadline exceeded
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = no active session for 7c6uxna8fa27v8oanfq8zcj3l: context deadline exceeded
Cleaning up file based variables 00:00
ERROR: Job failed: exit status 1