🐞 Dagger engine container is unable to refresh authorization for Google Artifact Registry

[davidwallacejackson]

What is the issue?

I was working on a script with the Go SDK, running it repeatedly to test pushes to a registry, when I suddenly got this error:

input:1: container.withDirectory.publish failed to solve: failed to push [PATH REDACTED]: unexpected status from HEAD request to https://[PATH REDACTED]: 401 Unauthorized
       
       Please visit https://dagger.io/help#go for troubleshooting guidance.

The time was just after midnight, so I figured some automated process had logged me out or revoked my permissions. I tried a push using the docker CLI and found that I was still able to push. On a hunch, I then deleted the docker container for the Dagger engine and re-ran -- at which point my script was able to push again.

Log output

No response

Steps to reproduce

Log into Google Artifact Registry with gcloud, e.g.

gcloud auth login
gcloud auth configure-docker us-central1-docker.pkg.dev

Run a dagger script that Publish()s a container to a repository that you have write access to. Then wait until after midnight (not sure if it's local midnight or PST), and run the script again.

SDK version

Go SDK 0.4.6

OS version

macOS 12.3.1 (ARM)

[jpadams]

@sipsma Looks like we're on
github.com/moby/buildkit v0.11.0-rc3.0.20230222163458-34a576c411ea perhaps?

Seems that v0.11.2 fixed issues related to "pusher" that seem related:
https://github.com/moby/buildkit/releases/tag/v0.11.2
moby/buildkit#3531
containerd/containerd#7990

containerd pusher issues with GCR previously:
containerd/containerd#4622 (comment)

[sipsma]

@jpadams v0.11.0-rc3.0.20230222163458-34a576c411ea is what the go.mod entry ends up being named after a go mod tidy, but the go toolchain is extremely misleading. We are on commit 34a576c which is actually way ahead of v0.11.0-rc3. I have no idea why go chooses that as the name in go.mod, but there's nothing we can do about it that I'm aware of.

Anyways, we have been on containerd-1.7.0-beta4 since last release:

dagger/go.mod

Line 17 in ac2a8ad

github.com/containerd/containerd v1.7.0-beta.4

And that tag appears to have the fix you linked to in place: containerd/containerd@c873647

[d3rp3tt3]

@sipsma, does ^^ mean that the issue should be fixed and that we need to validate by running the scenario?

[sipsma]

@d3rp3tt3 - David said he's on Go SDK 0.4.6, I just checked and that should have already had the fix in place, so I suspect that commit did not end up fixing the issue for him. I think it's worth it for us to see if we can repro it on our latest commits and if still present fix the problem.

[mircubed]

Assigning to @grouville to work on reproducing.