fix: Restart buildkit after containerd when provisioning

[pendo324]

Issue #, if available: Fixes #412

Description of changes:

• Previously, only containerd was restarted after configuring it to use the data on the persistent disk. This changes the UUID of the server worker. BuildKit also needs to be restarted to use the proper UUID. See issue for why this is important.

Testing done:

• Local testing

• I've reviewed the guidance in CONTRIBUTING.md

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[pendo324]

Seems like all of the errors on the tests are similar to this:

Finch Container Development E2E Tests save an image when the images exist should save multiple images with --output option
/Users/ec2-user/go/pkg/mod/github.com/runfinch/common-tests@v0.7.0/tests/save.go:73
  7ad7d303eef13a1d2f7cea1b13e6f5ad271e72836ed8b7104e705a8d60d74290
  No containers to be removed
  9977826e0d1d
  No images to be removed
  889f8d4032e4d9758db5fde95109c1028b52b51739f15b112690c6a150abf9a4
  bridge
  host
  none
  No networks to be removed
  2023-06-27 16:30:21.873538484 +0000 UTC finch /images/create {"name":"localhost:58043/alpine:latest"}
  cdfb4bbd2199
  2023-06-27 16:30:22.605963755 +0000 UTC finch /images/create {"name":"public.ecr.aws/docker/library/alpine:3.13"}
  469b6e04ee18
  time="2023-06-27T16:30:22Z" level=fatal msg="failed to get reader: content digest sha256:25f523f0e93b2b5fa676c15d91b90f08ee4de7a160874e6c52ea452929d5a7cc: not found"
  time="2023-06-27T16:30:22Z" level=fatal msg="exit status 1"
  [FAILED] in [It] - /Users/ec2-user/go/pkg/mod/github.com/runfinch/common-tests@v0.7.0/command/command.go:122 @ 06/27/23 16:30:22.866

Wondering if its because of the way the public.ecr.aws/docker/library/alpine:3.13 repo is cached in our local registry. Taking a look.

[austinvazquez]

Seems like all of the errors on the tests are similar to this:

Finch Container Development E2E Tests save an image when the images exist should save multiple images with --output option
/Users/ec2-user/go/pkg/mod/github.com/runfinch/common-tests@v0.7.0/tests/save.go:73
  7ad7d303eef13a1d2f7cea1b13e6f5ad271e72836ed8b7104e705a8d60d74290
  No containers to be removed
  9977826e0d1d
  No images to be removed
  889f8d4032e4d9758db5fde95109c1028b52b51739f15b112690c6a150abf9a4
  bridge
  host
  none
  No networks to be removed
  2023-06-27 16:30:21.873538484 +0000 UTC finch /images/create {"name":"localhost:58043/alpine:latest"}
  cdfb4bbd2199
  2023-06-27 16:30:22.605963755 +0000 UTC finch /images/create {"name":"public.ecr.aws/docker/library/alpine:3.13"}
  469b6e04ee18
  time="2023-06-27T16:30:22Z" level=fatal msg="failed to get reader: content digest sha256:25f523f0e93b2b5fa676c15d91b90f08ee4de7a160874e6c52ea452929d5a7cc: not found"
  time="2023-06-27T16:30:22Z" level=fatal msg="exit status 1"
  [FAILED] in [It] - /Users/ec2-user/go/pkg/mod/github.com/runfinch/common-tests@v0.7.0/command/command.go:122 @ 06/27/23 16:30:22.866

Wondering if its because of the way the public.ecr.aws/docker/library/alpine:3.13 repo is cached in our local registry. Taking a look.

Been seeing some of these failures across a few different PRs. Potentially not related to your changes.

[pendo324]

Been seeing some of these failures across a few different PRs. Potentially not related to your changes.

I reran the CI on #455, and it didn't hit these errors. I think its something related to this one

[pendo324]

The tests are failing here on this PR and on the finch-core repo. The similarity? Both are now running with synced containerd/buildkit worker UUIDs. In other words, the "correct" behavior appears to be that these tests fail. Looking into when/why this started happening.

[ginglis13]

I've pulled your changes down locally, verified that the UUIDs are matching, but can't reproduce the e2e test failures locally.

These tests have been failing on finch-core since e2e tests were added. The difference between finch and finch-core e2e tests is that finch-core e2e 1/ runs a slightly different Fedora VM and 2/ runs tests with nerdctl as the test subject rather than finch.

Both are now running with synced containerd/buildkit worker UUIDs

True, but for some reason the Fedora VM used in finch-core e2e tests doesn't require this buildkit systemd service ordering change to enforce that UUID match... see https://github.com/runfinch/finch-core/blob/eaeaa96443a633d49d51143373650a93ab83a467/e2e/fedora.yaml#L148-L150

Here's my results using https://github.com/runfinch/finch-core/blob/main/e2e/fedora.yaml:

[giinglis@lima-finch finch-core]$ uname -a
Linux lima-finch 6.1.18-200.fc37.aarch64 #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:03:54 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
[giinglis@lima-finch finch-core]$ sudo ctr info
{
    "server": {
        "uuid": "226df813-f172-43e2-ba4f-97d015bdca4d",
        "pid": 1794,
        "pidns": 4026531836
    }
}
[giinglis@lima-finch finch-core]$ sudo buildctl debug workers --format "{{json .}}" | jq -r '. | .[].labels'
{
  "org.mobyproject.buildkit.worker.containerd.namespace": "default",
  "org.mobyproject.buildkit.worker.containerd.uuid": "226df813-f172-43e2-ba4f-97d015bdca4d",
  "org.mobyproject.buildkit.worker.executor": "containerd",
  "org.mobyproject.buildkit.worker.hostname": "lima-finch",
  "org.mobyproject.buildkit.worker.network": "host",
  "org.mobyproject.buildkit.worker.selinux.enabled": "false",
  "org.mobyproject.buildkit.worker.snapshotter": "overlayfs"
}

The changes in this PR do what we expect, which is sync the UUIDs for buildkit and containerd. finch save is where we are seeing CI failures. The high level behavior of finch save -o file.tar image1 image2 should work - by that mean both nerdctl and docker have this feature. Still investigating why we're seeing these test failures in save once UUIDs are synced...

[ginglis13]

tl;dr for CI issues here is garbage collection labels not being set properly. I got to the bottom of it and opened an issue on nerdctl containerd/nerdctl#2372, but in reality this is a bug with buildkitv0.11.x . The bug has been patched in moby/buildkit#3972 which was included in buildkitv0.12.0 which was released just yesterday: https://github.com/moby/buildkit/releases/tag/v0.12.0.

Opened up containerd/nerdctl#2374 to upgrade nerdctl's buildkit to v0.12.0. Let's keep this PR open until we have both a new nerdctl release with the upgrade and a new Lima release pointing to the new nerdctl release.

[ginglis13]

Should be able to rebase and re-run CI now #521 has been merged with dependency upgrades

[pendo324]

Should be able to rebase and re-run CI now #521 has been merged with dependency upgrades

Just rebased 🤞

[pendo324]

Benchmarking steps:

1. git clone https://github.com/apache/couchdb-docker.git (just an example image with a lot of / large layers)
2. cd couchdb-docker/3.3.2
3. clear all layers / images finch system prune -a -f
4. regular build with Finch 0.7.0 time finch build -t couchdb .
   1. the time is 49s on my PC
5. clear all layers / images again finch system prune -a -f
6. do build with options that simulate the fix time finch build -t couchdb --output 1. type=image,unpack=true .
   1. time is 37s on my PC
7. 49/37 = 1.33, so 33% increase in performance