GHA cache image build dependencies by apostasie · Pull Request #3583 · containerd/nerdctl

apostasie · 2024-10-21T06:10:57Z

Folllow-on to discussion in #3580

What this does:

enable the use of github actions cache for docker build of the integration image
in the Dockerfile, separate third-party dependency build stage and nerdctl stage
in github workflows, separate out the step that builds these dependencies (read-write to cache) from the steps that prepare the final integration image (read-only from cache)

The result is:

with a cold cache, instead of building everything 9 times (4 rootful, 4 rootless, ipv6) , we only build the dependencies stage 4 times
the dependencies stage is then cached, and subsequent runs on the same branch will not rebuild it (unless the dependencies versions have changed, or the Dockerfile has been altered)
note that we SHOULD also get the cache from the main branch as well, so, once it is merged and the cache is hot, new PRs should benefit from the cache as well on their first run (eg: not build)

The key benefit here is about reducing the network traffic required to produce out test images (hence reducing the opportunity for failure due to third-party server hiccups). It is similar in intent to #3580 .

Incidentally, we will also get a small speed boost for the overall run.
Obviously, GHA cache is not "free": it takes time to retrieve and time to store - so, part of the speed gains from not-building are negated by the cache r/w.

Nevertheless, this looks promising for increased reliability (and reduced transactions with docker hub / debian / ubuntu).

This PR also has a couple of minor changes to the workflow file (reduced timeouts, cosmetic comments, along with bumping the size of the arm64 instance as previously discussed). If preferable, I can split these out.

Further refactoring / changes to the Dockerfile could bring more stuff in the dependencies stage.
This PR has been conservative on that front and staid with the minimal possible changes to the Dockerfile, so that we can decide separately if we want a more in-depth restructuring of it or not.

Finally note that GHA cache is rather limited (10G), and going over the limit will prune prior entries indiscriminately - we might want to keep an eye on that and check that this proposed implementation here stays under the limit to fully benefit from it.

apostasie · 2024-10-22T00:43:31Z

    libbtrfs-dev \
    libseccomp-dev \
    pkg-config
+RUN git config --global advice.detachedHead false


Silence useless / noisy git messages.

apostasie · 2024-10-22T00:43:56Z

 # We do not set CMD to `go test` here, because it requires systemd

-FROM build-base AS build-full
+FROM build-base AS build-deps


This is the part that splits out third-party dependencies from nerdctl.

apostasie · 2024-10-22T01:17:03Z

@AkihiroSuda @djdongjin this should not go in before #3535, but this looks really promising in term of minimizing transactions with Hub / third party services.

LMK your thoughts overall and I will polish it and rebase on top of ^ after it merges

apostasie · 2024-10-29T15:56:40Z

+FROM build-base AS build-dependencies
 ARG TARGETARCH
 ENV GOARCH=${TARGETARCH}
-RUN BINDIR=/out/bin make binaries install


Move down nerdctl, which is the part that changes (almost) all the time.

apostasie · 2024-10-29T15:57:15Z

  git checkout "${IMGCRYPT_VERSION}" && \
  CGO_ENABLED=0 make && DESTDIR=/out make install && \
  echo "- imgcrypt: ${IMGCRYPT_VERSION}" >> /out/share/doc/nerdctl-full/README.md
-ARG ROOTLESSKIT_VERSION


Move down rootless kit, as we support two versions.

apostasie · 2024-10-29T15:57:46Z

  mv /tmp/SHA256SUMS /out/share/doc/nerdctl-full/SHA256SUMS && \
  chown -R 0:0 /out

+FROM build-dependencies AS build-full


Restore nerdctl from above.

apostasie · 2024-10-29T15:58:05Z


  test-integration:
-    timeout-minutes: 60
+    timeout-minutes: 30


Lower timeouts.

apostasie · 2024-10-29T15:58:56Z

          go-version: ${{ env.GO_VERSION }}
          cache: true
          check-latest: true
-      - name: "Print docker info"


Debugging remnants.

apostasie · 2024-10-29T15:59:04Z

-
    steps:
      - uses: actions/checkout@v4.2.2
+        with:


apostasie · 2024-10-29T18:39:18Z

@AkihiroSuda at your convenience.

Would like to get this is in for a few PRs to better evaluate how the cache behaves.

AkihiroSuda · 2024-11-01T08:31:53Z

        run: |
          sudo mkdir -p /etc/docker
-          echo '{"ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "experimental": true, "ip6tables": true}' | sudo tee /etc/docker/daemon.json
+          echo '{"features": {"containerd-snapshotter": true}, "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "experimental": true, "ip6tables": true}' | sudo tee /etc/docker/daemon.json


containerd-snapshotter isn't necessary if you use docker buildx ? (probably with docker buildx create to create a standalone non-moby buildkitd instance)

Interesting.
That would certainly be better.
Let me look into that later today.

Done. Thanks for the tips.

AkihiroSuda · 2024-11-01T13:11:25Z

+          fetch-depth: 1
+      - name: "Expose GitHub Runtime variables for gha"
+        uses: crazy-max/ghaction-github-runtime@v3
+      - name: "Enable containerd to be able to use gha cache"


Not containerd

Right. Thanks.
Removed the separate step altogether with the latest push.

AkihiroSuda

Thanks

apostasie · 2024-11-01T14:08:15Z

@AkihiroSuda failure is likely #3556
Can you poke it?

ktock

LGTM, sorry for slow review.

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie · 2024-11-01T15:49:15Z

LGTM, sorry for slow review.

No problem @ktock.
Thanks!

apostasie · 2024-11-02T21:07:41Z

While working on something else, I am quickly getting 429s from gha cache, and jobs failing / timeouting because of throttled requests.

I am not optimistic about this overall.
GHA cache does not feel like a usable / scalable product - bad API, lilliputian limitations.

Suggesting we give it a couple of weeks and consider reverting this / tweak it / going back to other ideas (eg: proxy caching).

@AkihiroSuda @ktock

apostasie force-pushed the optimize-build-time branch 7 times, most recently from 4fd4147 to ed7df0b Compare October 22, 2024 00:36

apostasie commented Oct 22, 2024

View reviewed changes

apostasie force-pushed the optimize-build-time branch from ed7df0b to 3215ecc Compare October 22, 2024 00:54

apostasie force-pushed the optimize-build-time branch from 3215ecc to d38e4b5 Compare October 29, 2024 15:52

apostasie commented Oct 29, 2024

View reviewed changes

Comment thread .github/workflows/test.yml

steps:

- uses: actions/checkout@v4.2.2

with:

Copy link
Copy Markdown

Contributor Author

apostasie Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Harmonize

apostasie force-pushed the optimize-build-time branch from d38e4b5 to b187986 Compare October 29, 2024 16:19

apostasie changed the title ~~Experimental: GHA cache image build dependencies~~ GHA cache image build dependencies Oct 29, 2024

apostasie marked this pull request as ready for review October 29, 2024 16:47

AkihiroSuda added this to the v2.0.0 milestone Oct 29, 2024

AkihiroSuda requested a review from ktock October 29, 2024 18:47

AkihiroSuda added the area/ci e.g., CI failure label Nov 1, 2024

AkihiroSuda reviewed Nov 1, 2024

View reviewed changes

apostasie marked this pull request as draft November 1, 2024 12:15

apostasie force-pushed the optimize-build-time branch from aa5862c to 5d836d6 Compare November 1, 2024 12:15

apostasie marked this pull request as ready for review November 1, 2024 12:39

apostasie force-pushed the optimize-build-time branch from 5d836d6 to 8dc03b7 Compare November 1, 2024 12:41

AkihiroSuda reviewed Nov 1, 2024

View reviewed changes

apostasie force-pushed the optimize-build-time branch from 8dc03b7 to 3b1d250 Compare November 1, 2024 13:38

AkihiroSuda approved these changes Nov 1, 2024

View reviewed changes

ktock approved these changes Nov 1, 2024

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

Enable GHA cache for image build dependencies

de193b3

Signed-off-by: apostasie <spam_blackhole@farcloser.world>

apostasie force-pushed the optimize-build-time branch from 3b1d250 to de193b3 Compare November 1, 2024 15:46

ktock approved these changes Nov 2, 2024

View reviewed changes

ktock merged commit b00f8cd into containerd:main Nov 2, 2024

apostasie mentioned this pull request Dec 5, 2024

[CI]: github cache is too aggressive on limiting requests (429) #3733

Open

yankay mentioned this pull request Feb 7, 2025

Fix nerdctl-full shasum fails by move the sha256sum to build-full stage #3870

Merged

Conversation

apostasie commented Oct 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apostasie commented Oct 22, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apostasie commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

apostasie commented Nov 1, 2024

Uh oh!

ktock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

apostasie commented Nov 1, 2024

Uh oh!

apostasie commented Nov 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apostasie commented Oct 21, 2024 •

edited

Loading

apostasie commented Oct 29, 2024 •

edited

Loading