history api: support for logs of completed builds by tonistiigi · Pull Request #3339 · moby/buildkit

tonistiigi · 2022-11-30T05:18:08Z

buildctl debug logs <refid> now also works for completed builds. The logs are kept in contentstore with a simple length-based encoding of the proto format.

Signed-off-by: Tonis Tiigi tonistiigi@gmail.com

crazy-max · 2022-12-07T15:53:16Z

+	if req.NoWait {
+		return nil
+	}


I see OpenBlobWriter has been introduced to save tracing. Is the purpose of NoWait to avoid waiting for the history to be loaded in case it's being deleted/gc?

NoWait is if the client only wants to get a list of builds once and keep waiting on updates for them. Currently, the only use case is in the tests where because some tests check for cleaned contentstore the full build history also needs to be cleaned.

Could we rename it to Exit or EarlyExit or similar?

crazy-max · 2022-12-07T16:05:23Z

+	if s := trace.SpanFromContext(ctx); s.SpanContext().IsValid() {
+		if exp, err := detect.Exporter(); err == nil {
+			if rec, ok := exp.(*detect.TraceRecorder); ok {
+				stopTrace = rec.Record(s.SpanContext().TraceID())


We are gathering spans for the build and save them to the history now. TraceID() is used to be sure we capture the right traces for the build right?

We filter out the tracing info only for the same traceID that was set in the tracing context of the solve function. So two solves get their own individual traces.

jedevc · 2022-12-08T13:11:04Z

+	if req.NoWait {
+		return nil
+	}


Could we rename it to Exit or EarlyExit or similar?

thaJeztah · 2022-12-08T18:49:07Z

Will there be something on build to read back the logs? (see #1837)

tonistiigi · 2022-12-08T19:28:30Z

Will there be something on build to read back the logs?

buildctl debug logs <ref>

jedevc · 2022-12-12T18:59:27Z

CI is red 👀

jedevc

Provenance refactor looks good to me, few minor comments, but happy for this to merge once CI is green and comments are fixed.

jedevc · 2022-12-12T19:11:38Z

-	if err != nil {
-		return nil, err
-	}
+func (s *Solver) recordBuildHistory(ctx context.Context, id string, req frontend.SolveRequest, j *solver.Job) (func(*Result, exporter.DescriptorReference, error) error, error) {


We should have a tracking issue for SBOM support with build history if we're not doing it in this PR.

tonistiigi · 2022-12-13T01:41:32Z

TestIntegration/TestStargzLazyRegistryCacheImportExport/worker=containerd-snapshotter-stargz seems to be the only one left failing atm. @ktock Could you take a look as I don't really understand what it is testing and if the failing condition is valid. Otherwise, I suggest skipping this test for now, as it seems to be some stargz specific optimization behavior. The main difference with this PR is that ExportTo that was previously only called on --cache-to is now called with all builds.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

ktock · 2022-12-13T06:31:31Z

@tonistiigi Thank you for notifying me.

Is it an expected bahavior of this PR that BuildKit (with containerd worker configuration) doesn't cleanup containerd content store when pruning the cache all?
Master version (40bc1b3) cleans up the content store when doing buildctl prune --all but this PR doesn't seem to do so.
This is observed on overlayfs snapshotter as well.

buildkitd flags: --oci-worker=false --containerd-worker=true

# mkdir -p /tmp/ctx-a && cat <<EOF > /tmp/ctx-a/Dockerfile
FROM ubuntu:22.04
EOF
# buildctl build --progress=plain --frontend=dockerfile.v0 \
               --local context=/tmp/ctx-a --local dockerfile=/tmp/ctx-a \
               --output type=oci,dest=/tmp/test.tar
# buildctl prune --all
# buildctl du
ID	RECLAIMABLE	SIZE	LAST ACCESSED
Reclaimable:	0B
Total:		0B
# ctr --namespace=buildkit content ls
DIGEST									SIZE	AGE		LABELS
sha256:0585b60588963c4c76ac974991ddb8943760ea70c9e781b4d43ec023856dd37d	3.613kB	5 seconds	-
sha256:59329cca1b6c70654247470c3039b0076f3cffdc9ab32cce2f1006f404ab5642	481B	6 seconds	containerd.io/gc.ref.content.0=sha256:f35120b15f00b87e24dcf48439e45507e56802b121a7c8ebf7f1ddafe69d88e6,containerd.io/gc.ref.content.1=sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797
sha256:5e44ffe94256b613961a33356a6710f668e01e66ffd00000f535b10b92c6bac0	3.585kB	5 seconds	-
sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797	30.43MB	5 seconds	buildkit.io/blob/annotation.containerd.io/uncompressed=sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3,buildkit.io/blob/mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip,containerd.io/gc.ref.content.blob-sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797=sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797
sha256:7f9e7e014d150b47278fef38fe32896eb1e73c6c43bfe6a4842ade77ede515e6	258.5kB	2 seconds	-
sha256:f35120b15f00b87e24dcf48439e45507e56802b121a7c8ebf7f1ddafe69d88e6	870B	6 seconds	-

TestIntegration/TestStargzLazyRegistryCacheImportExport/worker=containerd-snapshotter-stargz assumes prune API cleans up containerd content store as well. The test failure seems to related to this difference.

Previously this was not issue as manifest was deleted after build completed but now the manifest is kept in build history API. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi · 2022-12-13T07:44:31Z

@ktock Thanks. That case should be fixed with 2be7e54

tonistiigi force-pushed the history-api-logs branch 2 times, most recently from b45a3a9 to f427054 Compare November 30, 2022 14:44

crazy-max reviewed Nov 30, 2022

View reviewed changes

Comment thread client/status.go

tonistiigi added this to the v0.11.0 milestone Dec 1, 2022

tonistiigi force-pushed the history-api-logs branch 2 times, most recently from 602e785 to dbfbbcd Compare December 6, 2022 05:54

tonistiigi marked this pull request as ready for review December 7, 2022 06:06

tonistiigi force-pushed the history-api-logs branch 2 times, most recently from e557e2e to f7ddfe1 Compare December 7, 2022 07:52

crazy-max reviewed Dec 7, 2022

View reviewed changes

tonistiigi force-pushed the history-api-logs branch 6 times, most recently from 0f91453 to c61c8e6 Compare December 8, 2022 02:31

jedevc reviewed Dec 8, 2022

View reviewed changes

tonistiigi force-pushed the history-api-logs branch from c61c8e6 to fceddbd Compare December 8, 2022 19:12

tonistiigi mentioned this pull request Dec 8, 2022

Sourcepolicy #3332

Merged

tonistiigi force-pushed the history-api-logs branch 2 times, most recently from 0717a26 to d1129c3 Compare December 12, 2022 08:14

tonistiigi force-pushed the history-api-logs branch from d1129c3 to 2c3fe26 Compare December 12, 2022 19:06

jedevc reviewed Dec 12, 2022

View reviewed changes

jedevc mentioned this pull request Dec 12, 2022

Embed Ref types directly into attestations #3389

Merged

tonistiigi force-pushed the history-api-logs branch 3 times, most recently from 93d336e to bbb57ea Compare December 12, 2022 22:22

tonistiigi force-pushed the history-api-logs branch from ce8e671 to 661b2df Compare December 13, 2022 00:48

tonistiigi added 10 commits December 12, 2022 22:15

history api: support for logs of completed builds

be6e1cf

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

history api: tracing storage support

8a2addb

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

history api: deletion and pinning support

301ea54

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

build history: refactor recording to a function

0994f41

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

history api: allow internal builds that are not tracked

fe0c8d3

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

provenance: allow builder ID to set with attrs

079505f

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

history api: save provenance to history

179bed6

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

build history: support for image descriptors

e5f9959

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

build history: fix provenance for lazy results

926451e

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

cache: make sure mutable refs are not finalized on exporting cache

ab0d1a6

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

tonistiigi force-pushed the history-api-logs branch from 661b2df to ab0d1a6 Compare December 13, 2022 06:15

containerimage: don't keep ref from manifest to layers

2be7e54

Previously this was not issue as manifest was deleted after build completed but now the manifest is kept in build history API. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

jedevc approved these changes Dec 13, 2022

View reviewed changes

crazy-max approved these changes Dec 13, 2022

View reviewed changes

tonistiigi merged commit ed041ab into moby:master Dec 13, 2022

thaJeztah mentioned this pull request Dec 22, 2022

FIXME: Got an API for which error does not match any expected type!!! moby/moby#38877

Closed

jedevc mentioned this pull request Jan 11, 2023

v0.11.0 produces blobs that can't be pruned (build histories, SLSA provenance blobs, etc.) #3492

Closed

AkihiroSuda mentioned this pull request Jan 12, 2023

[DNM] bisect BuildKit (0.11.0-rc1, rc2, ...) containerd/nerdctl#1833

Closed

thaJeztah mentioned this pull request Feb 10, 2023

Output of failed builds can be cumbersome to use for debugging docker/buildx#1619

Open

Conversation

tonistiigi commented Nov 30, 2022

Uh oh!

Uh oh!

crazy-max Dec 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tonistiigi Dec 7, 2022

Choose a reason for hiding this comment

Uh oh!

jedevc Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

crazy-max Dec 7, 2022

Choose a reason for hiding this comment

Uh oh!

tonistiigi Dec 7, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jedevc Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thaJeztah commented Dec 8, 2022

Uh oh!

tonistiigi commented Dec 8, 2022

Uh oh!

jedevc commented Dec 12, 2022

Uh oh!

jedevc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jedevc Dec 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tonistiigi commented Dec 13, 2022

Uh oh!

ktock commented Dec 13, 2022

Uh oh!

tonistiigi commented Dec 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

crazy-max Dec 7, 2022 •

edited

Loading