enable multi export by fahedouch · Pull Request #1575 · moby/buildkit

fahedouch · 2020-07-16T18:25:12Z

building #1555

tonistiigi · 2020-07-16T18:29:51Z

the proto files are generated, you should not change them manually. For backward compatibility new indexes are also needed. Switching the type from single to repeated may not need a new index iirc (@hinshun not sure if this was the case for all types).

update: make generated-files is what you need to run for proto changes.

hinshun · 2020-07-16T18:55:21Z

Switching the type from single to repeated may not need a new index iirc (@hinshun not sure if this was the case for all types).

Not true for all types, for example repeated int32 is not compatible with int32.

fahedouch · 2020-08-02T21:31:58Z

@tonistiigi @hinshun ok I will pay attention to backward compatibility ( but I don't think we have backward compatibility problem with string and ExporterAttrs types).

For the moment I need your help :
I don't know how to enables writing into multiple directories to be reachable through the grpc session here

I got this message Server.RegisterService found duplicate service registration for "moby.filesync.v1.FileSend" when

tonistiigi · 2020-08-03T18:25:09Z

There is no need to register filesync target server twice. I think for multi-export we are mainly talking about different types of exporters. Even if there is some use case for doing 2 local exports (that I think is just wasteful) there is one handler and splitting between targets is done by the call parameters.

fahedouch · 2020-08-08T19:52:01Z

@tonistiigi
Thank you for your reply. I think I am done with this PR. Do you see any additional changes? Should I do activate the multi-export in buildx project now ?

tonistiigi

Many failures in the CI, please take a look at the logs.

We will also need integration tests for this feature. PTAL https://github.com/moby/buildkit/blob/master/client/client_test.go and https://github.com/moby/buildkit/blob/master/.github/CONTRIBUTING.md#run-the-unit--and-integration-tests for running the tests.

tonistiigi · 2020-08-12T06:25:36Z

-	string Exporter = 3;
-	map<string, string> ExporterAttrs = 4;
+	repeated string Exporters = 3;
+	repeated ExporterAttrs ExportersAttrs = 4;


I'm pretty sure this is not backward compatible. @hinshun . At least it can't be compatible if we ever wish to extend ExporterAttrs in the future.

tonistiigi · 2020-08-12T06:27:08Z

+				if ex.OutputDir == "" {
+					return nil, errors.New("output directory is required for local exporter")
+				}
+				s.Allow(filesync.NewFSSyncTargetDir(ex.OutputDir))


Some things like output directory do not seem to support multiple afaics. That is fine but please add validation for not allowing 2 local exporters etc.

Done. Would you recheck please :) ?

tonistiigi · 2020-08-12T06:27:59Z

-	if req.Exporter != "" {
-		exp, err := w.Exporter(req.Exporter, c.opt.SessionManager)
+	if req.Exporters[0] != "" {
+		exp, err := w.Exporter(req.Exporters[0], c.opt.SessionManager)


Afaics this exports to first item of the array, not to all items.

Done. For the moment I commented some parts in code, I will reactive them.

AkihiroSuda · 2020-11-06T10:35:06Z

needs rebase

fahedouch · 2020-11-07T22:51:02Z

needs rebase

@AkihiroSuda after rebasing it is hard to fix DCO job by signing previous commits. Can you help plz!

AkihiroSuda · 2020-11-07T23:28:45Z

git rebase -i HEAD~N (N is a number)

more steps on enabling multi-export fix variable name fix variable name && lint code Signed-off-by: fahedouch <fahed.dorgaa@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Avoid hidden session passing and allow one session to drop when multiple builds share a vertex. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

update run/exec tests for stdin and expected failures move common tests for runc and container to shared tests package Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

This is the shim used by the containerd Runtime V2 on Linux, per the default setting of `io.containerd.runc.v2`. Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

…matting text does not appear to contain a placeholder Signed-off-by: Helen Xie <chenjg@harmonycloud.cn> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

According to documentation (https://docs.docker.com/engine/reference/builder/#dockerignore-file), absolute paths like `/foo/bar` should have the same effect as `foo/bar`. This is not the case today. This fix normalize paths when reading the .dockerignore file by removing leading slashes. Signed-off-by: Simon Ferquel <simon.ferquel@docker.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Daniel Nephin <dnephin@docker.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: wingkwong <wingkwong.code@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

…0ab3ab full diff: microsoft/go-winio@fc70bd9...5b44b70 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

full diff: golang/sys@2334cc1...aee5d88 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

…951081b35f full diff: moby/libnetwork@d8334cc...d095108 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

…adc38 full diff: docker/cli@2298e6a...1d20b15 relevant changes: - config: don't call homedir on init() - config: fix error message using incorrect filename - config: remove redundant os.Stat() - config: Handle errors on close in config file write. - config: ignore empty config file instead of printing warning - config: Fix ConfigFile.Save() replacing symlink with file - Config-file: remove User-Agent from config.json when saving Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

…352355d4 full diff: moby/moby@4634ce6...c2cc352 also adds github.com/cilium/ebpf as a dependency, which I set to the same version as is set in containerd/cgroups version Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Allows readonly passed cleanly. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Changes: - rootless-containers/rootlesskit@v0.9.5...v0.11.0 - containers/fuse-overlayfs@v1.1.2...v1.2.0 - containerd/fuse-overlayfs-snapshotter@v0.10.0...v1.0.0 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Same defaults already used on buildkitd side Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Currently moby/buildkit image misses fuse dependency (fusermount) so currently stargz support doesn't work on that image. This commit fixes this issue. Signed-off-by: ktock <ktokunaga.mail@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Signed-off-by: fahedouch <fahed.dorgaa@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

fahedouch · 2020-11-10T20:37:42Z

I close this PR for #1788

fahedouch changed the title ~~[WIP] enable multi export~~ enable multi export Aug 8, 2020

fahedouch marked this pull request as ready for review August 8, 2020 19:44

fahedouch force-pushed the enable-multi-export branch 2 times, most recently from 08c51cd to 269bad0 Compare August 8, 2020 20:07

tonistiigi reviewed Aug 12, 2020

View reviewed changes

fahedouch force-pushed the enable-multi-export branch from d107d5f to 7ef3c31 Compare September 23, 2020 20:42

fahedouch and others added 16 commits November 9, 2020 16:26

enable multi-export

024f85e

more steps on enabling multi-export fix variable name fix variable name && lint code Signed-off-by: fahedouch <fahed.dorgaa@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

enabling multi-export

58285c0

Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

secrets: allow providing secrets with env

89c63ad

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

secrets: remove file specifics from variable names

d1f5206

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

secrets: reuse constant on error message

b929bc1

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

solver: fix marking already cached vertex as cancelled

4765aba

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

session: track sessions with a group construct

bbee6c7

Avoid hidden session passing and allow one session to drop when multiple builds share a vertex. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

pull: allow separate sessions for different parts of pull

8ece6bf

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

pull: fix session updating on resolver

a10bbea

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

resolver: add credentials cache

1977ec2

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Update Executor interface for Run and Exec

90790d1

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

fix error handling for exec when container fails to start

f3e9a1c

update run/exec tests for stdin and expected failures move common tests for runc and container to shared tests package Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

simplify done channel handling, fix other pr comments.

450049f

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

update godoc for Executor interface

8b7ec2f

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

inline cache: fix handling of duplicate blobs

32dcf0d

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Build the containerd-shim-runc-v2 shim for tests

da2f8db

This is the shim used by the containerd Runtime V2 on Linux, per the default setting of `io.containerd.runc.v2`. Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

fate-grand-order and others added 25 commits November 9, 2020 16:34

use t.Fatal() to output the err message where the values used for for…

7faba79

…matting text does not appear to contain a placeholder Signed-off-by: Helen Xie <chenjg@harmonycloud.cn> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Add canonical import comment

51601bd

Signed-off-by: Daniel Nephin <dnephin@docker.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Fix typos

40c6ca7

Signed-off-by: wingkwong <wingkwong.code@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

dockerignore: remove import path enforcement comments

6a23424

Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

frontend/dockerfile: use dockerignore package from buildkit

868a95f

Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

vendor: github.com/Microsoft/go-winio v0.4.15-0.20200908182639-5b44b7…

2249bd2

…0ab3ab full diff: microsoft/go-winio@fc70bd9...5b44b70 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

vendor: github.com/gorilla/mux v1.8.0

9dc2466

Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

vendor: golang.org/x/sys v0.0.0-20200922070232-aee5d888a860

f8f1d61

full diff: golang/sys@2334cc1...aee5d88 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

vendor: github.com/docker/libnetwork v0.8.0-dev.2.0.20200917202933-d0…

e269ba1

…951081b35f full diff: moby/libnetwork@d8334cc...d095108 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

update runc used in integration tests so exec no longer panics

4afe93f

Signed-off-by: Cory Bennett <cbennett@netflix.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Fix race in CNI namespaces.

34a6746

Signed-off-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

Rename nativeID to nsPath. Simplify.

7420e92

Signed-off-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

add session injection to remote loading

ead9df8

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

executor: change mount to struct

67332ba

Allows readonly passed cleanly. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

progress: add log limits and clipping

b2616b8

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

control: split status messages if they grow big

b90486b

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

client: use containerd grpc message size defaults

c83cf8f

Same defaults already used on buildkitd side Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

logs: show log limit in the clipping message

023aa50

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

resolve MR conflics

d3fb750

Signed-off-by: fahedouch <fahed.dorgaa@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

lint code

2bab0e3

Signed-off-by: fahedouch <fahed.dorgaa@gmail.com> Signed-off-by: fahed dorgaa <fahed.dorgaa@gmail.com>

fahedouch force-pushed the enable-multi-export branch from f95b8b1 to 2bab0e3 Compare November 9, 2020 15:54

Merge branch 'master' into enable-multi-export

4facfdb

fahedouch mentioned this pull request Nov 10, 2020

enable multi exporters #1788

Closed

fahedouch closed this Nov 10, 2020

Conversation

fahedouch commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hinshun commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fahedouch commented Aug 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi commented Aug 3, 2020

Uh oh!

fahedouch commented Aug 8, 2020

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

tonistiigi Aug 12, 2020

Choose a reason for hiding this comment

Uh oh!

tonistiigi Aug 12, 2020

Choose a reason for hiding this comment

Uh oh!

fahedouch Sep 13, 2020

Choose a reason for hiding this comment

Uh oh!

tonistiigi Aug 12, 2020

Choose a reason for hiding this comment

Uh oh!

fahedouch Sep 13, 2020

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Nov 6, 2020

Uh oh!

fahedouch commented Nov 7, 2020

Uh oh!

AkihiroSuda commented Nov 7, 2020

Uh oh!

fahedouch commented Nov 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

fahedouch commented Jul 16, 2020 •

edited

Loading

tonistiigi commented Jul 16, 2020 •

edited

Loading

hinshun commented Jul 16, 2020 •

edited

Loading

fahedouch commented Aug 2, 2020 •

edited

Loading