set tracing socket path to runtime dir by crazy-max · Pull Request #4078 · moby/buildkit

crazy-max · 2023-07-29T01:04:33Z

AkihiroSuda · 2023-07-31T14:28:25Z


-func traceSocketPath(root string) string {
-	return filepath.Join(root, "otel-grpc.sock")
+func traceSocketPath(inUserNS bool) string {


This should be moved to https://github.com/moby/buildkit/blob/master/util/appdefaults/appdefaults_unix.go

crazy-max · 2023-07-31T15:03:04Z

In rootless mode in our integration tests that sounds legit:

2023/07/31 13:09:36 > startCmd 2023-07-31 13:09:36.150436901 +0000 UTC m=+0.639178596 /usr/bin/sudo -u #1000 -i -- exec nsenter -U --preserve-credentials -m -t 15717 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd3936068494/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --containerd-worker-snapshotter=native --containerd-worker-snapshotter=native --config=/tmp/bktest_config1926999853/buildkitd.toml --root /tmp/bktest_buildkitd2346048497 --addr unix:///tmp/bktest_buildkitd2346048497/buildkitd.sock --debug
2023/07/31 13:09:36 buildkitd: listen unix /run/user/1000/buildkit/otel-grpc.sock: bind: address already in use

https://github.com/moby/buildkit/actions/runs/5714997638/job/15483511608#step:7:1258

    sandbox.go:286: time="2023-07-31T13:15:57Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"/bin/sh -c mkdir -p /usr/bin && echo -n foo > /usr/bin/go\" did not complete successfully: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/buildkit/otel-grpc.sock\" to rootfs at \"/dev/otel-grpc.sock\": possibly malicious path detected -- refusing to operate on /tmp/bktest_containerd2293416339/state/io.containerd.runtime.v2.task/buildkit/yjdfcd29mfseoazkv7uopr7ne/rootfs/dev/otel-grpc.sock (deleted): unknown"
    sandbox.go:286: process "/bin/sh -c mkdir -p /usr/bin && echo -n foo > /usr/bin/go" did not complete successfully: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/buildkit/otel-grpc.sock" to rootfs at "/dev/otel-grpc.sock": possibly malicious path detected -- refusing to operate on /tmp/bktest_containerd2293416339/state/io.containerd.runtime.v2.task/buildkit/yjdfcd29mfseoazkv7uopr7ne/rootfs/dev/otel-grpc.sock (deleted): unknown
    sandbox.go:286: 35741 7fd9470 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd2293416339/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config3965535412/buildkitd.toml --root /tmp/bktest_buildkitd1187620594 --addr unix:///tmp/bktest_buildkitd1187620594/buildkitd.sock --debug

https://github.com/moby/buildkit/actions/runs/5714997636/job/15483510501#step:7:2316

Added BUILDKIT_TRACE_SOCKET env var so we can set the trace socket path to sandbox temp dir. Let me know if that sounds good.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max · 2023-07-31T17:24:02Z

Hum it still tries to write to xdg runtime dir in rootless mode for frontend integration tests, I might miss smth: https://github.com/moby/buildkit/actions/runs/5716370920/job/15487868441?pr=4078#step:7:1143

=== CONT  TestIntegration/TestOnBuildHeredoc/worker=oci-rootless/frontend=builtin
2023/07/31 15:15:27 stdout: /usr/bin/sudo -u #1000 -i -- exec rootlesskit buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config2668879534/buildkitd.toml --root /tmp/bktest_buildkitd3159267947 --addr unix:///tmp/bktest_buildkitd3159267947/buildkitd.sock --debug
2023/07/31 15:15:27 stderr: /usr/bin/sudo -u #1000 -i -- exec rootlesskit buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config2668879534/buildkitd.toml --root /tmp/bktest_buildkitd3159267947 --addr unix:///tmp/bktest_buildkitd3159267947/buildkitd.sock --debug
2023/07/31 15:15:27 > startCmd 2023-07-31 15:15:27.049972746 +0000 UTC m=+17.524187589 /usr/bin/sudo -u #1000 -i -- exec rootlesskit buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config2668879534/buildkitd.toml --root /tmp/bktest_buildkitd3159267947 --addr unix:///tmp/bktest_buildkitd3159267947/buildkitd.sock --debug
2023/07/31 15:15:27 buildkitd: listen unix /run/user/1000/buildkit/otel-grpc.sock: bind: address already in use

Edit: I think we are losing env context with sudo, I will take a look.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

tonistiigi

I'm not sure if this is completely safe to pick to v0.12 . Maybe there are some weird perm errors for /run/buildkit in some systems.

How about we merge this for master and pick https://github.com/moby/buildkit/pull/3483/files for v0.12?

crazy-max · 2023-08-02T10:05:52Z

How about we merge this for master and pick https://github.com/moby/buildkit/pull/3483/files for v0.12?

Fine to me

jedevc · 2023-08-03T11:27:50Z

 		}
 		// TODO: make sure the user exists and subuid/subgid are configured.
-		buildkitdArgs = append([]string{"sudo", "-u", fmt.Sprintf("#%d", s.UID), "-i", "--", "exec", "rootlesskit"}, buildkitdArgs...)
+		buildkitdArgs = append([]string{"sudo", "-E", "-u", fmt.Sprintf("#%d", s.UID), "-i", "--", "exec", "rootlesskit"}, buildkitdArgs...)


@crazy-max looks like this actually disables all the rootless tests 👀

See https://github.com/moby/buildkit/actions/runs/5749062460/job/15583230313:

time="2023-08-03T09:40:23Z" level=warning msg="rootless mode is not supported on this host: exit status 1 (sudo: you may not specify both the -i and -E options\nusage: sudo -h | -K | -k | -V\nusage: sudo -v [-ABkNnS] [-g group] [-h host] [-p prompt] [-u user]\nusage: sudo -l [-ABkNnS] [-g group] [-h host] [-p prompt] [-U user] [-u user]\n [command [arg ...]]\nusage: sudo [-ABbEHkNnPS] [-C num] [-D directory] [-g group] [-h host] [-p\n prompt] [-R directory] [-T timeout] [-u user] [VAR=value] [-i | -s]\n [command [arg ...]]\nusage: sudo -e [-ABkNnS] [-C num] [-D directory] [-g group] [-h host] [-p\n prompt] [-R directory] [-T timeout] [-u user] file ...\n)"

yikes test step didn't failed 😕

Opened an issue

Rootless tests are not running since 2023-08-02 (sudo: you may not specify both the -i and -E options) #4100

jedevc · 2023-08-03T11:37:10Z

Removing from the v0.12.1 milestone, see #4078 (comment).

AkihiroSuda · 2024-02-03T07:45:53Z

Regression

[v0.13.0-beta1 regression] /run/buildkit is inaccessible for socket users #4618

AkihiroSuda reviewed Jul 30, 2023

View reviewed changes

Comment thread cmd/buildkitd/main_unix.go Outdated

AkihiroSuda reviewed Jul 30, 2023

View reviewed changes

Comment thread cmd/buildkitd/main_unix.go Outdated

tonistiigi added this to the v0.12.1 milestone Jul 31, 2023

crazy-max force-pushed the fix-otel-sock branch from 7af2b3e to 27688e6 Compare July 31, 2023 12:47

crazy-max changed the title ~~fallback to runtime dir if tracing socket path too long~~ set tracing socket path to runtime dir Jul 31, 2023

crazy-max force-pushed the fix-otel-sock branch from 27688e6 to 32b3bcb Compare July 31, 2023 12:54

crazy-max requested review from AkihiroSuda and tonistiigi July 31, 2023 12:55

crazy-max marked this pull request as ready for review July 31, 2023 12:55

crazy-max added the needs-cherry-pick/v0.12 label Jul 31, 2023

crazy-max force-pushed the fix-otel-sock branch from 32b3bcb to 0e83310 Compare July 31, 2023 13:00

AkihiroSuda reviewed Jul 31, 2023

View reviewed changes

crazy-max force-pushed the fix-otel-sock branch from 0e83310 to 44dc4d7 Compare July 31, 2023 14:57

AkihiroSuda approved these changes Jul 31, 2023

View reviewed changes

set tracing socket path to runtime dir

a7690c5

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the fix-otel-sock branch from 44dc4d7 to a7690c5 Compare July 31, 2023 15:07

integration: preserve existing environment variables when using sudo

b57eeb2

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

tonistiigi approved these changes Aug 2, 2023

View reviewed changes

tonistiigi mentioned this pull request Aug 2, 2023

[v0.12.1] cherry-picks #4088

Merged

tonistiigi merged commit c7318df into moby:master Aug 2, 2023

crazy-max deleted the fix-otel-sock branch August 2, 2023 10:07

jedevc reviewed Aug 3, 2023

View reviewed changes

jedevc removed the needs-cherry-pick/v0.12 label Aug 3, 2023

jedevc removed this from the v0.12.1 milestone Aug 3, 2023

AkihiroSuda mentioned this pull request Aug 3, 2023

Rootless tests are not running since 2023-08-02 (sudo: you may not specify both the -i and -E options) #4100

Closed

crazy-max mentioned this pull request Aug 3, 2023

integration: fix rootless tests #4101

Merged

AkihiroSuda mentioned this pull request Feb 3, 2024

[v0.13.0-beta1 regression] /run/buildkit is inaccessible for socket users #4618

Closed

georgethebeatle mentioned this pull request Mar 15, 2024

[v0.13] OTEL socket mount requires new volumes when containerd is run in separate container #4764

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set tracing socket path to runtime dir#4078

set tracing socket path to runtime dir#4078
tonistiigi merged 2 commits intomoby:masterfrom
crazy-max:fix-otel-sock

crazy-max commented Jul 29, 2023

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda Jul 31, 2023

Uh oh!

crazy-max commented Jul 31, 2023

Uh oh!

crazy-max commented Jul 31, 2023 •

edited

Loading

Uh oh!

tonistiigi left a comment

Uh oh!

crazy-max commented Aug 2, 2023

Uh oh!

jedevc Aug 3, 2023

Uh oh!

crazy-max Aug 3, 2023

Uh oh!

AkihiroSuda Aug 3, 2023

Uh oh!

jedevc commented Aug 3, 2023

Uh oh!

AkihiroSuda commented Feb 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

crazy-max commented Jul 29, 2023

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

crazy-max commented Jul 31, 2023

Uh oh!

crazy-max commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

crazy-max commented Aug 2, 2023

Uh oh!

jedevc Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

crazy-max Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

jedevc commented Aug 3, 2023

Uh oh!

AkihiroSuda commented Feb 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

crazy-max commented Jul 31, 2023 •

edited

Loading