Debug gvisor

[Anjali05]

I am trying to debug gvisor with delve but it does not seem to work for me properly. Is there any documentation about debugging gvisor with delve or in general? I want to debug it by inserting breakpoints, I have seen this documentation: https://gvisor.dev/docs/user_guide/debugging/ but I am not sure how to go about it.

Also, if I am printing logs from any file(eg pkg/sentry/mm/syscalls.go) using log.Infof() where will it be written to?

[fvoznika]

Here is how I do it: 1. First, start the container you want to debug: docker run --runtime=runsc --name=test -d alpine sleep 1000 2. Find the PID for the sandbox: docker inspect test | grep Pid | head -n 1 3. Attach your favorite debugger: sudo dlv attach <PID> I hope it helps, Fabricio

…

On Wed, Jul 24, 2019 at 11:48 AM Anjali ***@***.***> wrote: I am trying to debug gvisor with delve but it does not seem to work for me properly. Is there any documentation about debugging gvisor with delve or in general? I want to debug it by inserting breakpoints, I have seen this documentation: https://gvisor.dev/docs/user_guide/debugging/ but I am not sure how to go about it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#590?email_source=notifications&email_token=AA27U7IXKFHFHG2B26IIFATQBCPXTA5CNFSM4IGS6ZLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HBI5AHA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA27U7JBZ2AGYOOQ2FZ7B6TQBCPXTANCNFSM4IGS6ZLA> .

[Anjali05]

@fvoznika I am getting this error:
could not attach to pid 14426: could not open debug info
Do I need to do any configuration changes?

[fvoznika]

'bazel build' strips debug symbols from the binary, so delve isn't able to find debug info. One option is to build runsc with debug option: bazel build -c dbg //runsc:runsc

…

On Fri, Jul 26, 2019 at 1:29 PM Anjali ***@***.***> wrote: @fvoznika <https://github.com/fvoznika> I am getting this error: could not attach to pid 14426: could not open debug info Do I need to do any configuration changes? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#590?email_source=notifications&email_token=AA27U7LSP7FN7LQSEPKJOJ3QBNND7A5CNFSM4IGS6ZLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25UKYA#issuecomment-515589472>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA27U7K4I5H2JOFFNQYMNILQBNND7ANCNFSM4IGS6ZLA> .

[fvoznika]

If you're using the nightly gVisor build <https://gvisor.dev/docs/user_guide/docker/#install-gvisor>, it will start including debug information <2762692> after tonight's build. On Fri, Jul 26, 2019 at 2:22 PM Fabricio Voznika <fvoznika@google.com> wrote:

…

'bazel build' strips debug symbols from the binary, so delve isn't able to find debug info. One option is to build runsc with debug option: bazel build -c dbg //runsc:runsc On Fri, Jul 26, 2019 at 1:29 PM Anjali ***@***.***> wrote: > @fvoznika <https://github.com/fvoznika> I am getting this error: > could not attach to pid 14426: could not open debug info > Do I need to do any configuration changes? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#590?email_source=notifications&email_token=AA27U7LSP7FN7LQSEPKJOJ3QBNND7A5CNFSM4IGS6ZLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25UKYA#issuecomment-515589472>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AA27U7K4I5H2JOFFNQYMNILQBNND7ANCNFSM4IGS6ZLA> > . >

[Anjali05]

@fvoznika If I want to debug a particular function( eg : Mmap() inside pkg/sentry/mm/syscalls.go), how do I set a break point in that? I tried with b Mmap <line_no> but it does not seem to work. Basically, I have a workload where I am doing some memory allocation and wants to see how many times this function is getting called.

[Anjali05]

Does the .boot file under the logs file contain syscalls to the host or just the syscalls within the container?

[fvoznika]

Try using the full method signature:

(dlv) b mm.MemoryManager.MMap
Breakpoint 1 set at 0x7353d8 for gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).MMap() pkg/sentry/mm/syscalls.go:75

The .boot file contains the syscalls issued by the processes running inside the sandbox (if --strace option is passed in). In case you want to see syscalls issued to the host, you can strace the sandbox process.

[Anjali05]

@fvoznika Does pkg/sentry/syscalls/linux contains syscalls that goes to the host from sentry or is it from the application to sentry?
Also, the strace for sentry does not seem to work for me even after enabling it in the daemon.json file. This is the content:

{
"default-runtime": "runc",
"data-root": "/my-disk/docker",
"runtimes": {
"runsc": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--debug-log=/my-disk/tmp/runsc-log/",
"--debug",
"--strace"
]
},
"runsc-kvm": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--platform=kvm",
"--debug-log=/my-disk/tmp/runsc-log/",
"--debug",
"--strace"
]
},
"runsc-ptrace": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--platform=ptrace",
"--debug-log=/my-disk/tmp/runsc-log/",
"--debug",
"--strace"
]
}
}
}

It starts tracing the process but I do not see any trace after I stop the tracing.
root@hp109:~# docker inspect memory | grep Pid | head -n 1
"Pid": 23036,~
root@hp109:~# strace -c -p 23036
strace: Process 23036 attached
^Cstrace: Process 23036 detached
root@hp109:~# strace -p 23036
strace: Process 23036 attached
futex(0x145dd88, FUTEX_WAIT_PRIVATE, 0, NULL^Cstrace: Process 23036 detached
<detached ...>
root@hp109:~#

I am running some mmap calls from the application and was hoping to see it at the host.

[nlacasse]

pkg/sentry/syscalls/linux contains sentry's syscall implementations. These are for the syscalls that the application makes to the sentry.

…

On Tue, Jul 30, 2019 at 11:05 AM Anjali ***@***.***> wrote: Does pkg/sentry/syscalls/linux contains syscalls that goes to the host from sentry or is it from the application to sentry? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#590?email_source=notifications&email_token=AABFN3KR6CN4PYLMJBCHIR3QCB7HNA5CNFSM4IGS6ZLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3EZ3AQ#issuecomment-516529538>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABFN3PDPYR4W3JR4BKE4T3QCB7HNANCNFSM4IGS6ZLA> .

[Anjali05]

@nlacasse Which are the files for syscalls the sentry makes to the host?

[nlacasse]

@Anjali05 The syscalls that the sentry makes to the host are just sprinkled around the codebase. Many come from within the Go runtime itself. You can find some of the syscalls we make by searching for syscall.Syscall and syscall.RawSyscall in our codebase.

The sentry runs in a restricted seccomp sandbox, and only a handful of syscalls are allowed to go to the host. The exact list of allowed syscalls is in the filter package:
https://github.com/google/gvisor/blob/67f2cefce02816307805699c3462d6fd7ce61b69/runsc/boot/filter/config.go

[Simon-Zh-0409]

@fvoznika Hi fvoznika, the attach works fine for me, but I found dlv exec can't work normally, it will crash when continue the program