kpatch script: better support for livepatch consistency model

[jpoimboe]

The kpatch script could use better support for livepatch modules. Some ideas below, roughly in descending order of usefulness.

1. signaling stalled tasks: I'm thinking this should be automatically done by "kpatch load" if the patch transition hasn't completed within a reasonable amount of time (10 seconds?). Basically, after loading, I think it should poll the transition state, and not exit the script until the transition is complete. Then if it's still not done after n seconds, it should do something like:

for s in /proc/*/task/*/patch_state; do
        pid=$(echo $s | sed 's;/.*task/\(.*\)/patch_state;\1;')
        if [[ $(cat /proc/$pid/patch_state) = 0 ]]; then
                echo "signaling pid $j $(cat /proc/$j/comm)"
                kill -SIGSTOP $j
                kill -SIGCONT $j
        fi
done

NOTE: Miroslav is working on an official way to do the above with a single write to sysfs.

2. reverting stalled patches: even if after signaling all the straggler tasks, the patch still fails to finish, I'm thinking "kpatch load" should reverse the patching process, unload the patch, and report an error.

3. EDIT: NEVER MIND. We could even mark patches with a "force" modinfo attribute which the kpatch script could read, for patches which are safe to force but might otherwise get stuck in transition because they patch a commonly slept on function (e.g., sys_poll). Then, in "kpatch load", it could force the patch - either immediately, or after the timeout described in update README with new OVERVIEW and DEMONSTRATION section #1 above. This will need Miroslav's force patch.

4. status of a transitioning patch: the "kpatch list" command should identify each patch as either "enabling...", "enabled", "disabling...", or "disabled". In the enabling/disabling case, it should list all the straggler tasks which are preventing the transition from completing.

5. kpatch signal: a command for signaling straggler tasks.

6. kpatch force: a command for forcing a patch as a last resort.

1-3 would be the "curated" approach -- just use "kpatch load"/"kpatch unload" and everything else is done for you. This will be used by most users, most of the time, to abstract away the details.

4-6 would be for those who want to load the module manually, without "kpatch load", in order to have more fine-grained control of the patching process.

[jpoimboe]

I'm actually not sure about the third one. Forcing a patch means that it may now be unsafe to rmmod the previous patch. So let's maybe forget that idea for now.

[jpoimboe]

Continuing my conversation with myself, the third one could work, if forcing a patch resulted in disallowing the previous patch from being unloaded. But I'm getting way ahead of myself. We can think about those scenarios later, let's still just ignore it.

[joe-lawrence]

Here's a work in progress branch.

This extends the kpatch list reporting to include the livepatch transition states, kpatch load waiting for the transition to complete, and periodically signalling the stalled tasks. I haven't tested with kpatch kernel/modules yet, but will do so to ensure backwards compatibility. I can expose more of the functions with command line options (like signalling a single PID, or waiting for a livepatch transition to finish, etc.) Feedback welcome here, as well as the reporting details and formatting.

Using the upstream livepatch callback "busymod" sample module, I can easily test a stubborn task (a kworker thread) that takes too long, doesn't care about signals, and triggers the patch revert code. Can you suggest an easy, positive test for the ordinary process signalling? Do I need to engineer a long running syscall that is to be patched? I didn't see any sample test programs in Miroslav's upstream kernel implementation.

[joe-lawrence]

What is the relationship between /proc/<pid>/patch_state and /proc/<pid>/task/<task>/patch_state? Your suggested code iterates over all of the tasks, but then checks the associated process task_state. Would it be sufficient to just iterate over processes and check their /proc/<pid>/patch_state?

[jpoimboe]

Can you suggest an easy, positive test for the ordinary process signalling? Do I need to engineer a long running syscall that is to be patched? I didn't see any sample test programs in Miroslav's upstream kernel implementation.

Try to patch a function which a lot of tasks are sleeping on in /proc/pid/stack, like ep_poll().

What is the relationship between /proc//patch_state and /proc//task//patch_state? Your suggested code iterates over all of the tasks, but then checks the associated process task_state. Would it be sufficient to just iterate over processes and check their /proc//patch_state?

For a multithreaded process, /proc is weird. When doing ls /proc, the individual threads don't show up in /proc, only the parent task does. But the individual thread directories are still there in /proc if you try to access them. They are also in in /proc/pid/task/tid. For example:

$ ps -efT |grep "gvim -v"
jpoimboe  6472  6472 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6504 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6505 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6506 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6507 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6508 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6509 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6510 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6511 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6512 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6513 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6472  6514 28733  0 14:23 pts/1    00:00:00 gvim -v fs/proc/base.c
jpoimboe  6705  6705  6568  0 14:27 pts/3    00:00:00 grep --color=auto gvim -v
[(2d23df4cf9d5...)] ~/git/linux $ ls /proc |grep 6472
6472
[(2d23df4cf9d5...)] ~/git/linux $ ls /proc |grep 6504
[(2d23df4cf9d5...)] ~/git/linux $ ls /proc/6504
attr       cgroup      comm             cwd      fd       io       loginuid   mem        mountstats  numa_maps  oom_score_adj  projid_map  schedstat  smaps  statm    task           uid_map
autogroup  clear_refs  coredump_filter  environ  fdinfo   latency  map_files  mountinfo  net         oom_adj    pagemap        root        sessionid  stack  status   timers         wchan
auxv       cmdline     cpuset           exe      gid_map  limits   maps       mounts     ns          oom_score  personality    sched       setgroups  stat   syscall  timerslack_ns
[(2d23df4cf9d5...)] ~/git/linux $ ls /proc/6472/task/6504/
attr  cgroup    clear_refs  comm    cwd      exe  fdinfo   io       limits    maps  mountinfo  net  numa_maps  oom_score      pagemap      projid_map  sched      sessionid  smaps  stat   status   uid_map
auxv  children  cmdline     cpuset  environ  fd   gid_map  latency  loginuid  mem   mounts     ns   oom_adj    oom_score_adj  personality  root        schedstat  setgroups  stack  statm  syscall  wchan

This kernel doesn't have CONFIG_LIVEPATCH enabled, but you get the idea.

So the following line

cat /proc/$pid/patch_state

could have instead been written as

cat /proc/$pid/task/$tid/patch_state

but I was too lazy to parse out the original pid and tid.

[joe-lawrence]

Interesting behavior for proc... the fact that you can access, but not see, /proc/<tid> for a thread is a little weird. I'll update the PR to iterate over the task ids and not just the process ids.

[joe-lawrence]

#753 implemented all but the module forcing (list item 3). That kernel feature is still under development by Miroslav upstream (should be merged soon though).