Potential bug in create-kpatch-module

[abk-code]

Potential bug in create-kpatch-module when running kpatch-build inside docker to build
against 4.4.27 kernel.

I did use the following link to install necessary dependencies.
https://github.com/dynup/kpatch

Since we are building inside docker against an older kernel, my understanding is
Kpatch-build will try to build older format of live patch using create-kpatch-module.
I do see both original .o and patched .o in the relevant directories inside docker.

But create-kpatch-module is not being called and I'm getting an error
indicating "Makefile target missing".

Upon further inspection (with -d oputput and from the script itself)

+ /usr/local/libexec/kpatch/create-kpatch-module /home/xyz/.kpatch/tmp/patch/tmp_output.o /home/xyz/.kpatch/tmp/patch/output.o
+ local to_stdout=1
+ [[ 1 -ge 2 ]]
+ [[ 1 -eq 1 ]]
+ tee -a /home/racker/.kpatch/build.log
+ check_pipe_status create-kpatch-module
+ rc=0
+ [[ 0 = 139 ]]
+ cd /home/xyz/.kpatch/tmp/patch
+ logger
+ KPATCH_BUILD=/source/linux
+ KPATCH_NAME=kpatch-4-4-0-2-modules-0-3-2-3-x86_64
+ KBUILD_EXTRA_SYMBOLS=/usr/lib/modules/4.4.0+2/extra/kpatch/Module.symvers
+ KPATCH_LDFLAGS=
+ make

This is where it's failing. The script comparison is with 139 which is SIGSEGV. I don't see
any core files either in the working directory.

I'm re-running the build again and will append any logs and more details.

• host kernel version

 uname -a
Darwin MVG0PGG8WL 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

• container OS :

Linux 15e882430758 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 15:12:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• container gcc version

gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

• patch-build line

kpatch-build -d -n kpatch-4.4.0+2-modules-0.3.2-3.x86_64 -c ./kernel-x86_64.config -s . -v /usr/lib/debug/lib/modules/4.4.0+2/vmlinux meminfo-string2.patch --skip-cleanup

• attach the input patch file

diff --git i/fs/proc/meminfo.c w/fs/proc/meminfo.c
index 9155a5a0d3b9..45dba271f01c 100644
--- i/fs/proc/meminfo.c
+++ w/fs/proc/meminfo.c
@@ -131,7 +131,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 		"Committed_AS:   %8lu kB\n"
 		"VmallocTotal:   %8lu kB\n"
 		"VmallocUsed:    %8lu kB\n"
-		"VmallocChunk:   %8lu kB\n"
+		"VmallocChunk temp:   %8lu kB\n"
 #ifdef CONFIG_MEMORY_FAILURE
 		"HardwareCorrupted: %5lu kB\n"
 #endif

edit: @joe-lawrence, formatting cleanup

[abk-code]

This is the build.log that you had asked for. I don't see anything in /var/log/messages.
build.log

[joe-lawrence]

Thanks for opening the bug report and providing the build.log file.

This is where it's failing. The script comparison is with 139 which is SIGSEGV. I don't see
any core files either in the working directory.

Oh, after re-reading the script, I think what you see in the output is that the ${PIPESTATUS[0]} is 0, and it's echoing the comparison against a SIGSEGV to the screen. In this case, there's no issue, so there are no complaints about core files anywhere.

What does the output of ls -al /home/xyz/.kpatch/tmp/patch look like? Is there a Makefile present in that directory?

[abk-code]

Thank you for the response.
Here is the info you requested.

ls -al /home/racker/.kpatch/tmp/patch
total 392
drwxr-xr-x 2 racker hypdev 4096 Mar 28 21:03 ./
drwxr-xr-x 6 racker hypdev 4096 Mar 28 21:03 ../
-rwxr-xr-x 1 racker hypdev 192776 Mar 28 21:03 output.o*
-rw-r--r-- 1 racker hypdev 193656 Mar 28 21:03 tmp_output.o

No Makefile in this directory.

[joe-lawrence]

Are you running kpatch-build from a git repo checkout, or did you install it to the system?

Also, do you have an instance of DATADIR=/some/path/ in your build output (doesn't get saved into the build.log unfortunately).

[abk-code]

1. I cloned kpatch and did make install-kpatch-build inside the docker.
2. I have some Linux kern version also cloned and point the kpatch-build command to use
   that Linux repo.

kpatch-build -d -n kpatch-4.4.0+2-modules-0.3.2-3.x86_64 -c ./kernel-x86_64.config -s . -v /usr/lib/debug/lib/modules/4.4.0+2/vmlinux meminfo-string2.patch --skip-cleanup

I don't see DATADIR but going to recheck again by capturing the output.

[joe-lawrence]

1. I cloned kpatch and did make install-kpatch-build inside the docker.

Ooo, that's an interesting make target. I think this only installed part of the tools. (You can double check to see if /usr/local/share/kpatch exists, at least on a RHEL system.)

I would try doing a full install via make install . That should create a template patch directory (example above) that gets copied into /home/xyz/.kpatch/tmp/patch as part of the kpatch-build. Do that full install and re-try the build.

[abk-code]

Thank you again:So that has a different issue.
kernel-devel-4.9.87-linuxkit-aufs - is docker related.
E.g

[15e882430758] racker@docker > uname -r
4.9.87-linuxkit-aufs
[15e882430758] racker@docker > sudo make
make -C kpatch-build
make[1]: Entering directory `/source/kpatch/kpatch-build'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/source/kpatch/kpatch-build'
make -C kpatch
make[1]: Entering directory `/source/kpatch/kpatch'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/source/kpatch/kpatch'
make -C kmod
make[1]: Entering directory `/source/kpatch/kmod'
make -C core clean
make[2]: Entering directory `/source/kpatch/kmod/core'
Makefile:7: *** /lib/modules/4.9.87-linuxkit-aufs/build doesn\'t exist.  Try installing the kernel-devel-4.9.87-linuxkit-aufs RPM or linux-headers-4.9.87-linuxkit-aufs DEB..  Stop.
make[2]: Leaving directory `/source/kpatch/kmod/core'
make[1]: *** [clean] Error 2
make[1]: Leaving directory `/source/kpatch/kmod'
make: *** [build-kmod] Error 2

& hence my question earlier, that is it ok to do kpatch-build within docker for a different kernel target.

[cardoe]

So ultimately the issue is can kpatch-build be used to build a patch file against the non-running kernel? I would expect that it could since this is often how kernel modules are built. Writing a kernel module 101 will use a makefile similar to:

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

But most people are going to really do something like:

KVER ?= $(uname -r)

all:
    make -C /lib/modules/$(KVER)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(KVER)/build M=$(PWD) clean

What @desisnacks is looking for is how to do make KVER=4.4.0+2 when uname -r is returning "4.9.87-linuxkit-aufs".

[joe-lawrence]

What @desisnacks is looking for is how to do make KVER=4.4.0+2 when uname -r is returning "4.9.87-linuxkit-aufs".

Yup, it's definitely possible. First you'll need to rebuild kpatch itself (specifically the kpatch core module) against the desired kernel, overriding KPATCH_BUILD path. For example, I'll run all of this from a git checkout of the repo:

% uname -r
4.4.0+
% make -s clean
% make KPATCH_BUILD=/lib/modules/3.10.0-693.el7.x86_64/build
...
make[2]: Entering directory `/root/kpatch.joe/kmod/core'
make -C /lib/modules/3.10.0-693.el7.x86_64/build M=/root/kpatch.joe/kmod/core kpatch.ko
make[3]: Entering directory `/usr/src/kernels/3.10.0-693.el7.x86_64'
...
% modinfo ./kmod/core/kpatch.ko | grep vermagic
vermagic:       3.10.0-693.el7.x86_64 SMP mod_unload modversions 

Then run kpatch-build and provide it with -s and -v options that point to your target kernel source directory and vmlinux path:

./kpatch-build/kpatch-build \
    -s /root/rpmbuild/BUILD/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.centos.x86_64 \
    -v /lib/debug/usr/lib/modules/3.10.0-693.el7.x86_64/vmlinux
    -t vmlinux \
    ./test/integration/centos-7/cmdline-string.patch
...
% modinfo kpatch-cmdline-string.ko | grep vermagic
vermagic:       3.10.0 SMP mod_unload modversions 

... reboot onto 3.10.0-693.el7.x86_64 ...

% uname -r
3.10.0-693.el7.x86_64
% ./kpatch/kpatch load kpatch-cmdline-string.ko 
loading core module: /root/kpatch.joe/kpatch/../kmod/core/kpatch.ko
loading patch module: kpatch-cmdline-string.ko
% cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64 root=/dev/mapper/centos-root ro console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8 kpatch=1

In my example, these paths are all over the place since I'm hacking it together with a distro kernel... interestingly the vermagic didn't match exactly, but I think that's because I only did an rpmbuild -bp on my kernel source rpm. Hopefully close enough to demonstrate the general idea.

[abk-code]

thanks @joe-lawrence. I will report you on this shortly. I did go past make/install for kpatch
instead of using install-kpatch-build as make target.

[abk-code]

+ cp -f /home/racker/.kpatch/tmp/patch/kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko /source/linux
+ [[ 1 -eq 0 ]]
+ echo SUCCESS
SUCCESS

Thanks @joe-lawrence , I am able to create lp. IF I may ask, can you share your specfile for building
an rpm for this lp? I have similar setup and just trying to leverage. Thanks again! Immensely helpful.

[abk-code]

[root@69-183-22-655721 ~]# kpatch info kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko
Patch information for kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko:
filename:       /root/kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko
license:        GPL
depends:        kpatch
vermagic:       4.4.0+2+ SMP mod_unload modversions
parm:           replace:replace all previously loaded patch modules (bool)
[root@69-183-22-655721 ~]# uname -a
Linux 69-183-22-655721 4.4.0+2 #1 SMP Tue Mar 13 16:59:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@69-183-22-655721 ~]# kpatch load kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko
loading patch module: kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko
insmod: ERROR: could not insert module kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko: Unknown symbol in module
kpatch: failed to load module kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko

Further the load/install of the kpatch fails. I think this is because of an extra +
symbol in the modinfo (vermagic).

[joe-lawrence]

Further the load/install of the kpatch fails. I think this is because of an extra +
symbol in the modinfo (vermagic).

I believe the kernel build adds the additional '+' symbol if there are any outstanding git changes in the kernel source directory. What does git status report from the kernel tree?

[joe-lawrence]

kpatch_root_kobj is provided by the kpatch core module, kpatch.ko which the kpatch script should have loaded when you tried to load kpatch-4-4-0-2-modules-0-3-2-3-x86_64.ko.

Can you provide the output from:
grep -e "T klp_register_patch" -e "T kpatch_register" /proc/kallsyms
and
modprobe kpatch

[abk-code]

[root@69-183-22-655721 ~]# grep -e "T klp_register_patch" -e "T kpatch_register" /proc/kallsyms
ffffffffa04e6c30 T kpatch_register	[kpatch]
[root@69-183-22-655721 ~]# modprobe kpatch
modprobe: FATAL: Module kpatch not found.

Also,from:

[root@69-183-22-655721 ~]# grep PATCH running.config
CONFIG_HAVE_LIVEPATCH=y
# CONFIG_LIVEPATCH is not set

I think I need to enable CONFIG_LIVEPATCH and rebuild the kernel.