add kernel config for gpu by Jimmy-Xu · Pull Request #938 · kata-containers/packaging

Jimmy-Xu · 2020-02-17T07:17:02Z

To support VFIO passthrough GPU (for example Nvidia Tesla P100)

enable mmio
enable load module(load NVIDIA Driver)

Usage example:

$ ./build-kernel.sh -v 4.19.86 -g nvidia -f setup
$ ./build-kernel.sh -v 4.19.86 -g nvidia build
$ sudo -E ./build-kernel.sh -v 4.19.86 -g nvidia install

-g <gpu_vendor> means enable gpu support, the option can be nvidia and intel

Output guest kernel:

/usr/share/kata-containers/vmlinux-nvidia-gpu.container -> vmlinux-4.19.86-70-nvidia-gpu
/usr/share/kata-containers/vmlinuz-nvidia-gpu.container -> vmlinuz-4.19.86-70-nvidia-gpu

To build NVIDIA Driver in kata container, kernel-devel is required:

$ cd kata-linux-4.19.86-70
$ make rpm-pkg

Output RPMs:

~/rpmbuild/RPMS/x86_64/kernel-devel-4.19.86_nvidia_gpu-1.x86_64.rpm

devimc

thanks @Jimmy-Xu

lgtm, but I'd like what other folks think about this

cc @grahamwhaley @amshinde @jcvenegas

jodh-intel · 2020-02-25T10:46:07Z

/test

devimc · 2020-02-25T14:41:01Z

@Jimmy-Xu CI is not happy

ERROR: Please bump version in kernel/kata_config_version

Jimmy-Xu · 2020-02-26T10:41:59Z

/test

chavafg · 2020-02-26T13:43:10Z

/AzurePipelines run

azure-pipelines · 2020-02-26T13:43:25Z

Azure Pipelines successfully started running 2 pipeline(s).

Jimmy-Xu · 2020-02-27T01:21:12Z

/AzurePipelines run

azure-pipelines · 2020-02-27T01:21:18Z

Commenter does not have sufficient privileges for PR 938 in repo kata-containers/packaging

jodh-intel · 2020-02-27T09:08:05Z

@grahamwhaley, @jcvenegas - any idea what's up with Azure? ^^

grahamwhaley · 2020-02-27T09:40:15Z

/AzurePipelines run

azure-pipelines · 2020-02-27T09:40:28Z

Azure Pipelines successfully started running 2 pipeline(s).

grahamwhaley · 2020-02-27T09:40:32Z

Let's try from me - it may be a perms/group thing.... that we can fix if we see that is true.

grahamwhaley · 2020-02-27T09:41:42Z

Sooo, @jcvenegas @chavafg - how does the Azure pipe check user perms - @Jimmy-Xu is in the kata org, I guess maybe he would need to be in one of the teams as well?

chavafg · 2020-02-27T15:48:19Z

@jcvenegas should know more, but I see the next note in the "Controlling the CI" wiki page

Note: Only works if you are a member of the packaging team and the phrase is the only text in the comment!

chavafg · 2020-02-27T15:51:06Z

I see @Jimmy-Xu is already part of the packaging team, so not sure what else is missing.

Jimmy-Xu · 2020-02-28T15:28:03Z

/test

amshinde · 2020-02-28T16:33:59Z

+# Support to load driver module
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODULE_SIG=y


cc @grahamwhaley @devimc Dont remember if we ran into issues with CONFIG_MODULE_SIG in the past.

I was wondering about this..... in theory I think MODULE_SIG is probably a good thing, as then you can only load modules actually built for your kernel.... but, it will add some overhead I think.
I was also thinking it would probably hard-tie the exact nvidia module used to the exact kernel you are loading... which, I'm not sure is the intention, or exactly what you don't want.

@Jimmy-Xu - can you tell us why you enabled it?

I don't remember, @Jimmy-Xu is this option really needed?

@grahamwhaley @devimc
There is a parameter CONFIG_CRYPTO_FIPS in kernel/configs/fragments/common/crypto.conf.
It depends on CRYPTO [=y] && (CRYPTO_ANSI_CPRNG [=y] || CRYPTO_DRBG [=n]) && !CRYPTO_MANAGER_DISABLE_TESTS [=n] && (MODULE_SIG [=n] || !MODULES [=y])
Therefore, if CONFIG_MODULES is enabled, CONFIG_MODULE_SIG is also required.

If CONFIG_CRYPTO_FIPS is removed, CONFIG_MODULE_SIG can also be removed.

I ran a test and after removing these two parameters the Nvidia GPU worked fine in a kata container.

Is the CONFIG_CRYPTO_FIPS really needed?

On the other hand, I think it's safer to use CONFIG_MODULE_SIG.

Oh, I remember now .... @amshinde added FIPS, which is somewhat entangled with MODULE. There is a conversation over at #891 (comment) where I dug into it...

@grahamwhaley No wonder that config looked familiar :) Now that I see the comments on the related issue, I am fine with having CONFIG_MODULE_SIG.

Jimmy-Xu · 2020-03-14T08:06:06Z

@devimc Ah, I see.
I can't put the gpu directory under x86_64 due to https://github.com/kata-containers/packaging/blob/master/obs-packaging/linux-container/debian.rules#L27
Fixed.
Thank you!

Jimmy-Xu · 2020-03-14T08:06:19Z

/test

Jimmy-Xu · 2020-03-15T09:37:10Z

/AzurePipelines run

azure-pipelines · 2020-03-15T09:37:17Z

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

jcvenegas · 2020-03-17T17:19:23Z

/AzurePipelines run

azure-pipelines · 2020-03-17T17:19:37Z

Azure Pipelines successfully started running 2 pipeline(s).

Jimmy-Xu · 2020-03-19T01:32:21Z

@sameo
kata-containers/documentation/pull/615 is blocked on this PR, can you PTAL?

grahamwhaley · 2020-03-19T10:01:04Z

I'm going to stick a DNM label on here whilst we do a little bit of review....

grahamwhaley

overall, looking very good @Jimmy-Xu :-)
A few minor comments.
I think we should probably add some documentation in https://github.com/kata-containers/packaging/blob/master/kernel/README.md about the command line arguments though - something we have been missing so far, but now I think with the addition of the gpu item, it is more important we have them.
/cc @jcvenegas

grahamwhaley · 2020-03-19T10:01:54Z

 kernel_config_path=""
+#Force generate config when setup
+force_setup_generate_config="false"
+#Nvidia kernel support


Is this Nvidia specific, or does it also work for other GPUs?
We should either fix the comment, or change the variable name I suspect.

I think GPU kernel support sounds more generic

updated. support Intel and Nvidia GPU, -g intel, -g nvidia

grahamwhaley · 2020-03-19T10:03:42Z

-	info "kernel path does not exist, will download kernel"
-	download_kernel="true"
-	[ -n "$kernel_version" ] || die "failed to get kernel version: Kernel version is emtpy"
+		if [[ "${force_setup_generate_config}" != "true" ]];then


nit: I would have added the 'force build' and 'bash debug' bits as separate commits in this PR, just to make it clearer for the reviewers ;-)

grahamwhaley · 2020-03-19T10:04:26Z

-	vmlinuz="vmlinuz-${kernel_version}-${config_version}"
-	vmlinux="vmlinux-${kernel_version}-${config_version}"
+
+	sufix=""


nit - I think you inherited this, but - s/sufix/suffix/

grahamwhaley · 2020-03-19T10:05:36Z

 				kernel_config_path="${OPTARG}"
 				;;
+			d)
+				PS4=' Line ${LINENO}: '


PS5 is coming 😉

The PS4 defined the $LINENO. So I used PS4 here.

devimc · 2020-03-19T14:19:49Z

thanks @Jimmy-Xu , just a couple of comments/questions and we could merge this

Add option '-g' in build-kernel.sh to build a guest kernel that supports Intel/Nvidia GPU Fixes: kata-containers#979 Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

Add option '-d' in build-kernel.sh to enable bash debug. Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

Add option '-f' in build-kernel.sh to force the generation of .config Signed-off-by: Jimmy Xu <junming.xjm@antfin.com> n 请为您的变更输入提交说明。以 '#' 开始的行将被忽略，而一个空的提交

Add usage of build-kernel.sh to the readme Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

Jimmy-Xu · 2020-03-19T19:24:02Z

/test

devimc · 2020-03-19T21:52:31Z

/AzurePipelines run

azure-pipelines · 2020-03-19T21:52:45Z

Azure Pipelines successfully started running 2 pipeline(s).

jodh-intel · 2020-03-20T09:38:14Z

Hi @Jimmy-Xu - when you are happy, please remove the do-not-merge label (as that is currently stopping this PR from being merged).

Jimmy-Xu · 2020-03-20T11:51:23Z

@jodh-intel do-not-merge label has been removed:)

Related: kata-containers/packaging#938 Fixes kata-containers#616 Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

Jimmy-Xu force-pushed the kernel-support-gpu branch from 1937954 to 6bfb355 Compare February 17, 2020 07:29

Jimmy-Xu mentioned this pull request Feb 17, 2020

Rescan PCI conflicts with shpchp / pciehp kata-containers/runtime#2460

Closed

bergwolf reviewed Feb 17, 2020

View reviewed changes

Comment thread kernel/configs/fragments/x86_64/gpu/version.conf Outdated

Jimmy-Xu force-pushed the kernel-support-gpu branch 2 times, most recently from f3503b8 to 673c94a Compare February 21, 2020 07:48

devimc approved these changes Feb 21, 2020

View reviewed changes

Jimmy-Xu mentioned this pull request Feb 25, 2020

Add support for hotplug PCIe device in q35 kata-containers/runtime#2432

Closed

jodh-intel approved these changes Feb 25, 2020

View reviewed changes

Jimmy-Xu force-pushed the kernel-support-gpu branch 2 times, most recently from f078453 to b292ee3 Compare February 26, 2020 10:35

amshinde reviewed Feb 27, 2020

View reviewed changes

Comment thread kernel/configs/fragments/x86_64/gpu/iommu.conf Outdated

Comment thread kernel/configs/fragments/x86_64/gpu/vfio.conf Outdated

Comment thread kernel/configs/fragments/x86_64/gpu/x86.conf Outdated

Jimmy-Xu force-pushed the kernel-support-gpu branch 2 times, most recently from bae6d24 to d4f4319 Compare February 28, 2020 14:47

amshinde reviewed Feb 28, 2020

View reviewed changes

Jimmy-Xu force-pushed the kernel-support-gpu branch from d4f4319 to 88f48d4 Compare February 29, 2020 06:50

Jimmy-Xu force-pushed the kernel-support-gpu branch from 88f48d4 to 5680e62 Compare March 14, 2020 07:56

Jimmy-Xu mentioned this pull request Mar 16, 2020

use-cases: Add documentation for using Nvidia GPU with Kata kata-containers/documentation#615

Merged

grahamwhaley added the do-not-merge PR has problems or depends on another label Mar 19, 2020

grahamwhaley reviewed Mar 19, 2020

View reviewed changes

Jimmy Xu added 3 commits March 20, 2020 02:18

kernel: support build guest kernel for gpu

d248e41

Add option '-g' in build-kernel.sh to build a guest kernel that supports Intel/Nvidia GPU Fixes: kata-containers#979 Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

kernel: support bash debug

7a17b50

Add option '-d' in build-kernel.sh to enable bash debug. Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

kernel: support force setup

1389500

Add option '-f' in build-kernel.sh to force the generation of .config Signed-off-by: Jimmy Xu <junming.xjm@antfin.com> n 请为您的变更输入提交说明。以 '#' 开始的行将被忽略，而一个空的提交

Jimmy-Xu force-pushed the kernel-support-gpu branch from 5680e62 to 0a875ff Compare March 19, 2020 18:49

kernel: add usage in readme

12d351d

Add usage of build-kernel.sh to the readme Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

Jimmy-Xu force-pushed the kernel-support-gpu branch from 0a875ff to 12d351d Compare March 19, 2020 18:54

Jimmy-Xu removed the do-not-merge PR has problems or depends on another label Mar 20, 2020

Jimmy-Xu pushed a commit to Jimmy-Xu/documentation that referenced this pull request Mar 20, 2020

use-cases: Add easy way to build guest kernel for Intel/Nvidia GPU

7c8c907

Related: kata-containers/packaging#938 Fixes kata-containers#616 Signed-off-by: Jimmy Xu <junming.xjm@antfin.com>

devimc merged commit 607931c into kata-containers:master Mar 20, 2020

jodh-intel mentioned this pull request Oct 7, 2020

Issue with NVIDIA GPU V100 and kata-containers kata-containers/kata-containers#835

Closed

Conversation

Jimmy-Xu commented Feb 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

devimc left a comment

Choose a reason for hiding this comment

Uh oh!

jodh-intel commented Feb 25, 2020

Uh oh!

devimc commented Feb 25, 2020

Uh oh!

Jimmy-Xu commented Feb 26, 2020

Uh oh!

chavafg commented Feb 26, 2020

Uh oh!

azure-pipelines Bot commented Feb 26, 2020

Uh oh!

Jimmy-Xu commented Feb 27, 2020

Uh oh!

azure-pipelines Bot commented Feb 27, 2020

Uh oh!

jodh-intel commented Feb 27, 2020

Uh oh!

grahamwhaley commented Feb 27, 2020

Uh oh!

azure-pipelines Bot commented Feb 27, 2020

Uh oh!

grahamwhaley commented Feb 27, 2020

Uh oh!

grahamwhaley commented Feb 27, 2020

Uh oh!

chavafg commented Feb 27, 2020

Uh oh!

chavafg commented Feb 27, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jimmy-Xu commented Feb 28, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jimmy-Xu Feb 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jimmy-Xu commented Mar 14, 2020

Uh oh!

Jimmy-Xu commented Mar 14, 2020

Uh oh!

Jimmy-Xu commented Mar 15, 2020

Uh oh!

azure-pipelines Bot commented Mar 15, 2020

Uh oh!

jcvenegas commented Mar 17, 2020

Uh oh!

azure-pipelines Bot commented Mar 17, 2020

Uh oh!

Jimmy-Xu commented Mar 19, 2020

Uh oh!

grahamwhaley commented Mar 19, 2020

Uh oh!

grahamwhaley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jimmy-Xu Mar 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jimmy-Xu commented Feb 17, 2020 •

edited

Loading

Jimmy-Xu Feb 29, 2020 •

edited

Loading

Jimmy-Xu Mar 19, 2020 •

edited

Loading

Jimmy-Xu Mar 19, 2020 •

edited

Loading

Jimmy-Xu commented Mar 20, 2020 •

edited

Loading