AArch64: Enable kernel fragment on aarch64 by Pennyzct · Pull Request #1006 · kata-containers/packaging

Pennyzct · 2020-04-09T06:15:56Z

Hi guys @kata-containers/kernel @grahamwhaley

I've been trying to enable kernel fragments on arm64. It's quite a large patch, plz review. ;)

Pennyzct · 2020-04-09T06:17:37Z

Hi guys
Could you plz add a WIP tag here?? I've waiting for #998 got merged and then do a few twist here. ;)

After backporting patch series of enabling memory hot remove on aarch64 to v5.4.x, we finally could enable nvdimm/dax on aarch64. Fixes: kata-containers#2601 Depends-on: github.com/kata-containers/packaging#1006 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

jodh-intel

Very nice! Thanks @Pennyzct !

lgtm

jodh-intel · 2020-04-09T08:19:02Z

@Pennyzct - label added. Note that you get the same behaviour (of stopping the PR from being merged) if you add "WIP" to the title of the PR.

Pennyzct · 2020-04-09T08:20:33Z

@jodh-intel thanks for the explanation. ;)

grahamwhaley

Overall, l g t m @Pennyzct - very nice.
Thanks for clearing up all the x86 specifics that were in common - sorry about that ;-)

Just a couple of bits of feedback we should discuss with some others - but overall, looking good!
Let's hope this does make future kernel updates easier!

grahamwhaley · 2020-04-09T09:50:35Z

-# FIXME - check if we should be setting this
-# https://github.com/kata-containers/packaging/issues/483
-#CONFIG_RANDOMIZE_BASE=y
+CONFIG_RANDOMIZE_BASE=y


This is a change to x86 as well I think? We should contemplate from the security pov then
@amshinde @egernst

@grahamwhaley Enabling KSLR is recommended from a security point of view, but I dont think you can leverage KSM after this is enabled.
cc @sameo

@amshinde thinking about this, KSM works by looking for hash identical pages - so, I suspect KSM will still work with RANDOMIZE_BASE - but, I suspect you cannot do any native address based sharing (as the page addresses will be different) - but then I'm not sure we do that between guest kernels loaded by the VM anyway?

I had a dig around to see if there were any documented and notable disadvantages to setting CONFIG_RANDOMIZE_BASE. The only real one I can identify as being real/current is that it will hinder trying to debug crashed kernels, as the address map will change between each boot. I think we debug guest kernels so infrequently that the gains from the positive security aspect probably outweigh the negatives of the debug situation.

We probably want to ask @kata-containers/architecture-committee if they have any thoughts here.

I remember noticing that the 'crash' tool had something for this (possibly x86 only and not sure how this works on arm and a guest):
commit ad1a44f5d9b9a41d2c43bb386c765913275f05bf
Author: Dave Anderson anderson@redhat.com
Date: Fri Jan 13 15:38:39 2017 -0500

Fix for support of /proc/kcore as the live memory source in Linux 4.8 and later x86_64 kernels configured with CONFIG_RANDOMIZE_BASE, which randomizes the unity-mapping PAGE_OFFSET value. Without the patch, the crash session fails during session initialization with the error message "crash: seek error: kernel virtual address: <address> type: page_offset_base". (anderson@redhat.com)

grahamwhaley · 2020-04-09T10:19:51Z

+# Compaction is the only memory management component to form high order
+# (larger physically contiguous) memory blocks reliably. The lack of the
+# feature can lead to unexpected OOM killer invocations for high order memory requests.
+CONFIG_COMPACTION=y


ooh, nice catch - yes, having checked, we probably do want this enabled.
Note to others, this is a config change for x86.

Nice catch @Pennyzct

grahamwhaley · 2020-04-09T10:20:40Z

 CONFIG_SCSI_VIRTIO=y
 CONFIG_VIRTIO_BLK=y
 CONFIG_TTY=y
+CONFIG_VIRTIO_BALLOON=y


I'm not sure if we actually use balloon - @jcvenegas can you check/config pls.

Dont think we use this today.

okay, I'll delete it. ;)

grahamwhaley · 2020-04-09T10:22:39Z

+# Use the maximum number of CPUs supported by KVM (255)
+CONFIG_NR_CPUS=255
+
+CONFIG_PERF_EVENTS=y


just to note, iirc, we are debating if we need perf on/off by default in the kata kernels I think - iirc, they may come with a startup perf cost?

IMO, we could make perf off by default.
But for x86_64, this config CONFIG_PERF_EVENTS is automatically selected by CONFIG_KVM, See https://github.com/torvalds/linux/blob/master/arch/x86/kvm/Kconfig#L41.
And we select CONFIG_CGROUP_PERF=y in common cgroup config file, which is dependent on CONFIG_PERF_EVENTS.
I'll try to dump kernel start up info to track the cost. ;)

Oh, that's interesting - didn't know KVM selected PERF_EVENTS. I wonder if they are needed for something specific - would you know @dagrh ?

I guess having them on may do no harm - but, I also guess that is now an architecture specific decision at some level - so probably your choice for this arch @Pennyzct :-)

That also surprises me; my reading of it is that it's used to emulate a PMU in the guest on x86 - although I thought the hardware PMU helped with that these days,

grahamwhaley · 2020-04-09T10:24:47Z

@@ -0,0 +1,498 @@
+From ba91422b18892bceacf3b4aa60354cf36fcabf9b Mon Sep 17 00:00:00 2001


I'm sure this has probably been discussed - but, do we do hot remove on Kata - I thought we pretty much only ever added, as remove can be very expensive (as you need to 'clear out' any used pages in the 'dimm' you want to remove).
/cc @jcvenegas for any thoughts.

Hi~ @grahamwhaley
The reason that I enabled memory hot remove is that ZONE_DEVICE is dependent on it, and
NVDIMM/DAX needs ZONE_DEVICE. ;). maybe I should elaborate more in commit message.

grahamwhaley · 2020-04-09T10:26:46Z

 # This can still be dynamically disabled on the kernel command line/kata config if needed.
 # Disable for now, as it upsets the entropy test, and we need to improve those: FIXME: see:
 # https://github.com/kata-containers/tests/issues/1543
-# CONFIG_RANDOM_TRUST_CPU is not set


I'm not sure that is/was a typo - iirc enabling it broke entropy - we need to review this one I think.

grahamwhaley · 2020-04-09T10:26:57Z

 # Estimated cost (detailed in the kernel config files)
 # is maybe 2.3% for both
 CONFIG_STACKPROTECTOR=y
-CONFIG_STACKPROTECTOR_STRONG


ouch! nice catch :-)

grahamwhaley · 2020-04-09T10:28:16Z

 #
 # Automatically generated file; DO NOT EDIT.
-# Linux/arm64 5.4.3 Kernel Configuration
+# Linux/arm64 5.4.15 Kernel Configuration


Nice to have the update! - ooi, have you done any metrics comparisons between the old and new config?

yep, I'll do the metrics comparisons asap. ;)

grahamwhaley · 2020-04-09T10:30:20Z

/test-ubuntu
because I'm curious...

jodh-intel · 2020-04-09T10:33:07Z

@Pennyzct - np, it's documented in the table here: https://github.com/kata-containers/community/blob/master/CONTRIBUTING.md#temporarily-blocking-a-pr.

Pennyzct · 2020-04-17T06:56:02Z

Hi guys @grahamwhaley @amshinde @kata-containers/kernel

I've re-based and addressed a few comments. ;)
The metrics report on boot-up time and memory footprint is here. For boot-up time, we have 3%~5% decrease after importing kernel fragments and enabling dax.
And for memory foortprint, we only could run a few tests, since we need a few extra configuration to run memory hotplug(need to boot from uefi). Still, we could see around 6% decrease on memory footprint with ksm and without ksm.

Still, we have a few configs left to discuss:
CONFIG_RANDOMIZE_BASE=y
CONFIG_PERF_EVENTS=y and CONFIG_CGROUP_PERF=y

grahamwhaley

Thanks for the metrics! Nice it all headed in the 'good' direction.
I'll drop my lgmt here in preparation for other feedback/decisions and the dropping of the WIP

grahamwhaley · 2020-04-20T14:41:11Z

-# FIXME - check if we should be setting this
-# https://github.com/kata-containers/packaging/issues/483
-#CONFIG_RANDOMIZE_BASE=y
+CONFIG_RANDOMIZE_BASE=y


@amshinde thinking about this, KSM works by looking for hash identical pages - so, I suspect KSM will still work with RANDOMIZE_BASE - but, I suspect you cannot do any native address based sharing (as the page addresses will be different) - but then I'm not sure we do that between guest kernels loaded by the VM anyway?

I had a dig around to see if there were any documented and notable disadvantages to setting CONFIG_RANDOMIZE_BASE. The only real one I can identify as being real/current is that it will hinder trying to debug crashed kernels, as the address map will change between each boot. I think we debug guest kernels so infrequently that the gains from the positive security aspect probably outweigh the negatives of the debug situation.

We probably want to ask @kata-containers/architecture-committee if they have any thoughts here.

grahamwhaley · 2020-04-20T14:43:45Z

+# Use the maximum number of CPUs supported by KVM (255)
+CONFIG_NR_CPUS=255
+
+CONFIG_PERF_EVENTS=y


Oh, that's interesting - didn't know KVM selected PERF_EVENTS. I wonder if they are needed for something specific - would you know @dagrh ?

I guess having them on may do no harm - but, I also guess that is now an architecture specific decision at some level - so probably your choice for this arch @Pennyzct :-)

For now, a few configs as follows in common acpi dir are truly x86-spcecific or disable by default on arm64. CONFIG_ACPI_CPU_FREQ_PSS=y CONFIG_ACPI_HOTPLUG_IOAPIC=y CONFIG_ACPI_LEGACY_TABLES_LOOKUP CONFIG_ACPI_LPIT=y CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y CONFIG_ACPI_PROCESSOR_CSTATE=y CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y CONFIG_HAVE_ACPI_APEI_NMI=y And I also add a few configs which are aarch64-specific. Like CONFIG_ACPI_REDUCED_HARDWARE_ONLY=y, since ARM64 can run properly in ACPI hardware reduced mode. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

There exists a few configs about linux guest support or optimization that are not supported on aarch64. CONFIG_HYPERVISOR_GUEST is only defined under arch/x86/Kconfig and unfortunately, CONFIG_KVM_GUEST is not supported on aarch64 for now. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

There exists a few security-related configs, which are x86-64 specific. CONFIG_LEGACY_VSYSCALL_NONE=y CONFIG_RETPOLINE=y CONFIG_RELOCATABLE and CONFIG_RANDOMIZE_BASE are kinds of tangled on aarch64, if CONFIG_RANDOMIZE_BASE=y, then CONFIG_RELOCATABLE will be selected automatically. CONFIG_RANDOMIZE_BASE will randomize the virtual address at which the kernel image is loaded, which as a security feature could deter exploit attempts relying on knowledge of the location of kernel internals. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

mmio devices are required in firecracker, and for now, x86_64 and aarch64 are all supporting kata containers with firecracker. So, we need to move mmio-related configs to common dir. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Compaction is the only memory management component to form high order (larger physically contiguous) memory blocks reliably. The page allocator relies on compaction heavily and the lack of the feature can lead to unexpected OOM killer invocations for high order memory requests. We shouldn't disable this option unless there really is a strong reason. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Add a few arm64-specific configs and classify them into seven new categories , that is, 1. base architecture-dependent options(base.conf) It also includes varient-specific features, like CONFIG_ARM64_PMEM is one ARMv8.2 arichitectural features. 2. crypto-related options(crypto.conf) ARMv8 adds cryptographic instructions that could significantly improve performance on tasks such as AES encryption and SHA1 and SHA256 hashing. 3. device tree related options(dt.conf) The "Open Firmware Device Tree", or simply Device Tree (DT), is a data structure and language for describing hardware, which is commonly used in arm architecture. 4. ARM errata workarounds options(errata.conf) There are many Kconfig entires under "Kernel Features" -> "ARM errata workarounds via the alternatives framework", which provides software workarounds to mitigate systems affected by those erratum. Vendor-specific option will be left to users to decide. 5. pci related options(pci.conf) a simplified pci host controller for mach-virt. 6. serial devices options(serial.conf) CONFIG_SERIAL_OF_PLATFORM is used for all 8250 compatible serial ports that are probed through device tree. 7. rtc related options(rtc.conf) we don't have KVM’s paravirtualized clock and ptp implementation is still under experimental mode, so we need rtc on aarch64. QEMU provides an emulated ARM AMBA PrimeCell PL031 RTC. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Backport Anshuman Khandual's patch series of Enabling memory hot remove on aarch64(https://patchwork.kernel.org/cover/11419305/) to v5.4.x. XONE_DEVICE is dependent on the implementation of memory hot remove. This patch series has already been merged, and queued for 5.7. After backporting this series, we could finally enable nvdimm/dax on arm64. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

fix a few typo errors. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Since we disable pci shpc hotplug for arm64, see kata-containers#498 for detailed reason. We need to move CONFIG_HOTPLUG_PCI_SHPC from common conf to x86_64-specific. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com> Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>

Pennyzct · 2020-04-21T06:59:24Z

Hi guys @grahamwhaley @dagrh
I will let config_perf_events on arm64 for consistency.
And BTW, in @justin-he's new PR, kata-containers/runtime@105673b, we will add pmu=off in qemu command line to deliberately turn off pmu, since pmu will cost significant overhead on boot time and virtualization context switch.

Pennyzct · 2020-04-21T07:04:13Z

Hi guys
I've added a new commit to enable kvm-ptp, plz re-review. ;)
And I've done all the work here, so maybe we could remove the wip flag here.
And btw, since we have no arm jenkins job here in packaging, I raised a PR kata-containers/runtime#2602 in runtime repo to run all tests.

kvm-ptp is critical for mitigating time drift between host and guest. This implementation in kernel side is still one experimental feature on aarch64, and see kata-containers#998 for detailed instructions. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com> Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>

The config file created by kernel fragments scheme is quite different with the old arm64_kata_kvm_5.4.x. So I will update arm64_kata_kvm_5.4.x for consistency. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Pennyzct · 2020-04-22T01:50:14Z

/test-ubuntu

Pennyzct · 2020-04-22T05:11:32Z

/AzurePipelines run

azure-pipelines · 2020-04-22T05:11:38Z

Commenter does not have sufficient privileges for PR 1006 in repo kata-containers/packaging

Pennyzct · 2020-04-22T05:13:35Z

Hi guys
obs ci is not responding. ;)
Need help trigger it~~~

jodh-intel · 2020-04-22T08:19:47Z

/AzurePipelines run

azure-pipelines · 2020-04-22T08:20:02Z

Azure Pipelines successfully started running 2 pipeline(s).

Pennyzct · 2020-04-23T02:24:42Z

Hi guys
CI is all happy now~~~~~~🎉🎊

The linux kernel feature RANDOMIZE_BASE improved the security and at the same time increased the memory footprint of a kata container, this feature was enabled in kata-containers/packaging#1006. In order to mitigate this increase in memory consumption, we can boot container using the uncompressed kernel. fixes # Signed-off-by: Julio Montes <julio.montes@intel.com>

The linux kernel feature RANDOMIZE_BASE improved the security and at the same time increased the memory footprint of a kata container, this feature was enabled in kata-containers/packaging#1006. In order to mitigate this increase in memory consumption, we can boot container using the uncompressed kernel. fixes kata-containers#669 Signed-off-by: Julio Montes <julio.montes@intel.com>

The linux kernel feature RANDOMIZE_BASE improved the security and at the same time increased the memory footprint of a kata container, this feature was enabled in kata-containers/packaging#1006. In order to mitigate this increase in memory consumption, we can boot container using the uncompressed kernel. Reduce boot time by ~5% Reduce KSM memory footprint by ~14% Reduce noKSM memory footprint by ~27% fixes kata-containers#669 Signed-off-by: Julio Montes <julio.montes@intel.com>

Pennyzct closed this Apr 9, 2020

Pennyzct reopened this Apr 9, 2020

Pennyzct mentioned this pull request Apr 9, 2020

AArch64: enable NVDIMM/DAX on AArch64 kata-containers/runtime#2601

Closed

Pennyzct mentioned this pull request Apr 9, 2020

dax: enable dax on arm64 kata-containers/runtime#2602

Merged

jodh-intel approved these changes Apr 9, 2020

View reviewed changes

jodh-intel added the wip Work in Progress (PR incomplete - needs more work or rework) label Apr 9, 2020

grahamwhaley reviewed Apr 9, 2020

View reviewed changes

Pennyzct force-pushed the kernel_fragment_on_aarch64 branch 2 times, most recently from c5b957f to 88899cb Compare April 17, 2020 06:41

grahamwhaley approved these changes Apr 20, 2020

View reviewed changes

Pennyzct added 9 commits April 21, 2020 14:47

config: fix typo error

f188ef7

fix a few typo errors. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Pennyzct force-pushed the kernel_fragment_on_aarch64 branch from 88899cb to d9b1e94 Compare April 21, 2020 06:49

config: update arm64_kata_kvm_5.4.x

14a281d

The config file created by kernel fragments scheme is quite different with the old arm64_kata_kvm_5.4.x. So I will update arm64_kata_kvm_5.4.x for consistency. Fixes: kata-containers#1004 Signed-off-by: Penny Zheng <penny.zheng@arm.com>

Pennyzct force-pushed the kernel_fragment_on_aarch64 branch from d9b1e94 to 14a281d Compare April 21, 2020 09:52

grahamwhaley removed the wip Work in Progress (PR incomplete - needs more work or rework) label Apr 21, 2020

jodh-intel merged commit 56d7074 into kata-containers:master Apr 23, 2020

chavafg mentioned this pull request Apr 24, 2020

Metrics CI started failing on memory-footprint test kata-containers/ci#273

Closed

devimc mentioned this pull request Sep 4, 2020

runtime: qemu: reduce boot time and memory footprint kata-containers/kata-containers#674

Merged

		@@ -0,0 +1,498 @@
		From ba91422b18892bceacf3b4aa60354cf36fcabf9b Mon Sep 17 00:00:00 2001

Conversation

Pennyzct commented Apr 9, 2020

Uh oh!

Pennyzct commented Apr 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jodh-intel left a comment

Choose a reason for hiding this comment

Uh oh!

jodh-intel commented Apr 9, 2020

Uh oh!

Pennyzct commented Apr 9, 2020

Uh oh!

grahamwhaley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grahamwhaley commented Apr 9, 2020

Uh oh!

jodh-intel commented Apr 9, 2020

Uh oh!

Pennyzct commented Apr 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grahamwhaley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Pennyzct commented Apr 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pennyzct commented Apr 21, 2020

Uh oh!

Pennyzct commented Apr 22, 2020

Uh oh!

Pennyzct commented Apr 22, 2020

Uh oh!

azure-pipelines Bot commented Apr 22, 2020

Uh oh!

Pennyzct commented Apr 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Pennyzct commented Apr 9, 2020 •

edited

Loading

Pennyzct commented Apr 17, 2020 •

edited

Loading

Pennyzct commented Apr 21, 2020 •

edited

Loading

Pennyzct commented Apr 22, 2020 •

edited

Loading