open /sys/devices/system/memory/block_size_bytes: no such file or directory

[cedriccchen]

Description of problem

run a kata container with a kernel compiled before

Expected result

setup successfully

Actual result

setup container failed:
docker: Error response from daemon: oci runtime error: rpc error: code = Unknown desc = open /sys/devices/system/memory/block_size_bytes: no such file or directory.

---

[grahamwhaley]

Is this an issue you will chase @clarecch ? - otherwise, we are going to need some more details :-) (like the output from the kata-env so we know which versions of which components you used - and also the command line you ran).

[cedriccchen]

This is a problem found by @caoruidong .
Ruidong please complete this issue later, thanks~

[caoruidong]

I compile the kernel long ago, recently kata cannot run successfully after adding memory hotplug feature. Because there is no memory under /sys/devices/system/ directory. We find that CONFIG_MEMORY_HOTPLUG is not set in guest kernel config.
This is to say if we use a custom kernel for kata, we need some specified configs.

[jodh-intel]

kata-containers/agent#358 introduced the agent GetGuestDetails() gRPC API. Then #624 started calling that from the runtime. It sounds then like we need to:

• Update GetGuestDetails() in the agent repo to set MemBlockSizeBytes to 0 if the block_size_bytes file doesn't exist in the guest. If the file doesn't exist, it should log a warning message (which could mention guest kernel lacks CONFIG_MEMORY_HOTPLUG).
• Check with @Pennyzct, @ydjainopensource and @alicefr if arm64 and s390 support CONFIG_MEMORY_HOTPLUG (neither architecture currently mentions that string at all) and if so update the relevant kernel configs over in https://github.com/kata-containers/packaging/tree/master/kernel/configs.

fwics, the runtime doesn't appear to need updating as memHotplugValid() already handles the zero case.

[caoruidong]

x86 and powerpc configs in packing repo set this option to y now. While arm64 doesn't mention it and s390 specifies "CONFIG_MEMORY_HOTPLUG is not set". We don't know if they can work well.
This issue may be useful if someone meets this problem later.

[cedriccchen]

@jodh-intel Do we force to use linux kernel config in https://github.com/kata-containers/packaging/tree/master/kernel/configs in katacontainer? If so, we could just update something as you comment. If not so, maybe we need to make some feature configurable, which may need some specific kernel config if some people want to use kernel config of their own.

[jodh-intel]

Hi @clarecch - No, we don't force users to use our kernels. But we do provide a default kernel via OBS which is built from the packaging repo.

Kernel config is a tricky problem to manage though - see:

• Create Linux kernel configuration script packaging#8
• Should we provide multiple guest kernel config files? packaging#55
• create script to validate kernel options across architectures packaging#56
• kernel: config: Improve CI for kernel config changes packaging#78

[cedriccchen]

Hi @jodh-intel If users use kernel config of their own, which is not open memory hotplug configure or cpu hotplug configure or other specific configures, many features of katacontainer will not be able to use. It means katacontainer may not run normally. Have we consider this situation?

[cedriccchen]

cc @linzichang

[jodh-intel]

Hi @clarecch - yes. We are trying to be as flexible as possible: we provide a working system but certain parts can be "tweaked" (config settings, osbuilder image, kernel, etc). It should be possible for a user to create a custom kernel based on the config we provide but with extra options enabled for example.

What we don't have currently though is a comprehensive list of kernel options that Kata will expect. All we have are those config files. See the links I pasted above.

On the topic if specific kernel features being available or not, I think this is something that might be worth bringing up at the Architecture Committee meeting and could be summarised as:

• Should the agent fail if specific userland and/or kernel features are not available in the guest?

It think the agent should be tolerant where possible but wdyt @sboeuf, @bergwolf, @grahamwhaley, @egernst ?

[grahamwhaley]

For some features, and in this specific case for the memory hotplug, maybe it should not fail. In this specific case, it should probably not fail if nobody tries to plug any memory in. That may mean we need to have options or a config on the runtime side to prevent it asking for hotplug.
I do think there will be situations where a user wants to disable a feature and not have things die. Let's cross those bridges as we come to them though....

which leaves the question - does anybody have a use case where they do not want hotplug?

[Weichen81]

@jodh-intel I have checked the kernel, both latest 4.14.x and latest stable kernel do not support CONFIG_MEMORY_HOTPLUG for Arm64. From the Kconfig, it seems only x64_64 supports it:
(SPARSEMEM [=y] || X86_64_ACPI_NUMA) && ARCH_ENABLE_MEMORY_HOTPLUG

And these is a "Memory hotplug support for arm64" patch series on mailing list, but it's pending on
reviewing. We'll add this feature to our TODO list.

[jodh-intel]

Hi @Weichen81 - thanks for checking. That means the current agent will fail here on arm64 I think?

• https://github.com/kata-containers/agent/blob/master/grpc.go#L1323

If so, we will definitely need to make that call non-fatal as mentioned in #796 (comment).

[Weichen81]

Hi @jodh-intel, although memory hotplug depends on arch_enable_memory_hotplug hasn’t been implemented on Arm64, but I think the block_size_bytes of memory doesn’t depends on memory hotplug, it’s a memory root attribute. I will check why this file can’t be access on Arm64 after 7th Oct, we are public holidays this week ; )

[linzichang]

@Weichen81 block_size_bytes depends on sparse memory.