RHCOS does not write kernel parameters into grub.conf

[jhrozek]

Description of problem:

The probes that check for kernel arguments such as whether the audit=1 kernel argument has been added do not work because they seem to be checking for /boot/grub2/grubenv. This is not what's being used on RHCOS, the file is more or less empty there. We need to adjust the probes to check for something different.

SCAP Security Guide Version:

as built from git as of today

Operating System Version:

RHCOS

Steps to Reproduce:

1. set kernel parameters with a MC
2. reboot the node
3. make sure the kernel parameters are set e.g. with cat /proc/cmdline

Actual Results:

the rule would still fail

Expected Results:

the rule should now pass

Addition Information/Debugging Steps:

I guess the question is how to check the kernel arguments. The first thing that comes to my mind is /proc/cmdline. Is there a reason not to check that in the first place?

I did a little digging how is the kernel argument set on RHCOS and it goes like this:

• a MachineConfig with an Ignition spec is defined by the admin
• the MC is read by the machineConfig deamon (MCD) that is running on each cluster node
• the MCD aggregates the resulting kernel arguments from all the applied MCs and calls rpm-ostree kargs and passes the list of the kernel arguments
• rpm-ostree seems to write the kernel arguments to /boot/loader/entries/ostree-$version-rhcos.conf but I'm unsure how to tell which is the "current" version

rpm-ostree also allows you to print the current kernel arguments with rpm-ostree kargs but I'm unsure if it's realistic to call this from the scanner because of library loading issues and all that.

Before digging even more about how the ostree works, is using /proc/cmdline acceptable? We seem to be using the proc filesystem for the sysctls, so..

[evgenyz]

Using /proc/cmdline should be acceptable. And, moreover, filecontent54 probe has support for "offline" mode (for actual /host/proc/cmdline).

[jhrozek]

Thanks for the reply. Do I read it right that you don't need any more info from me and would change the probes? Or would you like us to propose a patch? Either way is OK, if you want us to propose a patch, I might need some help though :-)

[redhatrises]

An auditor needs to check that a couple of things here...

1. that the kernel is configured correctly
2. that the system is running in the correct state

/proc/cmdline answers 2 but doesn't answer 1

[evgenyz]

• rpm-ostree seems to write the kernel arguments to /boot/loader/entries/ostree-$version-rhcos.conf but I'm unsure how to tell which is the "current" version

Is there /boot/grub2/grubenv or some other equivalent? It should contain something like that:

# GRUB Environment Block
saved_entry=41174c3a27414f3ea5cead0dcf5b73b6-5.5.7-200.fc31.x86_64
menu_auto_hide=1
boot_success=1
kernelopts=root=/dev/mapper/fedora_localhost--live-root ro resume=...
boot_indeterminate=0

[jhrozek]

On Tue, Mar 17, 2020 at 08:15:58AM -0700, Gabe Alford wrote: An auditor needs to check that a couple of things here... 1) that the kernel is configured correctly 2) that the system is running in the correct state `/proc/cmdline` answers 2 but doesn't answer 1

Fair enough, but since configuration changes on RHCOS are applied during boot time, doesn't correctness of 2) imply that 1) is correct? The devil advocate in my would also say that the current method of checking the grub config only answers 2) but not 1) :-)

[jhrozek]

On Tue, Mar 17, 2020 at 08:54:04AM -0700, Evgeny Kolesnikov wrote: > * `rpm-ostree` seems to write the kernel arguments to `/boot/loader/entries/ostree-$version-rhcos.conf` but I'm unsure how to tell which is the "current" version Is there `/boot/grub2/grubenv` or some other equivalent? It should contain something like that: ``` # GRUB Environment Block saved_entry=41174c3a27414f3ea5cead0dcf5b73b6-5.5.7-200.fc31.x86_64 menu_auto_hide=1 boot_success=1 kernelopts=root=/dev/mapper/fedora_localhost--live-root ro resume=... boot_indeterminate=0 ```

There is /boot/loader/entries/ostree-$version-rhcos.conf: ``` sh-4.4# ls /boot/loader/entries/ostree-* /boot/loader/entries/ostree-1-rhcos.conf /boot/loader/entries/ostree-2-rhcos.conf sh-4.4# cat /boot/loader/entries/ostree-1-rhcos.conf title Red Hat Enterprise Linux CoreOS 44.81.202001241431.0 (Ootpa) (ostree:1) version 1 options $ignition_firstboot rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.0/rhcos/7305df0ba79e48eed6b6bcf3c2ba09ff0d4ab459c70b8560c7a1e2d0ffa6149b/0 ignition.platform.id=aws linux /ostree/rhcos-7305df0ba79e48eed6b6bcf3c2ba09ff0d4ab459c70b8560c7a1e2d0ffa6149b/vmlinuz-4.18.0-147.3.1.el8_1.x86_64 initrd /ostree/rhcos-7305df0ba79e48eed6b6bcf3c2ba09ff0d4ab459c70b8560c7a1e2d0ffa6149b/initramfs-4.18.0-147.3.1.el8_1.x86_64.img sh-4.4# cat /boot/loader/entries/ostree-2-rhcos.conf title Red Hat Enterprise Linux CoreOS 44.81.202002070830-0 (Ootpa) (ostree:0) version 2 options $ignition_firstboot rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.0/rhcos/69e98189a9e90206c11381785234c0617ffc3f4188a269f83ab8b11d67ac3f39/0 ignition.platform.id=aws linux /ostree/rhcos-69e98189a9e90206c11381785234c0617ffc3f4188a269f83ab8b11d67ac3f39/vmlinuz-4.18.0-147.5.1.el8_1.x86_64 initrd /ostree/rhcos-69e98189a9e90206c11381785234c0617ffc3f4188a269f83ab8b11d67ac3f39/initramfs-4.18.0-147.5.1.el8_1.x86_64.img ``` I can ask our RHCOS colleagues how do I find the most recent version..

[jhrozek]

I checked and the recommended way is to call 'rpm-ostree kargs'. This should be doable through D-bus as well. I'm going to find the exact invocation..

[redhatrises]

On Tue, Mar 17, 2020 at 08:15:58AM -0700, Gabe Alford wrote: An auditor needs to check that a couple of things here... 1) that the kernel is configured correctly 2) that the system is running in the correct state /proc/cmdline answers 2 but doesn't answer 1
Fair enough, but since configuration changes on RHCOS are applied during boot time, doesn't correctness of 2) imply that 1) is correct? The devil advocate in my would also say that the current method of checking the grub config only answers 2) but not 1) :-)

The problem lies in the fact that you can check how it is running in the moment by looking at /proc, but you also need to check and verify that this won't change to an uncompliant state when rebooted.

[jhrozek]

Got it. Let's try to probe the status with D-Bus, then.

[evgenyz]

@redhatrises Oh, then I have very bad news for you about all rules that use environmentvariable58 probe.

[evgenyz]

On the topic. Why can't we have two ANDed checks, one for checking the run-time state and another for checking the persistent configuration?

[redhatrises]

@redhatrises Oh, then I have very bad news for you about all rules that use environmentvariable58 probe.

Already aware of those. Thank you. Already aware of those rules needed to be expanded... we have more than just those that need runtime checks too.

On the topic. Why can't we have two ANDed checks, one for checking the run-time state and another for checking the persistent configuration?

This should be done in OVAL like many of the sysctl checks vs XCCDF. But there should be no problem, just need to make sure that the OVAL does both.

[jhrozek]

@redhatrises Oh, then I have very bad news for you about all rules that use environmentvariable58 probe.

Already aware of those. Thank you. Already aware of those rules needed to be expanded... we have more than just those that need runtime checks too.

On the topic. Why can't we have two ANDed checks, one for checking the run-time state and another for checking the persistent configuration?

This should be done in OVAL like many of the sysctl checks vs XCCDF. But there should be no problem, just need to make sure that the OVAL does both.

Sorry to be dense, but can you expand on this? For this particular check, would you like us to have an AND-ed check that checks both proc/cpuinfo and the kernel arguments from rpm-ostree or only one of them?

[redhatrises]

@redhatrises Oh, then I have very bad news for you about all rules that use environmentvariable58 probe.

Already aware of those. Thank you. Already aware of those rules needed to be expanded... we have more than just those that need runtime checks too.

On the topic. Why can't we have two ANDed checks, one for checking the run-time state and another for checking the persistent configuration?

This should be done in OVAL like many of the sysctl checks vs XCCDF. But there should be no problem, just need to make sure that the OVAL does both.

Sorry to be dense, but can you expand on this? For this particular check, would you like us to have an AND-ed check that checks both proc/cpuinfo and the kernel arguments from rpm-ostree or only one of them?

Yes exactly! An AND-ed OVAL check that ensures the runtime and static kernel configurations are the same.

[evgenyz]

Let's just back up a bit.

So, in the given example, there are two possible ways to boot the machine:

sh-4.4# ls /boot/loader/entries/ostree-*
/boot/loader/entries/ostree-1-rhcos.conf
/boot/loader/entries/ostree-2-rhcos.conf

Each file contain options entry, which we could analyse using filecontent54 probe. The only question left — which of these configurations would be loaded on the next boot?

Which bootloader is in use there? How the boot process of CoreOS looks like?