[featurerequest] systemd hardening

[juju4]

I did an ansible role to install kunai and added systemd hardening inside
https://github.com/juju4/ansible-kunai/blob/main/templates/systemd-override-kunai.service.j2
It worked fine on a local system (debian12) but not in github ci/runner (ubuntu 24.04, 22.04).

Any plan to have it upstream? else any advice?
Thanks

[qjerome]

Hey @juju4 !
Do you have any output log that may help identifying why it is failing ?
I may have an idea though ... maybe the Linux kernels used in GH CI don't have eBPF LSM enabled. This is straightforward to verify: one just have to read /sys/kernel/security/lsm if it contains bpf it means it should work, if not it won't work.

[juju4]

Example fail
https://github.com/juju4/ansible-kunai/actions/runs/13229350703/job/36924319727#step:11:39

Feb 09 20:30:29 fv-az1340-745 systemd[1]: Starting 00-kunai.service - Kunai Service...
Feb 09 20:30:29 fv-az1340-745 systemd[1]: Started 00-kunai.service - Kunai Service.
Feb 09 20:30:29 fv-az1340-745 kunai[3655]: thread 'main' panicked at kunai/src/bin/main.rs:2513:18:
Feb 09 20:30:29 fv-az1340-745 kunai[3655]: called `Result::unwrap()` on an `Err` value: OpenError { io_error: Os { code: 13, kind: PermissionDenied, message: "Permission denied" } }
Feb 09 20:30:29 fv-az1340-745 kunai[3655]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Feb 09 20:30:29 fv-az1340-745 systemd[1]: 00-kunai.service: Main process exited, code=exited, status=101/n/a

but it does not say PermissionDenied on what.

I got a similar error previously and solved it (adding @PRIVILEGED and @debug to SystemCallFilter) but likely for a different place.
All error messages where kept in the template as reference
https://github.com/juju4/ansible-kunai/blob/main/templates/systemd-override-kunai.service.j2

# Error: map error: failed to create map `BPF_PROG_TRACK` with code -1
# Error: map error: failed to create map `KUNAI_EVENTS` with code -1
#  called `Result::unwrap()` on an `Err` value: OpenError { io_error: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" } }
SystemCallFilter=@system-service @privileged @debug

Strange, I have kunai working on Debian-12 (Proxmox 8 kernel; at least have some output in events.log) but no bpf

$ cat /sys/kernel/security/lsm
lockdown,capability,yama,apparmor

That's one side where a health check along a config check option would be useful and highlight any gap in capabilities.

[qjerome]

After investigation, the panic occurs very early:

kunai/kunai/src/bin/main.rs

Line 2513 in dfad346

.unwrap();

I will fix that error so that it is more explicit :)

This is because Kunai fails at opening underlying perf event buffer ... My guess is that Ubuntu distros shipped by GH actions comes with a custom Linux kernel with a reduced feature set. Maybe to make them faster and/or reduce their attack surface. TBH, I never tried to run Kunai in GH runners ... I just have a workflow testing compatibility with several Linux kernel but for that I am using Qemu in full emulation mode (because GH runners don't have KVM either). If you really need to make a CI/CD integration with Kunai on GH I would recommend using a self-hosted runner and/or run it in Qemu (without acceleration) but it is going to be damn slow.

To clarify things, Kunai in classic mode can definitely run without bpf in /sys/kernel/security/lsm. However, if one wants to run in harden mode, bpf must (in theory) be in /sys/kernel/security/lsm. If you managed to make it run in harden mode without bpf in /sys/kernel/security/lsm probably I need to investigate and understand why !

[juju4]

added some extra check in current ci
I confirm than harden mode install fails
https://github.com/juju4/ansible-kunai/actions/runs/13619056283/job/38065917415#step:6:654

strangely the kernel config file has "CONFIG_BPF_LSM=y"
https://github.com/juju4/ansible-kunai/actions/runs/13619056283/job/38065917415#step:12:30

bpftool output
https://github.com/juju4/ansible-kunai/actions/runs/13619444475/job/38066794037#step:14:33

For github action, it seems mostly qemu in docker options in https://github.com/marketplace?query=qemu&type=actions
It seems nested virtualization is supported by github runner since 2024 (https://github.com/actions/runner-images/discussions/7191). Any good actions you know?

[qjerome]

Unfortunately no, I don't know any good qemu action ... I am using a custom script to make that test with qemu: https://github.com/kunai-project/kunai/blob/main/scripts/ci/test_kernel.sh

This CONFIG_BPF_LSM=y is just the kernel compile option, meaning the feature is available in you kernel but it is not activated, through kernel boot argument lsm=bpf....

[juju4]

Confirming no bpf in boot command line for github runner
https://github.com/juju4/ansible-kunai/actions/runs/13742228038/job/38432788462#step:12:49

what's strange imho is I still have data
https://github.com/juju4/ansible-kunai/actions/runs/13742228038/job/38432788462#step:19:20
just that kunai does not run anymore if systemd hardening is enabled (and --harden not set)
https://github.com/juju4/ansible-kunai/actions/runs/13751387986/job/38452779064#step:11:39

[qjerome]

Confirming no bpf in boot command line for github runner https://github.com/juju4/ansible-kunai/actions/runs/13742228038/job/38432788462#step:12:49

what's strange imho is I still have data https://github.com/juju4/ansible-kunai/actions/runs/13742228038/job/38432788462#step:19:20

Until here, I would say all good because in that test you don't run in harden mode (unless I missed something). lsm=bpf is needed ONLY AND ONLY IF we want to use harden mode via --harden CLI option or appropriate setting in config. That is to say, if running in regular mode, we DO NOT NEED lsm=bpf.

just that kunai does not run anymore if systemd hardening is enabled (and --harden not set) https://github.com/juju4/ansible-kunai/actions/runs/13751387986/job/38452779064#step:11:39

OK, if I understand correctly in here this test you are configuring the service with https://github.com/juju4/ansible-kunai/blob/main/templates/systemd-override-kunai.service.j2 ? Is this correct ?
I see that you apply a lot of restrictions to the service and probably (at least) one of them make kunai failing because it cannot do what it wants. Either it is a bug in kunai or your service configuration is too restrictive and needs to be changed. Anyway the panic you get is not explicit and should be addressed to understand exactly what the issue is.

[juju4]

OK, if I understand correctly in here this test you are configuring the service with https://github.com/juju4/ansible-kunai/blob/main/templates/systemd-override-kunai.service.j2 ? Is this correct ? I see that you apply a lot of restrictions to the service and probably (at least) one of them make kunai failing because it cannot do what it wants. Either it is a bug in kunai or your service configuration is too restrictive and needs to be changed. Anyway the panic you get is not explicit and should be addressed to understand exactly what the issue is.

yes. but those restrictions work as is in my test environments (debian12-proxmox8, ubuntu22.04) or at least, not a complete fail as it happens in github ci.
agreed on more explicit error message.