`-a exclude,always -F msgtype=NETFILTER_CFG` doesn't seem to work

[cgwalters]

Not sure if this pilot error, a kernel or userspace issue. Basically in the container world (docker/podman/Kubernetes) the norm is for launching a container to involve setting up some iptables rules, a veth device etc. And particularly in Kubernetes, the workloads are commonly highly dynamic and ephemeral. The combination of these floods the default audit log and quickly rotates out things we do care about like e.g. SELinux AVC denials.

I'm trying this on RHEL CoreOS (RHEL 8.4):

$ rpm -q audit
audit-3.0-0.17.20191104git1c2f876.el8.x86_64
$ uname -a
Linux cosa-devsh 4.18.0-305.3.1.el8_4.x86_64 #1 SMP Mon May 17 10:08:25 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
$ rpm-ostree status -b
State: idle
BootedDeployment:
* ostree://457db8ff03dda5b3ce1a8e242fd91ddbe6a82f838d1b0047c3d4aeaf6c53f572
                   Version: 48.84.202106091622-0 (2021-06-09T16:25:42Z)
$ cat /etc/audit/rules.d/10-no-iptables.rules
-a exclude,always -F msgtype=NETFILTER_CFG
-a exclude,always -F msgtype=PROCTITLE
$ augenrules && kill -HUP $(pidof auditd)

but am not succeeding at filtering those out of the audit log; every run of e.g.

$ podman run --rm -ti docker.io/busybox true

still involves a storm of audit logs like:

type=PROCTITLE msg=audit(1623866492.232:409): proctitle=2F7362696E2F69707461626C6573002D74006E6174002D4400504F5354524F5554494E47002D730031302E38382E302E3135002D6A00434E492D64303664396238633
4653734663862666430633961643864002D6D00636F6D6D656E74002D2D636F6D6D656E74006E616D653A2022706F646D616E222069643A202232663262
type=NETFILTER_CFG msg=audit(1623866492.252:410): table=nat:2;CNI-d06d9b8c4e74f8bfd0c9ad8d:59 family=2 entries=60 op=nft_unregister_rule pid=6273 subj=unconfined_u:system_r:iptables_t:s0-s0
:c0.c1023 comm="iptables"
type=NETFILTER_CFG msg=audit(1623866492.252:410): table=nat:2;CNI-d06d9b8c4e74f8bfd0c9ad8d:59 family=2 entries=61 op=nft_unregister_rule pid=6273 subj=unconfined_u:system_r:iptables_t:s0-s0
:c0.c1023 comm="iptables"
type=NETFILTER_CFG msg=audit(1623866492.252:410): table=?:0;?:0 family=0 entries=226 op=nft_register_gen pid=6273 subj=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 comm="iptables"
type=SYSCALL msg=audit(1623866492.252:410): arch=c000003e syscall=46 success=yes exit=140 a0=3 a1=7ffc89ac9be0 a2=0 a3=7ffc89ac9bcc items=0 ppid=6216 pid=6273 auid=1000 uid=0 gid=0 euid=0 s
uid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=3 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 key=(null)ARCH=x86_64 SYSCALL=sendm
sg AUID="core" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PROCTITLE msg=audit(1623866492.252:410): proctitle=2F7362696E2F69707461626C6573002D74006E6174002D4600434E492D643036643962386334653734663862666430633961643864002D2D77616974
type=NETFILTER_CFG msg=audit(1623866492.258:411): table=nat:2;CNI-d06d9b8c4e74f8bfd0c9ad8d:59 family=2 entries=0 op=nft_unregister_chain pid=6274 subj=unconfined_u:system_r:iptables_t:s0-s0
:c0.c1023 comm="iptables"
type=NETFILTER_CFG msg=audit(1623866492.258:411): table=?:0;?:0 family=0 entries=227 op=nft_register_gen pid=6274 subj=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 comm="iptables"
type=SYSCALL msg=audit(1623866492.258:411): arch=c000003e syscall=46 success=yes exit=120 a0=3 a1=7ffe6745dfd0 a2=0 a3=7ffe6745dfbc items=0 ppid=6216 pid=6274 auid=1000 uid=0 gid=0 euid=0 s
uid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=3 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 key=(null)ARCH=x86_64 SYSCALL=sendm
sg AUID="core" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PROCTITLE msg=audit(1623866492.258:411): proctitle=2F7362696E2F69707461626C6573002D74006E6174002D5800434E492D643036643962386334653734663862666430633961643864002D2D77616974

[stevegrubb]

I added -a exclude,always -F msgtype=NETFILTER_CFG to my system and did some things that alter the network configuration. This was on a 5.12.10 kernel. The suppression was working fine there. Removing the rule brought network config events back. Of course, this did not involve any containers.

[cgwalters]

OK here's a quick test I did on current Fedora too. I totally believe you that this works and it seems likely there's pilot error here, but I can't see it. I also tried with the -a always,exclude -F msgtype=CRYPTO_KEY_USER from /usr/share/audit/sample-rules/20-dont-audit.rules but I still see that every time I ssh localhost.

Are there any tests for this code? I looked at recent PRs and I don't see any CI setup, and I'm also not seeing any audit selftests in the kernel tree, nor package-level tests in the Fedora dist-git, etc.

root@cosa-devsh ~]# cat /etc/audit/audit.rules
## This file is automatically generated from /etc/audit/rules.d
-D


-a task,never
-a exclude,always -F msgtype=NETFILTER_CFG

[root@cosa-devsh ~]# uname -a
Linux cosa-devsh 5.12.11-300.fc34.x86_64 #1 SMP Wed Jun 16 15:47:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@cosa-devsh ~]# vi /etc/audit/rules.d/iptables-quiet.rules
[root@cosa-devsh ~]# cat /etc/audit/rules.d/iptables-quiet.rules
-a exclude,always -F msgtype=NETFILTER_CFG
[root@cosa-devsh ~]# cat /etc/audit/audit.rules
## This file is automatically generated from /etc/audit/rules.d
-D


-a task,never
-a exclude,always -F msgtype=NETFILTER_CFG
[root@cosa-devsh ~]# augenrules && kill -HUP $(pidof auditd)
[root@cosa-devsh ~]# iptables -I INPUT -s 1.1.1.1 -j ACCEPT
[root@cosa-devsh ~]# tail /var/log/audit/audit.log 
type=DAEMON_CONFIG msg=audit(1624298544.801:2128): op=reconfigure state=changed auid=1000 pid=1248 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 res=successAUID="core"
type=NETFILTER_CFG msg=audit(1624298554.945:322): table=filter family=2 entries=4 op=xt_replace pid=2022 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 comm="iptables"
[root@cosa-devsh ~]# 

If nothing else jumps out here I can try adding some debugging to the kernel/userspace bits here.

[rgbriggs]

On 2021-06-21 12:20, Colin Walters wrote: Are there any tests for this code? I looked at recent PRs and I don't see any CI setup, and I'm also not seeing any audit selftests in the kernel tree, nor package-level tests in the Fedora dist-git, etc.

https://github.com/linux-audit/audit-testsuite

***@***.*** ~]# cat /etc/audit/audit.rules ## This file is automatically generated from /etc/audit/rules.d -D -a task,never

Here is your problem. Remove the line above. It is in one of the files under /etc/audit/rules.d/

[rgbriggs]

On 2021-06-21 12:45, Richard Guy Briggs wrote: On 2021-06-21 12:20, Colin Walters wrote: > -a task,never Here is your problem. Remove the line above. It is in one of the files under /etc/audit/rules.d/

See for background: https://pagure.io/fesco/issue/1311

[cgwalters]

Here is your problem. Remove the line above. It is in one of the files under /etc/audit/rules.d/

Hmm. I read through that and internally distilled it into "this line skips all other audit rules and defers solely to the built-in ones in auditd, so any custom ones are ignored" or so which kind of made sense. But then, removing it still didn't help.

I should have tried this earlier, but what did work is rebooting. Until now I'd been assuming the augenrules && auditctl --signal=HUP would be sufficient to re-initialize the state, but apparently not....and ah yes, I just now noticed --load in man augenrules. And that's the missing piece, apparently I'd just been reloading auditd.config but not the kernel rules. I am sure this was an "of course it works that way" to you all but...hmm, maybe some sort of cross-reference from man audit.rules that mentions e.g. "you should write a new file in /etc/audit/rules.d and run augenrules --load"?

[cgwalters]

FYI openshift/machine-config-operator#2633