ICMPv6 packets getting sporadically malformed when fragmented over WAN

[Denton22]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [ x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [ x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

Setup:

Preface:
So in IPv6 intermediate Devices like Firewalls or Routers are not allowed to fragment packets. In the IPv6 specifications it is clearly stated that fragmentation should only occour on the end-devices (Server and Client) and they should avoid fragmentation with PMTUD. If a device however recievs an bigger packet, they will send an ICMPv6 Type 2 Code 0 Message "Too Big" wich will transmit the MTU of the Server and will trigger fragmentation on the Client.

This looks like this:

As you can see my Client is sending a packet with 1400bytes payload and the Servers responds with an "Too Big Message". My Client then fragments the icmp request but for reasons unknown to me, the Server does not respond to my then fragmented request and i get an error Message on my Client.

This however is fine, since opensene handles it correctly.

The BUG

If i ping the host ds.frankfurt.test-ipv6.com [2a01:7e01::f03c:94ff:fed0:4087] and the size of the ping is for example 2500bytes, sometimes it works and sometimes it does not. Actually most of the time it does not work and while I was writting this it suddenly started working. So first I was thinking that this is a clear cut case, but now i am very confused because of the sporadic nature of this bug and I would love some insight from someone that understands it better hopfully.

Here is an example of it Working:

And i can see in the Pcaps taken from WAN and LAN that the packet are the same on WAN and LAN, so ignoring the first ping getting lost, i can see no issue on the opensense side.

Pcap Download:
working_wan.zip

However most of the time, and reason for my report, ist that usually the packets on the WAN side, are getting malformed by opensense....

The first fragment, is missing 8 Bytes in the Payload:
LAN - 1448 bytes
WAN - 1440 bytes
This invalidates the icmpv6 checksum but most notably also adds a second ipv6 Fragmentation Header!

Pcap Download:
malformed_wan.zip

So either Opensense is not conform to the IPv6 specification and also tries to fragment the packet or there is some kind of error in the handling of this pakets.

To Reproduce

1. start capture via Opensense GUI on WAN and LAN:

2. run ping ds.frankfurt.test-ipv6.com -l 2500

If you get no response, you will most likely see in the WAN Pcap, that the packets are malformed.

Expected behavior

The packets on the WAN interface should not be malformed by opensene and opensense should not add a second ipv6 fragmentation header. This Ping should work all the time, nomatter the size specified.

Describe alternatives you considered

• It is not a driver / interface issue, since i can reproduce it on igc0 (Intel I225-V) and also on my ixl bond (Intel x-710)

-It is not a upstream ISP Router issue, since if the packet is malformed, it makes sense why it will drop the paket or can not handle the packet if there are multiple fragmentation headers

-Suricata and Traffic Shaping with FQ Codel seem to have no impact on this issue, the behaviour does not change if disabled or enabled

-The sporadic nature of this issue makes it hard to pinpoint.

Additional context

Previous Report: opnsense/core#8744

So this issue was hard for me to pinnpoint and I am no expert when it comes to ipv6, but something is clearly wrong here and I would appreciate it, if someone would test it on their end if it is reproducible. It did cost me alot of time to debug this because everything i looked at made no sense whatsoever or it started to work all of a sudden and then stopped again.

I really hope that someone here can reproduce it and find out why it is happening. I appreciate your time

Environment

Software version used and hardware type if relevant, e.g.:
Hardware: Minisofurm MS-01
i9-13900H + 32Gb ddr5
Intel x710
Intel I225-V
Bios Version
1.26

ISP Router: Fritzbox 7590
DSL Vectoring with PPPoe

Versions
OPNsense 25.1.7_4-amd64
FreeBSD 14.2-RELEASE-p3
OpenSSL 3.0.16

[Monviech]

Hello,

I have tested this with my own connection which is quite similiar to yours (IPv6, DTAG - VDSL250, Zyxel modem). I also pinged from a Windows client just like you.

I see the same thing as you, sometimes the ping works, and sometimes it doesn't. And when it doesn't work, I see the same duplicate fragmentation headers in Wireshark.

The question is now, under which circumstances does the ping stop working, and why does it work sometimes. That's the crucial piece of the puzzle to get this further. I have no idea yet, maybe you do?

[Denton22]

@Monviech I am unsure,I am just a Network Engineer and no developer.
However to me it looks like, that OpenSense or FreeBSD is not conform to the IPv6 Specifications mentioned in RFC 8200

4.5. Fragment Header
unlike IPv4, fragmentation in IPv6 is performed only by source nodes, not by routers along a packet's delivery path Source

The RFC mentions Routers, but also mentiones that the Fragmentation is only performend on the source nodes wich a Firewall is not. So I would argue that OpenSense should NEVER fragment IPv6 Traffic that is passing through.

However since this is only happening sporadically, it is probably just a bug and not ignoring the RFC....

What I find interessting, the malformed packets are missing 8 bytes in the payload, 8 bytes is exactly the missing MTU on the ISP Routers WAN Interface because of PPPOE.

So either opensense is trying to Fragment the packet because its PMTU Cache suggests that the MTU is 1492, but this would violate the RFC, but adding another icmp Fragmentation header is wrong in so many ways that I find it hard to find reason in it.

What I would expect to happen ist the following:
ping with 2000bytes

1. my Client sending one fragmented 1500bytes Payload ICMP packet and one 500bytes fragmented paket to its DFG
2. DFG is opensense, Opensense looks in Routing table and checks if MTU is fine with the paket size, wich it is since Opensense has WAN and LAN MTU at 1500 ( If MTU would not be enough, opensense SHOULD answer ICMP Too Big to my Client )
3. My ISP Router is recieving the pakets (without manipulation) and sees that the WAN MTU of 1492 is not enough to transfer the first fragment because its payload is 1500bytes and it will send to my client an ICMP Too Big
4. My Client recievs the ICMP Too Big Message with the ISP Routers WAN MTU of 1492 and adjusts its PMTU Cache to reflect this change and resends the ICMP Request with the first Fragment having a payload of 1492
5. Ping now works, if for example we ping the mtu1280 Host , the process is the same, but the Server will additionaly sent ICMP to Big with MTU 1280 and my client should reflect that in the PMTU Cache.

BTW you can check the PMTU Cache in Windows with
netsh interface ipv6 show destinationcache
and in more details for a ip:
netsh interface ipv6 show destinationcache address=2a01:7e01::f03c:94ff:fed0:4087

[Denton22]

@Monviech Ok, so changing the WAN MTU to 1400 has the effect, that the ping "ping ds.frankfurt.test-ipv6.com -l 2000" is working relliably, the first reply is lost downstream but that should be of no concern to us.

The reason as to why the connection works now is interessting....

Changing the MTU on the WAN Interface is actually changing the MTU of the Clients with the Router Advertisments on Tracked Interfaces.
You can verify this if you set the WAN MTU to 1400 and run the command "netsh interface ipv6 show subinterface" on the client. You will see that your client has changed its MTU to 1400 aswell for IPv6 only.
Leaving the MTU field empty on the WAN, just removes this information from the RA and the Client is stuck with the last setting that has been pushed.
Setting WAN MTU to 1500 again also pushes this to the Client aswell.

This behavior is questionable, but if this is communicated on the wiki or GUI, I would not mind it and I could see the benefit of it, I however didnt know that this is a thinq...

But now to the fun part :)

1. Setting WAN MTU to 1400
2. OverwrittingClient MTU from 1400 -> 1500 with "netsh interface IPv6 set subinterface "Ethernet 3" mtu=1500"
3. Start Capturing
4. Run ping ds.frankfurt.test-ipv6.com -l 2000 on the client
5. CHAOS

On the LAN side, i can verify that the pakets get sent how you expect them to:

On the WAN side, the "BIG" fragment is just gone, not malformed but just outright deleted:

What should happen, is that opensense sends an ICMP too BIG Message to my client to lower the PMTU for this connection to 1400 (since WAN MTU is 1400) so that my Client can resend it.

So in my opinion Opensense is trying to fragment IPv6 packets and if the Payload is bigger then the PMTU / MTU it will either delete the packet or malform it.

Also it seems like opensene has no implementation of ICMPv6 "Too Big" and IPv6 Fragmentation is not IPv6 RFC compliant.

I am unsure how deep this issue goes, is the whole implementation for IPv6 Fragmentation broken or only the icmpv6 fragmentation?

I am also unsure if this is a opensense issue or a FreeBSD issue, the only thing I know is, that I do not have the skill to fix the code and to get it fixed I need your help.

I will also try to get my hands on a Fortigate , Checkpoint or any other Enterprise Firewall to see how they handle this.

Would love to get some input from you

[Denton22]

Ok so my final remark for this topic for today. I have now tested how the AVM Fritzbox 7590 does it (My ISP Router). I have connected my PC to the Fritzbox and wanted to confirm what MTU it distributes via RA / SLAC and what happens when i force a MTU that is higher then the WAN MTU.

So the Fritzbox advertises an IPv6 MTU of 1492bytes to the Client, and with this setting fragmentation does not happen. If i run a ping like before it works fine. So as a workaround, if I set the WAN MTU in Opensense to 1492 it works identical like on the ISP Router.

This means that it is actually a configuration error to set the WAN MTU to 1500. But Opensene should if the MTU value is not set and IPv6 is set to DHCPv6 it should learn the correct MTU via the Router Advertisment from the Upstream Router. This is currently not the Case and if it is empty it assumes 1500bytes. (Maybe a Feature Request?!)

However we still have an Bug here, Opensene needs to stop messing with IPv6 Fragments if they exceed the MTU of the Interface and fragmentation is need, opensene need to inform the client with ICMPv6 Typ 2 "Too Big" Messages that fragmentation is necessary and send the correct MTU.

The Fritzbox handels it well:

1. Fritzbox advertises MTU of 1492 via RA / SLAC to the PC
2. I overwritte the MTU on the Client with "netsh interface IPv6 set subinterface "Ethernet 3" mtu=1500"
3. I ping an IPv6 host who allows fragmentation , for example :
   ping blog.fefe.de -l 2000
   or
   ping ds.frankfurt.test-ipv6.com -l 2000

The Result must look like this:

PCAP Download:
fritzbox_correct_ipv6.zip

As you can see the fritzbox implements IPv6 correctly and sents an ICMPv6 Typ 2 Message to my Client with MTU hint of 1492 so that my Client can resend the packet with the correct MTU.

to summarise

1. OpenSense mustn't fragment IPv6 Traffic that is passing through the Firewall (RFC 8200)
2. Opensene must implement ICMPv6 Typ 2 to inform clients, when a Frame has been recieved that is bigger then the MTU of the destination Interface so that the Client knows that it needs to resend the paket with the MTU value transmitted via ICMPv6 Type 2 (RFC 8200)
3. [Feature Request]Opensene should honor the IPv6 MTU that upstream routers transmitt via Router Advertisments when the Interface Configuration is set to DHCPv6 and no MTU has been specified for the Interface

If you can fix and implement these 3 points, it will work how it is intended

[AdSchellevis]

what effect has disable scrubbing on your issue? (Firewall: Settings: Normalization / Disable interface scrub)

[Denton22]

@AdSchellevis Disabling FW Normalization like you mentioned, changes nothing in the IPv6 flow:

ISP Router LAN MTU= 1500
ISP Router WAN MTU = 1492
Opensene WAN MTU = 1500 --> RA to Client 1500 MTU
Opensene LAN MTU = 1500

Client IPv6 MTU = 1500

Opensene LAN side:

Opensene WAN Side:

The packets on WAN are still missing 8 bytes and opensense still adds a second fragmentation header.

For IPv4 this makes everything wierd.... So if interface scrub is disabled the paket flow is fine, but opensene does not transmitt the recieved ICMP Reply from WAN to LAN and my client gets no answer.

Opensene LAN side:

Opensene WAN side:

This reply is just deleted on WAN, it makes no sense to me. Sure the ISP Gateway seems to send 3 Fragments instead of 2 but the sizes of these fragments do not exceed MTU on any interface.

Enabling Normalisatzion again fixes this, and the flow is working fine

EDIT

Ok so it does make sense to me, since disabling Normalisiatzion disables ICMP statefullness and I get FW Rule blocks:

[AdSchellevis]

For ipv4 I'm quite sure it works as designed, for ipv6 we might be chasing ghosts here, although I'm not sure. If you want to exclude the firewall in your tests, for ipv6, you should also be able to disable pf (pfct -d and pfctl -e to enable) to see if fragmentation still happens. To some degree I expect a logical explanation, but surprises exist.

[Denton22]

For ipv4 I'm quite sure it works as designed, for ipv6 we might be chasing ghosts here, although I'm not sure. If you want to exclude the firewall in your tests, for ipv6, you should also be able to disable pf (pfct -d and pfctl -e to enable) to see if fragmentation still happens. To some degree I expect a logical explanation, but surprises exist.

I am with you that it is correct in IPv4

But I am certain that Opensense (or FreeBSD) is fragmenting IPv6 packets and that is just plain wrong behaviour. In IPv6 the firewall / Router should NEVER mess with the packet. Instead it should send these ICMPv6 Typ 2 Messages to inform the Client.

I will test the commands (pfct) later today

[Denton22]

@AdSchellevis i can confirm that the issue is not happening when i disable the firewall with pfctl -d

I have also set the MTU on WAN to 1500 again

Without pf enabled the LAN Interface IPv6 sents ICMPv6 Typ 2 "Too Big" as seen in the following picture:

On the WAN Interface you will not see the first fragment, this is most likly the case, because the Firewall Device knows the Path MTU already, but i do not know how to check the PMTU cache in FreeBSD.

I would need to test this more precisly, what happens if there are no Gateway Monitoring enabled, what happens when i reboot the system and clear the caches etc...

But this works, the goal is that the client is informed that the Fragments are too big and this happens here, it does not matter if opensene or the isp router inform the client.

So does that mean it is a opensense issue and not a freebsd one?

[AdSchellevis]

well, if the behavior is different with pf disabled, you might try to create a very simple ruleset (pass all) to see if there's something in our default ruleset (although I don't expect that's the case).

[Denton22]

Well i added an ipv4+ipv6 rule with any on both LAN and WAN, but it didnt changed the outcome.

[AdSchellevis]

just to be sure, I mean a pf.conf file with only pass all in it to exclude any defaults from our end (loaded via pfctl -f /path/to/my/file)

[Denton22]

just to be sure, I mean a pf.conf file with only pass all in it to exclude any defaults from our end (loaded via pfctl -f /path/to/my/file)

Sure can do that, but would you be so kind to tell me the path of my current config so that i can back it up first? It is still my live Home Network and not a lab setup

[Monviech]

Your current configuration exists in /conf/config.xml

Loading a file into PF manually won't change the main configuration file of the firewall, though.

Any "Apply" in the Firewall/Rules pages, restarting the packet filter in the GUI under services, or a reboot restores the correct ruleset.