Performance drop when adding packetizer

[wnagele]

I've been banging my head against this for two days now. Reaching out in the hopes of some guidance/pointers. 🙇

I am using a ULXS3 board and have connected two LAN8720 PHY modules via RMII. So far so good - the idea was to try to do some basic packet mangling by ingesting packets on one port, mangle and output on the other.

I set up the PHY + MAC like so:

self.ethphy1 = LiteEthPHYRMII(
    clock_pads             = platform.request("eth_clocks", 0),
    pads                   = platform.request("eth", 0),
    refclk_cd              = None,
    with_hw_init_reset     = False,
)
self.ethmac1 = LiteEthMACCore(
    phy = self.ethphy1,
    dw  = 8,
)
self.ethmac1 = ClockDomainsRenamer({
    "eth_tx": "ethphy1_eth_tx",
    "eth_rx": "ethphy1_eth_rx",
})(self.ethmac1)

self.ethphy2 = LiteEthPHYRMII(
    clock_pads             = platform.request("eth_clocks", 1),
    pads                   = platform.request("eth", 1),
    refclk_cd              = None,
    with_hw_init_reset     = False,
)
self.ethmac2 = LiteEthMACCore(
    phy = self.ethphy2,
    dw  = 8,
)
self.ethmac2 = ClockDomainsRenamer({
    "eth_tx": "ethphy2_eth_tx",
    "eth_rx": "ethphy2_eth_rx",
})(self.ethmac2)

And then I try to create the stream as follows (in one direction to test).

This one works fine - full 100Mbit line rate.

self.eth_packetizer = LiteEthMACPacketizer(8)
self.eth_depacketizer = LiteEthMACDepacketizer(8)
self.comb += [
    self.ethmac1.source.connect(self.eth_depacketizer.sink),
    self.eth_depacketizer.source.connect(self.eth_packetizer.sink),
    self.eth_packetizer.source.connect(self.ethmac2.sink),
]

Here the performance drops to about half. From about 50Mbit onwards packet drops start to occur.

self.eth_packetizer = LiteEthMACPacketizer(8)
self.eth_depacketizer = LiteEthMACDepacketizer(8)
self.eth_packetizer2 = LiteEthMACPacketizer(8)
self.comb += [
    self.ethmac1.source.connect(self.eth_depacketizer.sink),
    self.eth_depacketizer.source.connect(self.eth_packetizer.sink),
    self.eth_packetizer.source.connect(self.eth_packetizer2.sink),
    self.eth_packetizer2.sink.sender_mac.eq(self.eth_packetizer.sink.sender_mac),
    self.eth_packetizer2.sink.target_mac.eq(self.eth_packetizer.sink.target_mac),
    self.eth_packetizer2.sink.ethernet_type.eq(self.eth_packetizer.sink.ethernet_type),
    self.eth_packetizer2.source.connect(self.ethmac2.sink),
]

[enjoy-digital]

Hi @wnagele,

can you explain the purpose of cascading the 2 packetizers? The packetizer has some idle cycles, adding a FIFO between the two packetizer could help getting better throughput.

[wnagele]

This was just an example with no clear purpose to highlight the issue. I also tried adding a PacketFIFO directly after the ingress MAC but with no change.

[wnagele]

Basically what seems to happen here is an overrun as the output bandwidth is > input bandwidth. For bursts a FIFO works fine to compensate. But if I run the input hard the issue is that the whole stream/pipeline locks up and doesn't recover. Is there a default way in LiteEth/LiteX to drop further frames/packets into a stream when the output is overloaded?