bpi-r3 eth1 dropping packets

[gustavobsch]

Hi,

I'm running 6.2 kernel so I can use VLAN on the eth1 Fiber SFP and I'm observing a lot of packets getting dropped. Only the received ones. The SFP and Fiber were moved from another router that it was working just fine. So I don't think it's the tranceiver or fiber cable.

Has anyone else seen similar behavior on the bpi-r3?

# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::9065:f3ff:feec:b019  prefixlen 64  scopeid 0x20<link>
        ether 92:65:f3:ec:b0:19  txqueuelen 1000  (Ethernet)
        RX packets 11132122  bytes 10885212987 (10.1 GiB)
        RX errors 1  **dropped 875168**  overruns 0  frame 13784        <<<<<<< 875168 dropped packets
        TX packets 9577892  bytes 11333127550 (10.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 123  

# ethtool -S  eth1
NIC statistics:
     tx_bytes: 12214740723
     tx_packets: 10248274
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 10922473509
     rx_packets: 11473622
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 15184
     rx_checksum_errors: 1
     rx_flow_control_packets: 0
     rx_xdp_redirect: 0
     rx_xdp_pass: 0
     rx_xdp_drop: 0
     rx_xdp_tx: 0
     rx_xdp_tx_errors: 0
     tx_xdp_xmit: 0
     tx_xdp_xmit_errors: 0
     rx_pp_alloc_fast: 23145710
     rx_pp_alloc_slow: 17668
     rx_pp_alloc_slow_ho: 0
     rx_pp_alloc_empty: 17668
     rx_pp_alloc_refill: 388807
     rx_pp_alloc_waive: 0
     rx_pp_recycle_cached: 0
     rx_pp_recycle_cache_full: 0
     rx_pp_recycle_ring: 22421931
     rx_pp_recycle_ring_full: 219858
     rx_pp_recycle_released_ref: 909866

Thanks!

[frank-w]

I'm still doing basic tests and noticed that vlan aware bridge is not working on mt7531.

Only did ping tests with latest patches (series from felix). Have not yet seen packet drops.

Btw. I'm alone here,it will be better posting such questions in bananapi forum

[gustavobsch]

Thank you Frank, for now I will use another port for trunking. I realize this is a new board

[frank-w]

did a quick test using my H!fibre 1G SFP and kernel 6.2.0-rc1-bpi-r3:

root@bpi-r3:~# ip addr add 192.168.0.19/24 dev eth1
root@bpi-r3:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc mq state UP group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 92:65:f3:ec:b0:19 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.19/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::9065:f3ff:feec:b019/64 scope link 
       valid_lft forever preferred_lft forever
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: wan@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
6: lan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lanbr0 state UP group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
7: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master lanbr0 state DOWN group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
8: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master lanbr0 state DOWN group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
9: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master lanbr0 state DOWN group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
10: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1a:c2:57:ee:e2:1a brd ff:ff:ff:ff:ff:ff
11: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0c:43:26:60:00 brd ff:ff:ff:ff:ff:ff
12: wlan1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 82:0c:43:26:60:00 brd ff:ff:ff:ff:ff:ff
13: wwan0: <BROADCAST,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:d9:c2:2c:de:d2 brd ff:ff:ff:ff:ff:ff
14: lanbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 96:3f:c5:84:65:f0 brd ff:ff:ff:ff:ff:ff
15: wan.110@wan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether 10:c3:7b:4f:58:a4 brd ff:ff:ff:ff:ff:ff
root@bpi-r3:~# ping 192.168.0.10
PING 192.168.0.10 (192.168.0.10) 56(84) bytes of data.
64 bytes from 192.168.0.10: icmp_seq=1 ttl=64 time=0.861 ms
64 bytes from 192.168.0.10: icmp_seq=2 ttl=64 time=0.310 ms
64 bytes from 192.168.0.10: icmp_seq=3 ttl=64 time=0.294 ms
^C
--- 192.168.0.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2121ms
rtt min/avg/max/mdev = 0.294/0.488/0.861/0.263 ms
root@bpi-r3:~# iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.0.19 port 60022 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  79.9 MBytes   670 Mbits/sec    0   1.09 MBytes       
[  5]   1.00-2.00   sec  80.0 MBytes   671 Mbits/sec    0   1.14 MBytes       
[  5]   2.00-3.00   sec  77.5 MBytes   650 Mbits/sec    0   1.20 MBytes       
[  5]   3.00-4.00   sec  80.0 MBytes   671 Mbits/sec    0   1.33 MBytes       
[  5]   4.00-5.00   sec  77.5 MBytes   650 Mbits/sec    0   1.33 MBytes       
[  5]   5.00-6.00   sec  81.2 MBytes   682 Mbits/sec    0   1.33 MBytes       
[  5]   6.00-7.00   sec  80.0 MBytes   671 Mbits/sec    0   1.40 MBytes       
[  5]   7.00-8.00   sec  80.0 MBytes   671 Mbits/sec    0   1.40 MBytes       
[  5]   8.00-9.00   sec  80.0 MBytes   671 Mbits/sec    0   1.40 MBytes       
[  5]   9.00-10.00  sec  80.0 MBytes   671 Mbits/sec    0   1.40 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   796 MBytes   668 Mbits/sec    0             sender
[  5]   0.00-10.92  sec   793 MBytes   610 Mbits/sec                  receiver

iperf Done.
root@bpi-r3:~# ethtool -S  eth1
NIC statistics:
     tx_bytes: 872733324
     tx_packets: 574954
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 1770916
     rx_packets: 26026
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 0
     rx_checksum_errors: 0
     rx_flow_control_packets: 1021
     rx_xdp_redirect: 0
     rx_xdp_pass: 0
     rx_xdp_drop: 0
     rx_xdp_tx: 0
     rx_xdp_tx_errors: 0
     tx_xdp_xmit: 0
     tx_xdp_xmit_errors: 0
     rx_pp_alloc_fast: 26122
     rx_pp_alloc_slow: 9
     rx_pp_alloc_slow_ho: 0
     rx_pp_alloc_empty: 9
     rx_pp_alloc_refill: 407
     rx_pp_alloc_waive: 0
     rx_pp_recycle_cached: 0
     rx_pp_recycle_cache_full: 0
     rx_pp_recycle_ring: 26026
     rx_pp_recycle_ring_full: 0
     rx_pp_recycle_released_ref: 0
root@bpi-r3:~#
root@bpi-r3:~# ethtool -m eth1
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : Ethernet: 1000BASE-SX
        Encoding                                  : 0x01 (8B/10B)
        BR, Nominal                               : 1300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 550m
        Length (62.5um)                           : 270m
        Length (Copper)                           : 0m
        Length (OM3)                              : 0m
        Laser wavelength                          : 850nm
        Vendor name                               : OEM
        Vendor OUI                                : 00:90:65
        Vendor PN                                 : GLC-SX-MMD     _
        Vendor rev                                : _e
        Option values                             : 0x00 0x1a
        Option                                    : RX_LOS implemented
        Option                                    : TX_FAULT implemented
        Option                                    : TX_DISABLE implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : CSCGE1M14134
        Date code                                 : 220120
        Optical diagnostics support               : Yes
        Laser bias current                        : 3.684 mA
        Laser output power                        : 0.3136 mW / -5.04 dBm
        Receiver signal average optical power     : 0.3462 mW / -4.61 dBm
        Module temperature                        : 36.79 degrees C / 98.22 degrees F
        Module voltage                            : 3.3180 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : Off
        Laser rx power high warning               : Off
        Laser rx power low warning                : Off
        Laser bias current high alarm threshold   : 100.000 mA
        Laser bias current low alarm threshold    : 0.000 mA
        Laser bias current high warning threshold : 90.000 mA
        Laser bias current low warning threshold  : 0.100 mA
        Laser output power high alarm threshold   : 1.2589 mW / 1.00 dBm
        Laser output power low alarm threshold    : 0.0501 mW / -13.00 dBm
        Laser output power high warning threshold : 0.7943 mW / -1.00 dBm
        Laser output power low warning threshold  : 0.0631 mW / -12.00 dBm
        Module temperature high alarm threshold   : 90.00 degrees C / 194.00 degrees F
        Module temperature low alarm threshold    : -45.00 degrees C / -49.00 degrees F
        Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
        Module temperature low warning threshold  : -40.00 degrees C / -40.00 degrees F
        Module voltage high alarm threshold       : 3.8000 V
        Module voltage low alarm threshold        : 2.7000 V
        Module voltage high warning threshold     : 3.7000 V
        Module voltage low warning threshold      : 2.8000 V
        Laser rx power high alarm threshold       : 0.7943 mW / -1.00 dBm
        Laser rx power low alarm threshold        : 0.0079 mW / -21.02 dBm
        Laser rx power high warning threshold     : 0.5012 mW / -3.00 dBm
        Laser rx power low warning threshold      : 0.0126 mW / -19.00 dBm
root@bpi-r3:~# 

not very good throughput, but also no retransmitts and no error/dropped packets.

[frank-w]

i see you have no rx_flow_control_packets...maybe your other side/switch needs to enable pause??

[gustavobsch]

I did try enabling flow control on the other side, I also tried a different device and get the same result, lots of dropped packets. For now I will not use the SFP1