What is the expected behavior of the --skip-rx-copy flag for UDP and TCP?

[manedurphy]

I'd like to solidify my understanding of the expected throughput improvement as per the --skip-rx-copy. I can see that the buffer size for UDP is recalculated to size = sizeof(sec) + sizeof(usec) + sizeof(pcount);, which reduces the number of bytes that are copied into the input buffer, but I don't see the same approach for TCP. Are TCP still copied to the input buffer as before? Were any improvements seen for TCP at all? There are new concepts here that I am learning about, so I would appreciate keeping the conversation at a high level prior to diving deeper.

@davidBar-On tagging you since it was your feature.

[davidBar-On]

--skip-rx-copy uses MSG_TRUNC socket option, to skip copying the received data from the TCP/UDP stack buffer into the user's buffer. For iperf3 this is used for not including the overhead of data copy between the buffers.

For TCP, MSG_TRUNC does not copy any data, regardless of the size parameter value. For UDP, the size parameters specify how much data from the received datagram (packet) should be copied. iperf3 uses the first bytes of the UDP datagram for the time the packet was sent (sec, usec) and for the packet count (pcount) - total 16 bytes. Therefore iperf3 needs to read the header of the received UDP datagram and cannot ignore all of it.

The result is that with --skip-rx-copy, TCP completely skips copying data between the buffers. On the other hand, UDP does copy 16 bytes per datagram, although in most cases the overhead should be very small.

[manedurphy]

Understood. I have yet to observe an increase in max throughput for an iperf3 client, but I have now observed a significant reduction in CPU usage during the measurement. Thanks for clarifying the effect on TCP.