Handle audio packets received out of order

[softins]

I have started working on this, but I wanted to post it as an issue so it could be discussed too.

In the WorldJam community, there has recently been a significant problem with audio quality between clients using Virgin Media as an ISP and the WorldJam servers at OVH. The audio sent from client to server is severely burbled, although the client hears everyone else clearly.

Some tests were conducted on Monday by @sthenos and others using a test program to send streams of UDP packets containing serial numbers from the VM client to the OVH host. These tests demonstrated that the UDP packets were arriving out of order. Our theory is that there is some kind of multipath load-balancing routing happening, where one of the paths is slower than the other.

It is likely to be difficult to get either Virgin or OVH to investigate this deeply for us.

My own view is that out-of-order packet delivery is part and parcel of the unreliable nature of UDP.

Currently, the Jamulus jitter buffer is a simple ring buffer, and expects all audio packets to arrive in the order sent. In the protocol, there is no sequence numbering of audio packets to enable out-of-order packets to be re-ordered correctly.

My plan is to enhance the jitter buffer to re-order out of order packets instead of assuming they are all in the correct order. Protocols like RTP (used with SIP for VoIP calls) do indeed include sequence numbering for exactly that reason. I'm not thinking of actually using RTP, but something much simpler. Just a 3-byte header with 0xFF flag and 2-byte sequence number (counting in bytes) prepended to the existing audio data. The existing Jamulus data never has 0xFF as the first byte, so the server would only treat specially any packets starting with 0xFF. All other traffic, e.g. from older clients, would be processed exactly as at present. If the server sees the 0xFF, it would get the sequence number and use that to determine where in the jitter buffer to put the remaining audio data in the packet (and if it's too old, discard it). Also, once the server had seen the 0xFF flag from a client, it would know that it could send data back to that client in the same way. Otherwise (for older clients) it would send audio unsequenced as at present.

If the sequence number in a received audio packet was higher than expected, that would indicate missing data. We would need to fill the skipped space with silence, in case the missing data never arrived in time. If it did subsequently arrive, it could be placed in the JB at the correct place, overwriting the silence.

[WolfganP]

Interesting findings @softins. Good idea to add sequence numbers to the audio packets for better buffers' handling. Being jamulus custom UDP packets, I imagine that they even go to lower priority in the routing QoS rules of providers.

Saying so, I wouldn't disregard the use of RTP for the audio path, it maybe worthy to analyze pros and cons (ie low overhead, out of order & jitter mgmt & report, solid existing libraries, opus integration -RFC 7587-, ...) versus increase complexity in the jamulus protocol to achieve the same goals.
It's not that all protocol features needs to be used, and besides the inner properties, the stream can benefit for routing optimizations of ISPs and backbone providers.

[corrados]

Our theory is that there is some kind of multipath load-balancing routing happening, where one of the paths is slower than the other.

Who does this? That does not make any sense to me to split UDP packets of a continuous stream. Maybe Virgin is developing their own "Virginulus Realtime Jam Software" and want to turn Jamulus down ;-).

The audio sent from client to server is severely burbled, although the client hears everyone else clearly.

If it is just the connection from client to server which makes problems, maybe the UDP packets are sent in a bursted manner which leads to the routing issues. @hselasky had implemented some "traffic shaping" for the client since he found out that especially under Windows, the sound card callback does not have a good timing leading to UDP bursts. But I do not find that code in his repo anymore. Maybe he has removed it.

What you could try is to use Linux instead (or the Jamulus OS on an USB drive). Usually you have a better timing under Linux. If the problem does not occur under Linux, we know that the bursted UDP transmission causes the issue.

You could also try to increase the Buffer Delay in the Jamulus settings. Then the network packets are larger and may not suffer from the bad routing. If the small UDP packets are delayed by Virgin, you will get exactly the same overall latency by using larger buffers.

Just a 3-byte header with 0xFF flag and 2-byte sequence number (counting in bytes) prepended to the existing audio data.

Wow, 3 bytes? I usually run Jamulus with a Jitter buffer size of 2 or 3. For that small size a packet counter would gain nothing. I consider a Jitter buffer size of 8 as large. So a 3 bit counter would be sufficient for that size.
But anyway, I do not think a packet counter and a reassembling in the Jitter buffer would be a good solution for Jamulus since everything is designed for low latency. I would first try out other possibilities to solve the problem as mentioned above (buffer size, traffic shaping, etc.).

[softins]

What you could try is to use Linux instead (or the Jamulus OS on an USB drive). Usually you have a better timing under Linux. If the problem does not occur under Linux, we know that the bursted UDP transmission causes the issue.

You could also try to increase the Buffer Delay in the Jamulus settings. Then the network packets are larger and may not suffer from the bad routing. If the small UDP packets are delayed by Virgin, you will get exactly the same overall latency by using larger buffers.

The tests on Monday demonstrating out-of-order delivery of UDP packets in a single stream were performed without Jamulus, using a custom test program. I didn't conduct the tests (@sthenos did), but I watched the conversation. They demonstrated a general problem which would certainly result in degraded audio performance on Jamulus, with audio packets arriving out of order.

[sthenos]

I can comment on the tests we performed.

@softins , very kindly provided some scripts: https://github.com/softins/jamulus-wireshark, in order to perform tcpdumps whilst virgin and non-virgin (sounds dirty ;) ) clients were connected to the jamulus server running on our dedicated OVH server.

What we saw was that packets were arriving at the server in bursts every 5ms or so instead of a single packet every 3ms or so as most other clients were (depending on bufffer size, audio quality, etc). This was a consistent pattern for all the virgin clients we tested (we tested two). This led us initially to conclude that this queuing and flushing of packets could have been the issue. However @softins had an AWS VM and ran the same tests for the virgin clients (where there is no audio burbling for Virgin Jamulus clients) and could see the same behaviour of clumping of packets arriving every 5ms.

I then wrote a very simple java client/server app that streams 128 byte udp datagrams every 2ms( as jamulus would) and wrote a sequence number into the first 4 bytes. The server received the packets and printed out the sequence number. We could see clearly that for all virgin clients the packets were consistently inconsistently arriving at the OVH dedicated server.

I then ran the same test on a cloud VM at OVH, wondering if it was a problem with the baremetal server or the subnet it was in and the same behaviour occurred both for the UDP client/server app as well as when the users used Jamulus.

This behaviour has led us to conclude that packets are most likely being load balanced and split randomly by an ECMP router in the path, either within OVH or Virgin. My view is that it only affects jamulus instances at OVH, therefore the mutli-path routing must be occurring within OVH, but that it's only occuring for Virgin traffic, not any other ISP, therefore we are wondering if something is occurring on the header of the packets to make the ECMP router in OVH not be able to determine packets are from the same source and headed for the same destination thus should be kept on the same path.

Still all theories at this stage, and my next step is to test the java UDP client/server app with virgin clients to a non OVH server (in AWS for instance).

[gilgongo]

Would this also explain other strange behaviours? I'm on VM and have never been able to see more than about 15 servers per central server (while --showallservers shows a large number)

[corrados]

@sthenos Any updates on the network issues with Virgin? Have you found a solution/workaround which works for you?

[softins]

For the last couple of worldjams they have just used a different server. I don't know the current status with Virgin.

Although the last week has been busy, I still have plans to try a proof of concept with sequence numbering. Even if you are running on a short jitter buffer, sequence numbering allows a late, out-of-order packet to be discarded instead of being inserted into the jitter buffer in the wrong place, which should cause less aberration in the audio. And the affected client could always increase their jitter buffer setting to see if it improves things.

[interogativ]

In my experience, queuing and reassembling sequenced audio packets in a low latency environment is a very bad idea . We implemented a scheme to do that (in a teleconference system) years ago and it resulted in large jitter buffer sizes and noticeable audio delays. We found that if you're going to add a sequence number to packets, it should probably only be used to detect missing or out of sequence packets. These might be covered by initializing the jitter buffer to silence and playing the correctly sequenced streaming buffers when they arrive within the jitter window. Since Jamulus uses Opus audio compression, I'm not sure what effect that will have, but redesigning a working protocol based on one or two bad network path's is not a great idea in my opinion.

[softins]

@interogativ I largely agree with you, but I do think it is always wrong to process audio packets received out of order in the incorrect order that they were received, even if that situation is fairly rare. They should at least be detected and discarded, and if the user's jitter buffer size is such that the decoder hasn't yet needed the data from a late packet when it arrives, there is no harm in placing the data in the correct part of the buffer, and some benefit in doing so.

[interogativ]

Well, from the client's point of view, lets say it receives packet 10, then packet 12. Soon packet 11 might be arriving out of order, but because this is UDP, it might never arrive at all. I agree that packets should be placed into the correct order in the jitter buffer before they are played. I actually think we're in agreement here.

[softins]

Yes, I think we agree. Here is my plan to prove the concept (and I am taking account of forward/backward compatibility):

• A client, when connected to a compatible server, would maintain and attach a byte sequence number (not packet sequence number) to the outgoing stream packets.
• A server, when receiving from such a client, would maintain byte sequence numbers associated with the put and get pointers in the jitter buffer.
• When an inbound packet has a sequence number matching that for the put pointer, it will be added to the JB in the normal way.
• If the inbound packet has a higher sequence number than that for the put pointer, that means a packet has been missed. The missed portion will be filled with silence in the JB, and then the incoming packet written at the correct place.
• As the decode fetches from the JB, it will increment the get pointer and get sequence number in the normal way.
• If an inbound packet has a sequence number between the get and put, it will be put in the JB at the correct place, overwriting the silence that had been put in the gap.
• If an inbound packet is for earlier than the get sequence number, it is too late to be used and will be discarded.

The JB will be sized exactly as at present, according to the client settings. I am not proposing to make the JB artifically larger.

If a compatible server receives sequenced audio from a compatible client, it will also sequence its outgoing audio to that client in the same way.

[interogativ]

Its a pretty ambitious plan. Some thoughts and potential flies in the ointment:

1. Assuming 48000 samples per second and 128 samples per frame and that each Opus compressed frame in stereo takes 73 bytes, that means we have 48000 / 128 = 375 frames per second time 73 bytes = 27,375 bytes per second. Using a 16 bit byte sequence number it will last for all of 2 seconds, You'd need at least 32 bit byte sequence number (would last 43 hours). So you must add an additional 4 bytes (at least) to every audio frame.

2. Jamulus considers audio frames to be ("none of the above"), they are not considered connected or connectionless messages (or even the new special message type in 3.12), the only check on them is the frame size. Assuming we just appended the sequence number to the end of each audio message then the current audio frame check same scheme could be kept, and a new client could detect based on the frame size that this is a sequenced audio packet.

3. Right now the client sends audio packets to the server BEFORE it knows anything about the server. And as soon as the server has assigned a channel number it starts sending audio to the client. It might be possible to distinguish client and server versions as you discuss using the existing messaging, but it will require some pretty fancy state management for the server.

4. The Server will have to maintain different inbound sequence numbers for each channel. Right now the channel setup and take down is pretty straightforward, in your new scenario it will be a bit more complicated, especially when uses disconnect. On new connections, the server will have to either wait until it's sure of the client version before sending any audio, or risk those packets being thrown away. The server's outbound sequence numbers could be the same for all channels, but the new client would have to assume a starting index as the first numbered audio packet it received.

5. I'm still not sure what the Opus compression will make of a silent frame followed by an encoded one, does the Opus protocol carry state or is each frame independent?

As I said, this is pretty ambitious.

[softins]

All excellent points, and mostly ones that I've considered and taken into account.

I think for a low-latency application like Jamulus, a 2-second wrap around for the sequence number is more than enough. The jitter buffer would never be more than a hundred or two milliseconds at most, usually much less, and any UDP packet that didn't arrive in well under a second is probably lost rather than delayed. We are looking to reorder or discard packets delayed by only a few or few tens of milliseconds.

I still have a bit more research to do. My main obstacle has been the use of vectors, firstly understanding how they were used, and then thinking about how to avoid unnecessary copies. C-style buffers and pointers are so much easier to handle efficiently! C++ spans might be a reasonable way, but they are only in C++20.

[corrados]

Using a 16 bit byte sequence number it will last for all of 2 seconds, You'd need at least 32 bit byte sequence number (would last 43 hours)

What are you trying to achive? Using an audio packet for decoding which is 43 hours old? What latency is that? It will really hard to play with such a latency ;-).

What you are trying to do here is to solve a Virgin-specific problem. 99.5 % of the Jamulus users are happy with the current implementation. If a provider does nasty stuff with the network packet routing, I don't think you will get any good jam performance even if you correct the incorrect packet order.

I still have plans to try a proof of concept with sequence numbering

Ok, go ahead. It does not hurt to do investigations on a private fork. Merging such a code to the official repo is a completely different story...

[WolfganP]

As mentioned before, I think we're acting without having a sense on how prevalent is this case.

So my take would first insert sequence numbers and measure how many packets are lost, how many arrive in wrong order and get some stats to gauge if the development effort will be worthy (or at least have another measure of link quality to display besides the delay as a troubleshooting tool in case of bad audio quality).

In any case, I think the comment from @interogativ (#619 (comment)) on actual experience with streamed audio quality is worthy, measure it, insert silence and drop the out of order packet provides better perceived audio overall.