At about 70 client connections the client/server do not show/list all clients anymore

[corrados]

See the report in this post: #455 (comment)

[corrados]

I did a quick test where I created 100 artificial clients with the following code in socket.cpp:

            // server:

            int iCurChanID;

// TEST
for ( int i = 0; i < 100; i++ )
{
    if ( i > 0 )
    {
        RecHostAddr.iPort++;
    }

            if ( pServer->PutAudioData ( vecbyRecBuf, iNumBytesRead, RecHostAddr, iCurChanID ) )
            {
                // we have a new connection, emit a signal
                emit NewConnection ( iCurChanID, RecHostAddr );

                // this was an audio packet, start server if it is in sleep mode
                if ( !pServer->IsRunning() )
                {
                    // (note that Qt will delete the event object when done)
                    QCoreApplication::postEvent ( pServer,
                        new CCustomEvent ( MS_PACKET_RECEIVED, 0, 0 ) );
                }
            }
}

But I cannot reproduce the issue. I could see all 100 faders in the client and the server table was also complete. Maybe the problem was caused by the CPU being at 100% which can cause weird effects.

[storeilly]

We were doing undocumented tests last night on a server I've compiled to allow 255 clients and had similar issues, on one of my clients I could see 102 clients while the other was stuck at 53. I was watching htop and no cpu (4 total) was over 50% I plan to do this test properly with properly recorded results over the next few days. Audio at 102 clients was badly broken but intelligible.

[maallyn]

I just completed the following test and observation.

First, I started with 30 connections on the server. The client could see those 30 connections.

Then I added another 20 clients slowly, so as to not overrun the server with fast connection requests; I did each one every 5 seconds.

Then someone else came on (client with a different name)

However, with 50 connections, I could not see that person on my Jamulus client (Windows 10). However, I could see him on the master list as viewed with my connection status panel.

I then reduced the number of connections (by stopping each) until I got down to about 25 connections. Now I could see all of them without having to scroll sideways. However, I still could not see the one client whom I was talking with.

I then stopped and re-started the client and I did see him.

This suggests to me that there may be some data stuck in the client that had to be reset upon a restart of the client.

Throughout all of this, the sound from that one client was very good despite the fact that I could not see his fader on my panel.

Throughout all of this, there were only occasional instances when any of the four dedicated CPU's hit over fifty percent.

[corrados]

@softins Maybe the issue is related to #255. The client list for the audio mixer board gets quite big in case of 100 clients (similar to the server list). Is it possible that you support us with a wireshark analysis of that situation? Maybe if a test is running with more than 50 clients, you could log onto that session and check the protocol messages sent to your client?

[softins]

Sure, I'd be happy to help, although not available today (Sat).

[softins]

Just a couple of general comments before I disappear for the day:

• If some device in the path is causing fragmented IP packets to drop, then the whole list would be lost, not partially received. Like when a client sees no servers in Default or All Genres. So it looks to me more likely to be something got out of sync between the protocol and the GUI. That is also supported by Mark's comment "However, with 50 connections, I could not see that person on my Jamulus client (Windows 10). However, I could see him on the master list as viewed with my connection status panel."
• In the situation where one or more clients are not shown to another client, but are evidently connected, one can cross-check with the display from http://jamulus.softins.co.uk

[corrados]

So it looks to me more likely to be something got out of sync between the protocol and the GUI.

The protocol transmits messages "message by message" in the order the messages are scheduled. If one message does not make it through to the client because of fragmentation, the protocol mechanism will be stuck trying to transmit that one message and does not process any further messages. That would explain the described behaviour I think (or better, I guess... Therefore it would be good to get a proof of that by checking the network traffic with wireshark).

[maallyn]

Okay, thank you for the suggestion, Simon. Sometime next week when I have time, I will re-set up everything but install wireshark on both the server sunning the clients and the jamulus server itself and somehow capture the screens and record them as a video. I have a question. Can I attach a video or at least a link to a video on this ticket?

Or better yet, can I set up a jitsi or zoom session and show the stuff live?

[corrados]

Thanks for offering the test. I guess it would be easier if you talk to softins when you start your test so that he can connect to your server and capture the wireshark output on his side. So you should do the test when you and softins have time for it.

[softins]

Hi Mark, there's no Simon on this thread. I'm Tony (also on Facebook). Happy to liaise with you in the week. I'm on UK time, I think you are EDT? If you haven't seen it, check out my repo https://github.com/softins/jamulus-wireshark

[maallyn]

I'm embarrassed that I made this mistake. I keep thinking that Corrados is Simon. Thank you for straightening me out. If you are in UK, then the best time for me would be my morning, which is your late afternoon. So, if I try to do this around nine in the morning Pacific U.S. Time, that would be six in the evening for you, is that an okay time?

[corrados]

I was looking at the original error description again. Here's what I found:

And the client slider panel stopped showing sliders after about 60 or so connections. However, the total count (the number indicated adjacent to the server name on the master server listings showed the total count (81), along with the total number on the top of the client's panel.

I checked in the code of the client that the total count (in this case 81) is only shown if a client list protocol message for the audio mixer board was correctly and completely received. So maybe the issue is not related to #255. But I think it still does make sense to check the protocol anyway under this stress situation to check that everything works as expected.

[softins]

If you are in UK, then the best time for me would be my morning, which is your late afternoon. So, if I try to do this around nine in the morning Pacific U.S. Time, that would be six in the evening for you, is that an okay time?

Except for the week straddling Oct/Nov (we end DST a week earlier than the US), we are 8 hours ahead of Pacific time, so your 9am is UK 5pm. That would be fine for me. I normally go to eat about 6pm or so. I think any day this week works for me. You can find me on Discord as Tony M or on Facebook Messenger as Tony Mountifield