Support single audio mix server mode to support large ensembles and simplify the mixing

[corrados]

See here:

• At about 70 client connections the client/server do not show/list all clients anymore #547 (comment)
• At about 70 client connections the client/server do not show/list all clients anymore #547 (comment)
• Switch all Jamulus audio sample processing to use floats instead of a mix of double and int16_t #535 (comment)

A simple approach with only little changes in the Jamulus server code is wanted. Specification:

Adding a new command line argument to the server like --singlemix:

• No multithreading (since we only have one encoding and mixing so no multithreading needed)
• Only 128 sample frame size support
• Only Mono support (gives us the most possible number of connected clients which is what this modification is all about)
• The first connected client on that server is the "director", all other clients which connect afterwards get his mix. So you just have to make sure that the director is already connecting to the server before your session begins (this requirement should be very easily to be fulfilled).

There is a vecvecbyCodedData buffer which is used for both, encoding and decoding. I'll introduce a separate buffer so that I can re-use the output buffer of the first client for all other clients. So instead of calling MixEncodeTransmitData for all the other clients, they simply get vecChannels[iCurChanID].PrepAndSendPacket ( &Socket, vecvecbyCodedData[iDirectorID], iCeltNumCodedBytes );. I just did a quick hack: If I modify CreateChannelList so that no client is added, the audio mixer panel is just empty. This would be the case for the slave clients. But then they do not see how many clients are currently connected which is not a big issue. If "--singlemix" is given, "-F" and "-T" is deactivated and a warning is shown that these cannot be combined. In the OnNetTranspPropsReceived function we can check that the client uses 128 samples, if not, refuse to connect.

There is a new branch were the implementation is done:
https://github.com/corrados/jamulus/tree/feature_singlemixserver

[corrados]

@storeilly, @kraney, @maallyn I updated the code on the feature_singlemixserver branch with the following functionality:

• The master "director" client defines the mode, i.e. if mono/stereo and the audio quality.
• The slave clients have to use the same mono/stereo and audio quality settings as the director, otherwise their client will show "TRYING TO CONNECT" all the time until it automatically disconnects after a while.
• Only the master "director" sees the full audio mixer board faders and can control the audio mix. All slave clients will see an empty audio mixer board.

It would be great to get feedback from you:

• Is the source code stable?
• Does it work according to the above specification?
• How many clients does a fast server PC can serve in that mode (I would recommend to use Mono mode to get the most number of clients)?

[storeilly]

Will each client be able to balance their own input in the mix?, if not I can imagine one of two scenarios.
1 some singers will turn their gain too high to hear themselves against the mix and the director turning their fader back down to compensate ending with the singer clipping.
2 some singers turning their gain too low to reduce themselves and the director doing the reverse.
I'm not convinced this use case will be popular or workable sorry

[ann0see]

I really like this feature since it has a huge potential. If really only the "master client" can do the mix I don't see any problem @storeilly?

What I see as a problem is that the clients just see "TRYING TO CONNECT". I'd prefer (at least for newer clients) to show "Please set the audio quality/... as follows: [...]" or even a server message which sets them automatically. If you don't do that it would probably rise quite some support questions and problems.

[storeilly]

I don't like it. Sorry. Consider human nature and the fact that you are removing an element of control from them. If they can't adjust their own perception of their volume in the mix without affecting how others hear it, they will compensate and use the control that they have which does. They will sing too loud or too quiet (depending on personality type). Any attempt by an outside force (the director) to balance the mix will force them to increase the compensation to detrimental levels.

…

On Sat 19 Sep 2020, 13:16 ann0see, ***@***.***> wrote: I really like this feature. If really only the "master client" can do the mix I don't see any problem @storeilly <https://github.com/storeilly>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIJSQLRZPBGA6P3VVZFURLSGSOI7ANCNFSM4RS3AX3Q> .

[pljones]

Is there any way what's done could be tuned to having each participant (apart from the director) see only their own channel control? This wouldn't save server side processing - you'd still have n mixes to create. But it retains simplicity in the UI whilst preserving control.

The levels would be set by the director, apart from a participant's own level, which they could set for themselves. Perhaps it would still need a "balance" so you could raise your own level against the rest of the mix, or in this mode, your own slider becomes "balance" -- at the bottom, you only hear everyone else, at the top, you only hear yourself, in the middle you hear the director's mix.

(I'd also think a "list participants" option would be needed. Otherwise I'd be wondering "who's here?" all the time.)

---

Of course, my other question is -- how does this affect the jam recorder? Is the emit AudioFrame still done before the mix? This seems like it would answer one of my questions on the thread for recording a pre-mixed file - the director's mix is what you'd record. (Though I'd still advise a new signal/slot, it needs the filename handled properly and the file not included in the projects.)

[kraney]

To me the scenario that leads to clipping sounds pretty passive-aggressive and like it could best be resolved by the director mentioning out loud that he’s having to dial down (or up) someone’s volume. It doesn’t seem like an indictment of the entire feature.

I haven’t had a chance to try it yet for time zone reasons but I should be able to run it during this weekend.

I have a little concern with the “first to connect” solution for selecting a director only because, supposing it goes wrong and someone accidentally joins before the director, how can it be resolved? Everyone has to leave and then wait for confirmation by the director via some other channel before rejoining? However I understand the desire to minimize the disruption to the code.

One other idea for selection comes to mind - maybe the director is the first one to join who chooses a baton as his instrument?

[corrados]

If they can't adjust their own perception of their volume in the mix without affecting how others hear it, they will compensate and use the control that they have which does. They will sing too loud or too quiet (depending on personality type). Any attempt by an outside force (the director) to balance the mix will force them to increase the compensation to detrimental levels.

That may or may not be. I think this strongly depends on how the singers can adapt to the mix. For some it may be simple to adapt, for some it may not be possible to sing. But this can only be evaluated on a real test with a lot of singers, I guess.

I'd prefer (at least for newer clients) to show "Please set the audio quality/... as follows: [...]" or even a server message which sets them automatically.

Sure, a lot things could be done. The current code implements the basic functionality. If it turns out that this mode is useless, all additional effort would be wasted. So the first step is to verify that this mode works with a lot of singers (which are all instructed before entering the server what they have to set in the Jamulus client). If that was successful, more time could be spent to improve that server mode.

Is there any way what's done could be tuned to having each participant (apart from the director) see only their own channel control? This wouldn't save server side processing - you'd still have n mixes to create.

Yes, exactly. If you want any individual mix, you have to use the normal mode.

I'd also think a "list participants" option would be needed. Otherwise I'd be wondering "who's here?" all the time.

The server could send a Chat message when a new client enters the server. But as I wrote above, before any more time is spent, it has to be verified that this new mode works in a real scenario and turns out to be useful.

Of course, my other question is -- how does this affect the jam recorder?

The jam recorder is not affected. It works the same as in the normal server mode.

I have a little concern with the “first to connect” solution for selecting a director only because, supposing it goes wrong and someone accidentally joins before the director, how can it be resolved?

No problem. You can simply swap the places at the server. The one who currently has control leaves the server. Then the director leaves the server and immediately re-connects to the server. Now the director has the control.

[kraney]

The swap sounds simple for a small group, but really problematic with a group of 60 people who are all joining a rehearsal that is starting, each according to his own timing.

But I do want to say I really appreciate the very fast turnaround on this, and the approach to try with the simplest change first then refine once the principle is demonstrated.

[kraney]

I ran a test using the current master first to serve as a baseline. For my test I connect once using a normal client I can listen to, then add a bunch of headless clients that are just silent and produce no audio out. (Connected to a jack dummy audio.) Because of this I can't be 100% sure that they work equivalent to a normal client. However I did check that if one of those is "master", I can connect successfully with a normal client and get an empty mixer, but hear audio. I'm using 2vCPUs on a cloud server. Intel Skylake architecture I believe.

In this test I was able to connect 62 clients, but the last few didn't go smoothly. The user name took a long time to appear after the channel appeared. The sound quality degraded somewhat. CPU usage was around 75-80% out of 100 on a two core system. That is, about 160% usage out of 200 in the linux way of reporting it.

I then ran a test using singlemix. With this, I was able to connect 80 clients with no immediate issues. Sound quality was good. CPU usage was around 46% out of 100, or about 92% / 200 in the linux way of reporting it.

However, it only stayed stable for about a minute. After time passed, without any new clients, it shifted into a very degraded mode. Sound came in short, regular choppy bursts, about half on and half off, like 120 bpm pulses. CPU usage dropped to about 13%. Stopping the synthetic clients does not immediately resolve it, but it does go back to normal once the server times the clients out of its list.

This was in normal quality, 128 byte buffer, Mono.

[kraney]

There's one longer term use case I'd like to support that might suggest some requirements for this mode. I'm looking into the ability to re-transmit the Jamulus audio into a YouTube Live or Facebook Live stream. It seems pretty straightforward to do this using SoundFlower on a mac so you can capture what is coming from Jamulus.

In such a situation, the director might likely want to send a click track to the band members to control the tempo and dynamics. So it would be useful to have a second mix not tied to the director one, where that one click track can be muted for the audio that will be retransmitted.