Central servers: issues and possible approaches

[gustebeast]

The server documentation says that "Default" has been deprecated in favor of "All Genres". Can we merge these two lists in the client?

[pljones]

The reason they were split still exists.

At the moment, the list can get fragmented if too large and, where UDP fragmentation causes routing issues, this can cause the list to become only partially visible.

I'd also note that All Genres is pretty close to full, so something is going to need to be done one way or another.

As a further point, having the client poll a very large list of servers, most of which they won't be using is a waste of resources both on that client and on every server in the list.

[gustebeast]

It feels strange that we would use genre as a means of solving this scalability issue. Normally I would rely on pagination for that. The client only gets 50 servers or so from the server, then if they want they can request the next 50 and so forth.

The problem is that we can't send them the 50 servers with the lowest ping since the client needs to compute the ping for all the servers before it can sort them. Would it be possible to sort by distance instead of ping? The user experience should be comparable even if the ordering is slightly different. We can get/cache the locations using the client and server IPs. This would also get rid of the ordering issues when pings fluctuate, and the time required for the servers to sort when switching genres.

I'm also curious how much of this scalability issue is client vs server (jamulus.fischvolk.de).

[pljones]

Would it be possible to sort by distance instead of ping?

Distance from the server list? That wouldn't help much. To sort per client, it would have to have some way to compute -- on every client request, every two seconds -- the distance between the client and every server. That sounds like a lot of work. And physical distance isn't much use when it's the ping time that's important. I get half the ping to Frankfurt than Cardiff from London. Even Zurich is better than some places in England.

[gustebeast]

Sort per client yes. We wouldn't necessarily need it to update every two seconds though. The initial request (to central server) would just run once (with a refresh button to re-query). It would require looping through every server, but if you can cache the IP -> location mapping, then you'll just have to do a bit of basic math per server per request: https://stackoverflow.com/a/27943. You may even be able to avoid checking every server if they are stored in a data structure that is organized by location.

Once the client has the 50 servers, it can poll them every 2 seconds for the ping/musicians.

For the Frankfurt vs Cardiff example, my hope is that all those servers would appear near the top of the list, so while the lowest ping may not appear first, it would be easy enough to find.

[gilgongo]

The genre-based system is not ideal and has some clear disadvantages. So if the CS limit issue could be solved in a different way that would be nice.

so while the lowest ping may not appear first, it would be easy enough to find.

That's an an interesting observation I think because most people will only have a handful of fast enough servers to use anyway, and once they find one they're probably not going to care about sort order after that. So a "rough" sort on location (however that's done) should be fine even in a list of 50 I would think. Similarly with a fresh button: servers and players don't churn very fast, so I doubt you'd need to hit the button more than a couple of times before joining a server for a jam.

I take it also that servers would attempt to register on all central servers and go with the first free slot they were given?

[gustebeast]

with a fresh button: servers and players don't churn very fast

It may be advantageous to separate the query that gets the 50 servers from the query that checks players. With the current system, does each server report the players to the central server, then the central server reports the players to the client, or does it work like ping where each client asks the server directly?

I take it also that servers would attempt to register on all central servers and go with the first free slot they were given?

What do you mean by a "free slot"? The simplest set up would be to have one central server with a single unified list of servers. Depending on the load it gets, this could be done with a single beefy computer or require load balancing between multiple computers (still with a single domain and shared unified list). Load balancing may be tricky, although we could tackle it piece by piece, start by implementing the distance sorting, then unify the genres.

We need a way to avoid the inevitable issue of a genre having too many servers in it (unless we want to have a jazz1 and jazz2). However, I think this proposal does more than that: it paves the way for some nice improvements to the user experience:

1. User opens Jamulus for the first time, opens server list.
2. User is quickly presented with the 50 closest servers. The server name, location and genre columns appear right away, the ping and musicians populate after a second or two (and continue to update every couple seconds).
3. The user opens up the "Genre" filter dropdown and selects "Jazz" and "All Genres". The list then updates to show the 50 closest servers that have their genre set to "Jazz" or "All Genres".

If the user opens Jamulus back up, it would remember the filters so the servers would appear in the same order right away. We could still support filtering by server name or occupied, the server would just find the 50 closest servers that match the name/have musicians.

[gene96817]

Please don't implement a "nearest 50 server" list. I recently moved across 5 timezones. I now have Jamulus friends across 8 timezones. I won't see my friends from my old place with only the nearest 50 servers displayed. Also, my old friends won't see my new server and visit. It has been really great to have old friends dropping in, even if the total delay is >140ms.

[gene96817]

I just checked Jamulus Explorer and there are 78 servers under Genre Classical/Folk/Choir. My client shows 38. Any clue why the number of servers are different?

Maybe Jamulus should display all servers (ip addrs Or fqdn Or whatever can be displayed) in the list before pinging them instead of not displaying the unpinged servers. Then the list can be sorted by the client as it pings (the way it is now). I think adding heuristics etc. is just adding bloat.
Edit:
I just found another Undocumented Jamulus feature that solves this problem:
Jamulus --showallservers
please add this documentation to Jamulus --help AND the documentation page.

[gustebeast]

Would these options be sufficient for your use case?

1. At the bottom of the connection window, there would be a "Next Page" button so you can see servers that are further than the closest 50.
2. We could add the ability to "favorite" a server so you could quickly see those even if they span large distances.
3. You could filter out empty servers to keep the list under 50 (provided you are looking for online friends).

Additionally, we can make the page size larger than 50. Could probably make it the current max server count per genre without hurting performance.

[gustebeast]

The problem with sorting by ping is that you need to ping every server in the list. Creating separate lists per genre has worked for now, but that isn't a long term solution. We shouldn't need to separate "default" and "All Genres", and it should be possible to see all genres in one list if you don't care about what genre you want to play/listen to. This is made especially painful because it takes a while for the servers to sort when switching genres.

The location heuristic I think has the potential to actually remove bloat both in the user experience and in the code. For example, we could get rid of the logic that prevents ping sort order changes on mouse hover.

[gene96817]

@gustebeast I would prefer the connection window scroll. "Next" would work also work. It would be nice for the connection window to indicate the number of servers detected. Keep the list simple and just sort by ping times.

[gene96817]

If you don't sort the server list by ping, how will I know how long it will take my packets to reach the server? Also, the ping times could change by network traffic (e.g. time of day). Of course, we don't want the list to churn because of jitter in the ping times. I am sensitive to this issue because I don't live in the center of a metropolitan city. Locations at the edge of the city or further see more delay from fluctuations in network trafic.

[gustebeast]

The ping would still be displayed in the same place. It's just a matter of how the list is sorted. If distance is a good enough approximation of ping, you'll still see the low ping servers at the top of the list. Per pljones' example above, there will be cases where a server with a lower ping appears lower in the list, but so long as it is still close to the top, it should be easy enough to look at the pings and pick out the best one.

[gene96817]

What if the client keeps a single list of all servers, only the selected subset would be displayed. I agree that we don't want to be pinging all the servers frequently. We could ping the serves in a slow background process. If we keep the list between sessions, the old data should be a good starting approximation. Then only new users starting the client for the first time would need to have all the servers pinged.

I often see some servers that are farther away with shorter ping times. It depends on whether those servers are close to a major backbone.