Add pagination support to publicRooms

[erikjohnston]

Changes:

• Remove support for secondary directory servers, now that clients can request remote directories themselves.
• start and end keys have been removed from the response to /publicRooms, and replaced by next_batch and prev_batch (whose existence depends on if there is a next or prev batch).
• The rooms are now returned in order of number of users.
• /publicRooms accept limit and since parameters.
• Clients can paginate forwards and backwards by passing the {next,prev}_batch tokens to since=. There is no direction parameter.
• The room list is not updated with newly rooms published rooms when specifying a since token. new_rooms is returned in the response when a since token is specified and is a boolean indicating if there are new rooms. Unpublished rooms do get removed, however.
• The limit param is advisory, in particular synapse will return fewer rooms if they have been unpublished.
• Tokens are not valid indefinitely. Synapse return an error if the tokens are over a month old.

The response now looks something like:

{
    "chunk": [
        {
            "canonical_alias": "#wiiiiiiiiimble:localhost:8480",
            "world_readable": false,
            "num_joined_members": 1,
            "room_id": "!AyxdOWyQKnFhBrCyfS:localhost:8480",
            "guest_can_join": false,
            "aliases": [
                "#wiiiiiiiiimble:localhost:8480"
            ]
        }
    ],
    "prev_batch": "hKFwA6FzzQYfoWTCoW4I",
    "new_rooms": false
}

Question: If we don't specify a limit, should we return the full list (current behaviour) or should we cap it?

[NegativeMjark]

LGTM <- not sure why I typed that yet. I should probably actually read the PR... :(

[NegativeMjark]

Will this be quick on postgres, or will you need to use window functions there?

[NegativeMjark]

Not that it really matters since the table will probably be small to start with.

[erikjohnston]

I've been testing on postgres, and it appears to use the right indices.

[erikjohnston]

@matrixbot retest this please

[NegativeMjark]

LGTM