feat: Improve Engagement Dashboard's "Channels" tab performance

[matheusbsilva137]

Proposed changes (including videos or screenshots)

• Replace duplicated aggregation execution (which was used only to count documents) by a single request for counting all rooms;
• Apply limit as soon as possible in the aggregation so as to carry less data to the next steps;
• Only return channels, groups and DM rooms from this endpoint. Livechat rooms have been removed since we don't track their messages data.
• Added a hideRoomsWithNoActivity param to the engagement-dashboard/channels/list endpoint, which enables a faster search by not returning rooms that had no message sent in the given period. This param is being used as true by default in our client, which will be the default behavior starting on 7.0 (when this param will be removed).

Issue(s)

Steps to test or reproduce

The Engagement Dashboard's "Channels" tab should keep working just like before, but faster when there's a huge amount of rooms in the workspace. Channels should always be listed sorted by their amount of messages when no sort param is provided.
What should be validated:

• Rooms returned by the engagement-dashboard/channels/list endpoint should be sorted by their amount of new messages in the analyzed timespan (decreasingly);
• The engagement-dashboard/channels/list endpoint should accept a new boolean hideRoomsWithNoActivity param, which enables a faster search by removing rooms that had no messages sent in the analyzed timespan. This is specially relevant for workspaces that have a huge amount of rooms (this option is being used in the UI);
• A deprecation warning log should be displayed whenever the engagement-dashboard/channels/list endpoint is called (so that users are aware that the new param will be deprecated);
• Livechat rooms should not be returned by the engagement-dashboard/channels/list endpoint anymore.

Further comments

SUP-513
SUP-558

Aggregation explain (before): explain-aggregation-before.json

Aggregation explain (after): explain-aggregation-after.json

Aggregation explain (after and using hideRoomsWithNoActivity param): aggregation-explain-after-hide-rooms-with-no-activity.json

[dionisio-bot]

Looks like this PR is ready to merge! 🎉
If you have any trouble, please check the PR guidelines

[changeset-bot]

🦋 Changeset detected

Latest commit: f3ca6f0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 32 packages

Name	Type
@rocket.chat/meteor	Minor
@rocket.chat/model-typings	Minor
@rocket.chat/apps	Patch
@rocket.chat/models	Patch
@rocket.chat/account-service	Patch
@rocket.chat/authorization-service	Patch
@rocket.chat/ddp-streamer	Patch
@rocket.chat/omnichannel-transcript	Patch
@rocket.chat/presence-service	Patch
@rocket.chat/queue-worker	Patch
@rocket.chat/stream-hub-service	Patch
@rocket.chat/omnichannel-services	Patch
rocketchat-services	Patch
@rocket.chat/core-services	Patch
@rocket.chat/cron	Patch
@rocket.chat/instance-status	Patch
@rocket.chat/presence	Patch
@rocket.chat/core-typings	Minor
@rocket.chat/rest-typings	Minor
@rocket.chat/fuselage-ui-kit	Major
@rocket.chat/gazzodown	Major
@rocket.chat/livechat	Patch
@rocket.chat/ui-contexts	Major
@rocket.chat/api-client	Patch
@rocket.chat/license	Patch
@rocket.chat/pdf-worker	Patch
@rocket.chat/ddp-client	Patch
@rocket.chat/uikit-playground	Patch
@rocket.chat/ui-avatar	Major
@rocket.chat/ui-client	Major
@rocket.chat/ui-video-conf	Major
@rocket.chat/web-ui-registration	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.52%. Comparing base (e6ce4ee) to head (f3ca6f0).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #32493      +/-   ##
===========================================
- Coverage    55.58%   55.52%   -0.06%     
===========================================
  Files         2634     2634              
  Lines        57281    57255      -26     
  Branches     11861    11858       -3     
===========================================
- Hits         31838    31790      -48     
- Misses       22750    22770      +20     
- Partials      2693     2695       +2     

Flag	Coverage Δ
e2e	54.26% <ø> (-0.08%)	⬇️
unit	72.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[KevLehman]

༼ つ ◕_◕ ༽つ API tests

[geekgonecrazy]

@matheusbsilva137 since adjusting the aggregation as part of improving can you please attach the mongo explain of the query before the optimization and after?

ideally with some real amount of data being processed so we can get an idea if the query is going to utilize appropriate indexes and be any better.

[matheusbsilva137]

I've just attached the old and new aggregations explains to the PR description.
The new aggregation uses the t index for rooms, while the old one doesn't use indexes at all (based on what Compass told me 🤓 ). For now I'm using my local env, which doesn't have that much data, but we're planning to run this on a bigger DB for testing before this is merged.

cc @MarcosSpessatto @geekgonecrazy

[geekgonecrazy]

Thanks for adding 🙏 this sort of data is important. Might be worth just creating a bunch of records. We should make it easier to fake a lot of data for optimizations like this. Having to wait on larger environments seems like adding in delays that we could avoid