feat: Apps-Engine method to read multiple messages from a room

[Dnouv]

Proposed changes (including videos or screenshots)

Added a new method, getMessages to fetch messages from a specific room. This method accepts a roomId and an options object as parameters. The options object can contain limit, skip, and sort properties to control the query. The method uses the Messages.find function to fetch the messages from the database and converts them using a message raw converter obtained from the orch object.

Since a room can have a large number of messages, this method returns a maximum of 100 messages in a single call, to read more messages one can make another call passing the skip value. The default sort order is the descending order of timestamp (meaning the most recent messages will be returned), which can be overridden by passing the sort parameter.

Added a new converter convertMessageRaw it removes a few fields from the following (reason: comment ):

sender: {
  id: string,
  username: string,
  name?: string
},
editor: {
  id: string,
  username: string
},
room: {
  id: string
}

Additionally, this new method removes the delete since the delete operator is generally slow and can cause performance issues. It modifies the underlying object, which can lead to deoptimization in JavaScript engines.

Issue(s)

Steps to test or reproduce

To test as well as reproduce the issue and solution, try running the below in the App.

// before this PR
const roomMessage = await read
                .getRoomReader()
                .getMessages(message.room.id)

// after this PR 
const roomMessage = await read
                .getRoomReader()
                .getMessages(message.room.id, {
                    limit: 3,
                    sort: { createdAt: 'asc' },
                    skip: 1,
                });

Output:
(Before this PR):

• Error method not implemented

(After this PR):

• Array of messages

You can pass your own query to filter the messages you want to retrieve. The available options are:

• limit: The number of messages to retrieve. (Default: 100 and Max: 100)
• skip: The number of messages to skip.
• sort: The sort order of the messages. e.g. { ts: -1 } will sort the messages by timestamp in descending order.

Further comments

• ADR {For Internal Use}
• JIRA-28

Execution Stats on a room with 3M messages

{
  explainVersion: '1',
  queryPlanner: {
    namespace: 'meteor.rocketchat_message',
    indexFilterSet: false,
    parsedQuery: {
      '$and': [
        {
          rid: {
            '$eq': 'GENERAL'
          }
        },
        {
          _hidden: {
            '$not': {
              '$eq': true
            }
          }
        },
        {
          t: {
            '$not': {
              '$exists': true
            }
          }
        }
      ]
    },
    queryHash: '19FB9CF7',
    planCacheKey: 'E315B320',
    maxIndexedOrSolutionsReached: false,
    maxIndexedAndSolutionsReached: false,
    maxScansToExplodeReached: false,
    winningPlan: {
      stage: 'LIMIT',
      limitAmount: 100,
      inputStage: {
        stage: 'FETCH',
        filter: {
          '$and': [
            {
              _hidden: {
                '$not': {
                  '$eq': true
                }
              }
            },
            {
              t: {
                '$not': {
                  '$exists': true
                }
              }
            }
          ]
        },
        inputStage: {
          stage: 'IXSCAN',
          keyPattern: {
            rid: 1,
            ts: 1,
            _updatedAt: 1
          },
          indexName: 'rid_1_ts_1__updatedAt_1',
          isMultiKey: false,
          multiKeyPaths: {
            rid: [],
            ts: [],
            _updatedAt: []
          },
          isUnique: false,
          isSparse: false,
          isPartial: false,
          indexVersion: 2,
          direction: 'forward',
          indexBounds: {
            rid: [
              '["GENERAL", "GENERAL"]'
            ],
            ts: [
              '[MinKey, MaxKey]'
            ],
            _updatedAt: [
              '[MinKey, MaxKey]'
            ]
          }
        }
      }
    },
    rejectedPlans: [
      {
        stage: 'LIMIT',
        limitAmount: 100,
        inputStage: {
          stage: 'FETCH',
          filter: {
            '$and': [
              {
                t: {
                  '$not': {
                    '$exists': true
                  }
                }
              },
              {
                _hidden: {
                  '$not': {
                    '$eq': true
                  }
                }
              }
            ]
          },
          inputStage: {
            stage: 'IXSCAN',
            keyPattern: {
              rid: 1,
              t: 1,
              'u._id': 1
            },
            indexName: 'rid_1_t_1_u._id_1',
            isMultiKey: false,
            multiKeyPaths: {
              rid: [],
              t: [],
              'u._id': []
            },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: {
              rid: [
                '["GENERAL", "GENERAL"]'
              ],
              t: [
                '[null, null]'
              ],
              'u._id': [
                '[MinKey, MaxKey]'
              ]
            }
          }
        }
      },
      {
        stage: 'LIMIT',
        limitAmount: 100,
        inputStage: {
          stage: 'FETCH',
          filter: {
            '$and': [
              {
                _hidden: {
                  '$not': {
                    '$eq': true
                  }
                }
              },
              {
                t: {
                  '$not': {
                    '$exists': true
                  }
                }
              }
            ]
          },
          inputStage: {
            stage: 'IXSCAN',
            keyPattern: {
              rid: 1,
              tcount: 1
            },
            indexName: 'rid_1_tcount_1',
            isMultiKey: false,
            multiKeyPaths: {
              rid: [],
              tcount: []
            },
            isUnique: false,
            isSparse: false,
            isPartial: false,
            indexVersion: 2,
            direction: 'forward',
            indexBounds: {
              rid: [
                '["GENERAL", "GENERAL"]'
              ],
              tcount: [
                '[MinKey, MaxKey]'
              ]
            }
          }
        }
      }
    ]
  },
  executionStats: {
    executionSuccess: true,
    nReturned: 100,
    executionTimeMillis: 4,
    totalKeysExamined: 100,
    totalDocsExamined: 100,
    executionStages: {
      stage: 'LIMIT',
      nReturned: 100,
      executionTimeMillisEstimate: 0,
      works: 101,
      advanced: 100,
      needTime: 0,
      needYield: 0,
      saveState: 0,
      restoreState: 0,
      isEOF: 1,
      limitAmount: 100,
      inputStage: {
        stage: 'FETCH',
        filter: {
          '$and': [
            {
              _hidden: {
                '$not': {
                  '$eq': true
                }
              }
            },
            {
              t: {
                '$not': {
                  '$exists': true
                }
              }
            }
          ]
        },
        nReturned: 100,
        executionTimeMillisEstimate: 0,
        works: 100,
        advanced: 100,
        needTime: 0,
        needYield: 0,
        saveState: 0,
        restoreState: 0,
        isEOF: 0,
        docsExamined: 100,
        alreadyHasObj: 0,
        inputStage: {
          stage: 'IXSCAN',
          nReturned: 100,
          executionTimeMillisEstimate: 0,
          works: 100,
          advanced: 100,
          needTime: 0,
          needYield: 0,
          saveState: 0,
          restoreState: 0,
          isEOF: 0,
          keyPattern: {
            rid: 1,
            ts: 1,
            _updatedAt: 1
          },
          indexName: 'rid_1_ts_1__updatedAt_1',
          isMultiKey: false,
          multiKeyPaths: {
            rid: [],
            ts: [],
            _updatedAt: []
          },
          isUnique: false,
          isSparse: false,
          isPartial: false,
          indexVersion: 2,
          direction: 'forward',
          indexBounds: {
            rid: [
              '["GENERAL", "GENERAL"]'
            ],
            ts: [
              '[MinKey, MaxKey]'
            ],
            _updatedAt: [
              '[MinKey, MaxKey]'
            ]
          },
          keysExamined: 100,
          seeks: 1,
          dupsTested: 0,
          dupsDropped: 0
        }
      }
    }
  },
  command: {
    find: 'rocketchat_message',
    filter: {
      rid: 'GENERAL',
      _hidden: {
        '$ne': true
      },
      t: {
        '$exists': false
      }
    },
    limit: 100,
    '$db': 'meteor'
  },
  serverParameters: {
    internalQueryFacetBufferSizeBytes: 104857600,
    internalQueryFacetMaxOutputDocSizeBytes: 104857600,
    internalLookupStageIntermediateDocumentMaxSizeBytes: 104857600,
    internalDocumentSourceGroupMaxMemoryBytes: 104857600,
    internalQueryMaxBlockingSortMemoryUsageBytes: 104857600,
    internalQueryProhibitBlockingMergeOnMongoS: 0,
    internalQueryMaxAddToSetBytes: 104857600,
    internalDocumentSourceSetWindowFieldsMaxMemoryBytes: 104857600,
    internalQueryFrameworkControl: 'trySbeRestricted'
  },
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1721412111, i: 1 }),

  },
  operationTime: Timestamp({ t: 1721412111, i: 1 })
}

[dionisio-bot]

Looks like this PR is ready to merge! 🎉
If you have any trouble, please check the PR guidelines

[changeset-bot]

🦋 Changeset detected

Latest commit: 8a0e0fa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 32 packages

Name	Type
@rocket.chat/meteor	Minor
@rocket.chat/core-typings	Minor
@rocket.chat/rest-typings	Minor
@rocket.chat/apps	Patch
@rocket.chat/core-services	Patch
@rocket.chat/cron	Patch
@rocket.chat/fuselage-ui-kit	Major
@rocket.chat/gazzodown	Major
@rocket.chat/livechat	Patch
@rocket.chat/model-typings	Patch
@rocket.chat/ui-contexts	Major
@rocket.chat/account-service	Patch
@rocket.chat/authorization-service	Patch
@rocket.chat/ddp-streamer	Patch
@rocket.chat/omnichannel-transcript	Patch
@rocket.chat/presence-service	Patch
@rocket.chat/queue-worker	Patch
@rocket.chat/stream-hub-service	Patch
@rocket.chat/api-client	Patch
@rocket.chat/license	Patch
@rocket.chat/omnichannel-services	Patch
@rocket.chat/pdf-worker	Patch
@rocket.chat/presence	Patch
rocketchat-services	Patch
@rocket.chat/ddp-client	Patch
@rocket.chat/uikit-playground	Patch
@rocket.chat/models	Patch
@rocket.chat/ui-avatar	Major
@rocket.chat/ui-client	Major
@rocket.chat/ui-video-conf	Major
@rocket.chat/web-ui-registration	Major
@rocket.chat/instance-status	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 55.58%. Comparing base (4e8aa57) to head (8a0e0fa).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #32176   +/-   ##
========================================
  Coverage    55.57%   55.58%           
========================================
  Files         2634     2634           
  Lines        57273    57279    +6     
  Branches     11860    11860           
========================================
+ Hits         31828    31837    +9     
+ Misses       22759    22751    -8     
- Partials      2686     2691    +5     

Flag	Coverage Δ
e2e	54.33% <ø> (+0.03%)	⬆️
unit	72.23% <0.00%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[debdutdeb]

I am also taking some questions back to the adr.

[Dnouv]

Depends on: RocketChat/Rocket.Chat.Apps-engine#770

Approved on BE

Change applied