Message limit/message expiration - message retention policy

[fjorgemota]

Hello!

I'm researching chat tools that are opensource and Rocket.Chat seems like a great possibility.

However, one thing that I am searching is someway to limit the number of messages posted in the service and/or expiring old messages. In the RocketChat's documentation, I found in Schema Definition the field 'expireAt': https://github.com/RocketChat/Rocket.Chat/wiki/Schema-Definition, but, when installing the version 0.5.0 here, it seems to not create that, and there's no more references to message expiration in RocketChat's platform.

Is there a hidden setting to expire messages? Or it's just a plan for now? And, yet, it's there a plan for message limit too? I didn't found anything for that in RocketChat documentation.

Thanks. And sorry for my bad english. ✌️

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[geekgonecrazy]

Curious what is the application for this? This is one of the reasons why a lot of people move away from a lot of other services..

[fjorgemota]

Just to avoid excessive use of storage space. =)

I have a "team" of colleagues on the university where I study and we discuss a lot about the various disciplines of our course and about a lot of other nerd topics and these type of things.

Up to now, we're using Hipchat's service, but, as there a problems that cause downtime in the last days/months, we are searching to use other services. We thought about migrating to Slack, but we really don't want to depend too much about third party services that can have uptime problems as Hipchat is having recently, just because we generally talk in periods that do not include normal work times (like weekends, for example). Whatsapp isn't a possibilty because not everyone in our group have a smartphone (as we're just students), and Telegram, well, can suffer from downtime as well as we can't install a server for it.

So, that's my use case. Seems like a joke, but it's not, and for our study the use of a chat service improves resolutions of problems, allow us to study together to exams and many other things. (we have a group at facebook too, but it's virtually abandoned after we started using a chat service like Hipchat)

As MongoDB have support to expiring documents automatically, and RocketChat seems to have a field just for that - at least in the documentation, I thought that RocketChat have that support, and I don't found any other issues about that. (which seems to confirm your comment about this being a reason why a lot of people move away from a lot of other services)

Thanks.

What you think about?

PS: I thought about capped collections. But, I don't know about the possible behavior of Meteor/RocketChat with it.

PS 2: Note that, because we're students, we really don't need all the messages. At maximum the messages of a semester or two is sufficient for us. (and it probably will be cheaper to host, too)

[geekgonecrazy]

Space wise your biggest limitation is going to be file uploads. I've hosted forums and stuff in the past that had many thousands of lines of text and was only a few megs. Infact I think I saw a slack export of 10k messages that was under 1Mb.

That aside. I don't think its necessarily a bad idea. What about a tool that could prune messages older then a certain date? Lets you pick the date and prunes everything older then that?

Expiring is obviously an option. Just seems a bit scary to give that much power to the database. Also we would have to switch that functionality on and off for the db when the user changed. Idk how much of an issue that would or wouldn't be.

[fjorgemota]

Yep, maybe the messages will not ocuppy so much storage, my main concern is about some texts and longer codes that we post on the chat, that have a bit of frequency..I don't have a idea, however, of how much storage space that occupies

A tool to prune messages is a good option too. I think initially about 'expiring' or 'limiting', but I think that a tool with that utility is good too. (specially if it delete the attachments of the messages too)

Nice idea. =)

[fjorgemota]

Well.

I exported the messages of the group and we that generated a folder with 21MB of chat (in JSON format). Considering that Hipchat's probably cropped that export at the limit of 25k messages, and our biggest room have 81k messages, well...It's a bit. :P (and that without considering attachments..)

[geekgonecrazy]

@fjorgemota so it could be big :) I can see where small teams after a while would need some way of pruning down their disk footprint.

[Sing-Li]

👍 Great idea! It will make Rocket.Chat adaptable to many more installation scenarios (fixed sized SD card on a Raspberry Pi 2 - for example).

In the longer term, with two classes of users in mind:

• those who want to host Rocket.Chat for the masses
• enterprise and corporate users faced with compliance and audit

Message retention and per-user-quota will be completely driven by policies. Policies will require a 'roles and permissions' system being in place (there is a ticket for that somewhere). Basically, a policy may say - users in a student role will have only 5 MB storage quota, and their message retention is 3 weeks. The system will then enforce the policy.

Meanwhile, any implementation of pruning or per-user capping of storage compatible with the above will add value immediately.

[rodrigok]

O liked the idea, we can do this in 2 steps

1. General quantity/size limit or date expiration (we can do this direct in MongoDB)
2. Limits for quantity or time based on user's role (depends on role implementation and needs to be implemented in the application layer) and we can't limit by size.

The first step is relatively easy to do.

[AdrianoCahete]

enterprise and corporate users faced with compliance and audit

For audit, we need that ALL the operations are logged. From "giving permissions" to "leave channel".

[Sing-Li]

😀 @AdrianoCahete I'd say you're about a year ahead of your time.

[AdrianoCahete]

@Sing-Li Just habit.... I work in company who developing a system for Governance, Risks and Compliance. :P