Skip to content

Comments

fix: add TTL index to statistics collection for automatic cleanup#38568

Merged
kodiakhq[bot] merged 4 commits intodevelopfrom
fix/statistics-ttl
Feb 18, 2026
Merged

fix: add TTL index to statistics collection for automatic cleanup#38568
kodiakhq[bot] merged 4 commits intodevelopfrom
fix/statistics-ttl

Conversation

@ricardogarim
Copy link
Contributor

@ricardogarim ricardogarim commented Feb 9, 2026

The rocketchat_statistics collection grows indefinitely as statistics are collected daily but never cleaned up.

Customer reported: After running since 2017 (~7 years), their statistics collection reached 496MB - larger than their messages collection (454MB).

Proposed changes (including videos or screenshots)

This PR adds a MongoDB TTL index to automatically expire statistics documents after 1 year (365 days), capping storage to approximately ~70MB regardless of how long the server runs.

Why TTL index?

  • Already used across 10+ collections in the codebase (IntegrationHistory, Sessions, AppLogs, NotificationQueue, OAuth tokens, Trash, etc.)
  • Zero runtime code - MongoDB handles expiration automatically
  • No admin configuration needed - 1 year retention is sufficient for operational/debugging needs

Issue(s)

Steps to test or reproduce

  1. Start Rocket.Chat server
  2. Connect to MongoDB and verify the TTL index was created: db.rocketchat_statistics.getIndexes()
  3. Confirm the output includes an index with expireAfterSeconds: 31536000: { "key": { "createdAt": 1 }, "name": "createdAt_1", "expireAfterSeconds": 31536000 }

Further comments

Alternatives considered:

  • Configurable setting + cron job: Over-engineered for this use case; stats retention doesn't need runtime configuration
  • Keep last N documents (count-based): Requires cleanup job code; time-based is more intuitive
  • Size-based cap (e.g., 50MB): MongoDB doesn't support natively; would require complex implementation

Important: in the future, when we reach the next major, we should merge PR #38572. It contains the index consolidation and a migration that we can’t merge right now, since migrations are only included in major releases. The work there is ready to go.

Summary by CodeRabbit

  • New Features

    • Automatic cleanup of historical statistics: statistics older than one year are now automatically expired, reducing retained historical data.
  • Chores

    • Patch release recorded and changelog entry added.

@dionisio-bot
Copy link
Contributor

dionisio-bot bot commented Feb 9, 2026

Looks like this PR is ready to merge! 🎉
If you have any trouble, please check the PR guidelines

@changeset-bot
Copy link

changeset-bot bot commented Feb 9, 2026

🦋 Changeset detected

Latest commit: fe92468

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 40 packages
Name Type
@rocket.chat/models Patch
@rocket.chat/meteor Patch
@rocket.chat/core-services Patch
@rocket.chat/cron Patch
@rocket.chat/instance-status Patch
@rocket.chat/omni-core Patch
@rocket.chat/ui-client Patch
@rocket.chat/account-service Patch
@rocket.chat/authorization-service Patch
@rocket.chat/ddp-streamer Patch
@rocket.chat/omnichannel-transcript Patch
@rocket.chat/presence-service Patch
@rocket.chat/queue-worker Patch
@rocket.chat/abac Patch
@rocket.chat/federation-matrix Patch
@rocket.chat/media-calls Patch
@rocket.chat/omni-core-ee Patch
@rocket.chat/omnichannel-services Patch
@rocket.chat/presence Patch
rocketchat-services Patch
@rocket.chat/network-broker Patch
@rocket.chat/core-typings Patch
@rocket.chat/rest-typings Patch
@rocket.chat/uikit-playground Patch
@rocket.chat/api-client Patch
@rocket.chat/apps Patch
@rocket.chat/ddp-client Patch
@rocket.chat/fuselage-ui-kit Patch
@rocket.chat/gazzodown Patch
@rocket.chat/http-router Patch
@rocket.chat/livechat Patch
@rocket.chat/model-typings Patch
@rocket.chat/ui-avatar Patch
@rocket.chat/ui-contexts Patch
@rocket.chat/ui-voip Patch
@rocket.chat/web-ui-registration Patch
@rocket.chat/license Patch
@rocket.chat/pdf-worker Patch
@rocket.chat/mock-providers Patch
@rocket.chat/ui-video-conf Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 9, 2026

Walkthrough

Adds a TTL index to the MongoDB statistics collection so documents expire after 365 days and updates the changelog to document the new automatic cleanup behavior.

Changes

Cohort / File(s) Summary
Changelog
.changeset/young-humans-stare.md
Adds release notes documenting automatic cleanup of the statistics collection with a 1-year TTL retention.
Statistics Model
packages/models/src/models/Statistics.ts
Adds an ascending index on createdAt with expireAfterSeconds: 31536000 (365 days) while preserving the existing descending index on createdAt.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 I hop through bytes and tidy boons,

Old stats depart beneath the moons,
One year to rest, then off they go,
My burrow’s clean, the logs run slow. 🥕

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding a TTL index to the statistics collection for automatic cleanup of old documents.
Linked Issues check ✅ Passed The code changes directly implement the core requirements from both linked issues: automatic cleanup via TTL index on rocketchat_statistics collection with 365-day retention, addressing unbounded growth.
Out of Scope Changes check ✅ Passed All changes are in-scope: changelog entry and TTL index addition to Statistics model are directly related to the linked issues' objectives of automatic statistics cleanup.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ricardogarim ricardogarim added this to the 8.2.0 milestone Feb 9, 2026
@codecov
Copy link

codecov bot commented Feb 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.47%. Comparing base (42151e2) to head (fe92468).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop   #38568      +/-   ##
===========================================
+ Coverage    70.45%   70.47%   +0.01%     
===========================================
  Files         3176     3176              
  Lines       111476   111476              
  Branches     20130    20156      +26     
===========================================
+ Hits         78545    78559      +14     
+ Misses       30879    30872       -7     
+ Partials      2052     2045       -7     
Flag Coverage Δ
e2e 60.40% <ø> (+0.02%) ⬆️
e2e-api 47.76% <ø> (ø)
unit 71.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

📦 Docker Image Size Report

📈 Changes

Service Current Baseline Change Percent
sum of all images 1.1GiB 1.1GiB +11MiB
rocketchat 360MiB 349MiB +11MiB
omnichannel-transcript-service 134MiB 134MiB -855B
queue-worker-service 134MiB 134MiB -1.3KiB
ddp-streamer-service 128MiB 128MiB -220B
account-service 115MiB 115MiB +271B
authorization-service 112MiB 112MiB +311B
presence-service 112MiB 112MiB -102B

📊 Historical Trend

---
config:
  theme: "dark"
  xyChart:
    width: 900
    height: 400
---
xychart
  title "Image Size Evolution by Service (Last 30 Days + This PR)"
  x-axis ["11/27 22:32", "11/28 19:05", "12/01 23:01", "12/02 21:57", "12/03 21:00", "12/04 18:17", "12/05 21:56", "12/08 20:15", "12/09 22:17", "12/10 23:26", "12/11 21:56", "12/12 22:45", "12/13 01:34", "12/15 22:31", "12/16 22:18", "12/17 21:04", "12/18 23:12", "12/19 23:27", "12/20 21:03", "12/22 18:54", "12/23 16:16", "12/24 19:38", "12/25 17:51", "12/26 13:18", "12/29 19:01", "12/30 20:52", "02/12 22:57", "02/13 22:38", "02/16 14:04", "02/18 21:49", "02/18 22:02 (PR)"]
  y-axis "Size (GB)" 0 --> 0.5
  line "account-service" [0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11]
  line "authorization-service" [0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11]
  line "ddp-streamer-service" [0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12]
  line "omnichannel-transcript-service" [0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13]
  line "presence-service" [0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11]
  line "queue-worker-service" [0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13]
  line "rocketchat" [0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.34, 0.35]
Loading

Statistics (last 30 days):

  • 📊 Average: 1.5GiB
  • ⬇️ Minimum: 1.4GiB
  • ⬆️ Maximum: 1.6GiB
  • 🎯 Current PR: 1.1GiB
ℹ️ About this report

This report compares Docker image sizes from this build against the develop baseline.

  • Tag: pr-38568
  • Baseline: develop
  • Timestamp: 2026-02-18 22:02:44 UTC
  • Historical data points: 30

Updated: Wed, 18 Feb 2026 22:02:44 GMT

Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Co-authored-by: Diego Sampaio <chinello@gmail.com>
@sampaiodiego sampaiodiego added the stat: QA assured Means it has been tested and approved by a company insider label Feb 18, 2026
@dionisio-bot dionisio-bot bot added the stat: ready to merge PR tested and approved waiting for merge label Feb 18, 2026
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e64adb7 and 58aa7be.

📒 Files selected for processing (1)
  • .changeset/young-humans-stare.md
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-11-19T12:32:29.696Z
Learnt from: d-gubert
Repo: RocketChat/Rocket.Chat PR: 37547
File: packages/i18n/src/locales/en.i18n.json:634-634
Timestamp: 2025-11-19T12:32:29.696Z
Learning: Repo: RocketChat/Rocket.Chat
Context: i18n workflow
Learning: In this repository, new translation keys should be added to packages/i18n/src/locales/en.i18n.json only; other locale files are populated via the external translation pipeline and/or fall back to English. Do not request adding the same key to all locale files in future reviews.

Applied to files:

  • .changeset/young-humans-stare.md
📚 Learning: 2025-11-10T19:06:20.146Z
Learnt from: MartinSchoeler
Repo: RocketChat/Rocket.Chat PR: 37408
File: apps/meteor/client/views/admin/ABAC/useRoomAttributeOptions.tsx:53-69
Timestamp: 2025-11-10T19:06:20.146Z
Learning: In the Rocket.Chat repository, do not provide suggestions or recommendations about code sections marked with TODO comments. The maintainers have already identified these as future work and external reviewers lack the full context about implementation plans and timing.

Applied to files:

  • .changeset/young-humans-stare.md
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
  • GitHub Check: 📦 Build Packages
  • GitHub Check: CodeQL-Build
  • GitHub Check: CodeQL-Build
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.changeset/young-humans-stare.md:
- Around line 2-3: The changeset currently marks '@rocket.chat/models' and
'@rocket.chat/meteor' as 'patch' but the addition of a TTL index is a
data-destructive feature and must be surfaced as at least a 'minor' (or 'major')
bump; update the .changeset entry to use 'minor' (or 'major' if you prefer the
stronger signal) for those package entries so releases and release notes
highlight the automatic one‑year expiry change and operators are warned.

@kodiakhq kodiakhq bot merged commit 123aebe into develop Feb 18, 2026
44 checks passed
@kodiakhq kodiakhq bot deleted the fix/statistics-ttl branch February 18, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stat: QA assured Means it has been tested and approved by a company insider stat: ready to merge PR tested and approved waiting for merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collected statistics are not automatically rotated / cleaned

2 participants