[BUG] errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key

[ankar84]

Description:

From time to time rocket chat instances show that error:
errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "6mkv7tvrJAmi4AJrR", : "pE4ztmmZFNJ5TZXFv", : "raufRbopbb9NDcGsQ" }',

But @frdmn advised me that it is an old bug, that was already fixed in #10061 by @sampaiodiego
But our server version is 3.0.3 and we have that bug on a regular basis.
How to fix it? Delete duplicate from mongodb? How to do it correctly?

Expected behavior:

Not have error E11000 duplicate key error index

Actual behavior:

Error and maybe rocket chat instance crash and server resource consume.

Server Setup Information:

• Version of Rocket.Chat Server: 3.0.3
• Operating System: CentOS7
• Deployment Method: docker
• Number of Running Instances: 12
• DB Replicaset Oplog: Enabled
• NodeJS Version: v12.14.0
• MongoDB Version: 4.0.13

Client Setup Information

• Desktop App Version: 2.17.9
• Operating System:

Additional context

Show Read Receipts and Detailed Read Receipts are enabled

Allow Message Editing and Allow Message Deleting are enabled for 10 minutes

Relevant logs:

Logs from Kibana:

Details

	March 27th 2020, 12:51:55.623	s-rc-srv-03	rocketchat_rocketchat_3_1	=================================
	March 27th 2020, 12:51:55.622	s-rc-srv-03	rocketchat_rocketchat_3_1	Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
	March 27th 2020, 12:51:55.622	s-rc-srv-03	rocketchat_rocketchat_3_1	Errors like this can cause oplog processing errors.
	March 27th 2020, 12:51:55.622	s-rc-srv-03	rocketchat_rocketchat_3_1	Future node.js versions will automatically exit the process
	March 27th 2020, 12:51:55.621	s-rc-srv-03	rocketchat_rocketchat_3_1	[Symbol(mongoErrorContextSymbol)]: {}
	March 27th 2020, 12:51:55.621	s-rc-srv-03	rocketchat_rocketchat_3_1	---------------------------------
	March 27th 2020, 12:51:55.621	s-rc-srv-03	rocketchat_rocketchat_3_1	}
	March 27th 2020, 12:51:55.620	s-rc-srv-03	rocketchat_rocketchat_3_1	},
	March 27th 2020, 12:51:55.620	s-rc-srv-03	rocketchat_rocketchat_3_1	}
	March 27th 2020, 12:51:55.615	s-rc-srv-03	rocketchat_rocketchat_3_1	lastOp: [Object]
	March 27th 2020, 12:51:55.614	s-rc-srv-03	rocketchat_rocketchat_3_1	nModified: 0,
	March 27th 2020, 12:51:55.614	s-rc-srv-03	rocketchat_rocketchat_3_1	nRemoved: 0,
	March 27th 2020, 12:51:55.614	s-rc-srv-03	rocketchat_rocketchat_3_1	nMatched: 0,
	March 27th 2020, 12:51:55.614	s-rc-srv-03	rocketchat_rocketchat_3_1	upserted: [],
	March 27th 2020, 12:51:55.613	s-rc-srv-03	rocketchat_rocketchat_3_1	nUpserted: 0,
	March 27th 2020, 12:51:55.613	s-rc-srv-03	rocketchat_rocketchat_3_1	nInserted: 0,
	March 27th 2020, 12:51:55.613	s-rc-srv-03	rocketchat_rocketchat_3_1	insertedIds: [Array],
	March 27th 2020, 12:51:55.612	s-rc-srv-03	rocketchat_rocketchat_3_1	writeConcernErrors: [],
	March 27th 2020, 12:51:55.607	s-rc-srv-03	rocketchat_rocketchat_3_1	writeErrors: [Array],
	March 27th 2020, 12:51:55.606	s-rc-srv-03	rocketchat_rocketchat_3_1	result: {
	March 27th 2020, 12:51:55.606	s-rc-srv-03	rocketchat_rocketchat_3_1	ok: 1,
	March 27th 2020, 12:51:55.605	s-rc-srv-03	rocketchat_rocketchat_3_1	}
	March 27th 2020, 12:51:55.605	s-rc-srv-03	rocketchat_rocketchat_3_1	result: BulkWriteResult {
	March 27th 2020, 12:51:55.605	s-rc-srv-03	rocketchat_rocketchat_3_1	},
	March 27th 2020, 12:51:55.604	s-rc-srv-03	rocketchat_rocketchat_3_1	messageId: 'raufRbopbb9NDcGsQ',
	March 27th 2020, 12:51:55.604	s-rc-srv-03	rocketchat_rocketchat_3_1	ts: 2020-03-27T09:51:55.555Z
	March 27th 2020, 12:51:55.604	s-rc-srv-03	rocketchat_rocketchat_3_1	userId: 'pE4ztmmZFNJ5TZXFv',
	March 27th 2020, 12:51:55.603	s-rc-srv-03	rocketchat_rocketchat_3_1	roomId: '6mkv7tvrJAmi4AJrR',
	March 27th 2020, 12:51:55.601	s-rc-srv-03	rocketchat_rocketchat_3_1	_id: 'DyBPDuqasguFt2BCD',
	March 27th 2020, 12:51:55.597	s-rc-srv-03	rocketchat_rocketchat_3_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "6mkv7tvrJAmi4AJrR", : "pE4ztmmZFNJ5TZXFv", : "raufRbopbb9NDcGsQ" }',
	March 27th 2020, 12:51:55.597	s-rc-srv-03	rocketchat_rocketchat_3_1	op: {
	March 27th 2020, 12:51:55.596	s-rc-srv-03	rocketchat_rocketchat_3_1	err: {
	March 27th 2020, 12:51:55.596	s-rc-srv-03	rocketchat_rocketchat_3_1	index: 0,
	March 27th 2020, 12:51:55.596	s-rc-srv-03	rocketchat_rocketchat_3_1	code: 11000,
	March 27th 2020, 12:51:55.595	s-rc-srv-03	rocketchat_rocketchat_3_1	name: 'BulkWriteError',
	March 27th 2020, 12:51:55.595	s-rc-srv-03	rocketchat_rocketchat_3_1	driver: true,
	March 27th 2020, 12:51:55.595	s-rc-srv-03	rocketchat_rocketchat_3_1	},
	March 27th 2020, 12:51:55.594	s-rc-srv-03	rocketchat_rocketchat_3_1	ts: 2020-03-27T09:51:55.555Z
	March 27th 2020, 12:51:55.592	s-rc-srv-03	rocketchat_rocketchat_3_1	messageId: 'raufRbopbb9NDcGsQ',
	March 27th 2020, 12:51:55.590	s-rc-srv-03	rocketchat_rocketchat_3_1	userId: 'pE4ztmmZFNJ5TZXFv',
	March 27th 2020, 12:51:55.589	s-rc-srv-03	rocketchat_rocketchat_3_1	_id: 'DyBPDuqasguFt2BCD',
	March 27th 2020, 12:51:55.589	s-rc-srv-03	rocketchat_rocketchat_3_1	roomId: '6mkv7tvrJAmi4AJrR',
	March 27th 2020, 12:51:55.588	s-rc-srv-03	rocketchat_rocketchat_3_1	code: 11000,
	March 27th 2020, 12:51:55.588	s-rc-srv-03	rocketchat_rocketchat_3_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "6mkv7tvrJAmi4AJrR", : "pE4ztmmZFNJ5TZXFv", : "raufRbopbb9NDcGsQ" }',
	March 27th 2020, 12:51:55.588	s-rc-srv-03	rocketchat_rocketchat_3_1	op: {
	March 27th 2020, 12:51:55.587	s-rc-srv-03	rocketchat_rocketchat_3_1	at processTicksAndRejections (internal/process/task_queues.js:75:11) {
	March 27th 2020, 12:51:55.587	s-rc-srv-03	rocketchat_rocketchat_3_1	index: 0,
	March 27th 2020, 12:51:55.587	s-rc-srv-03	rocketchat_rocketchat_3_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:397:18
	March 27th 2020, 12:51:55.586	s-rc-srv-03	rocketchat_rocketchat_3_1	at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/replset.js:1155:22)
	March 27th 2020, 12:51:55.581	s-rc-srv-03	rocketchat_rocketchat_3_1	at resultHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:501:23)
	March 27th 2020, 12:51:55.580	s-rc-srv-03	rocketchat_rocketchat_3_1	BulkWriteError: E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "6mkv7tvrJAmi4AJrR", : "pE4ztmmZFNJ5TZXFv", : "raufRbopbb9NDcGsQ" }
	March 27th 2020, 12:51:55.580	s-rc-srv-03	rocketchat_rocketchat_3_1	at OrderedBulkOperation.handleWriteError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:1112:11)
	March 27th 2020, 12:51:55.566	s-rc-srv-03	rocketchat_rocketchat_3_1	=== UnHandledPromiseRejection ===
	March 27th 2020, 12:39:26.826	s-rc-srv-01	rocketchat_rocketchat_2_1	Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
	March 27th 2020, 12:39:26.826	s-rc-srv-01	rocketchat_rocketchat_2_1	=================================
	March 27th 2020, 12:39:26.826	s-rc-srv-01	rocketchat_rocketchat_2_1	Future node.js versions will automatically exit the process
	March 27th 2020, 12:39:26.822	s-rc-srv-01	rocketchat_rocketchat_2_1	Errors like this can cause oplog processing errors.
	March 27th 2020, 12:39:26.820	s-rc-srv-01	rocketchat_rocketchat_2_1	[Symbol(mongoErrorContextSymbol)]: {}
	March 27th 2020, 12:39:26.820	s-rc-srv-01	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:39:26.820	s-rc-srv-01	rocketchat_rocketchat_2_1	---------------------------------
	March 27th 2020, 12:39:26.819	s-rc-srv-01	rocketchat_rocketchat_2_1	lastOp: [Object]
	March 27th 2020, 12:39:26.819	s-rc-srv-01	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:39:26.819	s-rc-srv-01	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:39:26.819	s-rc-srv-01	rocketchat_rocketchat_2_1	upserted: [],
	March 27th 2020, 12:39:26.818	s-rc-srv-01	rocketchat_rocketchat_2_1	nRemoved: 0,
	March 27th 2020, 12:39:26.818	s-rc-srv-01	rocketchat_rocketchat_2_1	nMatched: 0,
	March 27th 2020, 12:39:26.818	s-rc-srv-01	rocketchat_rocketchat_2_1	nModified: 0,
	March 27th 2020, 12:39:26.815	s-rc-srv-01	rocketchat_rocketchat_2_1	nUpserted: 0,
	March 27th 2020, 12:39:26.814	s-rc-srv-01	rocketchat_rocketchat_2_1	writeConcernErrors: [],
	March 27th 2020, 12:39:26.814	s-rc-srv-01	rocketchat_rocketchat_2_1	insertedIds: [Array],
	March 27th 2020, 12:39:26.814	s-rc-srv-01	rocketchat_rocketchat_2_1	nInserted: 0,
	March 27th 2020, 12:39:26.814	s-rc-srv-01	rocketchat_rocketchat_2_1	writeErrors: [Array],
	March 27th 2020, 12:39:26.813	s-rc-srv-01	rocketchat_rocketchat_2_1	result: BulkWriteResult {
	March 27th 2020, 12:39:26.813	s-rc-srv-01	rocketchat_rocketchat_2_1	ok: 1,
	March 27th 2020, 12:39:26.813	s-rc-srv-01	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:39:26.813	s-rc-srv-01	rocketchat_rocketchat_2_1	result: {
	March 27th 2020, 12:39:26.806	s-rc-srv-01	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:39:26.804	s-rc-srv-01	rocketchat_rocketchat_2_1	roomId: 'GMqJTNEx74S3oGGLZxGRrsJoTqEi9cFYbN',
	March 27th 2020, 12:39:26.804	s-rc-srv-01	rocketchat_rocketchat_2_1	userId: 'GMqJTNEx74S3oGGLZ',
	March 27th 2020, 12:39:26.804	s-rc-srv-01	rocketchat_rocketchat_2_1	messageId: 'gu9BrisadJMaEcFRn',
	March 27th 2020, 12:39:26.804	s-rc-srv-01	rocketchat_rocketchat_2_1	ts: 2020-03-27T09:39:26.770Z
	March 27th 2020, 12:39:26.803	s-rc-srv-01	rocketchat_rocketchat_2_1	code: 11000,
	March 27th 2020, 12:39:26.803	s-rc-srv-01	rocketchat_rocketchat_2_1	op: {
	March 27th 2020, 12:39:26.803	s-rc-srv-01	rocketchat_rocketchat_2_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "GMqJTNEx74S3oGGLZxGRrsJoTqEi9cFYbN", : "GMqJTNEx74S3oGGLZ", : "gu9BrisadJMaEcFRn" }',
	March 27th 2020, 12:39:26.803	s-rc-srv-01	rocketchat_rocketchat_2_1	_id: 'o9W4gbNQCnWvPa23R',
	March 27th 2020, 12:39:26.802	s-rc-srv-01	rocketchat_rocketchat_2_1	index: 0,
	March 27th 2020, 12:39:26.802	s-rc-srv-01	rocketchat_rocketchat_2_1	err: {
	March 27th 2020, 12:39:26.798	s-rc-srv-01	rocketchat_rocketchat_2_1	driver: true,
	March 27th 2020, 12:39:26.797	s-rc-srv-01	rocketchat_rocketchat_2_1	name: 'BulkWriteError',
	March 27th 2020, 12:39:26.796	s-rc-srv-01	rocketchat_rocketchat_2_1	userId: 'GMqJTNEx74S3oGGLZ',
	March 27th 2020, 12:39:26.796	s-rc-srv-01	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:39:26.796	s-rc-srv-01	rocketchat_rocketchat_2_1	messageId: 'gu9BrisadJMaEcFRn',
	March 27th 2020, 12:39:26.796	s-rc-srv-01	rocketchat_rocketchat_2_1	ts: 2020-03-27T09:39:26.770Z
	March 27th 2020, 12:39:26.795	s-rc-srv-01	rocketchat_rocketchat_2_1	_id: 'o9W4gbNQCnWvPa23R',
	March 27th 2020, 12:39:26.795	s-rc-srv-01	rocketchat_rocketchat_2_1	roomId: 'GMqJTNEx74S3oGGLZxGRrsJoTqEi9cFYbN',
	March 27th 2020, 12:39:26.795	s-rc-srv-01	rocketchat_rocketchat_2_1	op: {
	March 27th 2020, 12:39:26.794	s-rc-srv-01	rocketchat_rocketchat_2_1	code: 11000,
	March 27th 2020, 12:39:26.794	s-rc-srv-01	rocketchat_rocketchat_2_1	index: 0,
	March 27th 2020, 12:39:26.794	s-rc-srv-01	rocketchat_rocketchat_2_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "GMqJTNEx74S3oGGLZxGRrsJoTqEi9cFYbN", : "GMqJTNEx74S3oGGLZ", : "gu9BrisadJMaEcFRn" }',
	March 27th 2020, 12:39:26.791	s-rc-srv-01	rocketchat_rocketchat_2_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:397:18
	March 27th 2020, 12:39:26.791	s-rc-srv-01	rocketchat_rocketchat_2_1	at processTicksAndRejections (internal/process/task_queues.js:75:11) {
	March 27th 2020, 12:39:26.790	s-rc-srv-01	rocketchat_rocketchat_2_1	at resultHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:501:23)
	March 27th 2020, 12:39:26.790	s-rc-srv-01	rocketchat_rocketchat_2_1	at OrderedBulkOperation.handleWriteError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:1112:11)
	March 27th 2020, 12:39:26.790	s-rc-srv-01	rocketchat_rocketchat_2_1	at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/replset.js:1155:22)
	March 27th 2020, 12:39:26.789	s-rc-srv-01	rocketchat_rocketchat_2_1	BulkWriteError: E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "GMqJTNEx74S3oGGLZxGRrsJoTqEi9cFYbN", : "GMqJTNEx74S3oGGLZ", : "gu9BrisadJMaEcFRn" }
	March 27th 2020, 12:39:26.777	s-rc-srv-01	rocketchat_rocketchat_2_1	=== UnHandledPromiseRejection ===
	March 27th 2020, 12:36:13.659	s-rc-srv-02	rocketchat_rocketchat_2_1	Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
	March 27th 2020, 12:36:13.659	s-rc-srv-02	rocketchat_rocketchat_2_1	=================================
	March 27th 2020, 12:36:13.659	s-rc-srv-02	rocketchat_rocketchat_2_1	Errors like this can cause oplog processing errors.
	March 27th 2020, 12:36:13.659	s-rc-srv-02	rocketchat_rocketchat_2_1	Future node.js versions will automatically exit the process
	March 27th 2020, 12:36:13.658	s-rc-srv-02	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:36:13.658	s-rc-srv-02	rocketchat_rocketchat_2_1	---------------------------------
	March 27th 2020, 12:36:13.658	s-rc-srv-02	rocketchat_rocketchat_2_1	[Symbol(mongoErrorContextSymbol)]: {}
	March 27th 2020, 12:36:13.657	s-rc-srv-02	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:36:13.655	s-rc-srv-02	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:36:13.654	s-rc-srv-02	rocketchat_rocketchat_2_1	lastOp: [Object]
	March 27th 2020, 12:36:13.653	s-rc-srv-02	rocketchat_rocketchat_2_1	nModified: 0,
	March 27th 2020, 12:36:13.653	s-rc-srv-02	rocketchat_rocketchat_2_1	upserted: [],
	March 27th 2020, 12:36:13.653	s-rc-srv-02	rocketchat_rocketchat_2_1	nRemoved: 0,
	March 27th 2020, 12:36:13.652	s-rc-srv-02	rocketchat_rocketchat_2_1	nMatched: 0,
	March 27th 2020, 12:36:13.652	s-rc-srv-02	rocketchat_rocketchat_2_1	nInserted: 0,
	March 27th 2020, 12:36:13.652	s-rc-srv-02	rocketchat_rocketchat_2_1	nUpserted: 0,
	March 27th 2020, 12:36:13.651	s-rc-srv-02	rocketchat_rocketchat_2_1	writeConcernErrors: [],
	March 27th 2020, 12:36:13.651	s-rc-srv-02	rocketchat_rocketchat_2_1	insertedIds: [Array],
	March 27th 2020, 12:36:13.650	s-rc-srv-02	rocketchat_rocketchat_2_1	writeErrors: [Array],
	March 27th 2020, 12:36:13.646	s-rc-srv-02	rocketchat_rocketchat_2_1	ok: 1,
	March 27th 2020, 12:36:13.645	s-rc-srv-02	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:36:13.645	s-rc-srv-02	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:36:13.645	s-rc-srv-02	rocketchat_rocketchat_2_1	result: BulkWriteResult {
	March 27th 2020, 12:36:13.645	s-rc-srv-02	rocketchat_rocketchat_2_1	result: {
	March 27th 2020, 12:36:13.644	s-rc-srv-02	rocketchat_rocketchat_2_1	userId: 'QFNqm28joYJqhmBCA',
	March 27th 2020, 12:36:13.644	s-rc-srv-02	rocketchat_rocketchat_2_1	ts: 2020-03-27T09:36:13.620Z
	March 27th 2020, 12:36:13.644	s-rc-srv-02	rocketchat_rocketchat_2_1	_id: 'tz4GPFEmF2NmwHpzY',
	March 27th 2020, 12:36:13.644	s-rc-srv-02	rocketchat_rocketchat_2_1	roomId: 'xnGcER23hDhCaXS6G',
	March 27th 2020, 12:36:13.644	s-rc-srv-02	rocketchat_rocketchat_2_1	messageId: 'Ewi50RwMhZLGEJfv6',
	March 27th 2020, 12:36:13.643	s-rc-srv-02	rocketchat_rocketchat_2_1	op: {
	March 27th 2020, 12:36:13.641	s-rc-srv-02	rocketchat_rocketchat_2_1	code: 11000,
	March 27th 2020, 12:36:13.641	s-rc-srv-02	rocketchat_rocketchat_2_1	index: 0,
	March 27th 2020, 12:36:13.641	s-rc-srv-02	rocketchat_rocketchat_2_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "xnGcER23hDhCaXS6G", : "QFNqm28joYJqhmBCA", : "Ewi50RwMhZLGEJfv6" }',
	March 27th 2020, 12:36:13.640	s-rc-srv-02	rocketchat_rocketchat_2_1	name: 'BulkWriteError',
	March 27th 2020, 12:36:13.640	s-rc-srv-02	rocketchat_rocketchat_2_1	driver: true,
	March 27th 2020, 12:36:13.640	s-rc-srv-02	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:36:13.640	s-rc-srv-02	rocketchat_rocketchat_2_1	err: {
	March 27th 2020, 12:36:13.639	s-rc-srv-02	rocketchat_rocketchat_2_1	ts: 2020-03-27T09:36:13.620Z
	March 27th 2020, 12:36:13.639	s-rc-srv-02	rocketchat_rocketchat_2_1	userId: 'QFNqm28joYJqhmBCA',
	March 27th 2020, 12:36:13.639	s-rc-srv-02	rocketchat_rocketchat_2_1	messageId: 'Ewi50RwMhZLGEJfv6',
	March 27th 2020, 12:36:13.638	s-rc-srv-02	rocketchat_rocketchat_2_1	roomId: 'xnGcER23hDhCaXS6G',
	March 27th 2020, 12:36:13.634	s-rc-srv-02	rocketchat_rocketchat_2_1	op: {
	March 27th 2020, 12:36:13.634	s-rc-srv-02	rocketchat_rocketchat_2_1	code: 11000,
	March 27th 2020, 12:36:13.634	s-rc-srv-02	rocketchat_rocketchat_2_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "xnGcER23hDhCaXS6G", : "QFNqm28joYJqhmBCA", : "Ewi50RwMhZLGEJfv6" }',
	March 27th 2020, 12:36:13.634	s-rc-srv-02	rocketchat_rocketchat_2_1	_id: 'tz4GPFEmF2NmwHpzY',
	March 27th 2020, 12:36:13.633	s-rc-srv-02	rocketchat_rocketchat_2_1	at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/replset.js:1155:22)
	March 27th 2020, 12:36:13.633	s-rc-srv-02	rocketchat_rocketchat_2_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:397:18
	March 27th 2020, 12:36:13.633	s-rc-srv-02	rocketchat_rocketchat_2_1	at processTicksAndRejections (internal/process/task_queues.js:75:11) {
	March 27th 2020, 12:36:13.633	s-rc-srv-02	rocketchat_rocketchat_2_1	index: 0,
	March 27th 2020, 12:36:13.632	s-rc-srv-02	rocketchat_rocketchat_2_1	at OrderedBulkOperation.handleWriteError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:1112:11)
	March 27th 2020, 12:36:13.632	s-rc-srv-02	rocketchat_rocketchat_2_1	at resultHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:501:23)
	March 27th 2020, 12:36:13.627	s-rc-srv-02	rocketchat_rocketchat_2_1	BulkWriteError: E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "xnGcER23hDhCaXS6G", : "QFNqm28joYJqhmBCA", : "Ewi50RwMhZLGEJfv6" }
	March 27th 2020, 12:36:13.626	s-rc-srv-02	rocketchat_rocketchat_2_1	=== UnHandledPromiseRejection ===
	March 27th 2020, 12:34:14.506	s-rc-srv-02	rocketchat_rocketchat_2_1	=================================
	March 27th 2020, 12:34:14.505	s-rc-srv-02	rocketchat_rocketchat_2_1	Future node.js versions will automatically exit the process
	March 27th 2020, 12:34:14.503	s-rc-srv-02	rocketchat_rocketchat_2_1	Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
	March 27th 2020, 12:34:14.502	s-rc-srv-02	rocketchat_rocketchat_2_1	Errors like this can cause oplog processing errors.
	March 27th 2020, 12:34:14.502	s-rc-srv-02	rocketchat_rocketchat_2_1	---------------------------------
	March 27th 2020, 12:34:14.501	s-rc-srv-02	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:34:14.501	s-rc-srv-02	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:34:14.501	s-rc-srv-02	rocketchat_rocketchat_2_1	[Symbol(mongoErrorContextSymbol)]: {}
	March 27th 2020, 12:34:14.501	s-rc-srv-02	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:34:14.500	s-rc-srv-02	rocketchat_rocketchat_2_1	nRemoved: 0,
	March 27th 2020, 12:34:14.500	s-rc-srv-02	rocketchat_rocketchat_2_1	lastOp: [Object]
	March 27th 2020, 12:34:14.500	s-rc-srv-02	rocketchat_rocketchat_2_1	upserted: [],
	March 27th 2020, 12:34:14.498	s-rc-srv-02	rocketchat_rocketchat_2_1	nModified: 0,
	March 27th 2020, 12:34:14.495	s-rc-srv-02	rocketchat_rocketchat_2_1	nMatched: 0,
	March 27th 2020, 12:34:14.495	s-rc-srv-02	rocketchat_rocketchat_2_1	nInserted: 0,
	March 27th 2020, 12:34:14.495	s-rc-srv-02	rocketchat_rocketchat_2_1	nUpserted: 0,
	March 27th 2020, 12:34:14.494	s-rc-srv-02	rocketchat_rocketchat_2_1	writeErrors: [Array],
	March 27th 2020, 12:34:14.494	s-rc-srv-02	rocketchat_rocketchat_2_1	writeConcernErrors: [],
	March 27th 2020, 12:34:14.494	s-rc-srv-02	rocketchat_rocketchat_2_1	ok: 1,
	March 27th 2020, 12:34:14.494	s-rc-srv-02	rocketchat_rocketchat_2_1	insertedIds: [Array],
	March 27th 2020, 12:34:14.493	s-rc-srv-02	rocketchat_rocketchat_2_1	result: {
	March 27th 2020, 12:34:14.493	s-rc-srv-02	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:34:14.493	s-rc-srv-02	rocketchat_rocketchat_2_1	result: BulkWriteResult {
	March 27th 2020, 12:34:14.492	s-rc-srv-02	rocketchat_rocketchat_2_1	}
	March 27th 2020, 12:34:14.490	s-rc-srv-02	rocketchat_rocketchat_2_1	ts: 2020-03-27T09:34:14.457Z
	March 27th 2020, 12:34:14.489	s-rc-srv-02	rocketchat_rocketchat_2_1	roomId: 'A9ag6QqqTqYnKDnafsh2tfEBdr4nZoqSH5',
	March 27th 2020, 12:34:14.489	s-rc-srv-02	rocketchat_rocketchat_2_1	op: {
	March 27th 2020, 12:34:14.489	s-rc-srv-02	rocketchat_rocketchat_2_1	userId: 'A9ag6QqqTqYnKDnaf',
	March 27th 2020, 12:34:14.489	s-rc-srv-02	rocketchat_rocketchat_2_1	messageId: 'fSgY8fg4BL9zWTBxi',
	March 27th 2020, 12:34:14.489	s-rc-srv-02	rocketchat_rocketchat_2_1	_id: 'FAL7JejWpQcvCHKoK',
	March 27th 2020, 12:34:14.488	s-rc-srv-02	rocketchat_rocketchat_2_1	code: 11000,
	March 27th 2020, 12:34:14.488	s-rc-srv-02	rocketchat_rocketchat_2_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "A9ag6QqqTqYnKDnafsh2tfEBdr4nZoqSH5", : "A9ag6QqqTqYnKDnaf", : "fSgY8fg4BL9zWTBxi" }',
	March 27th 2020, 12:34:14.488	s-rc-srv-02	rocketchat_rocketchat_2_1	index: 0,
	March 27th 2020, 12:34:14.487	s-rc-srv-02	rocketchat_rocketchat_2_1	err: {
	March 27th 2020, 12:34:14.486	s-rc-srv-02	rocketchat_rocketchat_2_1	driver: true,
	March 27th 2020, 12:34:14.484	s-rc-srv-02	rocketchat_rocketchat_2_1	ts: 2020-03-27T09:34:14.457Z
	March 27th 2020, 12:34:14.484	s-rc-srv-02	rocketchat_rocketchat_2_1	},
	March 27th 2020, 12:34:14.484	s-rc-srv-02	rocketchat_rocketchat_2_1	name: 'BulkWriteError',
	March 27th 2020, 12:34:14.484	s-rc-srv-02	rocketchat_rocketchat_2_1	messageId: 'fSgY8fg4BL9zWTBxi',
	March 27th 2020, 12:34:14.483	s-rc-srv-02	rocketchat_rocketchat_2_1	op: {
	March 27th 2020, 12:34:14.483	s-rc-srv-02	rocketchat_rocketchat_2_1	_id: 'FAL7JejWpQcvCHKoK',
	March 27th 2020, 12:34:14.483	s-rc-srv-02	rocketchat_rocketchat_2_1	roomId: 'A9ag6QqqTqYnKDnafsh2tfEBdr4nZoqSH5',
	March 27th 2020, 12:34:14.483	s-rc-srv-02	rocketchat_rocketchat_2_1	userId: 'A9ag6QqqTqYnKDnaf',
	March 27th 2020, 12:34:14.482	s-rc-srv-02	rocketchat_rocketchat_2_1	code: 11000,
	March 27th 2020, 12:34:14.482	s-rc-srv-02	rocketchat_rocketchat_2_1	errmsg: 'E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "A9ag6QqqTqYnKDnafsh2tfEBdr4nZoqSH5", : "A9ag6QqqTqYnKDnaf", : "fSgY8fg4BL9zWTBxi" }',
	March 27th 2020, 12:34:14.481	s-rc-srv-02	rocketchat_rocketchat_2_1	index: 0,
	March 27th 2020, 12:34:14.478	s-rc-srv-02	rocketchat_rocketchat_2_1	at processTicksAndRejections (internal/process/task_queues.js:75:11) {
	March 27th 2020, 12:34:14.478	s-rc-srv-02	rocketchat_rocketchat_2_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:397:18
	March 27th 2020, 12:34:14.477	s-rc-srv-02	rocketchat_rocketchat_2_1	at resultHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:501:23)
	March 27th 2020, 12:34:14.477	s-rc-srv-02	rocketchat_rocketchat_2_1	at OrderedBulkOperation.handleWriteError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/bulk/common.js:1112:11)
	March 27th 2020, 12:34:14.477	s-rc-srv-02	rocketchat_rocketchat_2_1	at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/replset.js:1155:22)
	March 27th 2020, 12:34:14.476	s-rc-srv-02	rocketchat_rocketchat_2_1	BulkWriteError: E11000 duplicate key error index: rocketchat.rocketchat_message_read_receipt.$roomId_1_userId_1_messageId_1 dup key: { : "A9ag6QqqTqYnKDnafsh2tfEBdr4nZoqSH5", : "A9ag6QqqTqYnKDnaf", : "fSgY8fg4BL9zWTBxi" }
	March 27th 2020, 12:34:14.465	s-rc-srv-02	rocketchat_rocketchat_2_1	=== UnHandledPromiseRejection ===
	March 27th 2020, 12:31:17.852	s-rc-srv-01	rocketchat_rocketchat_1_1	at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/replset.js:1155:22)
	March 27th 2020, 12:31:17.852	s-rc-srv-01	rocketchat_rocketchat_1_1	at processTicksAndRejections (internal/process/task_queues.js:75:11)
	March 27th 2020, 12:31:17.852	s-rc-srv-01	rocketchat_rocketchat_1_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:397:18
	March 27th 2020, 12:31:17.849	s-rc-srv-01	rocketchat_rocketchat_1_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/collection_ops.js:1259:5
	March 27th 2020, 12:31:17.849	s-rc-srv-01	rocketchat_rocketchat_1_1	at handleCallback (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:128:55)
	March 27th 2020, 12:31:17.848	s-rc-srv-01	rocketchat_rocketchat_1_1	at executeCallback (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:395:17)
	March 27th 2020, 12:31:17.848	s-rc-srv-01	rocketchat_rocketchat_1_1	at deleteCallback (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/collection_ops.js:349:17)
	March 27th 2020, 12:31:17.848	s-rc-srv-01	rocketchat_rocketchat_1_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/collection_ops.js:364:54
	March 27th 2020, 12:31:17.848	s-rc-srv-01	rocketchat_rocketchat_1_1	at ClientSession.endSession (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/sessions.js:134:41)
	March 27th 2020, 12:31:17.847	s-rc-srv-01	rocketchat_rocketchat_1_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/gridfs-stream/index.js:188:25
	March 27th 2020, 12:31:17.847	s-rc-srv-01	rocketchat_rocketchat_1_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:410:17
	March 27th 2020, 12:31:17.847	s-rc-srv-01	rocketchat_rocketchat_1_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:398:11
	March 27th 2020, 12:31:17.846	s-rc-srv-01	rocketchat_rocketchat_1_1	Error: FileNotFound: no file with id JwNf9f6ePkM7tKM8M found
	March 27th 2020, 12:13:18.506	s-rc-srv-02	rocketchat_rocketchat_4_1	at processTicksAndRejections (internal/process/task_queues.js:75:11)
	March 27th 2020, 12:13:18.506	s-rc-srv-02	rocketchat_rocketchat_4_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/collection_ops.js:1259:5
	March 27th 2020, 12:13:18.506	s-rc-srv-02	rocketchat_rocketchat_4_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:397:18
	March 27th 2020, 12:13:18.506	s-rc-srv-02	rocketchat_rocketchat_4_1	at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/topologies/replset.js:1155:22)
	March 27th 2020, 12:13:18.505	s-rc-srv-02	rocketchat_rocketchat_4_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/collection_ops.js:364:54
	March 27th 2020, 12:13:18.505	s-rc-srv-02	rocketchat_rocketchat_4_1	at handleCallback (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:128:55)
	March 27th 2020, 12:13:18.503	s-rc-srv-02	rocketchat_rocketchat_4_1	at deleteCallback (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/collection_ops.js:349:17)
	March 27th 2020, 12:13:18.503	s-rc-srv-02	rocketchat_rocketchat_4_1	at executeCallback (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:395:17)
	March 27th 2020, 12:13:18.499	s-rc-srv-02	rocketchat_rocketchat_4_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/gridfs-stream/index.js:188:25
	March 27th 2020, 12:13:18.499	s-rc-srv-02	rocketchat_rocketchat_4_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:410:17
	March 27th 2020, 12:13:18.499	s-rc-srv-02	rocketchat_rocketchat_4_1	at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:398:11
	March 27th 2020, 12:13:18.499	s-rc-srv-02	rocketchat_rocketchat_4_1	at ClientSession.endSession (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/sessions.js:134:41)
	March 27th 2020, 12:13:18.498	s-rc-srv-02	rocketchat_rocketchat_4_1	Error: FileNotFound: no file with id HC2JgAwfeCAyfGEZ6 found`

[ankar84]

Temporary disabled Detailed Read Receipts.
I think it is a main issue area

[sampaiodiego]

thanks for reporting @ankar84

as you mentioned, on #10061 the issue with message editings, do you happen to know what action causes the error you're seeing? even on the logs it is not clear what caused it.. if you know that will help a lot on finding a solution, thx

[ankar84]

thanks for reporting @ankar84

as you mentioned, on #10061 the issue with message editings, do you happen to know what action causes the error you're seeing? even on the logs it is not clear what caused it.. if you know that will help a lot on finding a solution, thx

Now true cause of that error is unknown. I did try to edit message after it was read by a few users and that error did not appear.
I will try to find what actually happened when that error appear in logs, because understand that it will help to handle that issue gratefully.

[emikolajczak]

Hi, check time synchro on all you servers. I had simillar issue when servers didn't synchronize time with NTP.

[dusatvoj]

I have same issue on 3.1.3. Is there anything I can help with? (Logs, etc.)

[sampaiodiego]

@dusatvoj yes. are you able to reproduce it every time you want or it just happen some times? I still don't know what actions cause the error to happen

[dusatvoj]

@sampaiodiego I'm sorry, I don't know how to reproduce it :/ but as it's already written here I think it could be caused with message edit and detailed read receipt. I'm not a programmer but how it's handled by clients when message is edited?
I'm thinking about the case when a man write some message. I send a read receipt. After that is message edited and I read it again and send another read receipt.

[galshiff]

Any update on this?
We have the same issue and it crashes the rocket chat instance...

[galshiff]

Hi, our team successfully found fix for this bug!
Could someone approve my account, so ill be able to push a pull request in order to close this issue and fix this bug?

Thanks ;)

[sampaiodiego]

that's good news @galshiff .. you just need to fork the repo and submit a pull request, no need to approval account.. see https://guides.github.com/activities/forking/

[ankar84]

Hi, our team successfully found fix for this bug!

Hi @galshiff
Do you know the way to reproduce that bug?

[sampaiodiego]

@ankar84 pls see #18560

[sampaiodiego]

Closed by #18560