Upgrade from 0.63.3 to 0.65.0 fails with 'call' is not a function.-> Restore from backup only solution to get it back up and working on 0.63.3.

[Lawri-van-Buel]

Description:

Server Setup Information:

• Version of Rocket.Chat Server: 0.63.3 --> 0.65.0
• Operating System: Ubuntu 16.04 LTS
• Deployment Method(snap/docker/tar/etc): Ansible - Tar
• Number of Running Instances: 5
• DB Replicaset Oplog: Present
• Node Version: 8.9.4
• mongoDB Version: 3.4.6

Steps to Reproduce:

1. Have a running Rocketchat at version 0.63.3 with a slackbridge attached and log errors to a channel
2. Upgrade Rockechat to 0.65.0 using my ansible script (Merged changes of 3.1.0 and the development branch. Rocket.Chat.Ansible#84)
3. Get the following error "

May 28 10:17:14 Rocketchat1 RocketChat[757]: rocketchat:lib ➔ oauth_updated Accounts_OAuth_Wordpress_scope
May 28 10:17:15 Rocketchat1 RocketChat[757]: packages/rocketchat_monitoring.js:3549
May 28 10:17:15 Rocketchat1 RocketChat[757]: return originalRun.call(this, val);
May 28 10:17:15 Rocketchat1 RocketChat[757]: ^
May 28 10:17:15 Rocketchat1 RocketChat[757]: TypeError: Apps.isLoaded is not a function
May 28 10:17:15 Rocketchat1 RocketChat[757]: at EventEmitter.RocketChat.sendMessage (/var/lib/rocket.chat/bundle/programs/server/packages/rocketchat_lib.js:5421:31)
May 28 10:17:15 Rocketchat1 RocketChat[757]: at ErrorHandler.trackError (/var/lib/rocket.chat/bundle/programs/server/packages/rocketchat_error-handler.js:89:18)
May 28 10:17:15 Rocketchat1 RocketChat[757]: at Object.Meteor._debug (/var/lib/rocket.chat/bundle/programs/server/packages/rocketchat_error-handler.js:59:12)
May 28 10:17:15 Rocketchat1 RocketChat[757]: at onException (packages/meteor.js:1220:14)
May 28 10:17:15 Rocketchat1 RocketChat[757]: at runWithEnvironment (packages/meteor.js:1243:9)
May 28 10:17:15 Rocketchat1 systemd[1]: rocketchat.service: Main process exited, code=exited, status=7/NOTRUNNING
May 28 10:17:15 Rocketchat1 systemd[1]: rocketchat.service: Unit entered failed state.
May 28 10:17:15 Rocketchat1 systemd[1]: rocketchat.service: Failed with result 'exit-code'.
"

Expected behavior:

A runing instance of Rocketchat at version 0.65.0

Actual behavior:

Rocketchat Crashes and locks up the Mongo upgrade.
I had to revert to a Backup to get it back up and running.

Relevant logs:

Snippet is in ticket already. if more is needed request in comment.

[TwizzyDizzy]

I'm wondering how even your 0.63.3 setup was working (spoiler: it shouldn't have). To quote the release notes of Rocket.Chat 0.60.0:

0.60.0 and higher requires Node.js 8.x to be installed. If you are upgrading from a previous version, you will have to update your Node.js version before running.

So please upgrade to the latest nodejs 8.x version (currently 8.11.2) and report what happened.

Cheers
Thomas

[sampaiodiego]

@TwizzyDizzy looking at his Ansible script, node version seems correct.. he might have only typed it wrong

but @Lawri-van-Buel I'm unable to reproduce the error. the error seems to happen on startup, is it correct?

does it work if you disable slackbridge (on previous version) ?

[Lawri-van-Buel]

it happens on startup. I did not check if it works when slackbridge is disabled before upgrade, can it really affect the upgrade this much?

[Lawri-van-Buel]

Yes itys 8.9.4 not 4.9.4 ( typo on my end)

[Lawri-van-Buel]

I can try the upgrade again tomorrow with slackbridge disabled (its a live server and it does work on my stage which does not have a slackbridge)

[sampaiodiego]

@Lawri-van-Buel I'm currently trying to find a cause, since I was not able to reproduce on my environment (with either slackbridge enable or disabled)..

another question, do you have Log Exceptions to Channel enabled? we've verified a bug on it when enabled and something happens during startup and something is calling sendMessage during your server's startup.

[Lawri-van-Buel]

Yes we do.

[sampaiodiego]

@Lawri-van-Buel I'm working on a fix for this. In the meanwhile you guys can turn Log Exceptions to Channel off and update to latest version.

[sampaiodiego]

I've done #10934 which will not prevent this issue to happen, but if this happen it will then show where the real issue is.

[Lawri-van-Buel]

I can confirm #10934 prevents this issue. (it will require a double restart of rocketchat but thats fine)