Perf issues after m9 deployment

[rafrombrc]

Folks are reporting major lag and STMO performance issues that started conspicuously after the last deployment went out, in bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=1394468) as well as multiple times in various IRC and slack channels. This is impacting people's work, so hopefully we can figure this out and get a 9.1 (or whatever the next minor rev is) out ASAP.

[madalincm]

The issue is most visible in the new query page and appears to be reproducing for all the data sources in production. In stage the issue is only visible for the Athena data source. Here is a link for a profile for a query written for against the Athena data source: http://bit.ly/2vrU6Ag

Also I have observed another issue that might be related. When writing a query the first letter introduced by the user is duplicated. This is only reproducing in production. Please view this screencast:

[alison985]

I can reproduce slowness in prod on OS X in Nightly and 55.0.3 (64-bit). Nightly is giving a javascript error, though it's a minified variable name. I'm going to create/update a local test environment on my nightly machine.

[alison985]

So... in nightly with postgres - current Mozilla master autocomplete works fine and has the drop-down on my local and production.

Production misses the dropdown and has a speed issue on Athena and Presto.

[alison985]

Just merged PR #233 to staging. Don't think it will fix the issue but it will be helpful for diagnostic purposes since Athena and Presto are the same on production and Athena is on staging and production.

[wlach]

I did a quick profile yesterday of this, and it seems like the client-side lag is coming from some long-running javascript in angular's digest cycle: https://www.sitepoint.com/understanding-angulars-apply-digest/

Several possibilities here come to mind:

1. We're processing more data in the digest cycle than we were before due to a server-side change
2. The digest cycle is being triggered more often than before due to a client-side change
3. We're doing more in the digest cycle due to a client-side change

[alison985]

So the JS in redash has a 5000 token limit set right now to turn the auto completer on and off with a note about performance. Also this isn't reproducible in nightly on staging or prod for the metadata data source.

I lowered the redash javascript token count check to 3000 as part of PR #234 and will see if that helps on staging. Given that the problem doesn't happen in Chrome (and at least 1 older version of Firefox), I currently think this was happening for a combination of:

1. the list of fields/tables in presto and athena got longer.
2. firefox latest/firefox nightly is doing something differently with javascript interpretation that was making it take longer.