Performance degradation investigation

[djbrooke]

In our retrospective today, @raprasad mentioned that there have been pages throwing performance alerts in New Relic. At first glance, there does not appear to be a common thread with these alerts.

We should investigate what's causing these alerts and determine small steps we can take to keep our performance where we want it.

[raprasad]

Some numbers for Jan. 21 to Feb. 21 from Google Analytics:

All pages: 5.77* seconds

• (*all pages, overall average)
• 6+ seconds: 23.49% of pageviews
• 7+ seconds: 17.21% of views

Dataset page: 8.53** seconds

• (**all dataset pages, overall average)

---

It is not "definitive" whether page slowness is a combination of slow-loading individual pages, overall system load, an unknown bug, or a combination of these.

However, there are known issues previously identified via tools such as google PageSpeeds.

As a first cut for examining issues:

• Narrow google analytics (GA) to a single day
• Choose a slow loading page. Example: On 2/21, GA reports 118 seconds to load this page: https://dataverse.harvard.edu/dataset.xhtml?persistentId=hdl:1902.1/10766
• Analyze the actual server logs for the selected slow-loading page. Looking for all the requests prior to and including that page request. e.g. Start with 1 hour of previous requests
  • As per @pameyer, include postgres logs, glassfish logs, solr logs
• Examine logs for unusual activity
• Replay requests via a script to examine system load and possible issues.

Barring a "silver bullet"/bad bug being found. Site performance enhancements may be made via:

• CDN use: Performance: Use a CDN for css, javascript, and static images #2647 (basic task)
• Following Google PageSpeed suggestions. These include:
  • Optimizing CSS ("no brainer; need to include in dev/deploy process")
  • Minifying javascript ("no brainer; need to include in dev/deploy process")
  • Optimizing server response time
    • Finding bugs. Query optimization. Caching. Storing metadata in JSON format,
  • Removing render-blocking javascript ("dev. work b/c primefaces related")
    • For a list of these files, use the PageSpeed links below.
    • May be harder in that they appear to be primefaces related.
  • See
    • homepage: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fdataverse.harvard.edu&tab=desktop
    • dataset page: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fdataverse.harvard.edu%2Fdataset.xhtml%3FpersistentId%3Dhdl%3A1902.1%2F10766

---

[pameyer]

It might be helpful to have some system-level information from the various servers involved (glassfish application servers, postgres servers, solr servers, apache servers) for the same time window. Assuming sysstat is installed, the contents of /var/log/sa would be a good starting point (possibly with more specific postgres level info if that looks to be the source of the problem).

[raprasad]

Important to note that stuff below is likely going to come into play--prepare to optimize

• Finding bugs. Query optimization. Caching. Storing metadata in JSON format

Guidelines for server response time:

• Target number established by google: 200 ms
• Example
  • CKAN page: 270 ms
  • open scholar. gking page: sub 200ms
• Sample DV page with 19 small files: 5700 ms yesterday, 6100 ms today
• Other sample DV pages are more in the 2000 ms range (10x the guideline)
  • This is only server response time--not time for js/css/rendering/etc

[raprasad]

• Beginning prototype that meets 200ms standard (metadata queries reduced to 20 queries 1st time; 1 query 2nd time):
  - https://services.dataverse.harvard.edu/miniverse/dvobjects/api/v1/datasets/by-persistent-id?persistentId=doi:10.7910/DVN1/22691
• In pagespeed:
  • https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fservices.dataverse.harvard.edu%2Fminiverse%2Fdvobjects%2Fapi%2Fv1%2Fdatasets%2Fby-persistent-id%3FpersistentId%3Ddoi%3A10.7910%2FDVN1%2F22691

[raprasad]

note: "prototype" above is not full-featured by any means. But thought is:

• Most of our traffic is "reads" vs. "writes"
• Dataset info, especially published dataset info is not going to change. e.g. Use JSON docs instead of re-querying each time

[djbrooke]

Great - it looks like we now have enough information to work on this when we pull it into a future sprint.

[djbrooke]

Closing in favor of #5824

[pdurbin]

@djbrooke I'm catching up on email and hope you don't mind that I'm clicking "close" for you. 😄