Support for collectd and Grafana

[pdurbin]

Now that pull request #96 has been merged, Prometheus is installed and running.

From discussion at http://irclog.iq.harvard.edu/dataverse/2019-08-22#i_102865 @pmauduit recommends adding collectd and Grafana next.

Over at http://irclog.iq.harvard.edu/dataverse/2019-08-23#i_103066 I suggested creating this issue so @pmauduit can leave comments in here to indicate whatever commands he ran on an EC2 instance I spun up to install and configure collectd and Grafana. Thanks!!!

[pmauduit]

It seems that a vanilla setup of prometheus will gather by default some metrics related to its own process and golang. My idea then is to setup collectd with the prometheus writer:

$ yum install collectd
$ yum install collectd-write_prometheus.x86_64

# vim /etc/collectd.d/prometheus.conf
LoadPlugin write_prometheus

<Plugin "write_prometheus">
  Port "9103"
</Plugin>

And see if we can gather default metrics from collectd (memory usage, %CPU, at least it is the default ones we have under debian-like distros, I expect centos to provide the same).

[pmauduit]

Unfortunately, I stumble upon this issue:
collectd/collectd#2715

[pmauduit]

Unfortunately, I stumble upon this issue:
collectd/collectd#2715

Workaround discussed on IRC:

$ setenforce Permissive

[pmauduit]

Proometheus configuration to scrape Collectd:

# cat /usr/local/prometheus/prometheus.yml
[...]
scrape_configs:
  - job_name: 'dataverse'
    static_configs:
    - targets: ['localhost:9090']
- job_name: 'collectd'
  static_configs:
  - targets: ['localhost:9103']

[pmauduit]

Still need to setup grafana, and configure a dashboard. As discussed earlier on IRC, we could also optionally launch the Java J2EE container with some JVM options to collect extra metrics via JMX.

[pdurbin]

@pmauduit I hacked on http.proxy.conf and prometheus.service a bit and now I can get to Prometheus in a web browser, but I don't know if I've broken anything with regard to the communication between collectd and Prometheus.

When I say the web stuff is working better I mean that now (with my changes below), you can go to...

http://ec2-3-81-53-52.compute-1.amazonaws.com/prometheus

... and be redirected to...

http://ec2-3-81-53-52.compute-1.amazonaws.com/prometheus/graph

... and make a graph like this:

To do this I ended up adding --web.external-url=http://localhost:9090/prometheus/ to /usr/lib/systemd/system/prometheus.service like this:

[root@ip-172-31-36-60 conf.d]# cd /usr/lib/systemd/system
[root@ip-172-31-36-60 system]# diff -u prometheus.service.orig prometheus.service
--- prometheus.service.orig	2019-08-23 13:33:26.870668227 +0000
+++ prometheus.service	2019-08-23 17:17:50.677749685 +0000
@@ -11,6 +11,7 @@
 #Prometheus on different path user
 ExecStart=/usr/local/prometheus/prometheus \
   --config.file=/usr/local/prometheus/prometheus.yml \
+  --web.external-url=http://localhost:9090/prometheus/ \
   --storage.tsdb.path=/usr/local/prometheus/data
 
 [Install]
[root@ip-172-31-36-60 system]# 

This means that http://localhost:9090/config for example has moved to http://localhost:9090/prometheus/config (which is why I'm a bit concerned I broke something between collectd and prometheus).

With the change above in place, getting Apache working wasn't too bad. Here's the config change:

[root@ip-172-31-36-60 conf.d]# cd /etc/httpd/conf.d
[root@ip-172-31-36-60 conf.d]# diff -u http.proxy.conf.orig http.proxy.conf
--- http.proxy.conf.orig	2019-08-23 13:17:03.870938915 +0000
+++ http.proxy.conf	2019-08-23 17:17:07.532539631 +0000
@@ -2,6 +2,11 @@
 
   ServerName ec2-3-81-53-52.compute-1.amazonaws.com
 
+  <Location "/prometheus">
+    ProxyPass "http://localhost:9090/prometheus"
+    ProxyPassReverse "http://localhost:9090/prometheus"
+  </Location>
+
   # custom error document when Glassfish isn't responding
   ErrorDocument 503 /503.html
   ProxyPassMatch ^/503.html$ !
[root@ip-172-31-36-60 conf.d]# 

If we go with the change above I'm wondering if we also need to edit these files:

• /etc/collectd.d/prometheus.conf
• /usr/local/prometheus/prometheus.yml
• other files I'm not aware of 😄

Please let me know what you think! (You too @donsizemore .) Thanks!

[pmauduit]

• /etc/collectd.d/prometheus.conf

No, because it just defines an endpoint to be harvested by prometheus, it does not imply where prometheus actually is.

• /usr/local/prometheus/prometheus.yml

Yes because you have to add the endpoint "collectd" at the end of the file, but keeping as it is currently defined, i.e.:

  - job_name: 'collectd'
    static_configs:
    - targets: ['localhost:9103']

• other files I'm not aware of

As far as I remember, I did not have to modify anything else.

[pmauduit]

Grafana can be easily installed using the yum repo advised in the official doc: https://grafana.com/docs/installation/rpm/

# cat /etc/yum.repos.d/grafana.repo 
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

# yum install grafana
# service grafana-server start

The service would then be available on the 3000 port.

[pmauduit]

Grafana by default expects to be at the root of the webserver (as prometheus), we do need to reconfigure it in regards to the expected endpoint:

# cat /etc/grafana/grafana.ini
[...]
# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = http://ec2-3-81-53-52.compute-1.amazonaws.com/grafana

# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = true

And we also need to reconfigure the httpd server:

# cat /etc/httpd/conf.d/http.proxy.conf
[...]
  <Location "/grafana">
    ProxyPass "http://localhost:3000"
    ProxyPassReverse "http://localhost:3000"
  </Location>

[...]

[pmauduit]

Adding a datasource once connected onto the grafana ui (admin/admin by default):

[pmauduit]

Grafana with anonymous auth:

# cat /etc/grafana/grafana.ini
[...]
#################################### Anonymous Auth ######################
[auth.anonymous]
# enable anonymous access
enabled = true
[...]

[pmauduit]

Configuring datasource & dashboard in grafana using ansible is described here:

• https://docs.ansible.com/ansible/latest/modules/grafana_dashboard_module.html
• https://docs.ansible.com/ansible/latest/modules/grafana_datasource_module.html

a json spec of a very basic dashboard is attached: overall-metrics-dataverse.json.txt

[pmauduit]

monitoring glassfish via JMX

As the prometheus exporter, the jmx plugin for collectd is packaged separately, so we first need to install collectd-generic-jmx.x86_64 (which also requires collectd-java.x86_64):

$ yum install collectd-generic-jmx.x86_64 collectd-java.x86_64

A JMX endpoint has been already configured by @pdurbin into /usr/local/glassfish4/glassfish/domains/domain1/config/domain.xml.

A simple configuration for collectd can be done in /etc/collectd.conf.d/jmx.conf:
jmx.conf.txt

[pdurbin]

@pmauduit awesome to have Glassfish heap usage! Here's a screenshot for others:

Here's the updated dashboard config:dashboard.json.txt I hope I'm saving it correctly. 😄