add logging stack

[adriansalamon]

Adds:

• Loki, a log aggregation service/database. Stores logs in S3
• Prometheus, a metrics collection service. Uses service discovery to find metrics for systems in nomad. Also collects node metrics for all nodes, and nomad metrics for all clients.
• Grafana, a nice frontend to visualize/build dashboards for both.
• Vector, a log scraping/processing service. Deployed on each nomad client.

Some implementation details:

We use vector as a nomad job, where it mounts to the local docker socket to read docker logs, and ships them to loki.

Prometheus can use Nomad service discovery to find scraping targets. For example, you could add a new endpoint to your service that exposes metrics (different than the public port ofc.), and add metrics like:

service {
        port     = "metrics_http"
        provider = "nomad"
        tags = [
          "prometheus.scrape=true",
          "traefik.enable=true",
          "traefik.http.routers.tiki-metrics.rule=Host(`tiki-metrics.nomad.dsekt.internal`)",
          "traefik.http.routers.tiki-metrics.entrypoints=web-internal",
        ]
      }
}

For service discovery for node metrics and similar, we can use DNS based service discovery. Ie. prometheus does a query to _node._tcp.monitoring.dsekt.internal, and gets SRV records for IPs and ports to scrape.

Note: Grafana, Loki, and Prometheus jobs are already deployed and working.

PS. I have no idea of how we effectively deploy/test this.

[adriansalamon]

On another note, we could consider using consul for service discovery and as a distributed kv-store because it is very nice ☺️

[foodelevator]

On the same other note: the current solution, i.e. using traefik and manual dns config for service "discovery" was picked because nomads built in service discovery doesn't support discovering services in other namespaces, and when I tried consul I tried to use the service mesh which I did not get working so I threw out consul entirely. But using it for service discovery but not service mesh might be nice.

[adriansalamon]

This is all deployed and working. Still todo:

• Setup OIDC and permission mappings for Grafana. Maybe this is blocked by Add some type of "group" or "roles" scope sso#24.
• Add some auth for reading logs from Loki. Should probably be locked down so only Grafana can read them.

[Poizon7]

lgtm