feat(observability): add runner VM hostmetrics Grafana dashboard#187
Merged
feat(observability): add runner VM hostmetrics Grafana dashboard#187
Conversation
Adds a read-only Grafana dashboard (editable: false) for runner VM host-level metrics to be served via cos-configuration-k8s using the grafana-dashboard relation, which provisions it as an immutable filesystem dashboard in Grafana. The dashboard covers: - CPU utilisation by state and load averages - Memory usage by state - Disk I/O throughput and operations - Filesystem usage % by mount point - Network traffic, errors and drops Template variables: - github_job_id: filter by GitHub Actions workflow run job ID - instance: filter by runner hostname Metric names follow the OpenTelemetry hostmetrics receiver prometheus convention (e.g. system_cpu_time_seconds_total). The github_job_id label is expected to be set as a resource attribute by the otelcol pipeline collecting metrics from the runner VMs. Related: ISD-5152
Rename grafana_dashboards/ to runner_grafana_dashboards/ to make the purpose explicit at the repo root level (runner VM host metrics, not charm workload metrics). Update README with: - Repository layout overview - Observability section explaining the cos-configuration-k8s delivery mechanism and the immutability guarantee - Table of conventions for where dashboards live and what grafana_dashboards_path value to use in Terraform
Replace github_job_id with github_job and instance with github_runner to match the actual attribute labels set by the pre-job OTel config (see canonical/github-runner-operator#781). Add github_repository and github_workflow template variables so the dashboard can be filtered the same way as the existing PS6 hostmetrics dashboard.
…ayout Restructure the runner VM hostmetrics dashboard to follow the upstream OpenTelemetry hostmetrics dashboard (Grafana gnetId 24638): Overview row of CPU/Memory/Root FS gauges plus Load/Cores/Total Memory stats, then CPU, Memory, Disk I/O, Filesystem and Network sections with read/write and rx/tx split axes. Make every templating variable support "All" via includeAll, multi-select and allValue ".*", and switch all label matchers to =~ so regex interpolation works.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new provisioned (read-only) Grafana dashboard for GitHub Actions runner VM host-level metrics and documents how dashboards are laid out and delivered via cos-configuration-k8s.
Changes:
- Add
runner_grafana_dashboards/runner_vm_hostmetrics.jsonwith CPU/memory/disk/filesystem/network panels based on OTel hostmetrics Prometheus metrics. - Update
README.mdto document repository layout and the Grafana dashboard delivery mechanism/conventions.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| runner_grafana_dashboards/runner_vm_hostmetrics.json | New Grafana dashboard JSON for runner VM host metrics with templating variables and PromQL queries. |
| README.md | Documents dashboard locations and provisioning conventions via cos-configuration-k8s / grafana-dashboard. |
…dashboard
When the runner variable resolves to multiple series (multi-select or
"All"), several panels previously produced misleading values:
- CPU Cores stat / System Load "cores" reference: count(count by (cpu) ...)
collapses cpu indexes across runners, returning the max-cores-on-any-host
rather than fleet total. Group by github_runner so cpu indexes stay
distinct, then expose total cores in the stat panel and per-runner cores
on the load panel (so the reference aligns with the averaged load lines).
- System Load 1m/5m/15m: bare metric returns one series per runner with
identical legends ("1m"/"5m"/"15m"), making the chart unreadable. Wrap
in avg() to get one fleet-average line per period.
- Disk Busy %: sum by (device) of fractional busy time can exceed 1 with
multiple runners and gets silently clamped by max:1. Switch to
avg by (device) so the value stays a meaningful 0-1 fleet average.
Also soften the README guidance on editable: false. cos-configuration-k8s
provisions dashboards from the filesystem, which makes them read-only in
Grafana regardless of the flag, so the explicit "must" requirement was
contradicted by existing dashboards in charms/planner-operator/.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new filesystem-provisioned Grafana dashboard for GitHub Actions runner VM host-level metrics and documents how dashboards are organized and delivered to Grafana via cos-configuration-k8s.
Changes:
- Add
runner_vm_hostmetrics.jsondashboard covering CPU, memory, disk I/O, filesystem, network, load. - Document repository layout and Grafana dashboard delivery/conventions in
README.md.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| runner_grafana_dashboards/runner_vm_hostmetrics.json | New hostmetrics dashboard with templated filters and multiple panels for VM resource metrics |
| README.md | Adds repo layout + explains dashboard provisioning via cos-configuration-k8s and dashboard authoring conventions |
Expand the dashboard description to spell out the expected usage of the github_runner variable: scope it to a flavor regex (e.g. flavor-x-.*) when comparing fleets, or pick a single runner for per-host inspection. Aggregating by device/mountpoint without grouping by github_runner is intentional — it produces meaningful fleet totals/averages when the matched runners share device semantics — but assumes operators don't mix heterogeneous flavors under "All".
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a filesystem-provisioned (read-only) Grafana dashboard for GitHub Actions runner VM host-level metrics, and documents how dashboards are delivered via cos-configuration-k8s.
Changes:
- Add
runner_vm_hostmetrics.jsondashboard covering CPU, memory, disk I/O, filesystem, and network metrics with GitHub-context templating variables. - Document dashboard delivery mechanism and repository/dashboard layout conventions in
README.md.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| runner_grafana_dashboards/runner_vm_hostmetrics.json | New hostmetrics dashboard for runner VMs with PromQL queries and templating variables. |
| README.md | Documents dashboard provisioning via cos-configuration-k8s and expected dashboard directory conventions. |
- Root FS gauge: restrict the denominator to state=~"used|free" to match the Filesystem Utilization bargauge and df semantics. Without this, reserved blocks (e.g. ext4's 5% root reservation) inflate the denominator and the gauge reads artificially low. - System Load cores override: the field matcher still pointed at the old "cores" legend after the per-runner rename, so the red dashed styling never applied. Update the matcher to "cores (per runner)".
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a filesystem-provisioned, read-only Grafana dashboard to visualize GitHub Actions runner VM host-level metrics, and documents the dashboard delivery/layout conventions for this monorepo.
Changes:
- Add a new runner VM hostmetrics dashboard JSON with CPU, memory, disk I/O, filesystem, and network panels plus GitHub-context template variables.
- Document the repository’s Grafana dashboard layout and
cos-configuration-k8sprovisioning conventions in the README.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| runner_grafana_dashboards/runner_vm_hostmetrics.json | New Grafana dashboard for runner VM hostmetrics (Prometheus/OTel hostmetrics naming + GitHub-context filters). |
| README.md | Documents dashboard directory conventions and delivery mechanism via cos-configuration-k8s and the grafana-dashboard relation. |
Drop multi-select on the GitHub-context variables (kept includeAll + allValue: ".*" so picking "All" still widens the scope as a regex). Single-select matches the upstream OpenTelemetry hostmetrics design and makes per-host attribution work — multi-runner aggregations under sum by (device) collapsed identically-named devices across hosts and hid which runner was responsible for any given spike. With single-select assured, simplify the dense per-device/per-mountpoint panels back to bare metrics (drop sum by device on disk I/O throughput, disk IOPS, disk busy %, memory usage, filesystem usage, network throughput/packets/errors). Revert the multi-runner-defensive variants of CPU Cores, System Load 1m/5m/15m and the cores reference series. Aggregations are kept where they are inherent to the metric: overview gauges (CPU/Memory/Root FS), Memory Utilization (sum/sum ratio), Filesystem Utilization (sum by mountpoint ratio) and TCP Connections (sum by state). Drop the cross-host-aggregation note from the dashboard description since the design no longer relies on it.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Grafana dashboard for GitHub Actions runner VM host-level metrics and documents how dashboards are organized and delivered via cos-configuration-k8s.
Changes:
- Add a filesystem-provisioned Grafana dashboard JSON for runner VM hostmetrics (CPU, memory, disk, filesystem, network, load).
- Document repository layout and the Grafana dashboard delivery mechanism/conventions in
README.md.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
runner_grafana_dashboards/runner_vm_hostmetrics.json |
New hostmetrics dashboard JSON with templated GitHub-context filters and PromQL queries. |
README.md |
Documents dashboard locations and delivery conventions via cos-configuration-k8s. |
cbartz
requested review from
javierdelapuente,
yanksyoon and
yhaliaw
as code owners
yhaliaw
approved these changes
Apr 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a read-only Grafana dashboard for runner VM host-level metrics, served via
cos-configuration-k8susing thegrafana-dashboardrelation. Provisioned dashboards are filesystem-managed in Grafana, so they cannot be edited through the UI regardless of user role.Changes
runner_grafana_dashboards/runner_vm_hostmetrics.json: new dashboard covering CPU, memory, disk I/O, filesystem, network traffic and load averagesgithub_repository,github_workflow,github_job,github_runner— all single-select withincludeAll: trueandallValue: ".*". Picking a value scopes panels to one host (the upstream design); picking "All" widens the scope as a regex shortcut.=~so the regex from "All" interpolates correctlysystem_cpu_time_seconds_total,system_memory_usage_bytes, …); the four GitHub-context labels are added by the OTel collector pipeline in github-runner-operator#781__inputsdeclares the Prometheus / Mimir datasource andeditable: falseis set for clarity (provisioned dashboards are read-only either way)README.md: documents the repo layout, the observability dashboard delivery mechanism, and the conventions for both per-charm and runner-VM dashboardsNotes
cos-configuration-k8s) is in platform-engineering-deploymentsfeat/runner-hostmetrics-cos-configurationCloses / relates to: ISD-5152