No revision data shown in Grafana dashboards

[ahmetb]

/area monitoring
/kind bug

Potentially the same as #4167 #4229 #4507.

I just installed Knative serving v0.11.0 with monitoring.yaml. I used istio-lean instuctions from here to install Istio from HEAD (near 1.4.2).

It seems most of the useful Grafana dashboards are broken.

(If so, please consider taking Grafana out of docs for now. If it's broken for 6 months, no point of documenting it as if it's working.)

When I look at Grafana dashboards, I see some telemetry about autoscaling metrics but the Revision Request Count metrics are empty. It doesn't even show any namespaces, configurations or revisions to choose from in the dropdowns.

See these pictures below (first one is a working dashboard for autoscaler), the rest doesn't seem to be working.

---

---

[dgerd]

cc @mdemirhan

[eallred-google]

cc @eallred-google

[anniefu]

Revision request count metrics are taken from queue-proxy now per #4418. However, there was an issue with the Prometheus scrape config that was recently fixed a couple of weeks ago by #6526.

If you pick up the new changes it should work. Note that the Prometheus pods need to be restarted to pick up the ConfigMap changes.

[ahmetb]

@anniefu thanks.
/close

[knative-prow-robot]

@ahmetb: Closing this issue.

Details

In response to this:

@anniefu thanks.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gadelkareem]

@anniefu I am still facing the same problem with the latest release. revision_request_count has no metrics in Prometheus.

[gadelkareem]

it seems that the labels changed again

          - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_release, __meta_kubernetes_pod_container_port_name]
            action: keep
            regex: .+;.*metrics