Migrate dynamic scripted Grafana Dashboards to Scenes

CHANGELOG.md

@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - [#7812](https://github.com/apache/trafficcontrol/pull/7812) *Traffic Portal*: Expose the `configUpdateFailed` and `revalUpdateFailed` fields on the server table.
 - [#7870](https://github.com/apache/trafficcontrol/pull/7870) *Traffic Portal*: Adds a hyperlink to the DSR page to the DS itself for ease of navigation.
 - [#7896](https://github.com/apache/trafficcontrol/pull/7896) *ATC Build system*: Count commits since the last release, not commits
+- [#7927](https://github.com/apache/trafficcontrol/pull/7927) *Traffic Stats*: Migrate dynamic scripted Grafana Dashboards to Scenes
 
 ### Changed
 - [#7614](https://github.com/apache/trafficcontrol/pull/7614) *Traffic Ops* The database upgrade process no longer overwrites changes users may have made to the initially seeded data.

docs/source/admin/traffic_stats.rst

@@ -91,10 +91,8 @@ To easily create databases, retention policies, and continuous queries, run :pro
 
 Configuring Grafana
 -------------------
-In Traffic Portal the :menuselection:`Other --> Grafana` menu item can be configured to display Grafana graphs using InfluxDB data (when not configured, this menu item will not appear). In order for this to work correctly, you will need two things:
-
-#. A :term:`Parameter` with the graph URL (more information below)
-#. The graphs created in Grafana. See below for how to create some simple graphs in Grafana. These instructions assume that InfluxDB has been configured and that data has been written to it. If this is not true, you will not see any graphs.
+Grafana can be configured to display graphs using InfluxDB data.
+See below for how to create some simple graphs in Grafana. These instructions assume that InfluxDB has been configured and that data has been written to it. If this is not true, you will not see any graphs.
 
 To create a graph in Grafana, you can follow these basic steps:
 
@@ -115,11 +113,11 @@ To create a graph in Grafana, you can follow these basic steps:
 #. Once you have the graph the way you want it, click the :guilabel:`Save Dashboard` button at the top
 #. You should now have a new saved graph
 
-In order for Traffic Portal users to see Grafana graphs, Grafana will need to allow anonymous access. Information on how to configure anonymous access can be found on the configuration page of the `Grafana Website  <http://docs.grafana.org/installation/configuration/#authanonymous>`_.
+Grafana uses Grafana Scenes to display information about individual :term:`Delivery Services` or :term:`Cache Groups`. In order for the custom graphs to display correctly, the built files of :atc-file:`traffic_stats/trafficcontrol-scenes/`  need to be placed in the :file:`/var/lib/grafana/plugins/trafficcontrol-scenes-app` directory on the Grafana server. If your Grafana server is the same as your Traffic Stats server the RPM install process will take care of putting the files in place. If your Grafana server is different from your Traffic Stats server, you will need to manually copy the files to the correct directory.
 
-Traffic Portal uses custom dashboards to display information about individual :term:`Delivery Services` or :term:`Cache Groups`. In order for the custom graphs to display correctly, the Javascript files in :atc-file:`traffic_stats/grafana/`  need to be in the :file:`/usr/share/grafana/public/dashboards/` directory on the Grafana server. If your Grafana server is the same as your Traffic Stats server the RPM install process will take care of putting the files in place. If your Grafana server is different from your Traffic Stats server, you will need to manually copy the files to the correct directory.
+To view dynamic dashboards from Grafana Scenes, visit: ``https://grafanaHost/a/trafficcontrol-scenes-app``
 
-.. seealso:: More information on custom scripted graphs can be found in the `scripted dashboards <http://docs.grafana.org/reference/scripting/>`_ section of the Grafana documentation.
+.. seealso:: More information on Grafana Scenes can be found in the `blog post <https://grafana.com/blog/2023/09/12/grafana-scenes-is-generally-available-start-building-highly-interactive-apps-today/>`_ of Grafana.
 
 Configuring Traffic Portal for Traffic Stats
 --------------------------------------------

infrastructure/cdn-in-a-box/optional/grafana/Dockerfile

@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-FROM grafana/grafana:10.1.5-ubuntu
+FROM grafana/grafana:10.2.3-ubuntu
 
 USER root
 ARG TRAFFIC_TS_RPM=traffic_stats/traffic_stats.rpm
@@ -28,13 +28,15 @@ ADD enroller/server_template.json \
     traffic_ops/to-access.sh \
     optional/grafana/run-grafana.sh \
     optional/grafana/datasources.yml.template \
+    optional/grafana/app.yaml.template \
     $TRAFFIC_TS_RPM \
     /
 
 RUN cd ~ \
     && 7z x /traffic_stats.rpm \
-    && 7z e traffic_stats-*.cpio *.js -r \
-    && mv *.js /usr/share/grafana/public/dashboards \
+    && 7z e traffic_stats-*.cpio module.js plugin.json module.js.map -r \
+    && mkdir -p /var/lib/grafana/plugins/trafficcontrol-scenes-app/ \
+    && mv plugin.json module.js.map module.js /var/lib/grafana/plugins/trafficcontrol-scenes-app/ \
     && rm /traffic_stats.rpm \
     && rm ~/traffic_stats-*.cpio
 

infrastructure/cdn-in-a-box/optional/grafana/app.yaml.template

@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Refer to "http://docs.grafana.org/administration/provisioning/#example-datasource-config-file"
+# for the variables usages
+
+apiVersion: 1
+
+apps:
+  - type: 'trafficcontrol-scenes-app'
+    org_id: 1
+    org_name: 'apache'
+    disabled: false
+    jsonData:
+      apiUrl: https://trafficcontrol.apache.org/
+      isApiKeySet: true

infrastructure/cdn-in-a-box/optional/grafana/datasources.yml.template

@@ -51,3 +51,14 @@ datasources:
   database: deliveryservice_stats
   basicAuth: false
   isDefault: false
+
+- name: telegraf
+  type: influxdb
+  access: proxy
+  orgId: 1
+  url: http://$INFLUXDB_HOST:$INFLUXDB_PORT
+  password: $INFLUXDB_ADMIN_PASSWORD
+  user: $INFLUXDB_ADMIN_USER
+  database: telegraf
+  basicAuth: false
+  isDefault: false

infrastructure/cdn-in-a-box/optional/grafana/run-grafana.sh

@@ -58,6 +58,8 @@ export GF_SERVER_PROTOCOL="https"
 export GF_SERVER_HTTP_PORT=$GRAFANA_PORT
 export GF_SERVER_CERT_FILE=$X509_INFRA_CERT_FILE
 export GF_SERVER_CERT_KEY=$X509_INFRA_KEY_FILE
+export GF_DEFAULT_APP_MODE="development"
 envsubst < "/datasources.yml.template" > "$GF_PATHS_PROVISIONING/datasources/datasources.yml"
+envsubst < "/app.yaml.template" > "$GF_PATHS_PROVISIONING/plugins/app.yaml"
 
 /run.sh

infrastructure/docker/build/Dockerfile-traffic_stats

@@ -42,6 +42,9 @@ RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-* && \
 ### traffic_stats specific requirements
 FROM common-dependencies AS traffic-stats
 
+RUN curl -sL https://rpm.nodesource.com/setup_20.x | bash - && \
+    yum -y install nodejs
+
 COPY GO_VERSION /
 RUN set -o nounset -o errexit; \
 	rpm_arch="$(rpm --eval %_arch)" && \

infrastructure/docker/build/docker-compose.yml

@@ -167,6 +167,7 @@ services:
           - linux/arm64
     volumes:
       - ../../..:/trafficcontrol:z
+      - ../../../.npm:/root/.npm:z
 
   grove_build:
     image: apache/traffic_grove_builder:master

traffic_stats/build/build_rpm.sh

@@ -78,6 +78,19 @@ initBuildArea() {
 	go build -v -gcflags "$gcflags" -ldflags "$ldflags" create/create_ts_databases.go || \
 								{ echo "Could not build create_ts_databases binary"; return 1; })
 
+  # compile trafficcontrol-scenes
+  echo "Installing grafana scenes npm dependencies"
+  (cd trafficcontrol-scenes
+    npm i || \
+        { echo "Could not install packages from $TS_DIR/trafficcontrol-scenes: $?"; return 1; }
+  )
+
+  echo "Build grafana scenes"
+  (cd trafficcontrol-scenes
+    npm run build || \
+      	{ echo "Could not build $TS_DIR/trafficcontrol-scenes: $?"; return 1; }
+  )
+
 	rsync -aLv ./ "$ts_dest"/ || \
 		 { echo "Could not copy to $ts_dest: $?"; return 1; }
 	cp "$TS_DIR"/build/*.spec "$RPMBUILD"/SPECS/. || \
@@ -103,6 +116,6 @@ preBuildChecks() {
 
 importFunctions
 preBuildChecks
-checkEnvironment -i go,rsync
+checkEnvironment -i npm,go,rsync
 initBuildArea
 buildRpm traffic_stats

traffic_stats/build/traffic_stats.spec

@@ -50,6 +50,13 @@ godir=src/github.com/apache/trafficcontrol/traffic_stats/influxdb_tools
 	cp -R "$TC_DIR"/traffic_stats/influxdb_tools/* .
 ) || { echo "Could not copy go program at $(pwd): $!"; exit 1; }
 
+# copy trafficcontrol-scenes
+scenesdir=src/github.com/apache/trafficcontrol/traffic_stats/trafficcontrol-scenes
+( mkdir -p "$scenesdir" && \
+	cd "$scenesdir" && \
+	cp -R "$TC_DIR"/traffic_stats/trafficcontrol-scenes/* .
+) || { echo "Could not copy trafficcontrol-scenes at $(pwd): $!"; exit 1; }
+
 %install
 mkdir -p "${RPM_BUILD_ROOT}"/opt/traffic_stats
 mkdir -p "${RPM_BUILD_ROOT}"/opt/traffic_stats/bin
@@ -60,15 +67,15 @@ mkdir -p "${RPM_BUILD_ROOT}"/opt/traffic_stats/var/run
 mkdir -p "${RPM_BUILD_ROOT}"/opt/traffic_stats/var/log/traffic_stats
 mkdir -p "${RPM_BUILD_ROOT}"/etc/init.d
 mkdir -p "${RPM_BUILD_ROOT}"/etc/logrotate.d
-mkdir -p "${RPM_BUILD_ROOT}"/usr/share/grafana/public/dashboards/
+mkdir -p "${RPM_BUILD_ROOT}"/var/lib/grafana/plugins/trafficcontrol-scenes-app
 
 src=src/github.com/apache/trafficcontrol/traffic_stats
 cp -p "$src"/traffic_stats         "${RPM_BUILD_ROOT}"/opt/traffic_stats/bin/traffic_stats
 cp "$src"/traffic_stats.cfg        "${RPM_BUILD_ROOT}"/opt/traffic_stats/conf/traffic_stats.cfg
 cp "$src"/traffic_stats_seelog.xml "${RPM_BUILD_ROOT}"/opt/traffic_stats/conf/traffic_stats_seelog.xml
 cp "$src"/traffic_stats.init       "${RPM_BUILD_ROOT}"/etc/init.d/traffic_stats
 cp "$src"/traffic_stats.logrotate  "${RPM_BUILD_ROOT}"/etc/logrotate.d/traffic_stats
-cp "$src"/grafana/*.js             "${RPM_BUILD_ROOT}"/usr/share/grafana/public/dashboards/
+cp -r "$src"/trafficcontrol-scenes/dist/* "${RPM_BUILD_ROOT}"/var/lib/grafana/plugins/trafficcontrol-scenes-app/
 cp "$src"/influxdb_tools/sync_ts_databases  "${RPM_BUILD_ROOT}"/opt/traffic_stats/influxdb_tools/
 cp "$src"/influxdb_tools/create_ts_databases  "${RPM_BUILD_ROOT}"/opt/traffic_stats/influxdb_tools/
 
@@ -125,15 +132,12 @@ fi
 %dir /opt/traffic_stats/var/log
 %dir /opt/traffic_stats/var/run
 %dir /opt/traffic_stats/var/log/traffic_stats
-%dir /usr/share/grafana/public/dashboards
+%dir /var/lib/grafana/plugins/trafficcontrol-scenes-app
 %dir /opt/traffic_stats/influxdb_tools
 
 %attr(755, traffic_stats, traffic_stats) /opt/traffic_stats/bin/traffic_stats
 %attr(755, traffic_stats, traffic_stats) /etc/init.d/traffic_stats
-%attr(644, traffic_stats, traffic_stats) /usr/share/grafana/public/dashboards/traffic_ops_cachegroup.js
-%attr(644, traffic_stats, traffic_stats) /usr/share/grafana/public/dashboards/traffic_ops_deliveryservice.js
-%attr(644, traffic_stats, traffic_stats) /usr/share/grafana/public/dashboards/traffic_ops_scripted.js
-%attr(644, traffic_stats, traffic_stats) /usr/share/grafana/public/dashboards/traffic_ops_server.js
+%attr(755, traffic_stats, traffic_stats) /var/lib/grafana/plugins/trafficcontrol-scenes-app
 %attr(755, traffic_stats, traffic_stats) /opt/traffic_stats/influxdb_tools/create_ts_databases
 %attr(755, traffic_stats, traffic_stats) /opt/traffic_stats/influxdb_tools/sync_ts_databases