[AIP-49] Breeze OpenTelemetry Integration

.pre-commit-config.yaml

@@ -711,7 +711,10 @@ repos:
         language: python
         pass_filenames: true
         files: ^scripts/ci/docker-compose/.+\.ya?ml$|docker-compose\.ya?ml$
-        exclude: ^scripts/ci/docker-compose/grafana/.|^scripts/ci/docker-compose/prometheus.yml
+        exclude: >
+          (?x)
+          ^scripts/ci/docker-compose/grafana/.|
+          ^scripts/ci/docker-compose/.+-config\.ya?ml
         require_serial: true
         additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'PyYAML==5.3.1', 'requests==2.25.0']
       - id: lint-json-schema

BREEZE.rst

@@ -569,9 +569,11 @@ Those are all available flags of ``shell`` command:
   :width: 100%
   :alt: Breeze shell
 
+Running Breeze with Metrics
+---------------------------
 
 Running Breeze with a StatsD Metrics Stack
-------------------------------------------
+..........................................
 
 You can launch an instance of Breeze pre-configured to emit StatsD metrics using
 ``breeze start-airflow --integration statsd``.  This will launch an Airflow webserver
@@ -592,6 +594,37 @@ You can connect to these ports/databases using:
 * Prometheus Targets: http://127.0.0.1:29090/targets
 * Grafana Dashboards: http://127.0.0.1:23000/dashboards
 
+Running Breeze with an OpenTelemetry Metrics Stack
+..................................................
+
+----
+
+[Work in Progress]
+NOTE:  This will launch the stack as described below but Airflow integration is
+still a Work in Progress.  This should be considered experimental and likely to
+change by the time Airflow fully supports emitting metrics via OpenTelemetry.
+
+----
+
+You can launch an instance of Breeze pre-configured to emit OTel metrics using
+``breeze start-airflow --integration otel``.  This will launch an Airflow webserver
+within the Breeze environment as well as containers running OpenTelemetry-Collector,
+Prometheus, and Grafana.  The integration configures the "Targets" in Prometheus,
+the "Datasources" in Grafana, and includes a default dashboard in Grafana.
+
+When you run Airflow Breeze with this integration, in addition to the standard ports
+(See "Port Forwarding" below), the following are also automatically forwarded:
+
+* 28888 -> forwarded to OpenTelemetry Collector -> breeze-otel_collector:8888
+* 29090 -> forwarded to Prometheus -> breeze-prometheus:9090
+* 23000 -> forwarded to Grafana -> breeze-grafana:3000
+
+You can connect to these ports/databases using:
+
+* OpenTelemetry Collector: http://127.0.0.1:28888/metrics
+* Prometheus Targets: http://127.0.0.1:29090/targets
+* Grafana Dashboards: http://127.0.0.1:23000/dashboards
+
 
 Stopping the environment
 ------------------------

dev/breeze/src/airflow_breeze/global_constants.py

@@ -54,6 +54,7 @@
     [
         *ALL_INTEGRATIONS,
         "all",
+        "otel",
         "statsd",
     ]
 )

images/breeze/output-commands-hash.txt

@@ -1,7 +1,7 @@
 # This file is automatically generated by pre-commit. If you have a conflict with this file
 # Please do not solve it but run `breeze setup regenerate-command-images`.
 # This command should fix the conflict and regenerate help images that you have conflict with.
-main:2b27f16d64b13766bdbc9e7979f75601
+main:2e1b4e5838838f513750d1352798b006
 build-docs:18235f12f85f8df82f3eb245e429f62d
 ci:find-newer-dependencies:8fa2b57f5f0523c928743b235ee3ab5a
 ci:fix-ownership:fee2c9ec9ef19686792002ae054fecdd
@@ -53,12 +53,12 @@ setup:regenerate-command-images:15215e52342dd2f2e27a85726f40a820
 setup:self-upgrade:d02f70c7a230eae3463ceec2056b63fa
 setup:version:123b462a421884dc2320ffc5e54b2478
 setup:56a2ef337c354362760d247df5d05365
-shell:42c48eb6f0b9d2b5e3e7602de2600341
-start-airflow:91a1c8ef99573266e669acb678aaa857
+shell:e34d2512360503b70e6a05e5b8b59986
+start-airflow:e096fa493a810935c012ef3d95775aa2
 static-checks:12e8fed2acbed0d823efc5121fd0eb58
 stop:e5aa686b4e53707ced4039d8414d5cd6
 testing:docker-compose-tests:b86c044b24138af0659a05ed6331576c
 testing:helm-tests:94a442e7f3f63b34c4831a84d165690a
-testing:integration-tests:eee580f73de750c1229c5f2b0d9e500f
-testing:tests:200916b63e40d9c72b58236c925b2589
-testing:3627d67696a9a2a31aec76ec745f7004
+testing:integration-tests:fa4f9be38ee380f6f8feeb9dd8ba7669
+testing:tests:8dd0496fddb37b207fc2b74add201d43
+testing:1e4ea0f58b4814342911355d35efd966