Make Open Telemetry the default instead of StatsD for monitoring #40800
Description
Currently, Apache Airflow uses StatsD for metrics collection and monitoring. To modernize our observability stack and align with industry standards, we should adopt OpenTelemetry as the primary metrics collection and monitoring tool for Airflow 3 while keeping Statsd as an option. This is made easier with AIP-49 Open Telemetry Support.
Backward Compatibility / Migration
Most of the enterprise Airflow deployments rely on StatsD for monitoring & alerting so we should make sure that there is a smooth migration path. Ideally, all the StatsD metrics mentioned in https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#metric-descriptions should have a like-by-like replacement.
High-level Objectives:
- Use OpenTelemetry as the default metrics collection tool. This includes setting up configurations, dependencies, and code integration.
- Documentation Update: Update the official documentation to reflect the change from StatsD to OpenTelemetry as the default, including setup, configuration, and usage instructions. Include migration guide for users transitioning from StatsD to OpenTelemetry if there are breaking changes. This includes the Helm chart.