From 38bfe1cbaf0c2d0075ee0684d7625640ce29da23 Mon Sep 17 00:00:00 2001 From: nate contino Date: Sun, 18 Dec 2022 21:30:12 -0500 Subject: [PATCH 1/4] Remove extra H1 & improve formatting of Listeners docs page I noticed that the documentation has an unclickable "Usage" page in the TOC. A little digging later, I discovered that this page contains an extra H1, and since this page is in the top level of the TOC, all the H1s on this page show up in the left docs sidebar. Demoted the "Usage" section to an H2, and fixed the other headers on this page to use consistent underlining with most other docs pages in this repo. I also took the liberty of sprucing up the language on the page to follow docs best practices, like shorter, highly readable sentences, title case in section titles, and bulleted lists to draw attention to important collections. --- .../listeners.rst | 70 ++++++++++++------- 1 file changed, 43 insertions(+), 27 deletions(-) diff --git a/docs/apache-airflow/administration-and-deployment/listeners.rst b/docs/apache-airflow/administration-and-deployment/listeners.rst index 2678e12e5ef66..7c3ab93a76efa 100644 --- a/docs/apache-airflow/administration-and-deployment/listeners.rst +++ b/docs/apache-airflow/administration-and-deployment/listeners.rst @@ -18,43 +18,59 @@ Listeners ========= -Airflow gives you an option to be notified of events happening in Airflow -by writing listeners. Listeners are powered by `pluggy `__ +You can write listeners to enable Airflow to notify you when events happen. +`Pluggy `__ powers these listeners. -Right now Airflow exposes few types of events. +Airflow supports notifications for the following events: -Lifecycle events -^^^^^^^^^^^^^^^^ -Those events - ``on_starting`` and ``before_stopping`` allow you to react to -lifecycle to an Airflow ``Job``, like ``SchedulerJob`` or ``BackfillJob``. +Lifecycle Events +---------------- -TaskInstance state change events -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Those events - ``on_task_instance_running``, ``on_task_instance_success`` and ``on_task_instance_failed`` -once ``TaskInstance`` state changes to one of the respective states. This generally happens on ``LocalTaskJob``. +- ``on_starting`` +- ``before_stopping`` -DagRun state change events -^^^^^^^^^^^^^^^^^^^^^^^^^^ -Those events - ``on_dag_run_running``, ``on_dag_run_success`` and ``on_dag_run_failed`` -once ``DagRun`` state changes to one of the respective states. This generally happens on ``SchedulerJob`` or ``BackfillJob``. +Lifecycle events allow you to react to start and stop events for an Airflow ``Job``, like ``SchedulerJob`` or ``BackfillJob``. + +TaskInstance State Change Events +-------------------------------- + +- ``on_task_instance_running`` +- ``on_task_instance_success`` +- ``on_task_instance_failed`` + +TaskInstance state change events occur when a ``TaskInstance`` changes state. +You can use these events to react to ``LocalTaskJob`` state changes. + +DagRun State Change Events +-------------------------- + +- ``on_dag_run_running`` +- ``on_dag_run_success`` +- ``on_dag_run_failed`` + +DagRun state change events occur when a ``DagRun`` changes state. +You can use these events to react to ``SchedulerJob`` or ``BackfillJob`` state changes. Usage -===== +----- + +To create a listener: -To create a listener you will need to derive the import -``airflow.listeners.hookimpl`` and implement the ``hookimpls`` for -events you want to be notified at. +- import ``airflow.listeners.hookimpl`` +- implement the ``hookimpls`` for events that you'd like to generate notifications -Their specification is defined as ``hookspec`` in ``airflow/listeners/spec`` directory. -Your implementation needs to accept the same named parameters as defined in hookspec, or Pluggy will complain about your plugin. -On the other hand, you don't need to implement every method - it's perfectly fine to have a listener that implements just one method, or any subset of methods. +Airflow defines the specification as `hookspec `__. +Your implementation must accept the same named parameters as defined in hookspec. +If you don't use the same parameters as hookspec, Pluggy throws an error when you try to use your plugin. +But you don't need to implement every method. +Many listeners only implement one method, or a subset of methods. -To include listener in your Airflow installation, include it as a part of an :doc:`Airflow Plugin ` +To include the listener in your Airflow installation, include it as a part of an :doc:`Airflow Plugin ` -Listener API is meant to be called across all dags, and all operators - in contrast to methods like -``on_success_callback``, ``pre_execute`` and related family which are meant to provide callbacks -for particular dag authors, or operator creators. There is no possibility to listen on events generated -by particular dag. +Listener API is meant to be called across all DAGs and all operators. +You can't listen to events generated by specific DAGs. +For that behavior, try methods like ``on_success_callback`` and ``pre_execute``. +These provide callbacks for particular DAG authors or operator creators. |experimental| From 40a50809b7f9de13e9547496d209442ea682712a Mon Sep 17 00:00:00 2001 From: nate contino Date: Mon, 19 Dec 2022 10:15:07 -0500 Subject: [PATCH 2/4] Remove extra newlines from specification discussion --- .../administration-and-deployment/listeners.rst | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/apache-airflow/administration-and-deployment/listeners.rst b/docs/apache-airflow/administration-and-deployment/listeners.rst index 7c3ab93a76efa..432c5cdb2c840 100644 --- a/docs/apache-airflow/administration-and-deployment/listeners.rst +++ b/docs/apache-airflow/administration-and-deployment/listeners.rst @@ -59,11 +59,7 @@ To create a listener: - import ``airflow.listeners.hookimpl`` - implement the ``hookimpls`` for events that you'd like to generate notifications -Airflow defines the specification as `hookspec `__. -Your implementation must accept the same named parameters as defined in hookspec. -If you don't use the same parameters as hookspec, Pluggy throws an error when you try to use your plugin. -But you don't need to implement every method. -Many listeners only implement one method, or a subset of methods. +Airflow defines the specification as `hookspec `__. Your implementation must accept the same named parameters as defined in hookspec. If you don't use the same parameters as hookspec, Pluggy throws an error when you try to use your plugin. But you don't need to implement every method. Many listeners only implement one method, or a subset of methods. To include the listener in your Airflow installation, include it as a part of an :doc:`Airflow Plugin ` From 30f8f528b3d406dc6998c2b40fb3da74be212568 Mon Sep 17 00:00:00 2001 From: nate contino Date: Mon, 19 Dec 2022 10:15:36 -0500 Subject: [PATCH 3/4] Remove single newlines from listener API discussion --- .../administration-and-deployment/listeners.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/apache-airflow/administration-and-deployment/listeners.rst b/docs/apache-airflow/administration-and-deployment/listeners.rst index 432c5cdb2c840..77a40f5ac16c1 100644 --- a/docs/apache-airflow/administration-and-deployment/listeners.rst +++ b/docs/apache-airflow/administration-and-deployment/listeners.rst @@ -63,10 +63,7 @@ Airflow defines the specification as `hookspec ` -Listener API is meant to be called across all DAGs and all operators. -You can't listen to events generated by specific DAGs. -For that behavior, try methods like ``on_success_callback`` and ``pre_execute``. -These provide callbacks for particular DAG authors or operator creators. +Listener API is meant to be called across all DAGs and all operators. You can't listen to events generated by specific DAGs. For that behavior, try methods like ``on_success_callback`` and ``pre_execute``. These provide callbacks for particular DAG authors or operator creators. |experimental| From 9840ec4cd617996305b252b0e4e527506952b661 Mon Sep 17 00:00:00 2001 From: nate contino Date: Mon, 19 Dec 2022 10:16:32 -0500 Subject: [PATCH 4/4] Remove nonexistent DagRun events from listeners page --- .../administration-and-deployment/listeners.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/apache-airflow/administration-and-deployment/listeners.rst b/docs/apache-airflow/administration-and-deployment/listeners.rst index 77a40f5ac16c1..c34689c53fa49 100644 --- a/docs/apache-airflow/administration-and-deployment/listeners.rst +++ b/docs/apache-airflow/administration-and-deployment/listeners.rst @@ -41,15 +41,6 @@ TaskInstance State Change Events TaskInstance state change events occur when a ``TaskInstance`` changes state. You can use these events to react to ``LocalTaskJob`` state changes. -DagRun State Change Events --------------------------- - -- ``on_dag_run_running`` -- ``on_dag_run_success`` -- ``on_dag_run_failed`` - -DagRun state change events occur when a ``DagRun`` changes state. -You can use these events to react to ``SchedulerJob`` or ``BackfillJob`` state changes. Usage -----