From d55c814760bdfbe5918a81c4d5a12c888180affc Mon Sep 17 00:00:00 2001 From: XD-DENG Date: Thu, 20 Sep 2018 09:45:14 +0800 Subject: [PATCH] [AIRFLOW-XXX] Fix a wrong sample bash command, a display issue & a few typos --- docs/concepts.rst | 6 +++--- docs/howto/executor/use-celery.rst | 2 +- docs/howto/manage-connections.rst | 6 +++--- docs/howto/secure-connections.rst | 2 +- docs/kubernetes.rst | 3 ++- docs/ui.rst | 2 +- 6 files changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/concepts.rst b/docs/concepts.rst index 50c18c9b98daa..73164772251cc 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -320,7 +320,7 @@ Connections =========== The connection information to external systems is stored in the Airflow -metadata database and managed in the UI (``Menu -> Admin -> Connections``) +metadata database and managed in the UI (``Menu -> Admin -> Connections``). A ``conn_id`` is defined there and hostname / login / password / schema information attached to it. Airflow pipelines can simply refer to the centrally managed ``conn_id`` without having to hard code any of this @@ -353,7 +353,7 @@ See :doc:`howto/manage-connections` for how to create and manage connections. Queues ====== -When using the CeleryExecutor, the celery queues that tasks are sent to +When using the CeleryExecutor, the Celery queues that tasks are sent to can be specified. ``queue`` is an attribute of BaseOperator, so any task can be assigned to any queue. The default queue for the environment is defined in the ``airflow.cfg``'s ``celery -> default_queue``. This defines @@ -361,7 +361,7 @@ the queue that tasks get assigned to when not specified, as well as which queue Airflow workers listen to when started. Workers can listen to one or multiple queues of tasks. When a worker is -started (using the command ``airflow worker``), a set of comma delimited +started (using the command ``airflow worker``), a set of comma-delimited queue names can be specified (e.g. ``airflow worker -q spark``). This worker will then only pick up tasks wired to the specified queue(s). diff --git a/docs/howto/executor/use-celery.rst b/docs/howto/executor/use-celery.rst index fd6db96a95872..0e1bab060b0b2 100644 --- a/docs/howto/executor/use-celery.rst +++ b/docs/howto/executor/use-celery.rst @@ -44,4 +44,4 @@ Some caveats: - Make sure to use a database backed result backend - Make sure to set a visibility timeout in [celery_broker_transport_options] that exceeds the ETA of your longest running task -- Tasks can and consume resources, make sure your worker as enough resources to run `worker_concurrency` tasks +- Tasks can consume resources. Make sure your worker has enough resources to run `worker_concurrency` tasks diff --git a/docs/howto/manage-connections.rst b/docs/howto/manage-connections.rst index f5203157aca8d..f869a08b3c061 100644 --- a/docs/howto/manage-connections.rst +++ b/docs/howto/manage-connections.rst @@ -3,7 +3,7 @@ Managing Connections Airflow needs to know how to connect to your environment. Information such as hostname, port, login and passwords to other systems and services is -handled in the ``Admin->Connection`` section of the UI. The pipeline code you +handled in the ``Admin->Connections`` section of the UI. The pipeline code you will author will reference the 'conn_id' of the Connection objects. .. image:: ../img/connections.png @@ -17,7 +17,7 @@ more information. Creating a Connection with the UI --------------------------------- -Open the ``Admin->Connection`` section of the UI. Click the ``Create`` link +Open the ``Admin->Connections`` section of the UI. Click the ``Create`` link to create a new connection. .. image:: ../img/connection_create.png @@ -34,7 +34,7 @@ to create a new connection. Editing a Connection with the UI -------------------------------- -Open the ``Admin->Connection`` section of the UI. Click the pencil icon next +Open the ``Admin->Connections`` section of the UI. Click the pencil icon next to the connection you wish to edit in the connection list. .. image:: ../img/connection_edit.png diff --git a/docs/howto/secure-connections.rst b/docs/howto/secure-connections.rst index f9e252c4c365a..bb13b1bb08bba 100644 --- a/docs/howto/secure-connections.rst +++ b/docs/howto/secure-connections.rst @@ -26,7 +26,7 @@ variable over the value in ``airflow.cfg``: .. code-block:: bash # Note the double underscores - EXPORT AIRFLOW__CORE__FERNET_KEY = your_fernet_key + export AIRFLOW__CORE__FERNET_KEY=your_fernet_key 4. Restart Airflow webserver. 5. For existing connections (the ones that you had defined before installing ``airflow[crypto]`` and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it. diff --git a/docs/kubernetes.rst b/docs/kubernetes.rst index cb064cbaf8d1d..8a4cea5a733f5 100644 --- a/docs/kubernetes.rst +++ b/docs/kubernetes.rst @@ -4,7 +4,8 @@ Kubernetes Executor The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance. Example helm charts are available at `scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml` in the source distribution. The volumes are optional and depend on your configuration. There are two volumes available: -- Dags: by storing all the dags onto the persistent disks, all the workers can read the dags from there. Another option is using git-sync, before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod/ + +- Dags: by storing all the dags onto the persistent disks, all the workers can read the dags from there. Another option is using git-sync, before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod. - Logs: by storing the logs onto a persistent disk, all the logs will be available for all the workers and the webserver itself. If you don't configure this, the logs will be lost after the worker pods shuts down. Another option is to use S3/GCS/etc to store the logs. diff --git a/docs/ui.rst b/docs/ui.rst index 4b232fa1aea18..5a09ce5aa0a92 100644 --- a/docs/ui.rst +++ b/docs/ui.rst @@ -1,6 +1,6 @@ UI / Screenshots ================= -The Airflow UI make it easy to monitor and troubleshoot your data pipelines. +The Airflow UI makes it easy to monitor and troubleshoot your data pipelines. Here's a quick overview of some of the features and visualizations you can find in the Airflow UI.