From b65254dc867aaca6bc7bb638f02b46d6b35e8483 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Sun, 4 May 2025 19:09:40 +0530 Subject: [PATCH 01/11] Add setup guide for Apache Airflow on Mac M1 and above --- README.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/README.md b/README.md index 5a78ca94428d6..97cffe963e3c8 100644 --- a/README.md +++ b/README.md @@ -188,6 +188,69 @@ pip install 'apache-airflow[postgres,google]==3.0.0' \ For information on installing provider distributions, check [providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). +3. Setting up Apache Airflow on Mac M1 or above (Apple Silicon) users + +- **Create and Activate a Virtual Environment**: Since Apache Airflow requires multiple dependencies, it's a good practice to isolate these dependencies in a virtual environment. + +- Create a virtual environment: + +```bash +python3 -m venv airflow_venv +``` + +- Activate the virtual environment: + +```bash +source airflow_venv/bin/activate +``` + +- **Install Python 3.9 or later**:Airflow requires Python 3.9 or higher, which is supported on M1 Macs or above. You can check your Python version with: + +```bash +python3 --version +``` + +- If Python is not installed or you need to install a different version, I recommend using Homebrew to manage Python installations. + +```bash +brew install python@3.9 +``` + +- **Install Homebrew (if not already installed)**: If you haven’t installed Homebrew on your Mac yet, you can install it with the following command: + +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +``` + +- **Install Apache Airflow**: Apache Airflow is available on PyPI. To install it, you can use the following command in your terminal: + +```bash +pip install apache-airflow +``` + +- **Set the AIRFLOW_HOME Environment Variable**: Apache Airflow requires a directory to store configuration files, logs, and other data. Set the AIRFLOW_HOME variable to specify this directory. + +- Set the Airflow home directory: + +```bash +export AIRFLOW_HOME=~/airflow +``` + +- **Start the Airflow Components**: Apache Airflow runs several components, like the scheduler, web server, and API server, to manage workflows and show the UI. + +- To run Airflow in standalone mode (which will automatically start the required components): + +```bash +airflow standalone +``` + +- **Access the Airflow Web UI**: Once the components are up and running, you can access the Airflow UI through your browser: + +- Open your browser and go to: + +```arduino +http://localhost:8080 +``` ## Official source code From 5b3c3f935a75122109a526051fe913fd8d776f86 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Mon, 5 May 2025 01:21:23 +0530 Subject: [PATCH 02/11] Add Apple Silicon setup instructions for Apache Airflow --- README.md | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 97cffe963e3c8..507832d842528 100644 --- a/README.md +++ b/README.md @@ -188,47 +188,52 @@ pip install 'apache-airflow[postgres,google]==3.0.0' \ For information on installing provider distributions, check [providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). -3. Setting up Apache Airflow on Mac M1 or above (Apple Silicon) users +## Setting up Apache Airflow on Apple Silicon (Mac M1/M2/M3/M4) -- **Create and Activate a Virtual Environment**: Since Apache Airflow requires multiple dependencies, it's a good practice to isolate these dependencies in a virtual environment. +> ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. -- Create a virtual environment: +### ✅ Recommended Setup (using pyenv): + +1. **Install pyenv**: ```bash -python3 -m venv airflow_venv +brew install pyenv ``` -- Activate the virtual environment: +2. **Install Python**: ```bash -source airflow_venv/bin/activate +pyenv install 3.11.9 +pyenv global 3.11.9 ``` -- **Install Python 3.9 or later**:Airflow requires Python 3.9 or higher, which is supported on M1 Macs or above. You can check your Python version with: +3. **Check Python version**: ```bash -python3 --version +python --version ``` -- If Python is not installed or you need to install a different version, I recommend using Homebrew to manage Python installations. +4. **Create and Activate a Virtual Environment**: Since Apache Airflow requires multiple dependencies, it's a good practice to isolate these dependencies in a virtual environment. + +- Create a virtual environment: ```bash -brew install python@3.9 +python -m venv airflow_venv ``` -- **Install Homebrew (if not already installed)**: If you haven’t installed Homebrew on your Mac yet, you can install it with the following command: +- Activate the virtual environment: ```bash -/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +source airflow_venv/bin/activate ``` -- **Install Apache Airflow**: Apache Airflow is available on PyPI. To install it, you can use the following command in your terminal: +5. **Install Apache Airflow**: Apache Airflow is available on PyPI. To install it, you can use the following command in your terminal: ```bash pip install apache-airflow ``` -- **Set the AIRFLOW_HOME Environment Variable**: Apache Airflow requires a directory to store configuration files, logs, and other data. Set the AIRFLOW_HOME variable to specify this directory. +6. **Set the AIRFLOW_HOME Environment Variable**: Apache Airflow requires a directory to store configuration files, logs, and other data. Set the AIRFLOW_HOME variable to specify this directory. - Set the Airflow home directory: @@ -236,7 +241,7 @@ pip install apache-airflow export AIRFLOW_HOME=~/airflow ``` -- **Start the Airflow Components**: Apache Airflow runs several components, like the scheduler, web server, and API server, to manage workflows and show the UI. +7. **Run Airflow in standalone mode**: Apache Airflow runs several components, like the scheduler, web server, and API server, to manage workflows and show the UI. - To run Airflow in standalone mode (which will automatically start the required components): @@ -244,13 +249,14 @@ export AIRFLOW_HOME=~/airflow airflow standalone ``` -- **Access the Airflow Web UI**: Once the components are up and running, you can access the Airflow UI through your browser: +8. **Access the Airflow Web UI**: Once the components are up and running, you can access the Airflow UI through your browser: - Open your browser and go to: ```arduino http://localhost:8080 ``` + ## Official source code From b55b283660ab6a564a8b61a93d590dabf200a7e1 Mon Sep 17 00:00:00 2001 From: praneeth552 <112244566+praneeth552@users.noreply.github.com> Date: Mon, 5 May 2025 13:53:18 +0530 Subject: [PATCH 03/11] Update README.md Co-authored-by: Jarek Potiuk --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 507832d842528..1e2ac84c53c44 100644 --- a/README.md +++ b/README.md @@ -192,7 +192,13 @@ For information on installing provider distributions, check > ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. -### ✅ Recommended Setup (using pyenv): +You can use other ways to install Python and airflow. Airflow development setup requires ``uv`` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. + +If you are just installing airflow to run it locally, You can use other ways to setup your Python and virtualenv: `uv` is one of the options (refer to `uv` documentation, but you can also use more traditional tools - for example `pyenv`. Note that it is recommmended to install airflow with constraints - at least initially - because this way you can reproducibly install airflow. See [Installation from PyPI[(https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) for more details. + +Herr + +### ✅ Setup using pyenv: 1. **Install pyenv**: From db096a93091c3a3e7142ee6eb4819a78bd99062c Mon Sep 17 00:00:00 2001 From: praneeth552 <112244566+praneeth552@users.noreply.github.com> Date: Mon, 5 May 2025 13:53:31 +0530 Subject: [PATCH 04/11] Update README.md Co-authored-by: Jarek Potiuk --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1e2ac84c53c44..ebdd871e29237 100644 --- a/README.md +++ b/README.md @@ -236,7 +236,14 @@ source airflow_venv/bin/activate 5. **Install Apache Airflow**: Apache Airflow is available on PyPI. To install it, you can use the following command in your terminal: ```bash -pip install apache-airflow +pip install apache-airflow==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" + +Note that installing with constraints - at least initially - is recommended for reproducible installation. It might sometimes happen that 3rd-party distributions are relased and their latest versions break airflow. Using constraints makes the installation reproducible with versions of dependencies that were "frozen" at the time of releasing airflow. Note you have to specify both - airflow version and Python version you are using. + +You can also specify additional extras - when you want to install airflow with addtional providers: + +```bash +pip install apache-airflow[amazon,google]==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" ``` 6. **Set the AIRFLOW_HOME Environment Variable**: Apache Airflow requires a directory to store configuration files, logs, and other data. Set the AIRFLOW_HOME variable to specify this directory. From c2b9c898fac878176ed35f7586fa6b85203c4ae9 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Mon, 5 May 2025 14:36:22 +0530 Subject: [PATCH 05/11] clean up the doc --- README.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index ebdd871e29237..adf98319c46e9 100644 --- a/README.md +++ b/README.md @@ -194,9 +194,7 @@ For information on installing provider distributions, check You can use other ways to install Python and airflow. Airflow development setup requires ``uv`` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. -If you are just installing airflow to run it locally, You can use other ways to setup your Python and virtualenv: `uv` is one of the options (refer to `uv` documentation, but you can also use more traditional tools - for example `pyenv`. Note that it is recommmended to install airflow with constraints - at least initially - because this way you can reproducibly install airflow. See [Installation from PyPI[(https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) for more details. - -Herr +If you are just installing airflow to run it locally, You can use other ways to set up your Python and virtualenv: `uv` is one of the options (refer to `uv` documentation), but you can also use more traditional tools - for example `pyenv`. Note that it is recommended to install airflow with constraints - at least initially - because this way you can reproducibly install airflow. See [Installation from PyPI](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) for more details. ### ✅ Setup using pyenv: @@ -237,10 +235,11 @@ source airflow_venv/bin/activate ```bash pip install apache-airflow==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" +``` -Note that installing with constraints - at least initially - is recommended for reproducible installation. It might sometimes happen that 3rd-party distributions are relased and their latest versions break airflow. Using constraints makes the installation reproducible with versions of dependencies that were "frozen" at the time of releasing airflow. Note you have to specify both - airflow version and Python version you are using. +Note that installing with constraints - at least initially - is recommended for reproducible installation. It might sometimes happen that 3rd-party distributions are released and their latest versions break airflow. Using constraints makes the installation reproducible with versions of dependencies that were "frozen" at the time of releasing airflow. Note you have to specify both - airflow version and Python version you are using. -You can also specify additional extras - when you want to install airflow with addtional providers: +You can also specify additional extras - when you want to install airflow with additional providers: ```bash pip install apache-airflow[amazon,google]==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" @@ -266,7 +265,7 @@ airflow standalone - Open your browser and go to: -```arduino +```text http://localhost:8080 ``` From 9e9c15640056b4fe6028aabfba599085c9765c17 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Mon, 5 May 2025 14:42:30 +0530 Subject: [PATCH 06/11] further polished and refined the doc --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index adf98319c46e9..929aa31383dfa 100644 --- a/README.md +++ b/README.md @@ -192,7 +192,7 @@ For information on installing provider distributions, check > ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. -You can use other ways to install Python and airflow. Airflow development setup requires ``uv`` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. +You can use other ways to install Python and airflow. Airflow development setup requires `uv` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. If you are just installing airflow to run it locally, You can use other ways to set up your Python and virtualenv: `uv` is one of the options (refer to `uv` documentation), but you can also use more traditional tools - for example `pyenv`. Note that it is recommended to install airflow with constraints - at least initially - because this way you can reproducibly install airflow. See [Installation from PyPI](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) for more details. From fff3a698edfd84b22776b7a7212a0a2988cd7826 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Mon, 5 May 2025 15:44:45 +0530 Subject: [PATCH 07/11] Moved installation instructions to INSTALLING.md --- INSTALLING.md | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 79 +-------------------------------------------------- 2 files changed, 79 insertions(+), 78 deletions(-) create mode 100644 INSTALLING.md diff --git a/INSTALLING.md b/INSTALLING.md new file mode 100644 index 0000000000000..0ab71a2605729 --- /dev/null +++ b/INSTALLING.md @@ -0,0 +1,78 @@ +> ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. + +You can use other ways to install Python and airflow. Airflow development setup requires `uv` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. + +If you are just installing airflow to run it locally, You can use other ways to set up your Python and virtualenv: `uv` is one of the options (refer to `uv` documentation), but you can also use more traditional tools - for example `pyenv`. Note that it is recommended to install airflow with constraints - at least initially - because this way you can reproducibly install airflow. See [Installation from PyPI](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) for more details. + +### ✅ Setup using pyenv: + +1. **Install pyenv**: + +```bash +brew install pyenv +``` + +2. **Install Python**: + +```bash +pyenv install 3.11.9 +pyenv global 3.11.9 +``` + +3. **Check Python version**: + +```bash +python --version +``` + +4. **Create and Activate a Virtual Environment**: Since Apache Airflow requires multiple dependencies, it's a good practice to isolate these dependencies in a virtual environment. + +- Create a virtual environment: + +```bash +python -m venv airflow_venv +``` + +- Activate the virtual environment: + +```bash +source airflow_venv/bin/activate +``` + +5. **Install Apache Airflow**: Apache Airflow is available on PyPI. To install it, you can use the following command in your terminal: + +```bash +pip install apache-airflow==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" +``` + +Note that installing with constraints - at least initially - is recommended for reproducible installation. It might sometimes happen that 3rd-party distributions are released and their latest versions break airflow. Using constraints makes the installation reproducible with versions of dependencies that were "frozen" at the time of releasing airflow. Note you have to specify both - airflow version and Python version you are using. + +You can also specify additional extras - when you want to install airflow with additional providers: + +```bash +pip install apache-airflow[amazon,google]==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" +``` + +6. **Set the AIRFLOW_HOME Environment Variable**: Apache Airflow requires a directory to store configuration files, logs, and other data. Set the AIRFLOW_HOME variable to specify this directory. + +- Set the Airflow home directory: + +```bash +export AIRFLOW_HOME=~/airflow +``` + +7. **Run Airflow in standalone mode**: Apache Airflow runs several components, like the scheduler, web server, and API server, to manage workflows and show the UI. + +- To run Airflow in standalone mode (which will automatically start the required components): + +```bash +airflow standalone +``` + +8. **Access the Airflow Web UI**: Once the components are up and running, you can access the Airflow UI through your browser: + +- Open your browser and go to: + +```text +http://localhost:8080 +``` \ No newline at end of file diff --git a/README.md b/README.md index 929aa31383dfa..4eef03ada8b92 100644 --- a/README.md +++ b/README.md @@ -190,84 +190,7 @@ For information on installing provider distributions, check ## Setting up Apache Airflow on Apple Silicon (Mac M1/M2/M3/M4) -> ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. - -You can use other ways to install Python and airflow. Airflow development setup requires `uv` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. - -If you are just installing airflow to run it locally, You can use other ways to set up your Python and virtualenv: `uv` is one of the options (refer to `uv` documentation), but you can also use more traditional tools - for example `pyenv`. Note that it is recommended to install airflow with constraints - at least initially - because this way you can reproducibly install airflow. See [Installation from PyPI](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) for more details. - -### ✅ Setup using pyenv: - -1. **Install pyenv**: - -```bash -brew install pyenv -``` - -2. **Install Python**: - -```bash -pyenv install 3.11.9 -pyenv global 3.11.9 -``` - -3. **Check Python version**: - -```bash -python --version -``` - -4. **Create and Activate a Virtual Environment**: Since Apache Airflow requires multiple dependencies, it's a good practice to isolate these dependencies in a virtual environment. - -- Create a virtual environment: - -```bash -python -m venv airflow_venv -``` - -- Activate the virtual environment: - -```bash -source airflow_venv/bin/activate -``` - -5. **Install Apache Airflow**: Apache Airflow is available on PyPI. To install it, you can use the following command in your terminal: - -```bash -pip install apache-airflow==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" -``` - -Note that installing with constraints - at least initially - is recommended for reproducible installation. It might sometimes happen that 3rd-party distributions are released and their latest versions break airflow. Using constraints makes the installation reproducible with versions of dependencies that were "frozen" at the time of releasing airflow. Note you have to specify both - airflow version and Python version you are using. - -You can also specify additional extras - when you want to install airflow with additional providers: - -```bash -pip install apache-airflow[amazon,google]==3.0.0 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.11.txt" -``` - -6. **Set the AIRFLOW_HOME Environment Variable**: Apache Airflow requires a directory to store configuration files, logs, and other data. Set the AIRFLOW_HOME variable to specify this directory. - -- Set the Airflow home directory: - -```bash -export AIRFLOW_HOME=~/airflow -``` - -7. **Run Airflow in standalone mode**: Apache Airflow runs several components, like the scheduler, web server, and API server, to manage workflows and show the UI. - -- To run Airflow in standalone mode (which will automatically start the required components): - -```bash -airflow standalone -``` - -8. **Access the Airflow Web UI**: Once the components are up and running, you can access the Airflow UI through your browser: - -- Open your browser and go to: - -```text -http://localhost:8080 -``` +For detailed instructions on setting up Apache Airflow on macOS (Apple Silicon), please refer to the [INSTALLING.md](./INSTALLING.md) file. From 6bc7433f0a04f5382fd29430c2dfd2fe8a8fa952 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Mon, 5 May 2025 16:00:17 +0530 Subject: [PATCH 08/11] Refined entire installation section in README.MD and moved to INSTALLING.md --- INSTALLING.md | 35 +++++++++++++++++++++++++++++++++++ README.md | 37 ++----------------------------------- 2 files changed, 37 insertions(+), 35 deletions(-) diff --git a/INSTALLING.md b/INSTALLING.md index 0ab71a2605729..83d48d19b8662 100644 --- a/INSTALLING.md +++ b/INSTALLING.md @@ -1,3 +1,38 @@ +1. Installing just Airflow: + +> Note: Only `pip` installation is currently officially supported. + +While it is possible to install Airflow with tools like [Poetry](https://python-poetry.org) or +[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as +`pip` - especially when it comes to constraint vs. requirements management. +Installing via `Poetry` or `pip-tools` is not currently supported. + +There are known issues with ``bazel`` that might lead to circular dependencies when using it to install +Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing +the problem in `this PR `_ so it might be that +newer versions of ``bazel`` will handle it. + +If you wish to install Airflow using those tools, you should use the constraint files and convert +them to the appropriate format and workflow that your tool requires. + + +```bash +pip install 'apache-airflow==3.0.0' \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" +``` + +2. Installing with extras (i.e., postgres, google) + +```bash +pip install 'apache-airflow[postgres,google]==3.0.0' \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" +``` + +For information on installing provider distributions, check +[providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). + +## Setting up Apache Airflow on Apple Silicon (Mac M1/M2/M3/M4) + > ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. You can use other ways to install Python and airflow. Airflow development setup requires `uv` and if you want to setup environment for development, `uv` is the only supported local development environment setup, because we are using `uv workspace` extensively. See [local virtualenv setup in contributing docs](https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst) for details. diff --git a/README.md b/README.md index 4eef03ada8b92..9c4f0eb7dc4e1 100644 --- a/README.md +++ b/README.md @@ -155,42 +155,9 @@ You can use them as constraint files when installing Airflow from PyPI. Note tha correct Airflow tag/version/branch and Python versions in the URL. -1. Installing just Airflow: +1. Installation Guide -> Note: Only `pip` installation is currently officially supported. - -While it is possible to install Airflow with tools like [Poetry](https://python-poetry.org) or -[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as -`pip` - especially when it comes to constraint vs. requirements management. -Installing via `Poetry` or `pip-tools` is not currently supported. - -There are known issues with ``bazel`` that might lead to circular dependencies when using it to install -Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing -the problem in `this PR `_ so it might be that -newer versions of ``bazel`` will handle it. - -If you wish to install Airflow using those tools, you should use the constraint files and convert -them to the appropriate format and workflow that your tool requires. - - -```bash -pip install 'apache-airflow==3.0.0' \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" -``` - -2. Installing with extras (i.e., postgres, google) - -```bash -pip install 'apache-airflow[postgres,google]==3.0.0' \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" -``` - -For information on installing provider distributions, check -[providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). - -## Setting up Apache Airflow on Apple Silicon (Mac M1/M2/M3/M4) - -For detailed instructions on setting up Apache Airflow on macOS (Apple Silicon), please refer to the [INSTALLING.md](./INSTALLING.md) file. +See [INSTALLING.md](./INSTALLING.md) for setup instructions across all platforms, including Apple Silicon Macs. From c80dbcf9d1c8821528f803b2756ba91743ad17fa Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Mon, 5 May 2025 16:02:13 +0530 Subject: [PATCH 09/11] further refined --- INSTALLING.md | 15 +++++++++++++++ README.md | 17 +---------------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/INSTALLING.md b/INSTALLING.md index 83d48d19b8662..84df2e4081151 100644 --- a/INSTALLING.md +++ b/INSTALLING.md @@ -1,3 +1,18 @@ +## Installing from PyPI + +We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky +because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and +applications usually pin them, but we should do neither and both simultaneously. We decided to keep +our dependencies as open as possible (in `pyproject.toml`) so users can install different versions of libraries +if needed. This means that `pip install apache-airflow` will not work from time to time or will +produce unusable Airflow installation. + +To have repeatable installation, however, we keep a set of "known-to-be-working" constraint +files in the orphan `constraints-main` and `constraints-2-0` branches. We keep those "known-to-be-working" +constraints files separately per major/minor Python version. +You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify +correct Airflow tag/version/branch and Python versions in the URL. + 1. Installing just Airflow: > Note: Only `pip` installation is currently officially supported. diff --git a/README.md b/README.md index 9c4f0eb7dc4e1..ce2ff8bd374b1 100644 --- a/README.md +++ b/README.md @@ -139,23 +139,8 @@ Documentation for dependent projects like provider distributions, Docker image, -## Installing from PyPI -We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky -because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and -applications usually pin them, but we should do neither and both simultaneously. We decided to keep -our dependencies as open as possible (in `pyproject.toml`) so users can install different versions of libraries -if needed. This means that `pip install apache-airflow` will not work from time to time or will -produce unusable Airflow installation. - -To have repeatable installation, however, we keep a set of "known-to-be-working" constraint -files in the orphan `constraints-main` and `constraints-2-0` branches. We keep those "known-to-be-working" -constraints files separately per major/minor Python version. -You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify -correct Airflow tag/version/branch and Python versions in the URL. - - -1. Installation Guide +## Installation Guide See [INSTALLING.md](./INSTALLING.md) for setup instructions across all platforms, including Apple Silicon Macs. From eabb51a4830336abb5959dfa0b8b9303fcb2a2c5 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Wed, 7 May 2025 16:41:08 +0530 Subject: [PATCH 10/11] separated the installation steps --- INSTALLING.md | 48 ----------------------------------------------- README.md | 52 +++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 50 insertions(+), 50 deletions(-) diff --git a/INSTALLING.md b/INSTALLING.md index 84df2e4081151..60fcf8a8c00d5 100644 --- a/INSTALLING.md +++ b/INSTALLING.md @@ -1,51 +1,3 @@ -## Installing from PyPI - -We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky -because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and -applications usually pin them, but we should do neither and both simultaneously. We decided to keep -our dependencies as open as possible (in `pyproject.toml`) so users can install different versions of libraries -if needed. This means that `pip install apache-airflow` will not work from time to time or will -produce unusable Airflow installation. - -To have repeatable installation, however, we keep a set of "known-to-be-working" constraint -files in the orphan `constraints-main` and `constraints-2-0` branches. We keep those "known-to-be-working" -constraints files separately per major/minor Python version. -You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify -correct Airflow tag/version/branch and Python versions in the URL. - -1. Installing just Airflow: - -> Note: Only `pip` installation is currently officially supported. - -While it is possible to install Airflow with tools like [Poetry](https://python-poetry.org) or -[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as -`pip` - especially when it comes to constraint vs. requirements management. -Installing via `Poetry` or `pip-tools` is not currently supported. - -There are known issues with ``bazel`` that might lead to circular dependencies when using it to install -Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing -the problem in `this PR `_ so it might be that -newer versions of ``bazel`` will handle it. - -If you wish to install Airflow using those tools, you should use the constraint files and convert -them to the appropriate format and workflow that your tool requires. - - -```bash -pip install 'apache-airflow==3.0.0' \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" -``` - -2. Installing with extras (i.e., postgres, google) - -```bash -pip install 'apache-airflow[postgres,google]==3.0.0' \ - --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" -``` - -For information on installing provider distributions, check -[providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). - ## Setting up Apache Airflow on Apple Silicon (Mac M1/M2/M3/M4) > ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. diff --git a/README.md b/README.md index ce2ff8bd374b1..647222adee42d 100644 --- a/README.md +++ b/README.md @@ -140,9 +140,57 @@ Documentation for dependent projects like provider distributions, Docker image, -## Installation Guide +## Installing from PyPI -See [INSTALLING.md](./INSTALLING.md) for setup instructions across all platforms, including Apple Silicon Macs. +We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky +because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and +applications usually pin them, but we should do neither and both simultaneously. We decided to keep +our dependencies as open as possible (in `pyproject.toml`) so users can install different versions of libraries +if needed. This means that `pip install apache-airflow` will not work from time to time or will +produce unusable Airflow installation. + +To have repeatable installation, however, we keep a set of "known-to-be-working" constraint +files in the orphan `constraints-main` and `constraints-2-0` branches. We keep those "known-to-be-working" +constraints files separately per major/minor Python version. +You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify +correct Airflow tag/version/branch and Python versions in the URL. + +1. Installing just Airflow: + +> Note: Only `pip` installation is currently officially supported. + +While it is possible to install Airflow with tools like [Poetry](https://python-poetry.org) or +[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as +`pip` - especially when it comes to constraint vs. requirements management. +Installing via `Poetry` or `pip-tools` is not currently supported. + +There are known issues with ``bazel`` that might lead to circular dependencies when using it to install +Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing +the problem in `this PR `_ so it might be that +newer versions of ``bazel`` will handle it. + +If you wish to install Airflow using those tools, you should use the constraint files and convert +them to the appropriate format and workflow that your tool requires. + + +```bash +pip install 'apache-airflow==3.0.0' \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" +``` + +2. Installing with extras (i.e., postgres, google) + +```bash +pip install 'apache-airflow[postgres,google]==3.0.0' \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.9.txt" +``` + +For information on installing provider distributions, check +[providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). + +## macOS Setup Instructions + +If you're a macOS user looking to install or run Airflow locally, see the [INSTALLING.md](./INSTALLING.md) file From 1597a783579ca6c5aff7f4a0595fecd08cfc9421 Mon Sep 17 00:00:00 2001 From: praneeth552 Date: Wed, 14 May 2025 15:50:23 +0530 Subject: [PATCH 11/11] moved to down of the PyPI --- INSTALLING.md | 29 ++++++++++++++++++++++++++--- README.md | 8 +++++--- generated/PYPI_README.md | 2 +- 3 files changed, 32 insertions(+), 7 deletions(-) diff --git a/INSTALLING.md b/INSTALLING.md index 60fcf8a8c00d5..e62b66a7a2a0e 100644 --- a/INSTALLING.md +++ b/INSTALLING.md @@ -1,4 +1,25 @@ -## Setting up Apache Airflow on Apple Silicon (Mac M1/M2/M3/M4) + + +## Local Development Setup + +This section outlines a recommended approach for setting up a local development environment for Apache Airflow on macOS and Linux, primarily using PyEnv for Python version management. > ⚠️ Avoid using either system-installed Python or Python from Homebrew, as these versions are often labeled `--externally-managed` resulting in restricted dependency installation. @@ -8,12 +29,14 @@ If you are just installing airflow to run it locally, You can use other ways to ### ✅ Setup using pyenv: -1. **Install pyenv**: +1. **Install pyenv (macOS and Linux)**: ```bash brew install pyenv ``` +(Note: Homebrew is the recommended method on macOS. For Linux, you can typically install pyenv using the `pyenv-installer` script as detailed in the official documentation: [https://github.com/pyenv/pyenv#installation](https://github.com/pyenv/pyenv#installation).) + 2. **Install Python**: ```bash @@ -77,4 +100,4 @@ airflow standalone ```text http://localhost:8080 -``` \ No newline at end of file +``` diff --git a/README.md b/README.md index 647222adee42d..e0b37e3a5c899 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,7 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The - [Requirements](#requirements) - [Getting started](#getting-started) - [Installing from PyPI](#installing-from-pypi) +- [Installation](#installation) - [Official source code](#official-source-code) - [Convenience packages](#convenience-packages) - [User Interface](#user-interface) @@ -188,11 +189,12 @@ pip install 'apache-airflow[postgres,google]==3.0.0' \ For information on installing provider distributions, check [providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html). -## macOS Setup Instructions + -If you're a macOS user looking to install or run Airflow locally, see the [INSTALLING.md](./INSTALLING.md) file +## Installation + +For comprehensive instructions on setting up your local development environment and installing Apache Airflow, please refer to the [INSTALLING.md](INSTALLING.md) file. - ## Official source code diff --git a/generated/PYPI_README.md b/generated/PYPI_README.md index 618b9985a7e5d..947b9d6f5cd76 100644 --- a/generated/PYPI_README.md +++ b/generated/PYPI_README.md @@ -93,6 +93,7 @@ the [Airflow Wiki](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+I Documentation for dependent projects like provider distributions, Docker image, Helm Chart, you'll find it in [the documentation index](https://airflow.apache.org/docs/). + ## Installing from PyPI We publish Apache Airflow as `apache-airflow` package in PyPI. Installing it however might be sometimes tricky @@ -108,7 +109,6 @@ constraints files separately per major/minor Python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and Python versions in the URL. - 1. Installing just Airflow: > Note: Only `pip` installation is currently officially supported.