From 03d53921ae51c3c105d73992803317d0e432bc5b Mon Sep 17 00:00:00 2001 From: Kaxil Naik Date: Tue, 13 May 2025 20:54:41 +0530 Subject: [PATCH 1/2] Add Execution API versioning docs for Contributors (#50539) Added our versioning strategy for Task Execution API in Contributing Guide. (cherry picked from commit 188e22530cd3067d1c6ad7291834dad1ebe23732) --- .../19_execution_api_versioning.rst | 103 ++++++++++++++++++ contributing-docs/README.rst | 3 + 2 files changed, 106 insertions(+) create mode 100644 contributing-docs/19_execution_api_versioning.rst diff --git a/contributing-docs/19_execution_api_versioning.rst b/contributing-docs/19_execution_api_versioning.rst new file mode 100644 index 0000000000000..189aed58a5221 --- /dev/null +++ b/contributing-docs/19_execution_api_versioning.rst @@ -0,0 +1,103 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Execution API Versioning +======================== + +Airflow's Task Execution API uses `Cadwyn `_ for API versioning with CalVer. +This allows us to maintain backward compatibility while evolving the API over time. + +Why? +---- + +Airflow components (e.g., workers, API servers) could be deployed independently. This can lead +to version mismatches—for example, a worker using Task SDK 1.0.1 (requiring Airflow >=3.0.1) while the +API server is still on 3.0.0. Without versioning, such mismatches can cause runtime failures or subtle bugs. +A strict versioning model allows older clients to continue working with newer servers, enabling safe, incremental upgrades. + +Versioning Principles +--------------------- + +1. Any change to the API schema or behavior that affects existing clients requires a new version. +2. All versions are maintained in the codebase +3. Migrations encapsulate all schema changes. +4. Core code logic remains version-agnostic. + +Version Changes +--------------- + +When making changes to the Execution API, you must: + +1. Add a new migration module (e.g., ``v2025_04_28.py``). Pick a likely date for when the Airflow version + will be released. + + - Use the ``vYYYY_MM_DD`` format. + - New migrations can be added to an existing unreleased version; not every migration needs a new version. + - For unreleased versions, pick a likely date for when it will be released. + - The version number should be unique and not conflict with any existing versions. +2. Document the changes in the migration. +3. Update tests to cover both old and new versions. +4. Ensure backward compatibility wherever possible. + +Example Version Migration +------------------------- + +Here's an example of how to add a new version migration that adds a new field ``consumed_asset_events`` to the +``DagRun`` model. + +.. code-block:: python + + from cadwyn import ResponseInfo, VersionChange, convert_response_to_previous_version_for, schema + + from airflow.api_fastapi.execution_api.datamodels.taskinstance import DagRun, TIRunContext + + + class AddConsumedAssetEventsField(VersionChange): + """Add the `consumed_asset_events` to DagRun model.""" + + description = __doc__ + + instructions_to_migrate_to_previous_version = (schema(DagRun).field("consumed_asset_events").didnt_exist,) + + @convert_response_to_previous_version_for(TIRunContext) # type: ignore + def remove_consumed_asset_events(response: ResponseInfo): # type: ignore + response.body["dag_run"].pop("consumed_asset_events") + +Directory Structure +------------------- + +The Execution API versioning code is organized as follows: + +- ``airflow-core/src/airflow/api_fastapi/execution_api/versions/`` - Contains version migrations +- ``airflow-core/src/airflow/api_fastapi/execution_api/versions/head/`` - Contains the latest version models +- ``airflow-core/src/airflow/api_fastapi/execution_api/versions/v2025_04_28/`` - Contains version-specific models and migrations +- ``airflow-core/tests/unit/api_fastapi/execution_api/versions/head`` - Contains tests for the latest version +- ``airflow-core/tests/unit/api_fastapi/execution_api/versions/v2025_04_28`` - Contains tests for the version-specific models + +Examples +-------- + +For an example of how to implement version changes, see: +- `PR #50528 `_ +- `PR #48125 `_ + +Resources +--------- + +- `Cadwyn Documentation `_ +- `Cadwyn GitHub Repository `_ +- `Cadwyn Version Changes Guide `_ diff --git a/contributing-docs/README.rst b/contributing-docs/README.rst index f26fd952bac77..9796df57a08e7 100644 --- a/contributing-docs/README.rst +++ b/contributing-docs/README.rst @@ -118,3 +118,6 @@ You can also dive deeper into more specific areas that are important for contrib * `Architecture diagram <17_architecture_diagrams.rst>`__ describes how to create and update the architecture diagrams embedded in Airflow documentation. + +* `Execution API versioning <19_execution_api_versioning.rst>`__ describes how to + version the Task Execution API and how to add new versions of the API. From 142b61347df0104ff342591c3ab114e149553cd2 Mon Sep 17 00:00:00 2001 From: Amogh Desai Date: Wed, 28 May 2025 20:20:59 +0530 Subject: [PATCH 2/2] Refine the execution api versioning doc to reflect reality (#51152) (cherry picked from commit f7ceec4b63b6e51e56b41826fe5c4fae410668b1) --- contributing-docs/19_execution_api_versioning.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/contributing-docs/19_execution_api_versioning.rst b/contributing-docs/19_execution_api_versioning.rst index 189aed58a5221..c9d4590950829 100644 --- a/contributing-docs/19_execution_api_versioning.rst +++ b/contributing-docs/19_execution_api_versioning.rst @@ -25,9 +25,14 @@ Why? ---- Airflow components (e.g., workers, API servers) could be deployed independently. This can lead -to version mismatches—for example, a worker using Task SDK 1.0.1 (requiring Airflow >=3.0.1) while the -API server is still on 3.0.0. Without versioning, such mismatches can cause runtime failures or subtle bugs. -A strict versioning model allows older clients to continue working with newer servers, enabling safe, incremental upgrades. +to version mismatches—for example, a worker using Task SDK 1.0.0 (requiring Airflow >=3.0.0) while the +API server has been upgraded to 3.0.1. Without versioning, such mismatches can cause runtime failures or subtle bugs. +Enforcing a clear versioning contract ensures backward compatibility and allows for safe, incremental upgrades +across different components. + +Note: We're prioritizing backward compatibility, older clients should continue to work seamlessly with newer servers. +Since the client (Task SDK) and server (API) are still coupled, we can look at forward compatibility once we have a +clear versioning strategy between Task SDK and Airflow Core. Versioning Principles ---------------------