Handle Serialized DAG Format from v3 to v2 when downgrading Airflow #55975

kaxil · 2025-09-23T00:08:07Z

This migration enables Airflow downgrades by converting v3 serialized DAGs back to v2 format. The upgrade() is a no-op since the server handles v1/v2/v3 at runtime, but downgrade() removes client_defaults sections and updates version numbers to ensure compatibility with older Airflow versions.

closes #55949

Note on dag_hash behavior:
The migration does not update the dag_hash column when modifying DAG data. The DAG processor will only fix the hash for the latest record on next processing - older records will remain with inconsistent
hashes permanently.

The hash mismatch is a natural consequence of modifying existing serialized DAG data and does not require manual intervention.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

jedcunningham

Self-correcting: On the next DAG processing cycle, the dag processor will detect the hash mismatch and update both the serialized data and dag_hash to maintain consistency

Pretty sure it will only fix the latest, if there are no runs... Older versions will be left in a not-consistent state long term. Might not break anything, but isn't great...

...ow-core/src/airflow/migrations/versions/0085_3_1_0_downgrade_serialized_dag_version_to_v2.py

This migration enables Airflow downgrades by converting v3 serialized DAGs back to v2 format. The `upgrade()` is a no-op since the server handles v1/v2/v3 at runtime, but `downgrade()` removes client_defaults sections and updates version numbers to ensure compatibility with older Airflow versions. closes apache#55949

Improves storage efficiency and query performance for DAG serialization data by using PostgreSQL's native `JSONB` type instead of `JSON`. `JSONB` provides better compression, faster equality comparisons, and removes whitespace/duplicate keys. Would have also made the query in apache#55975 simpler.

vatsrahul1001 · 2025-09-23T11:22:39Z

Testing

Steps for testing

Created 3.0.6 deployment and executed dag runs
Upgraded to db-downgrade branch in OSS and executed dag runs
Initiated downgrade
Verified migration job pod and confirmed patch downgrade version did execute

025-09-23T04:17:57.594277Z [info     [] Running downgrade cc92b33c6709 -> eaf332f43c7c, Add backward compatibility for serialized DAG format v3 to v2. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.764601Z [info     [] Running downgrade eaf332f43c7c -> a3c7f2b18d4e, add last_parse_duration to dag model. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.775642Z [info     [] Running downgrade a3c7f2b18d4e -> 7582ea3f3dd5, Add tables to store teams and associations with dag bundles. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.804898Z [info     [] Running downgrade 7582ea3f3dd5 -> a169942745c2, Make bundle_name not nullable. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.820755Z [info     [] Running downgrade a169942745c2 -> 808787349f22, Remove dag_id from Deadline. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.833496Z [info     [] Running downgrade 808787349f22 -> 3bda03debd04, Modify deadline's callback schema. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.845897Z [info     [] Running downgrade 3bda03debd04 -> f56f68b9e02f, Add url template and template params to DagBundleModel. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.857328Z [info     [] Running downgrade f56f68b9e02f -> 09fa89ba1710, Add callback_state to deadline. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.880375Z [info     [] Running downgrade 09fa89ba1710 -> 40f7c30a228b, Add trigger_id to deadline. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.892091Z [info     [] Running downgrade 40f7c30a228b -> ffdb0566c7c0, Add Human In the Loop Detail table. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.903529Z [info     [] Running downgrade ffdb0566c7c0 -> 66a7743fe20e, Add dag_favorite table. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.915499Z [info     [] Running downgrade 66a7743fe20e -> 583e80dfcef4, Add triggering user to dag_run. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.926852Z [info     [] Running downgrade 583e80dfcef4 -> 3ac9e5732b1f, Add task_inlet_asset_reference table. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.938975Z [info     [] Running downgrade 3ac9e5732b1f -> 0242ac120002, Change the on-delete behaviour of task_instance.dag_version_id foreign key constraint to RESTRICT. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.953119Z [info     [] Running downgrade 0242ac120002 -> dfee8bd5d574, Rename Deadline column in the Deadline table from deadline to deadline_time and change its type from DateTime to UTC DateTime. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.977139Z [info     [] Running downgrade dfee8bd5d574 -> fe199e1abd77, Add Deadline to Dag. [alembic.runtime.migration] loc=migration.py:622

Downgrade complete

Confirmed I am able to see runs executed in step2 and no error in API server logs

kaxil · 2025-09-23T12:25:03Z

fyi I checked dag_hash usage and found that dag_hash is primarily used in SerializedDagModel.write_dag() (lines 413-419) to detect if a DAG has changed and avoid unnecessary database writes. The scheduler doesn't use it anymore for change detection anymore - it uses dag_version_id. The hash is mainly for write optimization in the DAG processor. So I think it should be ok.

(fyi @ephraimbuddy )

…55975) This migration enables Airflow downgrades by converting v3 serialized DAGs back to v2 format. The `upgrade()` is a no-op since the server handles v1/v2/v3 at runtime, but `downgrade()` removes client_defaults sections and updates version numbers to ensure compatibility with older Airflow versions. closes #55949 (cherry picked from commit 3299567)

Improves storage efficiency and query performance for DAG serialization data by using PostgreSQL's native `JSONB` type instead of `JSON`. `JSONB` provides better compression, faster equality comparisons, and removes whitespace/duplicate keys. Would have also made the query in apache#55975 simpler.

Improves storage efficiency and query performance for DAG serialization data by using PostgreSQL's native `JSONB` type instead of `JSON`. `JSONB` provides better compression, faster equality comparisons, and removes whitespace/duplicate keys. Would have also made the query in #55975 simpler.

…pache#55975) This migration enables Airflow downgrades by converting v3 serialized DAGs back to v2 format. The `upgrade()` is a no-op since the server handles v1/v2/v3 at runtime, but `downgrade()` removes client_defaults sections and updates version numbers to ensure compatibility with older Airflow versions. closes apache#55949

…e#55979) Improves storage efficiency and query performance for DAG serialization data by using PostgreSQL's native `JSONB` type instead of `JSON`. `JSONB` provides better compression, faster equality comparisons, and removes whitespace/duplicate keys. Would have also made the query in apache#55975 simpler.