Airflow 2 to 3 Rest API Changes

[pierrejeambrun]

Rest API Breaking Change

The goal is to make an exhaustive list of of the breaking / noticeable changes for the Rest API moving from airflow 2 to airflow 3.

Some of those changes are inherent to FastAPI and hard to circumvent, some others are here just for code clarity or simplicity and can be 'fixed'. That being said I think that we should take the opportunity of Airflow 3 to clean/adjust the code allowing ourselves a few breaking changes when necessary.

• Th API now returns 422 instead of 400 in cases for validation/deserialization errors (body, query/path parameters).

• When listing a resource for instance on GET /dags, fields parameter is not supported anymore to obtain a partial response. Full objects will be returned by the endpoint.

• Passing list query parameter switched from form, non exploded to form, exploded, i.e before ?my_list=item1,item2 now ?my_list=item1&my_list=item2

• execution_date deprecated and is now removed. Any payload or query parameters mentioning this field has been removed.

• datetime format are RFC3339-compliant in FastAPI, more permissive than ISO8601. meaning that the API returns zulu datetime for responses, more info here. Both Z and 00+xx are supported for payload and params. Workaround here Unable to parse Zulu datetimes using standard library pydantic/pydantic#6028 (comment). This is due to pydantic v2 default behavior

• Patch on DagRun and TaskInstance are more generic and allow in addition to update the ressource state to update the note content. Therefore the two legacy dedicated endpoints to update DagRun note and TaskInstance note have been removed. Same for the set task instance state, it is now handled by the broader PATCH on task intances.

• assets/queuedEvent endpoints have moved to assets/queuedEvents for consistency.

• dag_parsing endpoint now returns a 409 when the DagPriorityParsingRequest already exists. (It was returning 201 before).

• clearTaskInstances endpoint, default value for reset_dag_runs has been updated from False to True.

• Pool name can't be modified in the PATCH pool endpoint anymore. Pool name shouldn't be updated via pool PATCH API call #46986

• Logical date is now a nullable required payload field for Triggering a DagRun endpoint

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pierrejeambrun]

Waiting for the newsfragment format to be settle to create associated significant newsfragments.

[rawwar]

@pierrejeambrun , regarding

When listing a resource for instance on GET /dags, fields parameter is not supported anymore to obtain a partial response. Full objects will be returned by the endpoint.

We can use model_serializer to decide what fields to return. Would this be ok?

[omkar-foss]

I suppose FastAPI throws 422 instead of 400 in cases where there are missing required parameters (or validation errors) in the request header or body.

@pierrejeambrun please see if this needs to be listed as a breaking change, thought to highlight it, just in case! :)

[pierrejeambrun]

I suppose FastAPI throws 422 instead of 400 in cases where there are missing required parameters (or validation errors) in the request header or body.
@pierrejeambrun please see if this needs to be listed as a breaking change, thought to highlight it, just in case! :)

Good idea, done