Support Pydantic 2 #32366

uranusjr · 2023-07-05T08:43:55Z

Fix #32311 eventually. For CI only at the moment.

This is a temporary measure to work around dependencies pinning Pydantic to 1.8.x. Will remove this once we fixed all the 2.x incompatibilities.

uranusjr · 2023-07-06T09:19:17Z

Hey, the CI passes.

If I can get some reviews to ensure the changes are OK, I’ll add a commit to remove the build script hack so this is merge-able.

scripts/docker/install_airflow_dependencies_from_branch_tip.sh

potiuk · 2023-07-06T10:51:45Z

airflow/serialization/pydantic/dag_run.py

    dag_id: str
    queued_at: Optional[datetime]
-    execution_date: DateTime
+    execution_date: datetime


Ah... So Pydantic just got more "picky" about those. Cool.

Yeah I think they basically switch from isinstance to type identity check, probably for performance concerns (we have similar designs in our own serialisation code). The old behaviour can be brought back with a flag, but this doesn’t need that.

Dockerfile

potiuk

Pending the pydantic>2 hack removal.

This reverts commit f17c6dc.

eladkal · 2023-07-08T12:24:57Z

setup.cfg

-    # Limit Pydantic to <2.0.0 because it breaks Kubernetes tests and building providers
-    # This limit should be removed after https://github.com/apache/airflow/issues/32311 is fixed
-    pydantic>=1.10.0,<2.0.0
+    pydantic>=1.10.0


Started to see

Unknown warnings generated: /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:210:Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/ /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:261:Valid config keys have changed in V2: * 'orm_mode' has been renamed to 'from_attributes' /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:210:Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/ /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:210:Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/ /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:210:Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/ /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:210:Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/ /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:210:Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/ /usr/local/lib/python3.8/site-packages/pydantic/_internal/_config.py:261:Valid config keys have changed in V2: * 'orm_mode' has been renamed to 'from_attributes' ERROR! There were 8 warnings generated during the import

Example:

https://github.com/apache/airflow/actions/runs/5494206967/jobs/10012899407?pr=32441#step:11:1600

#32444 will handle it

lmrlyv · 2023-08-18T13:47:21Z

Hi, @potiuk!

I'm interested in finding out when the next release is expected to be made available. Having support for Pydantic 2 would be highly beneficial.

potiuk · 2023-08-18T13:59:18Z

I'm interested in finding out when the next release is expected to be made available. Having support for Pydantic 2 would be highly beneficial.

Why? Could you explain why ?

Airlfow 2.7.0 is being released today as we speak (working on it). But in constraints Pydantic is still < 2 - because some of our dependencies limit us, But we have no requiments, and if you are not using any provider that limits pydantic through their dependencies, you might attempt to upgrade and see if it works for you. But you will need to test it and make sure it works.

lmrlyv · 2023-08-18T16:00:19Z

I'm interested in finding out when the next release is expected to be made available. Having support for Pydantic 2 would be highly beneficial.

Why? Could you explain why ?

Airlfow 2.7.0 is being released today as we speak (working on it). But in constraints Pydantic is still < 2 - because some of our dependencies limit us, But we have no requiments, and if you are not using any provider that limits pydantic through their dependencies, you might attempt to upgrade and see if it works for you. But you will need to test it and make sure it works.

If you're asking about the reasons for using Pydantic v2, I'd highlight its significant speed improvement of 5-50 times, along with its enhanced stability and flexibility when compared to v1 (according to its release notes).

Regarding why we are waiting for this release, it's because we encountered an error (although I don't remember the details now) while attempting to deploy our airflow (v2.6.3) project that relies on Pydantic v2 models into a Kubernetes cluster. There was an open issue in this repo, and this pull request is expected to address and resolve that problem.

potiuk · 2023-08-18T18:40:19Z

If you're asking about the reasons for using Pydantic v2, I'd highlight its significant speed improvement of 5-50 times, along with its enhanced stability and flexibility when compared to v1 (according to its release notes).

We are hardly using Pydantic, so improvements would be mariginal, but yes I am very well aware of it, just for "Airflow" it is not highly benefitial (yet). It might be in the future as we have plans for using Pydantic for internal API.

Regarding why we are waiting for this release, it's because we encountered an error (although I don't remember the details now) while attempting to deploy our airflow (v2.6.3) project that relies on Pydantic v2 models into a Kubernetes cluster. There was an open issue in this repo, and this pull request is expected to address and resolve that problem.

Just checked: 2.7.0 stil has no support for Pydantic:

# We limit Pydantic to <2.0.0 until we can upgrade - there are limitation for Pydantic in AWS provider
# dependency (aws-sam-translator) - also we need to change orm-mode to from_attributes in definitions
# of the ORM models. See for the previous attempt https://github.com/apache/airflow/pull/33220
pydantic>=1.10.0,<2.0.0

That was likely an overlook as part of the problem with orm-mode at least is solved. So you will have to wait for 2.7.1

potiuk · 2023-08-18T18:55:46Z

I added limitation removal and scheduled it for 2.7.1 #33507 - seems that the aws-sam-translator removed the limitation last week and are going to be released in a week or two, which should be good as then our CI will test it automatically and we will produce constraints with Pydantic 2.

uranusjr added 2 commits July 5, 2023 14:05

Remove Pydantic limit

855077f

Force-install Pydantic 2 in CI

f17c6dc

This is a temporary measure to work around dependencies pinning Pydantic to 1.8.x. Will remove this once we fixed all the 2.x incompatibilities.

boring-cyborg bot added area:dev-tools area:production-image Production image improvements and fixes labels Jul 5, 2023

Use base datetime class for Pydantic

52228da

uranusjr commented Jul 6, 2023

View reviewed changes

scripts/docker/install_airflow_dependencies_from_branch_tip.sh Outdated Show resolved Hide resolved

uranusjr marked this pull request as ready for review July 6, 2023 09:20

uranusjr requested review from ashb, bolkedebruin, kaxil and potiuk as code owners July 6, 2023 09:20

uranusjr added area:dependencies Issues related to dependencies problems and removed area:production-image Production image improvements and fixes area:dev-tools labels Jul 6, 2023

potiuk reviewed Jul 6, 2023

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

potiuk approved these changes Jul 6, 2023

View reviewed changes

Revert "Force-install Pydantic 2 in CI"

a952ea1

This reverts commit f17c6dc.

vincbeck approved these changes Jul 7, 2023

View reviewed changes

potiuk merged commit 9cb463e into apache:main Jul 7, 2023

eladkal added this to the Airflow 2.6.4 milestone Jul 8, 2023

eladkal reviewed Jul 8, 2023

View reviewed changes

uranusjr deleted the pydantic-2-support branch July 10, 2023 08:36

eladkal modified the milestones: Airflow 2.6.4, Airflow 2.7.0 Aug 1, 2023

ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Aug 2, 2023

hsheth2 mentioned this pull request Sep 27, 2023

acryl-datahub-airflow-plugin airflow 2.7 compatibility datahub-project/datahub#8892

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Pydantic 2 #32366

Support Pydantic 2 #32366

Uh oh!

uranusjr commented Jul 5, 2023

Uh oh!

uranusjr commented Jul 6, 2023

Uh oh!

Uh oh!

potiuk Jul 6, 2023

Uh oh!

uranusjr Jul 7, 2023

Uh oh!

Uh oh!

potiuk left a comment

Uh oh!

eladkal Jul 8, 2023

Uh oh!

eladkal Jul 8, 2023

Uh oh!

lmrlyv commented Aug 18, 2023

Uh oh!

potiuk commented Aug 18, 2023

Uh oh!

lmrlyv commented Aug 18, 2023

Uh oh!

potiuk commented Aug 18, 2023

Uh oh!

potiuk commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Support Pydantic 2 #32366

Support Pydantic 2 #32366

Uh oh!

Conversation

uranusjr commented Jul 5, 2023

Uh oh!

uranusjr commented Jul 6, 2023

Uh oh!

Uh oh!

potiuk Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

uranusjr Jul 7, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

eladkal Jul 8, 2023

Choose a reason for hiding this comment

Uh oh!

eladkal Jul 8, 2023

Choose a reason for hiding this comment

Uh oh!

lmrlyv commented Aug 18, 2023

Uh oh!

potiuk commented Aug 18, 2023

Uh oh!

lmrlyv commented Aug 18, 2023

Uh oh!

potiuk commented Aug 18, 2023

Uh oh!

potiuk commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants