python: generate Pydantic v2 + typing complete code by multani · Pull Request #16624 · OpenAPITools/openapi-generator

multani · 2023-09-19T20:30:43Z

was: python: Generate fully described Python types

Warning

Sorry, it's a rather large change, but I think it all helps to produce better type information.

I'm looking for feedback to know if this is the good way to integrate this kind of change, and if this has a chance of being merged.

If needed, I can try to make this big change shorter by splitting this into multiple PRs; it may be harder to see the big picture with smaller PRs though.

The goal of this change is to replace the mypy incompatible types (conlist, conint, constr, etc.) by their more modern versions using Annotated types with Pydantic v2.

To do so, the code generator changes in several ways:

The generator introduces several new inner classes to help the code generation:
- PythonType is a recursive object that hold the information about a particular type that the generator is trying to build
- PydanticType, directly translated from the previous getPydanticType methods, creates PythonType instances from the OpenAPI specifications and produces the final Python types as strings.
- An initial Imports class holds all the new Python's import statements that the generator produces along the way.

Via these new classes, the code generation gets improved by the following:

The string-based formater approach is now replaced by a tree of PythonType objects:
- Each object can hold all its information: type constraints, metadata, etc.
- By being a tree of itself, types can become naturally nested and allow to easily express complex types
Most of the code duplication between the 2 getPydanticType methods have been unified to remove duplication:
- Each OpenAPI type definition has a matching OpenAPI type -> Python type translation method
- Common type translations between CodegenParamerer and CodegenProperty have been unified
The new Imports class can be passed between calls to accumulate all the imports:
- This removes several flags that were present only to track if something needed to be imported or not.
- An instance of the class can be shared across multiple method calls to accumulate all the imports that have to be done
- It reduces the amount of objects to carry around as it can hold any kind of imports.
- Ultimately, it also allows to control better which Python resources are imported from which modules, and should help to reduce the number of unused imports in the generated code.

References I used to implement the new types:

The parent issue [REQ][python-nextgen] Use Annotated with constrained types #15464
A related issue in Pydantic, when Pydantic introduce support for Annotated: mypy annotation and constrained lists recommendation pydantic/pydantic#975)
(I will add more as I review the code)

I think once generalized, usage of the Imports class instead of the numerous Set instances could help to track all the import dependencies, reduce the generated code (and unused imports) and the code generator itself.
A whole part of the code generator has not been touched and could benefit from the same kind of improvements.

PR checklist

Read the contribution guidelines.
Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
Run the following to build the project and update samples:
```
./mvnw clean package 
./bin/generate-samples.sh ./bin/configs/*.yaml
./bin/utils/export_docs_generators.sh
```
Commit all changed files.
This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
These must match the expectations made by your contribution.
You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
For Windows users, please run the script in Git BASH.
File the PR against the correct branch: master (upcoming 7.1.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

@krjakbrjak as Python tech-comittee
@wing328 as the previous major editor of the code generator

samples/client/echo_api/python/openapi_client/api/body_api.py

multani · 2023-09-21T20:07:53Z

I removed the poetry.lock files I added during my experiment, and it divided the size of the pull request by 2 :D

wing328 · 2023-09-22T03:50:55Z

cc @OpenAPITools/generator-core-team

robertschweizer

Thanks a lot for the PR! I checked some of the generated classes.

modules/openapi-generator/src/main/resources/python/requirements.mustache

samples/client/echo_api/python/openapi_client/models/pet.py

robertschweizer · 2023-09-22T08:22:00Z

samples/openapi3/client/petstore/python-aiohttp/petstore_api/models/color.py

    """
    # data type: List[int]
-    oneof_schema_1_validator: Optional[conlist(conint(strict=True, le=255, ge=0), max_items=3, min_items=3)] = Field(None, description="RGB three element array with values 0-255.")
+    oneof_schema_1_validator: Optional[Annotated[List[Annotated[int, Field(le=255, strict=True, ge=0)]], Field(min_items=3, max_items=3)]] = Field(default=None, description="RGB three element array with values 0-255.")


I do not get an error if I construct this with oneof_schema_1_validator=[1, 2]. Maybe pydantic is thrown off by Field() being used on both sides of =? Also pydantic might not take into account Field() restrictions in the first argument of Annotated[].

Best add a unit-test to make sure validation works, or is there one already?

I will check this 👍

I added more tests: this works with Pydantic v2 but not with Pydantic v1. We'll focus on the former anyway and I'll merge my PR in a Pydantic v2-only generator 👍

This is great, thanks a lot for moving forward to pydantic v2! Then things work of course :)

multani · 2023-09-22T10:32:59Z

Thanks to the merge of #16643, I have plenty of test failures now :)

I will turn this back to draft while I'm fixing the comments and the tests 👍

wing328 · 2023-09-27T05:49:38Z

modules/openapi-generator/src/main/resources/python/api.mustache

 from pydantic import validate_arguments, ValidationError
-{{#asyncio}}
-from typing import overload, Optional, Union, Awaitable
-{{/asyncio}}


why these lines are removed?

2 reasons:

These imports are not asyncio-specific

The imported names are not used in the template

The names could be used by the code generated by the Java code, in which case the imports would be provided from the Java code instead.

wing328 · 2023-09-27T05:51:03Z

...penapi-generator/src/main/java/org/openapitools/codegen/languages/AbstractPythonCodegen.java


            // import models one by one
-            if (!modelImports.isEmpty()) {
+            {


why removed the check here?

I don't remember, I will add it back.

I need to change the place where x-py-model-imports (there could be resources to imports even if there no dependencies between models), I will move it outside the if block.

I re-added the condition and moved the otherImports.exports() imports out of the block.

b0g3r · 2023-12-12T21:44:46Z

modules/openapi-generator/src/main/resources/python/model_anyof.mustache

 {{/composedSchemas.anyOf}}
    if TYPE_CHECKING:
-        actual_instance: Union[{{#anyOf}}{{{.}}}{{^-last}}, {{/-last}}{{/anyOf}}]
+        actual_instance: Optional[Union[{{#anyOf}}{{{.}}}{{^-last}}, {{/-last}}{{/anyOf}}]] = None


@multani Hi Jonathan! Do you remember, by any chance, why did you introduce Optional here? Is it possible to have None value here even with one_of validator?

Hey 👋 Sorry, I don't remember the details of this particular change :/ Does it create a problem for you?

Yes, unfortunately. Before this changes non-optional anyOf had correct typing (Union[A, B]), but now it has additional None in the union, which isn't right and stops me from using proper typing. Should I try to change it back and create PR for it?

Yes, of course, feel free to create a pull request!

I just checked again, and there's also a is None check in the to_dict method at the end of the file, it's probably why I added the Optional type in the first place. If you remove Optional, can you also remove that check?

In OpenAPITools#16624, I introduced a new mechanism to record imports to other modules, instead of having specialized datetime/typing/pydantic objects to manage imports for these modules. This change reuses the mechanism from OpenAPITools#16624 and replace the specialized import managers by the generic one. Unused imports from various .mustache templates are also cleaned up.

In #16624, I introduced a new mechanism to record imports to other modules, instead of having specialized datetime/typing/pydantic objects to manage imports for these modules. This change reuses the mechanism from #16624 and replace the specialized import managers by the generic one. Unused imports from various .mustache templates are also cleaned up.

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624 (cherry picked from commit d176113)

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624 (cherry picked from commit d176113)

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624 (cherry picked from commit d176113630d6b7d4bc36511fe3f8cb013060d675) GitOrigin-RevId: 4e84838785f49f7d0a646018ba22992d82e65a76

Python Client now need pydantic as we use the new version of the OpenAPI python client generator: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/python/requirements.mustache#L14 It was added in OpenAPITools/openapi-generator#16624 GitOrigin-RevId: d176113630d6b7d4bc36511fe3f8cb013060d675

multani marked this pull request as draft September 19, 2023 20:30

This was referenced Sep 19, 2023

Draft: Update python generator with correct mypy types and remove sync endpoints in async generator #15878

Closed

[REQ][python-nextgen] Use Annotated with constrained types #15464

Closed

multani force-pushed the python-typing-1 branch 2 times, most recently from bdc8789 to e067ce3 Compare September 20, 2023 07:08

multani commented Sep 20, 2023

View reviewed changes

samples/client/echo_api/python/openapi_client/api/body_api.py Outdated Show resolved Hide resolved

python: improve type generation with more specific typing

23617df

multani force-pushed the python-typing-1 branch from e067ce3 to 23617df Compare September 21, 2023 19:34

multani added 5 commits September 21, 2023 21:48

Annotate function parameters

4587ad9

Remove unused imports

161f8b5

remove unused files

867436a

remove temporary hack

2424ac5

remove lock file

bdc72fe

multani marked this pull request as ready for review September 21, 2023 20:03

fix Annotated import

f3a358d

multani added 2 commits September 21, 2023 22:13

support Python 3.7

59619d6

Regenerate code with typing-extensions

fa5184f

multani mentioned this pull request Sep 21, 2023

python: run the Python samples tests with GitHub Actions #16643

Merged

5 tasks

wing328 added Client: Python Enhancement: General labels Sep 22, 2023

wing328 added this to the 7.1.0 milestone Sep 22, 2023

anis-campos mentioned this pull request Sep 22, 2023

[REQ] Overall python generators syntax and format improvements #16384

Closed

5 tasks

robertschweizer reviewed Sep 22, 2023

View reviewed changes

multani added 2 commits September 22, 2023 11:11

Merge remote-tracking branch 'origin/master' into python-typing-1

bf32b33

Fix setup.py

bdaf264

multani marked this pull request as draft September 22, 2023 10:33

Merge remote-tracking branch 'origin/master' into python-typing-1

2173266

Fix tests

a1c2973

wing328 reviewed Sep 27, 2023

View reviewed changes

multani added 5 commits September 27, 2023 09:34

split model imports from other modules imports

20faf8c

fix workflow

4d1d8dd

Comment the array unique items convertion, remove set translation

dd97b6b

Merge remote-tracking branch 'origin/master' into python-typing-1

4894c95

Replace alias usage

5b1b7a8

wing328 merged commit 04fa53b into OpenAPITools:master Sep 28, 2023

multani mentioned this pull request Sep 29, 2023

[PYTHON] generate code based on pydantic v2 #16685

Merged

5 tasks

multani deleted the python-typing-1 branch September 30, 2023 07:23

robertschweizer mentioned this pull request Oct 5, 2023

[PYTHON] update to pydantic v2 #16468

Closed

multani mentioned this pull request Oct 13, 2023

[python] Fix Circular imports on inherited discriminators. #16809

Closed

5 tasks

b0g3r reviewed Dec 12, 2023

View reviewed changes

multani mentioned this pull request Dec 31, 2023

python: simplify module imports #17507

Merged

5 tasks

multani mentioned this pull request Mar 2, 2024

Remove pydantic warnings when using model_dump as the default output is then overridden (#17939) #17941

Merged

5 tasks

kaxil mentioned this pull request May 19, 2025

Add Pydantic as dep for the Python Client apache/airflow#50765

Merged

Uh oh!

Conversation

multani commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

Uh oh!

multani commented Sep 21, 2023

Uh oh!

wing328 commented Sep 22, 2023

Uh oh!

robertschweizer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

multani commented Sep 22, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

b0g3r Dec 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

multani commented Sep 19, 2023 •

edited

Loading

b0g3r Dec 12, 2023 •

edited

Loading