Skip to content

Conversation

@Dong-yeong0
Copy link
Contributor

@Dong-yeong0 Dong-yeong0 commented Mar 13, 2025

This PR fixes an issue(#47555) where Airflow incorrectly generates a PostgreSQL connection URI using postgres:// instead of postgresql://, which causes SQLAlchemy to fail.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg
Copy link

boring-cyborg bot commented Mar 13, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.
@Dong-yeong0 Dong-yeong0 changed the title Fix apache#47555: PostgreSQL URI scheme issue Fix apache#47555: PostgreSQL URI driver issue Mar 13, 2025
Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.
Copy link
Member

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Only the test needs to be added.

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't thnk this is needed/useful. Get_uri - returns "airflow" URI format as described in https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#uri-format-example, the #38831 was indeed a way how to get another uri (sqlalchemy URI) - which is pretty different than AIrfow's has different potential parameters and uses postgres as prefix exclusively.

If anything - maybe it's worth (and change this PR into it) to add comment that this is NOT sqlalchemy URI and that sqlalchemy_url should be used instead.

It also makes it very fishy to add postgres-specific code in airflow models anyway.

@Dong-yeong0
Copy link
Contributor Author

@potiuk My intention was to allow get_uri() in the Connection model to directly return a SQLAlchemy-compatible URI. However, I now see that my approach was incorrect.

If using sqlalchemy_url is the proper way, would it be better to update the get_uri() in models/connection.py docstring instead?

I can modify the PR accordingly.

@Dong-yeong0 Dong-yeong0 requested a review from potiuk March 15, 2025 15:26
@potiuk
Copy link
Member

potiuk commented Mar 15, 2025

@potiuk My intention was to allow get_uri() in the Connection model to directly return a SQLAlchemy-compatible URI. However, I now see that my approach was incorrect.

If using sqlalchemy_url is the proper way, would it be better to update the get_uri() in models/connection.py docstring instead?

I can modify the PR accordingly.

Yep. That's better approach. It did cause confusion before so you are not the first one. It would be great to have it better decribed in docstring but also maybe even in documentation if we find a good place ?

@Dong-yeong0
Copy link
Contributor Author

@potiuk My intention was to allow get_uri() in the Connection model to directly return a SQLAlchemy-compatible URI. However, I now see that my approach was incorrect.
If using sqlalchemy_url is the proper way, would it be better to update the get_uri() in models/connection.py docstring instead?
I can modify the PR accordingly.

Yep. That's better approach. It did cause confusion before so you are not the first one. It would be great to have it better decribed in docstring but also maybe even in documentation if we find a good place ?

I’ll update the get_uri() docstring to clearly describe that it does not return a SQLAlchemy-compatible URI, also i’ll check the documentation to find a good place to add this clarification.

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NICE

Copy link
Member

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Dong-yeong0, but the ruff static check need fix.

@Dong-yeong0 Dong-yeong0 requested a review from jason810496 March 20, 2025 06:25
@jason810496 jason810496 merged commit 79733e9 into apache:main Mar 20, 2025
2 checks passed
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 20, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
* Fix apache#47555: PostgreSQL URI scheme issue

* Fix: allow configuring PostgreSQL connection driver

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.

* Fix apache#47555: PostgreSQL URI scheme issue

* Fix: allow configuring PostgreSQL connection driver

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.

* Add the documentation for the 'designator' parameter in the PostgreSQL extras section

* Remove the code that constructs the URI based on connection type

* Update get_uri method docstring

* Clarify that get_uri() does not return a SQLAlchemy URI

* Fix rst-backticks

* Remove designer description

* Fix format code with ruff

---------

Co-authored-by: LIU ZHE YOU <68415893+jason810496@users.noreply.github.com>
shubham-pyc pushed a commit to shubham-pyc/airflow that referenced this pull request Mar 22, 2025
* Fix apache#47555: PostgreSQL URI scheme issue

* Fix: allow configuring PostgreSQL connection driver

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.

* Fix apache#47555: PostgreSQL URI scheme issue

* Fix: allow configuring PostgreSQL connection driver

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.

* Add the documentation for the 'designator' parameter in the PostgreSQL extras section

* Remove the code that constructs the URI based on connection type

* Update get_uri method docstring

* Clarify that get_uri() does not return a SQLAlchemy URI

* Fix rst-backticks

* Remove designer description

* Fix format code with ruff

---------

Co-authored-by: LIU ZHE YOU <68415893+jason810496@users.noreply.github.com>
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
* Fix apache#47555: PostgreSQL URI scheme issue

* Fix: allow configuring PostgreSQL connection driver

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.

* Fix apache#47555: PostgreSQL URI scheme issue

* Fix: allow configuring PostgreSQL connection driver

Added support for customizing the PostgreSQL connection driver (`postgres` vs `postgresql`)
via the `postgres_driver` option in the extra JSON field.

* Add the documentation for the 'designator' parameter in the PostgreSQL extras section

* Remove the code that constructs the URI based on connection type

* Update get_uri method docstring

* Clarify that get_uri() does not return a SQLAlchemy URI

* Fix rst-backticks

* Remove designer description

* Fix format code with ruff

---------

Co-authored-by: LIU ZHE YOU <68415893+jason810496@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants