Change parameters conn_id into mongo_conn_id #28946
Conversation
It looks like all manuals speak about mongo_conn_id, but the code uses conn_id. Hence is ignoring the mongo_conn_id that is set and falls back to the default connection. Hope this helps.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
| hook_name = "MongoDB" | ||
|
|
||
| def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> None: | ||
| def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> None: |
There was a problem hiding this comment.
This add breaking changes in case if user provide arguments as keyword:
MongoHook(conn_id="awesome-conn-id")
There was a problem hiding this comment.
@Wesseldr Thanks for the contribution and your first PR! Definitely the docstring for the hook doesn't match what's actually required for use.
+1 on what @Taragolis mentioned.
We try to be very cognizant when thinking about backwards compat for users and try not to introduce breaking changes if avoidable. We really don't want users to upgrade a provider version and suddenly DAGs begin to break. In this situation, if the parameter name is changing, a DeprecationWarning should be raised that conn_id is deprecated and mongo_conn_id should be used. You can find a number of examples in other providers' hooks where a parameter is deprecated for reference on how to do this.
For the deprecation period, the hook should accept both conn_id and mongo_conn_id to maintain backwards compat and ultimately resolve to the self.mongo_conn_id value. Also the MongoSensor should ideally be updated to call this hook with keyword args.
The unit tests for the provider need to be updated accordingly too.
We're more than happy to help out if you are stuck. You can find us on Airflow Slack or we can communicate in this PR directly too.
| hook_name = "MongoDB" | ||
|
|
||
| def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> None: | ||
| def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> None: |
There was a problem hiding this comment.
@Wesseldr Thanks for the contribution and your first PR! Definitely the docstring for the hook doesn't match what's actually required for use.
+1 on what @Taragolis mentioned.
We try to be very cognizant when thinking about backwards compat for users and try not to introduce breaking changes if avoidable. We really don't want users to upgrade a provider version and suddenly DAGs begin to break. In this situation, if the parameter name is changing, a DeprecationWarning should be raised that conn_id is deprecated and mongo_conn_id should be used. You can find a number of examples in other providers' hooks where a parameter is deprecated for reference on how to do this.
For the deprecation period, the hook should accept both conn_id and mongo_conn_id to maintain backwards compat and ultimately resolve to the self.mongo_conn_id value. Also the MongoSensor should ideally be updated to call this hook with keyword args.
The unit tests for the provider need to be updated accordingly too.
We're more than happy to help out if you are stuck. You can find us on Airflow Slack or we can communicate in this PR directly too.
|
@josh-fell @Taragolis Totally agree with avoiding code breaking changes.. There was no wisdom from my side here.. I just made the assumption the manual was written from the design and since that named it There for I posted a pull request for a name change as I started to believe that was the norm.. Then either the manual is updated to represent Let me know what you pick, obviously dropping my PR is no problem if the current parameters are consistent with other providers and how the styleguide recommends these :-) Thank you for so quickly picking up on this and providing feedback&thoughts. Resources The example I followed that used the same Parameter |
Unfortunetly this is not Airflow documentation, and we can't change it easily. I also agree with the fact that there is some inconsistency between Hook and Operators/Sensors. But there is no problem to resolve it, see some simple snippet: import warnings
from airflow.hooks.base import BaseHook
from airflow.utils.types import NOTSET
class MongoHook(BaseHook):
conn_id = "mongo_conn_id"
default_conn_name = "mongo_default"
conn_type = "mongo"
hook_name = "MongoDB"
def __init__(self, mongo_conn_id: str = NOTSET, *args, **kwargs) -> None:
super().__init__()
conn_id = kwargs.pop("conn_id", NOTSET)
if conn_id is not NOTSET:
warnings.warn(
"Parameter `conn_id` is deprecated and will be removed "
"in a future releases. Please use `mongo_conn_id` instead.",
DeprecationWarning,
stacklevel=2,
)
if mongo_conn_id is NOTSET:
mongo_conn_id = conn_id
else:
raise ValueError("You cannot provide both `mongo_conn_id` and `conn_id`.")
elif mongo_conn_id is NOTSET:
mongo_conn_id = self.default_conn_name
self.mongo_conn_id = mongo_conn_id
...
@property
def conn_id(self):
warnings.warn(
"Property `conn_id` is deprecated and will be removed "
"in a future releases. Please use `mongo_conn_id` instead.",
DeprecationWarning,
stacklevel=2,
)
return self.mongo_conn_id
hook1 = MongoHook("conn_id_as_attr")
assert hook1.conn_id == "conn_id_as_attr"
assert hook1.mongo_conn_id == "conn_id_as_attr"
hook2 = MongoHook(conn_id="conn_id_as_legacy_kwarg")
assert hook2.mongo_conn_id == "conn_id_as_legacy_kwarg"
hook3 = MongoHook(mongo_conn_id="conn_id_as_new_kwarg")
assert hook3.mongo_conn_id == "conn_id_as_new_kwarg"
try:
MongoHook(mongo_conn_id="conn_id_as_new_kwarg", conn_id="conn_id_as_legacy_kwarg")
except ValueError:
pass
else:
raise ValueError("Unsupported")
hook_with_default_conn_id = MongoHook()
assert hook_with_default_conn_id.mongo_conn_id == MongoHook.default_conn_name |
|
Yep. Changin it in back-compatible way as @Taragolis explained is a good idea. |
|
@Wesseldr, do you have a plan to continue work on this PR? |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
4 participants
It looks like all manuals speak about mongo_conn_id, but the code uses conn_id. Hence is ignoring the mongo_conn_id that is set and falls back to the default connection. Hope this helps.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.