Accept task_key as an argument in DatabricksNotebookOperator
#44960
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hardeybisey
force-pushed
the
accept_task_key_in_databricks_taskoperator
branch
2 times, most recently
from
55100c3 to
11ff19e
Compare
hardeybisey
force-pushed
the
accept_task_key_in_databricks_taskoperator
branch
from
11ff19e to
0862491
Compare
hardeybisey
force-pushed
the
accept_task_key_in_databricks_taskoperator
branch
from
0862491 to
676e00f
Compare
rawwar
reviewed
Dec 18, 2024
providers/src/airflow/providers/databricks/operators/databricks.py
Outdated
Show resolved
Hide resolved
rawwar
reviewed
Dec 18, 2024
providers/src/airflow/providers/databricks/operators/databricks.py
Outdated
Show resolved
Hide resolved
rawwar
reviewed
Dec 18, 2024
hardeybisey
commented
Dec 18, 2024
hardeybisey
force-pushed
the
accept_task_key_in_databricks_taskoperator
branch
2 times, most recently
from
7560470 to
3d10268
Compare
hardeybisey
force-pushed
the
accept_task_key_in_databricks_taskoperator
branch
from
3d10268 to
fe641e4
Compare
pankajkoti
approved these changes
Dec 26, 2024
Member
pankajkoti
left a comment
pankajkoti
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the great fix and your efforts in resolving this @hardeybisey
got686-yandex
pushed a commit
to got686-yandex/airflow
that referenced
this pull request
Jan 30, 2025
…e#44960) This PR introduces the ability for users to explicitly specify databricks_task_key as a parameter for the DatabricksNotebookOperator. If databricks_task_key is not provided, a default value is generated using the hash of the dag_id and task_id. Key Changes: Users can now define databricks_task_key explicitly. When not provided, the key defaults to a deterministic hash based on dag_id and task_id. Fixes: apache#41816 Fixes: apache#44250 related: apache#43106
2 tasks
Contributor
|
@hardeybisey @rawwar this MR is creating the issue when there we are adding databricks task dependencies: #47983 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the ability for users to explicitly specify
databricks_task_keyas a parameter for theDatabricksNotebookOperator. Ifdatabricks_task_keyis not provided, a default value is generated using the hash of thedag_idandtask_id.Key Changes:
Fixes: #41816
Fixes: #44250
related: #43106
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.