Fix named parameters templating in Databricks operators #40864
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
Those failing tests should be DB tests. |
Member
|
I have created a PR #40869 for marking them as db tests. Hope it passes and then we can rebase it here once CI passes and we merge it after review. |
Member
|
@boraberke could you please rebase as the PR to fix mark the required tests as db tests is merged now. |
potiuk
force-pushed
the
fix/databricks-templated-params
branch
from
9b4fe65 to
3c6c175
Compare
Member
|
I rebased it (@pankajkoti -> the "drop-down" on "update branch" has rebase option - and in most cases maintainers have the permission to push to the branch of the fork of Airflow :). |
Member
Thank you @potiuk :) Yes, I was just unsure if authors feel comfortable/uncomfortable if I do a rebase and whether it's an accepted policy. Glad to know it can be done, thank you :) |
Contributor
Author
Thank you for the feedback @potiuk and the PR @pankajkoti, I wasn't really sure what to do with the failing tests! |
Member
I usually rebase and write a comment about it. Worst thing that can happen, the author will have to push their changes with |
potiuk
approved these changes
Jul 18, 2024
Member
|
LGTM. @pankajkoti ? |
pankajkoti
approved these changes
Jul 18, 2024
Member
There was a problem hiding this comment.
LGTM too.
@josh-fell I hope the issue you reported previously still is resolved together with this and the earlier PR #40471 from @boraberke .
@vatsrahul1001 I guess would be good if we could test this.
I will go ahead with merging the PR. But in case, you find something not working, please report here and we can try to resolve it asap.
Contributor
|
Verified named parameters issue is resolved now, however, I noticed Issue still persists. @josh-fell can you verify as well? |
Member
|
@boraberke would you have time to confirm if #35433 has resurfaced after this PR? We might have to reopen that issue in that case. |
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
This PR fixes the many named parameters that was templated and was broken with apache#40471. The following operators are affected: DatabricksCreateJobsOperator DatabricksSubmitRunOperator DatabricksRunNowOperator closes: apache#40788
pankajkoti
added a commit
that referenced
this pull request
Jul 27, 2024
* Revert "Fix named parameters templating in Databricks operators (#40864)" This reverts commit cfe1d53. * Revert "Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471)" This reverts commit 4fb2140. This reverts PR #40864 and PR #40471. Previously, PR #40471 was contributed to address issue #35433. However, that contribution gave rise to another issue #40788. Next #40788 was being attempted to be resolved in PR #40864. However, with the second PR, it appears that the previous old issue #35433 has [resurfaced](#40864 (comment)). So, at the moment, the case is that we have 2 PRs on top of the existing implementation eventually having nil effect and the previous issues persists. I believe it is better to revert those 2 PRs, reopen the earlier issue #35433 and peacefully address it by taking the needed time.
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Nov 9, 2024
* Revert "Fix named parameters templating in Databricks operators (#40864)" This reverts commit cfe1d53ed041ea903292e3789e1a5238db5b5031. * Revert "Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471)" This reverts commit 4fb2140f393b6332903fb833151c2ce8a9c66fe2. This reverts PR #40864 and PR #40471. Previously, PR apache/airflow#40471 was contributed to address issue apache/airflow#35433. However, that contribution gave rise to another issue apache/airflow#40788. Next apache/airflow#40788 was being attempted to be resolved in PR #40864. However, with the second PR, it appears that the previous old issue #35433 has [resurfaced](apache/airflow#40864 (comment)). So, at the moment, the case is that we have 2 PRs on top of the existing implementation eventually having nil effect and the previous issues persists. I believe it is better to revert those 2 PRs, reopen the earlier issue #35433 and peacefully address it by taking the needed time. GitOrigin-RevId: 4535e08b862e2b7ff4f2a76de7124983d4efe9db
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
May 6, 2025
* Revert "Fix named parameters templating in Databricks operators (#40864)" This reverts commit cfe1d53ed041ea903292e3789e1a5238db5b5031. * Revert "Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471)" This reverts commit 4fb2140f393b6332903fb833151c2ce8a9c66fe2. This reverts PR #40864 and PR #40471. Previously, PR apache/airflow#40471 was contributed to address issue apache/airflow#35433. However, that contribution gave rise to another issue apache/airflow#40788. Next apache/airflow#40788 was being attempted to be resolved in PR #40864. However, with the second PR, it appears that the previous old issue #35433 has [resurfaced](apache/airflow#40864 (comment)). So, at the moment, the case is that we have 2 PRs on top of the existing implementation eventually having nil effect and the previous issues persists. I believe it is better to revert those 2 PRs, reopen the earlier issue #35433 and peacefully address it by taking the needed time. GitOrigin-RevId: 4535e08b862e2b7ff4f2a76de7124983d4efe9db
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
May 27, 2025
* Revert "Fix named parameters templating in Databricks operators (#40864)" This reverts commit cfe1d53ed041ea903292e3789e1a5238db5b5031. * Revert "Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471)" This reverts commit 4fb2140f393b6332903fb833151c2ce8a9c66fe2. This reverts PR #40864 and PR #40471. Previously, PR apache/airflow#40471 was contributed to address issue apache/airflow#35433. However, that contribution gave rise to another issue apache/airflow#40788. Next apache/airflow#40788 was being attempted to be resolved in PR #40864. However, with the second PR, it appears that the previous old issue #35433 has [resurfaced](apache/airflow#40864 (comment)). So, at the moment, the case is that we have 2 PRs on top of the existing implementation eventually having nil effect and the previous issues persists. I believe it is better to revert those 2 PRs, reopen the earlier issue #35433 and peacefully address it by taking the needed time. GitOrigin-RevId: 4535e08b862e2b7ff4f2a76de7124983d4efe9db
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Sep 22, 2025
* Revert "Fix named parameters templating in Databricks operators (#40864)" This reverts commit cfe1d53ed041ea903292e3789e1a5238db5b5031. * Revert "Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471)" This reverts commit 4fb2140f393b6332903fb833151c2ce8a9c66fe2. This reverts PR #40864 and PR #40471. Previously, PR apache/airflow#40471 was contributed to address issue apache/airflow#35433. However, that contribution gave rise to another issue apache/airflow#40788. Next apache/airflow#40788 was being attempted to be resolved in PR #40864. However, with the second PR, it appears that the previous old issue #35433 has [resurfaced](apache/airflow#40864 (comment)). So, at the moment, the case is that we have 2 PRs on top of the existing implementation eventually having nil effect and the previous issues persists. I believe it is better to revert those 2 PRs, reopen the earlier issue #35433 and peacefully address it by taking the needed time. GitOrigin-RevId: 4535e08b862e2b7ff4f2a76de7124983d4efe9db
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 20, 2025
* Revert "Fix named parameters templating in Databricks operators (#40864)" This reverts commit cfe1d53ed041ea903292e3789e1a5238db5b5031. * Revert "Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471)" This reverts commit 4fb2140f393b6332903fb833151c2ce8a9c66fe2. This reverts PR #40864 and PR #40471. Previously, PR apache/airflow#40471 was contributed to address issue apache/airflow#35433. However, that contribution gave rise to another issue apache/airflow#40788. Next apache/airflow#40788 was being attempted to be resolved in PR #40864. However, with the second PR, it appears that the previous old issue #35433 has [resurfaced](apache/airflow#40864 (comment)). So, at the moment, the case is that we have 2 PRs on top of the existing implementation eventually having nil effect and the previous issues persists. I believe it is better to revert those 2 PRs, reopen the earlier issue #35433 and peacefully address it by taking the needed time. GitOrigin-RevId: 4535e08b862e2b7ff4f2a76de7124983d4efe9db
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the many named parameters that was templated and was broken with #40471.
The following operators are affected:
closes: #40788
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.