AIP-69: Add Executor to Edge Provider #42048
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jscheffl
requested review from
ashb,
jedcunningham and
potiuk
as code owners
jscheffl
changed the title
jscheffl
force-pushed
the
feature/aip-69-remote-executor-provider-executor
branch
3 times, most recently
from
45b7e33 to
dc70939
Compare
jscheffl
force-pushed
the
feature/aip-69-remote-executor-provider-executor
branch
from
dc70939 to
57775d3
Compare
o-nikolas
reviewed
Sep 16, 2024
jscheffl
force-pushed
the
feature/aip-69-remote-executor-provider-executor
branch
3 times, most recently
from
0fd2bbf to
dad57cd
Compare
o-nikolas
reviewed
Sep 24, 2024
jscheffl
force-pushed
the
feature/aip-69-remote-executor-provider-executor
branch
3 times, most recently
from
27da6c0 to
7338f09
Compare
jscheffl
force-pushed
the
feature/aip-69-remote-executor-provider-executor
branch
from
7338f09 to
7eb06d6
Compare
ashb
reviewed
Oct 17, 2024
providers/src/airflow/providers/edge/executors/edge_executor.py
Outdated
Show resolved
Hide resolved
ashb
approved these changes
Oct 17, 2024
Member
ashb
left a comment
ashb
left a comment
There was a problem hiding this comment.
Pre-emptively approving, but please either address https://github.com/apache/airflow/pull/42048/files#r1805240595 or tell me why it's not a problem.
(cherry picked from commit 4e9c2262f567a2511d02d4acd43b821fa0df45d2)
jscheffl
force-pushed
the
feature/aip-69-remote-executor-provider-executor
branch
from
7eb06d6 to
87a4188
Compare
Contributor
Author
Can confirm both: Addressed it and it is not a problem! |
harjeevanmaan
pushed a commit
to harjeevanmaan/airflow
that referenced
this pull request
Oct 23, 2024
* Add Executor to Edge Provider * Review feeedback + small adjustments from Niko * Add explicit note about non-performance in current state * Fix adoption of tasks when restarting scheduler (cherry picked from commit 4e9c2262f567a2511d02d4acd43b821fa0df45d2) * Optimize sync call for MVP executor * Adjust new folder structure from PR 42505 * Review feedback: removed cleanup_stuck_queued_tasks() and added notes about performance tests
PaulKobow7536
pushed a commit
to PaulKobow7536/airflow
that referenced
this pull request
Oct 24, 2024
* Add Executor to Edge Provider * Review feeedback + small adjustments from Niko * Add explicit note about non-performance in current state * Fix adoption of tasks when restarting scheduler (cherry picked from commit 4e9c2262f567a2511d02d4acd43b821fa0df45d2) * Optimize sync call for MVP executor * Adjust new folder structure from PR 42505 * Review feedback: removed cleanup_stuck_queued_tasks() and added notes about performance tests
ellisms
pushed a commit
to ellisms/airflow
that referenced
this pull request
Nov 13, 2024
* Add Executor to Edge Provider * Review feeedback + small adjustments from Niko * Add explicit note about non-performance in current state * Fix adoption of tasks when restarting scheduler (cherry picked from commit 4e9c2262f567a2511d02d4acd43b821fa0df45d2) * Optimize sync call for MVP executor * Adjust new folder structure from PR 42505 * Review feedback: removed cleanup_stuck_queued_tasks() and added notes about performance tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a split of #41729 and builds on-top of #42046 and #42047 to make review easier.
It only contains the new Edge Executor functions needed of the edge provider package. The first commit is from the other base PRs and if this is merged will go-away on rebase. If you want to see Executor implementation only, please take a look to the second commit only,