Skip to content

Add conditional logic for dataset triggering #37016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dstandish merged 23 commits into from
Feb 21, 2024
Merged

Add conditional logic for dataset triggering #37016

dstandish merged 23 commits into from
Feb 21, 2024

Conversation

@dstandish
Copy link
Contributor

@dstandish dstandish commented Jan 25, 2024 •
edited
Loading

Add conditional logic for dataset-triggered dags.

This means we can schedule based on dataset1 OR dataset1.

This PR only implements the underlying classes, DatasetAny and DatasetAll. In a followup PR we will add more convenient syntax for this, specifically the | and & symbols, e.g. (dataset1 | dataset2) & dataset3.

@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch from cd01fde to 859a0ac Compare January 29, 2024 23:20
@sunank200 sunank200 mentioned this pull request Jan 30, 2024
Merged
1 task
@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch 2 times, most recently from 0e1e5d8 to ecccd6f Compare February 1, 2024 16:21
@dstandish
Copy link
Contributor Author

dstandish commented Feb 2, 2024 •
edited
Loading

@sunank200

  1. fix test on main PR
  2. look at the tests on the PR and determine which should be added or removed
  3. performance
    • why 10 minutes in main?
    • why slower with this PR?
    • what's going on?

update: the performance concern appears to be invalid, due to randomness in task execution issues / unrelated scheduler restarts

@sunank200 sunank200 force-pushed the add-conditional-logic-for-dataset-triggering branch 5 times, most recently from 83087fc to 3cffcf4 Compare February 7, 2024 06:24
@kaxil kaxil added this to the Airflow 2.9.0 milestone Feb 7, 2024
@sunank200 sunank200 force-pushed the add-conditional-logic-for-dataset-triggering branch 3 times, most recently from 7514d44 to ab979da Compare February 8, 2024 07:16
@sunank200
Copy link
Collaborator

sunank200 commented Feb 8, 2024

@sunank200

  1. fix test on main PR

  2. look at the tests on the PR and determine which should be added or removed

  3. performance

    • why 10 minutes in main?
    • why slower with this PR?
    • what's going on?

Documentation changes are done here

@sunank200 sunank200 mentioned this pull request Feb 12, 2024
Closed
2 tasks
@dstandish dstandish changed the title DRAFT Add conditional logic for dataset triggering Add conditional logic for dataset triggering Feb 13, 2024
@dstandish dstandish marked this pull request as ready for review February 13, 2024 19:32
uranusjr
uranusjr reviewed Feb 20, 2024
View reviewed changes
uranusjr
uranusjr reviewed Feb 20, 2024
View reviewed changes
@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch from 870452c to 9753973 Compare February 20, 2024 22:14
sunank200 and others added 13 commits February 21, 2024 09:47
@sunank200 @dstandish
Fix the tests for dataset models
a511556
@sunank200 @dstandish
Remove clear_dataset fixture
d86cf1a
@sunank200 @dstandish
fix the test for dataset serialization
8d44b4c
@sunank200 @dstandish
Add clear dataset fixture
45d6969
@sunank200 @dstandish
Add example DAGs
c5bee76
@sunank200 @dstandish
Add strict in class serialization and remove example DAGs
341a91a
@dstandish
Query "existing dag runs" one at a time
fdb6bc6
@dstandish
Revert "Query "existing dag runs" one at a time"
6ab9d8f 
This reverts commit 2a2a195244b4e40882cd0f758b356d21167a5299.
@dstandish
enable serialization check of dataset_triggers attr
df77b93
@dstandish @jedcunningham
Update airflow/models/dataset.py
d068f57 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish @jedcunningham
Update airflow/models/dataset.py
9d204ad 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish @jedcunningham
Update airflow/models/dataset.py
9b5bac7 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish
remove unnecessary db test mark
71f6eba
@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch from 0137de6 to 71f6eba Compare February 21, 2024 17:47
dstandish and others added 5 commits February 21, 2024 09:49
@dstandish
remove unnec add
4721b6a
@dstandish @Lee-W @jedcunningham
Apply suggestions from code review
4df31d4 
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish
better comment
10909dc
@dstandish
remove unnec fixture
28db305
@dstandish
relocate fixture
01ccbc5
@dstandish
Copy link
Contributor Author

dstandish commented Feb 21, 2024

Do we plan to update object/next_run_datasets so the UI can show all of this logic?

Same for /next_run_datasets_summary, We should make sure the ready+total counts are still accurate. It might need to change from total to a min and max. so we can say something like 1 of 2-3 datasets updated

Created issue on our board @bbovenzi

@dstandish dstandish merged commit f971232 into apache:main Feb 21, 2024
@dstandish dstandish deleted the add-conditional-logic-for-dataset-triggering branch February 21, 2024 19:24
This was referenced Feb 22, 2024
Merged
Closed
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@uranusjr uranusjr uranusjr left review comments

@Lee-W Lee-W Lee-W left review comments

@jedcunningham jedcunningham jedcunningham approved these changes

@kaxil kaxil Awaiting requested review from kaxil

@ashb ashb Awaiting requested review from ashb

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

+1 more reviewer

@sunank200 sunank200 sunank200 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providers area:serialization provider:common-sql type:new-feature Changelog: New Features

Projects

None yet

Milestone

Airflow 2.9.0

Development

Successfully merging this pull request may close these issues.

8 participants

@dstandish @sunank200 @bbovenzi @uranusjr @Lee-W @jedcunningham @ephraimbuddy @kaxil