Skip to content

Add number of queries guard in public dag runs list endpoints #57450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pierrejeambrun merged 6 commits into from
Oct 31, 2025
Merged

Add number of queries guard in public dag runs list endpoints #57450

pierrejeambrun merged 6 commits into from
Oct 31, 2025

Conversation

@pierrejeambrun
Copy link
Member

@pierrejeambrun pierrejeambrun commented Oct 28, 2025

Fixes: #57432.

Multiple endpoints fixed and N+1 problem solved.

@pierrejeambrun pierrejeambrun added this to the Airflow 3.1.2 milestone Oct 28, 2025
@pierrejeambrun pierrejeambrun self-assigned this Oct 28, 2025
@pierrejeambrun pierrejeambrun added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Oct 28, 2025
@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Oct 28, 2025
@pierrejeambrun pierrejeambrun force-pushed the add-number-of-queries-guard-for-public-dag-runs branch from 2150a65 to 4fcbfa5 Compare October 28, 2025 17:21
@tirkarthi
Copy link
Contributor

tirkarthi commented Oct 28, 2025

Thanks @pierrejeambrun , any timings improvement with the changes?

@pierrejeambrun
Copy link
Member Author

pierrejeambrun commented Oct 29, 2025 •
edited
Loading

Thanks @pierrejeambrun , any timings improvement with the changes?

The number of queries went from 15 to 7, making sure that it won't grow with the number of records in the db, so I would say that this should solves the problem.

I don't have a big database to benchmark against, but if you have one it would be cool if you could test this change in your setup and confirm that this solves the problem.

@pierrejeambrun
Copy link
Member Author

pierrejeambrun commented Oct 29, 2025 •
edited
Loading

On main with limit 5000:
Screenshot 2025-10-29 at 12 23 24

On this branch with limit 5000:

Screenshot 2025-10-29 at 12 25 18

For only the first 50 items which is the default pagination size, we are around 35-50ms now, against 100 before. (This is with local testing, where db <=> server are on the same host).

@tirkarthi
Copy link
Contributor

tirkarthi commented Oct 29, 2025

Thanks @pierrejeambrun , I will try to apply it in our environment by patching it or will test the timings once Airflow 3.1.2 is released.

@pierrejeambrun pierrejeambrun force-pushed the add-number-of-queries-guard-for-public-dag-runs branch from 4dc2138 to d07b2fd Compare October 29, 2025 13:27
@pierrejeambrun pierrejeambrun requested a review from kaxil October 30, 2025 09:22
This was referenced Oct 30, 2025
Merged
Merged
kaxil
kaxil reviewed Oct 30, 2025
View reviewed changes
kaxil
kaxil reviewed Oct 31, 2025
View reviewed changes
@pierrejeambrun pierrejeambrun force-pushed the add-number-of-queries-guard-for-public-dag-runs branch from 014c191 to 549b1c9 Compare October 31, 2025 13:55
@pierrejeambrun pierrejeambrun force-pushed the add-number-of-queries-guard-for-public-dag-runs branch from 1988172 to 91bcc9a Compare October 31, 2025 16:17
@kaxil kaxil modified the milestones: Airflow 3.1.2, Airflow 3.1.3 Oct 31, 2025
@pierrejeambrun pierrejeambrun merged commit fb9dea5 into apache:main Oct 31, 2025
115 checks passed
@pierrejeambrun pierrejeambrun deleted the add-number-of-queries-guard-for-public-dag-runs branch October 31, 2025 18:25
@github-actions
Copy link

github-actions bot commented Oct 31, 2025

Backport failed to create: v3-1-test. View the failure log Run details

Status Branch Result
❌ v3-1-test Commit Link

You can attempt to backport this manually by running:

cherry_picker fb9dea5 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Nov 3, 2025
@pierrejeambrun
Add number of queries guard in public dag runs list endpoints (apache…
4fc84ca 
…#57450)

* WIP to revert

* Fix batch endpoint

* Fix CI

* Address review comments

* Fix CI

* Address PR comments

(cherry picked from commit fb9dea5)
@pierrejeambrun
Copy link
Member Author

pierrejeambrun commented Nov 3, 2025

Manual backport #57736

pierrejeambrun added a commit that referenced this pull request Nov 3, 2025
@pierrejeambrun
Add number of queries guard in public dag runs list endpoints (#57450) (
d53a00e 
#57736)

* WIP to revert

* Fix batch endpoint

* Fix CI

* Address review comments

* Fix CI

* Address PR comments

(cherry picked from commit fb9dea5)
ephraimbuddy pushed a commit that referenced this pull request Nov 3, 2025
@pierrejeambrun @ephraimbuddy
Add number of queries guard in public dag runs list endpoints (#57450) (
ae9ba12 
#57736)

* WIP to revert

* Fix batch endpoint

* Fix CI

* Address review comments

* Fix CI

* Address PR comments

(cherry picked from commit fb9dea5)
ephraimbuddy pushed a commit that referenced this pull request Nov 4, 2025
@pierrejeambrun @ephraimbuddy
Add number of queries guard in public dag runs list endpoints (#57450) (
20544a4 
#57736)

* WIP to revert

* Fix batch endpoint

* Fix CI

* Address review comments

* Fix CI

* Address PR comments

(cherry picked from commit fb9dea5)
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Nov 10, 2025
Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
@pierrejeambrun
Add number of queries guard in public dag runs list endpoints (apache…
f58e9ad 
…#57450)

* WIP to revert

* Fix batch endpoint

* Fix CI

* Address review comments

* Fix CI

* Address PR comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ephraimbuddy ephraimbuddy ephraimbuddy left review comments

@dstandish dstandish dstandish left review comments

@kaxil kaxil kaxil approved these changes

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@jscheffl jscheffl Awaiting requested review from jscheffl

@shubhamraj-git shubhamraj-git Awaiting requested review from shubhamraj-git shubhamraj-git is a code owner

@rawwar rawwar Awaiting requested review from rawwar rawwar is a code owner

@jason810496 jason810496 Awaiting requested review from jason810496 jason810496 is a code owner

@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@guan404ming guan404ming Awaiting requested review from guan404ming guan404ming is a code owner

Assignees

@pierrejeambrun pierrejeambrun

Labels

area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch type:bug-fix Changelog: Bug Fixes

Projects

None yet

Milestone

Airflow 3.1.3

Development

Successfully merging this pull request may close these issues.

n+1 queries in public dag runs endpoint

5 participants

@pierrejeambrun @tirkarthi @ephraimbuddy @kaxil @dstandish