Skip to content

Conversation

@ephraimbuddy
Copy link
Contributor

Updating a taskinstance's map_index etc should cascade to the task_map table
else we would get foreign key violation error when updating a task instance from unmapped
to mapped.

closes: #31431

@uranusjr
Copy link
Member

Would this be performant if the tables contain a lot of rows? I seem to recall creating a new index has performance issues.

Updating a taskinstance's map_index etc should cascade to the task_map table
else we would get foreign key violation error when updating a task instance from unmapped
to mapped.
@ephraimbuddy ephraimbuddy force-pushed the add-cascade-taskmap-fk branch from db35322 to 12348af Compare May 22, 2023 08:50
@ephraimbuddy
Copy link
Contributor Author

Would this be performant if the tables contain a lot of rows? I seem to recall creating a new index has performance issues.

Do you mean a performance issue when upgrading? Like taking a lot of time to run the upgrade? I don't think there would be a performance issue but it's worth testing

@uranusjr
Copy link
Member

Yes when upgrading.

Copy link
Member

@jedcunningham jedcunningham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(assuming no catastrophic performance issues are found)

@ephraimbuddy
Copy link
Contributor Author

ephraimbuddy commented May 24, 2023

I have tested the upgrade and downgrade with 17million rows in taskinstance and dagrun tables, 15million rows in task_map table.
The test was done with airflow 2.6.1.
Upgrade: airflow db upgrade 1.26s user 0.20s system 10% cpu 14.438 total
Downgrade: airflow db downgrade --to-version 2.6.1 0.78s user 0.17s system 5% cpu 15.969 total

So a total of 1.26s for upgrade and 0.78s for downgrade

Below is the full log of the test:
Screenshot 2023-05-24 at 11 44 09

@ephraimbuddy ephraimbuddy merged commit f6bb474 into apache:main May 24, 2023
@ephraimbuddy ephraimbuddy deleted the add-cascade-taskmap-fk branch May 24, 2023 10:54
@eladkal eladkal added this to the Airflow 2.6.2 milestone Jun 8, 2023
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Jun 8, 2023
@eladkal eladkal modified the milestones: Airflow 2.6.2, Airlfow 2.6.3 Jun 8, 2023
pierrejeambrun pushed a commit that referenced this pull request Jun 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clearing a task flow function executed earlier with task changed to mapped task crashes scheduler

5 participants