-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Fix random deadlocks in MSSQL database #18362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
cc: @aneesh-joseph |
29cdaaf to
e20fc4b
Compare
mik-laj
reviewed
Sep 19, 2021
mik-laj
reviewed
Sep 19, 2021
mik-laj
reviewed
Sep 19, 2021
d72c78a to
00ef7fe
Compare
mik-laj
approved these changes
Sep 19, 2021
mik-laj
reviewed
Sep 19, 2021
edf3c5a to
1eed784
Compare
mik-laj
reviewed
Sep 19, 2021
b788061 to
3b2656f
Compare
SamWheating
reviewed
Sep 20, 2021
df806d3 to
f3d2da7
Compare
Default isolation level for the MSSQL database is READ_COMMITTED but its implementation in MSSQL might cause random deadlocks for select queries - because by default MSSQL uses shared locks for READ_COMMITTED transaction isolation that might cause deadlocks with the UPDATE statements which create exclusive locks. This change switches the MSSQL DB used during tests to have READ_COMMITTED_SNAPSHOTS enabled and it adds check at the Airflow startup to make sure that if the MSSQL database is used, the READ_COMMITTED_SNAPSHOTS is set and documentation is added to describe it. It also allows to revert removal of 2017 version of mssql which was done in the faith that removal of 2017 will remove the flaky random test failures. Also Fixes: apache#17018
f3d2da7 to
bd2d435
Compare
Member
Author
|
@aneesh-joseph -> if you have setup MSSQL before, setting the "READ_COMMITTED_SNAPSHOT" database option is CRUCIAL to avoid deadlocks on read. It is apparently set on by default on Azure databases, but it is disabled by default on regular mssql databases. in 2.2.0 we are checking if the option is enabled for the airflow database and fail if not. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area:dev-tools
full tests needed
We need to run full set of tests for this PR to merge
kind:documentation
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Default isolation level for the MSSQL database is READ_COMMITTED
but its implementation in MSSQL might cause random deadlocks for
select queries - because by default MSSQL uses shared locks
for READ_COMMITTED transaction isolation that might cause
deadlocks with the UPDATE statements which create exclusive
locks.
This change switches the MSSQL DB used during tests to have
READ_COMMITTED_SNAPSHOTS enabled and it adds check at the Airflow
startup to make sure that if the MSSQL database is used, the
READ_COMMITTED_SNAPSHOTS is set and documentation is added to
describe it.
It also allows to revert removal of 2017 version of mssql which
was done in the faith that removal of 2017 will remove the
flaky random test failures.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.