Redshift task in running state but SQL execution does not start when OpenLineage is enabled #39969
Description
Apache Airflow Provider(s)
openlineage
Versions of Apache Airflow Providers
apache-airflow-providers-openlineage==1.8.0
apache-airflow-providers-common-sql==1.12.0
apache-airflow-providers-amazon==8.23.0
Apache Airflow version
2.9
Operating System
GNU Linux
Deployment
Astronomer
Deployment details
No response
What happened
A SQLExecuteQueryOperator in an environment with apache-airflow-providers-common-sql==1.12.0 ran into a problem where the SQL execution never starts but the task is running. This issue only presented itself in a specific task in the DAG and not other tasks executing different SQLs.
Once we disabled OpenLineage, the task instance ran successfully. We enabled it again to see if it would fail, and it did.
This issue was observed both when using Snowflake and Redfshift.
It seems #39113 #39890 solves the issue for Snowflake, but we're still missing some work to avoid the issue happening for Redshift.
What you think should happen instead
Enabling OpenLineage should not affect task execution, even if it parsing, processing and emitting OL events fail.
How to reproduce
.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct