Skip to content

Upgrade databricks provider dependency databricks-sql-connector to support version >= 3.0 #42626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rubanolha wants to merge 2 commits into from
Closed

Upgrade databricks provider dependency databricks-sql-connector to support version >= 3.0 #42626

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
25319a6
update databricks-sql-connector to 3.0.0 for databricks provider
rubanolha Oct 9, 2024
23a999d
fix: patches mypy no-redef error for duplicate Row definition
dcmshi Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion generated/provider_dependencies.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@
"aiohttp>=3.9.2, <4",
"apache-airflow-providers-common-sql>=1.10.0",
"apache-airflow>=2.8.0",
"databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0",
"databricks-sql-connector>=3.0.0",
"mergedeep>=1.3.4",
"pandas>=1.5.3,<2.2;python_version<\"3.9\"",
"pandas>=2.1.2,<2.2;python_version>=\"3.9\"",
Expand Down
5 changes: 1 addition & 4 deletions providers/src/airflow/providers/databricks/provider.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,7 @@ dependencies:
- apache-airflow>=2.8.0
- apache-airflow-providers-common-sql>=1.10.0
- requests>=2.27.0,<3
# The connector 2.9.0 released on Aug 10, 2023 has a bug that it does not properly declare urllib3 and
# it needs to be excluded. See https://github.com/databricks/databricks-sql-python/issues/190
# The 2.9.1 (to be released soon) already contains the fix
- databricks-sql-connector>=2.0.0, <3.0.0, !=2.9.0
- databricks-sql-connector>=3.0.0
- aiohttp>=3.9.2, <4
- mergedeep>=1.3.4
- pandas>=2.1.2,<2.2;python_version>="3.9"
Expand Down
4 changes: 3 additions & 1 deletion providers/tests/snowflake/operators/test_snowflake_sql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@
except ImportError:
# Row is used in the parametrize so it's parsed during collection and we need to have a viable
# replacement for the collection time when databricks is not installed (Python 3.12 for now)
def Row(*args, **kwargs):
def MockRow(*args, **kwargs):
return MagicMock()

Row = MockRow


from airflow.models.connection import Connection
from airflow.providers.common.compat.openlineage.facet import (
Expand Down