Move databricks provider to new structure

[josix]

related: #46045

• fix unit tests

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

conflicts again (but I hope this time things will be green).

[josix]

Thanks! Let me fix this first, I guess the patch for move_providers.py might affect other providers.

[potiuk]

Thanks! Let me fix this first, I guess the patch for move_providers.py might affect other providers.

Cool :)

[o-nikolas]

I think you just barely missed a fix in main when you rebased an hour ago 😢

[josix]

Hmmm... I rebased after #46291 merged, and it indeed fixed the issues that caused the previous failed tests in cncf.kubernetes and microsoft.azure related to time.sleep. However, it looks like this is a different issue, and all the failed tests are using caplog (not related 🤔) .

[josix]

I'll continue fixing the flaky caplog-related unit tests once the lazy consensus is reached.

[potiuk]

Rebased - and the consensus is reached so ..... you can fix it :)

[potiuk]

Some errors to look at :( . Happy to help if needed.

[josix]

Yeah, let me circle back to this. Sorry for being busy the past two days, but thankfully, I just finished it and can now focus on this issue.

[potiuk]

Yeah, let me circle back to this. Sorry for being busy the past two days, but thankfully, I just finished it and can now focus on this issue.

No worries at all :), It's open -source. We do stuff when we can !

[potiuk]

The wasb tests should be "fine" after this is merged #46539 -> then you can rebase and 🤞 we should get this one green.

[potiuk]

OK. you can rebase it now :)

[josix]

Thanks, @potiuk! It turns out that the issue is caused from caplog, just as my initially assumed. However, I still can't figure out why it is related to building connection, given that the error shows airflow.exceptions.AirflowNotFoundException: The conn_id `` isn't defined. How does caplog affect the Hooks 🤔

Disposing DB connection pool (PID 75)
Unable to retrieve connection from secrets backend (MetastoreBackend). Checking subsequent secrets backend.
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/connection.py", line 463, in get_connection_from_secrets
    conn = secrets_backend.get_connection(conn_id=conn_id)
  File "/opt/airflow/airflow/utils/session.py", line 100, in wrapper
    with create_session() as session:
  File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/opt/airflow/airflow/utils/session.py", line 39, in create_session
    raise RuntimeError("Session must be set before!")
RuntimeError: Session must be set before!
Exception when trying to check remote location: "The conn_id `` isn't defined"
Unable to retrieve connection from secrets backend (MetastoreBackend). Checking subsequent secrets backend.
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/connection.py", line 463, in get_connection_from_secrets
    conn = secrets_backend.get_connection(conn_id=conn_id)
  File "/opt/airflow/airflow/utils/session.py", line 100, in wrapper
    with create_session() as session:
  File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/opt/airflow/airflow/utils/session.py", line 39, in create_session
    raise RuntimeError("Session must be set before!")
RuntimeError: Session must be set before!
Could not write logs to wasb://container/remote/log/location/1.log
Traceback (most recent call last):
  File "/opt/airflow/providers/src/airflow/providers/microsoft/azure/log/wasb_task_handler.py", line 203, in wasb_write
    self.hook.load_string(log, self.wasb_container, remote_log_location, overwrite=True)
  File "/opt/airflow/providers/src/airflow/providers/microsoft/azure/hooks/wasb.py", line 361, in load_string
    self.upload(
  File "/opt/airflow/providers/src/airflow/providers/microsoft/azure/hooks/wasb.py", line 418, in upload
    blob_client = self._get_blob_client(container_name, blob_name)
  File "/opt/airflow/providers/src/airflow/providers/microsoft/azure/hooks/wasb.py", line 233, in _get_blob_client
    return self.blob_service_client.get_blob_client(container=container_name, blob=blob_name)
  File "/usr/local/lib/python3.9/functools.py", line 993, in __get__
    val = self.func(instance)
  File "/opt/airflow/providers/src/airflow/providers/microsoft/azure/hooks/wasb.py", line 157, in blob_service_client
    return self.get_conn()
  File "/opt/airflow/providers/src/airflow/providers/microsoft/azure/hooks/wasb.py", line 161, in get_conn
    conn = self.get_connection(self.conn_id)
  File "/opt/airflow/airflow/hooks/base.py", line 64, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
  File "/opt/airflow/airflow/models/connection.py", line 474, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `` isn't defined

[potiuk]

The conn_id `` isn't defined. How does caplog affect the Hooks 🤔

Beats me... No idea other than there are side effects from other tests. I know removing it and replacing with direct mocking helped in pretty much all cases where it happened (many) - I am not going to lose any more time on diving deeper :) - we all lost already far too much time on looking at caplog issues IMHO.

[potiuk]

Pushed a fixup with conflict resolution :)

[potiuk]

wooohooo