Update list_jobs function in DatabricksHook to token-based pagination

[oleksiidav]

Update Airflow to switch from default offset-based pagination (deprecated) to using a page token

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[eladkal]

What is JOBS-13077 ?

[oleksiidav]

What is JOBS-13077 ?

Tracking ticket from Databricks side, nevermind

[alexott]

code looks good - can we add a unit test for token usage?

[oleksiidav]

Added test @alexott, PTAL!

[alexott]

lgtm.

@eladkal WDYT?

[oleksiidav]

Hi @eladkal, could you please take another look?

[potiuk]

LGTM @eladkal ?

[oleksiidav]

Heyy @eladkal , I have updated the tests/formatting, can we pls merge?

[eladkal]

@oleksiidav static checks are failing you are missing import of AirflowProviderDeprecationWarning

[potiuk]

Static check to fix.

[oleksiidav]

Thanks @eladkal @potiuk ! Fixed it now. Did not have access to test/linter workflow to run checks

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.