Status of testing Providers that were prepared on June 22, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.25.0rc1

• Add Amazon Comprehend Document Classifier (#40287): @gopidesupavan
• Fix importlib_metadata import in aws utils (#40134): @eladkal
• openlineage, redshift: do not call DB for schemas below Airflow 2.10 (#40197): @mobuchowski
• Lazy match escaped quotes in RedshiftToS3Operator (#40206): @jasonspeck
• Use stdlib importlib.metadata for retrieve botocore package version (#40137): @Taragolis
  Linked issues:
  • Linked Issue #40134: @eladkal

Provider apache.drill: 2.7.2rc1

• remove deprecation drill (#39732): @Bowrna
  Linked issues:
  • Linked Issue #39485: @Taragolis

Provider apache.kafka: 1.5.0rc1

• Add delete_topic to KafkaAdminClientHook and teardown logic to Kafka integration tests (#40142): @shahar1
  Linked issues:
  • Linked Issue #40140: @shahar1
• Default client in KafkaBaseHook (#40284): @riccardoforzan

Provider cncf.kubernetes: 8.3.2rc1

• Fix Scheduler restarting due to too many completed pods in cluster (#40183): @ephraimbuddy
  Linked issues:
  • Linked Issue #22612: @dviru
• k8s version bump (#40253): @dirrao

Provider common.compat: 1.0.0rc1

• Add common.compat provider (#40374): @potiuk

Provider common.sql: 1.14.1rc1

• standardizes template fields for BaseSQLOperator and adds database as a templated field (#39826): @nyoungstudios

Provider databricks: 6.6.0rc1

• Add DatabricksTaskOperator (#40013): @pankajkoti
• Add DatabricksWorkflowTaskGroup (#39771): @pankajkoti
• Databricks: stop including user names in list_jobs (#40178): @stephenpurcell-db
• Lower log levels from INFO to DEBUG to reduce log verbosity - Databricks provider auth (#39941): @pankajkoti

Provider dbt.cloud: 3.9.0rc1

• Add retry_from_failure parameter to DbtCloudRunJobOperator (#38868): @boraberke
  Linked issues:
  • Linked Issue #35772: @sdurry
• Fix DbtCloudRunJobOperator to Use Correct Status Parameters for reuse_existing_run (#40048): @boraberke

Provider docker: 3.12.1rc1

• Bump docker version to 7.1.0 (#39839): @eladkal

Provider fab: 1.2.0rc1

• Add CSRF protection to "/logout" (#40145): @shahar1
  Linked issues:
  • Linked Issue #33030: @potiuk
• Upgrade to FAB 4.5.0 (#39851): @potiuk
• fix: sqa deprecations for airflow providers (#39293): @dondaum
  Linked issues:
  • Linked Issue #28723: @potiuk
• Add [webserver]update_fab_perms to deprecated configs (#40317): @ephraimbuddy

Provider ftp: 3.10.0rc1

• Allow FTPHook to change port number (#39465): @VelmiraPetkova
  Linked issues:
  • Linked Issue #39308: @arollet-decathlon

Provider google: 10.20.0rc1

• Add GenerationConfig and SafetySettings parameters to Google Cloud Multimodal Model Operators (#40126): @CYarros10
• Add missing location param to BigQueryUpdateTableSchemaOperator (#40237): @shahar1
  Linked issues:
  • Linked Issue #40235: @mariayanovskaya
• Add support for external IdP OIDC token retrieval for Google Cloud Operators. (#39873): @dybolo
  Linked issues:
  • Linked Issue #35899: @ChloeSheasby
• Add encryption_configuration parameter to BigQuery operators (#40063): @VladaZakharova
• add default gcp_conn_id to GoogleBaseAsyncHook (#40080): @Lee-W
• Added ordering key option for PubSubPublishMessageOperator GCP Operator (#39955): @mehdigati
  Linked issues:
  • Linked Issue #39940: @mehdigati
• Add method to get metadata from GCS blob in GCSHook (#38398): @jalengg
• Adding params. to create_auto_ml_forecasting_training_job in AutoMl hook (#39767): @vinay2242g
• Implement CloudComposerDAGRunSensor (#40088): @MaksYermak
• Implement CloudDataTransferServiceRunJobOperator (#39154): @boraberke
• Fetch intermediate log async GKEStartPod (#39348): @pankajastro
• feat: added OL support for AzureBlobStorageToGCSOperator in google provider package (#40290): @rahul-madaan
• Fix hive_partition_sensor system test (#40023): @e-galan
• Fix openai 1.32 breaking openai tests (#40110): @potiuk
• Fix credentials intialization revealed by mypy version of google auth (#40108): @potiuk
• Change regular expression to exclude double quote and newline (#39991): @kylase
• Fix replace parameter for BigQueryToPostgresOperator (#40278): @e-galan
• Fix GCSToGCSOperator behavior difference for moving single object (#40162): @boraberke
  Linked issues:
  • Linked Issue #37576: @nyoungstudios
• Refactor Datapipeline operators and fix system tests (#39716): @VladaZakharova
• openlineage: isolate metadata extraction by executing OL methods in separate, forked process (#40078): @mobuchowski
• Bump minimum version of google auth (#40190): @potiuk
  Linked issues:
  • Linked Issue #39873: @dybolo
• Bump google-ads version to use v17 by default (#40158): @molcay
• google: move bigquery openlineage imports inside methods (#40062): @ahidalgob
  Linked issues:
  • Linked Issue #40043: @ahidalgob
• Add template_field to DataplexGetDataQualityScanResultOperator (#40041): @amirmor1

Provider http: 4.12.0rc1

• Add retry_args parameter to HttpOperator (#40086): @boraberke
  Linked issues:
  • Linked Issue #39247: @lopezvit

Provider microsoft.azure: 10.1.2rc1

• Switch AzureDataLakeStorageV2Hook to use ManagedIdentityCredential for managed identity/workload auth (#38497): @TJaniF
• BUGFIX: Make sure XComs work correctly in MSGraphAsyncOperator with paged results and dynamic task mapping (#40301): @dabla
• Revert "refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers" (#40300): @dabla
• refactor: Make sure xcoms work correctly in MSGraphAsyncOperator with concurrent workers (#40297): @dabla

Provider microsoft.mssql: 3.7.2rc1

• Add mssql integration tests and relocate existing unit tests (#39831): @shahar1
  Linked issues:
  • Linked Issue #39768: @shahar1
  • Linked Issue #38577: @Taragolis

Provider openai: 1.2.2rc1

• Fix openai 1.32 breaking openai tests (#40110): @potiuk

Provider openlineage: 1.9.0rc1

• local task job: add overtime mechanism, to not kill task prematurely when auxiliary processes are running (#39890): @mobuchowski
• openlineage: add some debug logging around sql parser call sites (#40200): @mobuchowski
• openlineage: Add task SLA and queued datetime to AirflowRunFacet (#40091): @kacpermuda
• [openlineage] Add error stacktrace to task fail event (#39813): @kacpermuda
  Linked issues:
  • Linked Issue #38155: @vandonr
• [OPENLINEAGE] Introduce AirflowJobFacet and AirflowStateRunFacet (#39520): @kacpermuda
  Linked issues:
  • Linked Issue #39467: @kacpermuda
• Use UUIDv7 for OpenLineage runIds (#39889): @dolfinus
• openlineage: isolate metadata extraction by executing OL methods in separate, forked process (#40078): @mobuchowski
• openlineage, redshift: do not call DB for schemas below Airflow 2.10 (#40197): @mobuchowski
• Fix scheduler crashing with OpenLineage provider==1.8.0 on airflow standalone (#40353): @kacpermuda
  Linked issues:
  • Linked Issue #40309: @merobi-hub
• fix: sqa deprecations for airflow providers (#39293): @dondaum
  Linked issues:
  • Linked Issue #28723: @potiuk

Provider opensearch: 1.3.0rc1

• Opensearch operator set connection type (#39788): @Lukas1v

Provider sftp: 4.10.2rc1

• sftp_sensor: fixing resource management with sensor (#40022): @pateash

Provider snowflake: 5.5.2rc1

• openlineage: add some debug logging around sql parser call sites (#40200): @mobuchowski
• Bump Snowflake client driver versions per support policy (#39886): @josh-fell

Provider telegram: 4.5.2rc1

• Working fix for typing in telegram provider (#40258): @potiuk
  Linked issues:
  • Linked Issue #40255: @potiuk
• Fix typing in telegram provider (#40255): @potiuk

Provider teradata: 2.3.0rc1

• Enhancements to S3 and Azure to Teradata Cloud Transfer Operators (#40378): @satish-chinthanippu

Provider ydb: 1.0.0rc1

• add new YDB provider (#39996): @uzhastik

All users involved in the PRs:
@rahul-madaan @eladkal @Taragolis @nyoungstudios @uzhastik @jalengg @mobuchowski @e-galan @VladaZakharova @riccardoforzan @josh-fell @pankajastro @kacpermuda @ephraimbuddy @satish-chinthanippu @boraberk

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[uzhastik]

Hello. Could you please exclude from this release ydb provider? It’s too young for that ;) it has known bug related to auth and I’d like to add new feature there later. Thank you in advance.

[eladkal]

Hello. Could you please exclude from this release ydb provider? It’s too young for that ;) it has known bug related to auth and I’d like to add new feature there later. Thank you in advance.

In your PR you marked it as ready to be released

airflow/airflow/providers/ydb/provider.yaml

Line 24 in 6e5ae26

state: ready

Missing features are not a blocker for release. We can always add new features later on.

[uzhastik]

Hello. Could you please exclude from this release ydb provider? It’s too young for that ;) it has known bug related to auth and I’d like to add new feature there later. Thank you in advance.

In your PR you marked it as ready to be released

airflow/airflow/providers/ydb/provider.yaml

Line 24 in 6e5ae26

state: ready

Missing features are not a blocker for release. We can always add new features later on.

To be honest yaml was just copied and bug was found two days ago;) auth info from file does not work, but there is workaround to provide auth info inplace.

[potiuk]

Checked that all changes are there. The common.compat does not have any code yet, so we could skip releasing it, but there is no harm in doing so - having a 1.0.0 version released in PyPI is generally a good idea.

[gopidesupavan]

Tested #40287, working fine. Thanks

[rahul-madaan]

tested #40290, working fine 👍🏻

[mehdigati]

tested #39955, working fine, thanks !

[shahar1]

#40142, #40237, #39831 are ok

[jasonspeck]

Confirmed #40206 works as expected

[kylase]

I have tested #39991 and it works as expected.

[2024-06-23, 04:47:48 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - Dataflow SDK version: 2.56.0","severity":"INFO"}
[2024-06-23, 04:47:49 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/<redacted>","severity":"INFO"}
[2024-06-23, 04:47:49 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - Submitted job: <redacted>","severity":"INFO"}
[2024-06-23, 04:47:49 UTC] {beam.py:119} WARNING - {"message":"org.apache.beam.runners.dataflow.DataflowRunner - To cancel the job using the \u0027gcloud\u0027 tool, run:\n\u003e gcloud dataflow jobs --project\u003d<redacted> cancel --region\u003<redacted> <redacted>","severity":"INFO"}
[2024-06-23, 04:47:49 UTC] {beam.py:172} INFO - Process exited with return code: 0
[2024-06-23, 04:47:49 UTC] {dataflow.py:461} INFO - Start waiting for done.
[2024-06-23, 04:47:49 UTC] {dataflow.py:403} INFO - Google Cloud DataFlow job <redacted> is state: JOB_STATE_PENDING
[2024-06-23, 04:47:49 UTC] {dataflow.py:464} INFO - Waiting for done. Sleep 10 s
[2024-06-23, 04:47:59 UTC] {dataflow.py:403} INFO - Google Cloud DataFlow job <redacted> is state: JOB_STATE_PENDING
[2024-06-23, 04:47:59 UTC] {dataflow.py:464} INFO - Waiting for done. Sleep 10 s
[2024-06-23, 04:48:09 UTC] {dataflow.py:403} INFO - Google Cloud DataFlow job <redacted> is state: JOB_STATE_RUNNING
[2024-06-23, 04:48:09 UTC] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=<redacted>, task_id=start_streaming, map_index=0, execution_date=20240623T044637, start_date=20240623T044717, end_date=20240623T044809
[2024-06-23, 04:48:09 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 0
[2024-06-23, 04:48:09 UTC] {taskinstance.py:2781} INFO - 0 downstream tasks scheduled from follow-on schedule check

[dirrao]

Changes #40253 works as expected.