Status of testing Providers that were prepared on July 06, 2023

[eladkal]

Body

Issue title: Status of testing Providers that were prepared on July 06, 2023

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.5.0rc2

• Add Alibaba Cloud AnalyticDB Spark Support (#31787): @dcoliversun

Provider amazon: 8.3.0rc2

• Add ChimeWebhookHook (#31939): @cjames23
• Add ChimeNotifier (#32222): @cjames23
• Add deferrable mode to S3KeysUnchangedSensor (#31940): @sunank200
• Add deferrable mode to RdsCreateDbInstanceOperator and RdsDeleteDbInstanceOperator (#32171): @syedahsn
• Add deferrable mode for AthenaOperator (#32186): @vandonr-amz
• Add a deferrable mode to BatchCreateComputeEnvironmentOperator (#32036): @vandonr-amz
• Add deferrable mode in EMR operator and sensor (#32029): @pankajastro
• add async wait method to the "with logging" aws utils (#32055): @vandonr-amz
• Add custom waiters to EMR Serverless (#30463): @syedahsn
• Add an option to GlueJobOperator to stop the job run when the TI is killed (#32155): @hussein-awala
• deferrable mode for SageMakerTuningOperator and SageMakerEndpointOperator (#32112): @vandonr-amz
• EKS Create/Delete Nodegroup Deferrable mode (#32165): @syedahsn
• Deferrable mode for ECS operators (#31881): @vandonr-amz
• Add job_poll_interval argument to AWS GlueJobOperator (#32147): @raphaelauv
• Added AzureBlobStorageToS3Operator transfer operator (#32270): @Adaverse
• bugfix: break down run+wait method in ECS operator (#32104): @vandonr-amz
• Handle UnboundLocalError while parsing invalid s3_url (#32120): @Adaverse
• Fix LambdaInvokeFunctionOperator payload parameter type (#32259): @eladi99
• Bug fix GCSToS3Operator: avoid ValueError when replace=False with files already in S3 (#32322): @Adaverse
• Deprecate delimiter param and source object's wildcards in GCS, introduce match_glob param. (#31261): @shahar1
• aws waiter util: log status info with error level on waiter error (#32247): @vandonr-amz
• rewrite method used in ecs to fetch less logs (#31786): @vandonr-amz
• Refactor Eks Create Cluster Operator code (#31960): @syedahsn
• Use a waiter in AthenaHook (#31942): @vandonr-amz
• Add on_finish_action to KubernetesPodOperator (#30718): @hussein-awala
• Add default_deferrable config (#31712): @Lee-W
• deprecate arbitrary parameter passing to RDS hook (#32352): @vandonr-amz

Provider apache.hive: 6.1.2rc2

• Add proxy_user template check (#32334): @potiuk

Provider apache.impala: 1.1.2rc2

• Add kerberos dependency to Impala Provider (#32304): @Joffreybvn

Provider apache.kafka: 1.1.2rc2

• Break AwaitMessageTrigger execution when finding a message with the desired format (#31803): @hussein-awala

Provider apache.livy: 3.5.2rc2

• Fix LivyHook TypeError exception on session_id log format (#32051): @joaopamaral
• Add default_deferrable config (#31712): @Lee-W

Provider cncf.kubernetes: 7.2.0rc2

• Add on_finish_action to KubernetesPodOperator (#30718): @hussein-awala
• Fix KubernetesPodOperator validate xcom json and add retries (#32113): @aagateuip
• Fix KubernetesPodTrigger waiting strategy (#31348): @hussein-awala
• fix spark-kubernetes-operator compatibility (#31798): @blcksrx
• Add default_deferrable config (#31712): @Lee-W

Provider common.sql: 1.6.0rc2

• openlineage, common.sql: provide OL SQL parser as internal OpenLineage provider API (#31398): @JDarDagran
• openlineage: pass SQLAlchemy engine to construct information schema query. (#32371): @JDarDagran

Provider databricks: 4.3.1rc2

• Modify template_fields of DatabricksSqlOperator to support parent class fields (#32253): @eladkal
• Add default_deferrable config (#31712): @Lee-W

Provider dbt.cloud: 3.2.2rc2

• Add default_deferrable config (#31712): @Lee-W

Provider elasticsearch: 5.0.0rc2

• Deprecate the 2 non-official elasticsearch libraries (#31920): @Owen-CH-Leung

Provider google: 10.3.0rc2

• Add on_finish_action to KubernetesPodOperator (#30718): @hussein-awala
• Add deferrable mode to CloudSQLExportInstanceOperator (#30852): @bkossakowska
• Adding src_fmt_configs to the list of template fields. (#32097): @nathadfield
• Fix name format in the batch requests (#32070): @HaeSe0ng
• Fix BigQueryInsertJobOperator error handling in deferrable mode (#32034): @phanikumv
• Fix BIGQUERY_JOB_DETAILS_LINK_FMT in BigQueryConsoleLink (#31953): @bkossakowska
• Make the deferrable version of DataprocCreateBatchOperator handle a b… (#32216): @kristopherkane
• Switch Google Ads API version from v13 to v14 (#32028): @ahidalgob
• Deprecate delimiter param and source object's wildcards in GCS, introduce match_glob param. (#31261): @shahar1
• Refactor GKECreateClusterOperator's body validation (#31923): @moiseenkov
• Optimize deferrable mode execution for BigQueryValueCheckOperator (#31872): @phanikumv
• Add default_deferrable config (#31712): @Lee-W

Provider hashicorp: 3.4.2rc2

• Add default retry to hvac client requests (#31073): @notatallshaw-gts

Provider microsoft.azure: 6.2.0rc2

• Adds connection test for ADLS Gen2 (#32126): @Adaverse
• Add option to pass extra configs to ClientSecretCredential (#31783): @Adaverse
• Added AzureBlobStorageToS3Operator transfer operator (#32270): @Adaverse
• feat(providers/azure): cancel pipeline if unexpected exception caught (#32238): @Lee-W
• Fix where account url is build if not provided using login (account name) (#32082): @Adaverse
• refresh connection if an exception is caught in "AzureDataFactory" (#32323): @Lee-W
• Add default_deferrable config (#31712): @Lee-W

Provider postgres: 5.5.2rc2

• postgres: deprecate runtime_parameters in favor of options in hook_params (#32345): @JDarDagran

Provider snowflake: 4.3.0rc2

• Add Deferrable switch to SnowflakeSqlApiOperator (#31596): @utkarsharma2

Provider trino: 5.2.0rc2

• Trino Hook: Add ability to read JWT from file (#31950): @jhbigler

Provider vertica: 3.5.0rc2

• Add various Vertica connection parameters (#32089): @darkag

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@darkag @raphaelauv @blcksrx @vandonr-amz @jhbigler @potiuk @Lee-W @bkossakowska @nathadfield @dcoliversun @phanikumv @pankajastro @aagateuip @joaopamaral @Adaverse @JDarDagran @moiseenkov @Joffreybvn
@eladi99 @hussein-awala @notatallshaw-gts @Owen-CH-Leung @shahar1 @eladkal @utkarsharma2 @sunank200 @syedahsn @kristopherkane @cjames23 @ahidalgob @HaeSe0ng

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Joffreybvn]

#32304 Works fine. Kerberos is installed when doing pip install apache-airflow-providers-apache-impala[kerberos] ✅

[JDarDagran]

#32345 works fine - tested both with hook_params and deprecated runtime_parameters.
#31398 , #32371 are added too. There's however a bug fixed in #32400 which is not exposed though as there's no release of OpenLineage provider yet (method with a bug is never called without the OL provider). Would be nice to include this in this release cycle, I guess?

[eladkal]

There's however a bug fixed in #32400 which is not exposed though as there's no release of OpenLineage provider yet (method with a bug is never called without the OL provider). Would be nice to include this in this release cycle, I guess?

Thanks for verifying
This is not a blocker for this release since the OL integration doesn't work anyway (as OL provider is not yet available) the bug is locallized to OL part.
For the moment I see no reason to have RC3, The fix will be included in future release of the provider.

[Lee-W]

We tested our example, DAGs with the following rc without encountering any error

• apache-airflow-providers-amazon
• apache-airflow-providers-apache-hive
• apache-airflow-providers-apache-livy
• apache-airflow-providers-cncf-kubernetes
• apache-airflow-providers-databricks
• apache-airflow-providers-dbt-cloud
• apache-airflow-providers-google
• apache-airflow-providers-microsoft-azure
• apache-airflow-providers-snowflake

[pankajkoti]

For #31920 I have a question which I have asked here #31920 (comment)

[darkag]

tested #32089, works as expected

[vandonr-amz]

good for me, even if I'd prefer an RC3 to sneak some not-yet-breaking changes in there.

[Adaverse]

Tested #32126 #31783 #32082 #32120 #32322 works as expected.

[potiuk]

Checked #32334 👍

vs (6.2.1)