Status of testing Providers that were prepared on January 26, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 3.6.0rc2

• Add deferrable functionality to the AirbyteJobSensor and AirbyteTriggerSyncOperator (#36780): @chrishronek

Provider alibaba: 2.7.2rc2

• Fix assignment of template field in __init__ in analyticdb_spark.py (#36911): @romsharon98

Provider amazon: 8.17.0rc2

• add deferrable mode to RedshiftDataOperator (#36586): @Lee-W
• Adds support for capacity providers to ECS Executor (#36722): @ferruzzi
• add use_regex argument for allowing S3KeySensor to check s3 keys with regular expression (#36578): @Lee-W
• add deferrable mode to RedshiftClusterSensor (#36550): @Lee-W
• Implement AthenaSQLHook (#36171): @flolas
• Create CLI commands for AWS auth manager to create Amazon Verified Permissions related resources (#36799): @vincbeck
• Implement is_authorized_dag in AWS auth manager (#36619): @vincbeck
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis
• EC2 CreateInstance: terminate instances in on_kill (#36828): @AchimGaedkeLynker
• Fallback to default value if [aws] cloudwatch_task_handler_json_serializer not set (#36851): @Taragolis
• Raise AirflowOptionalProviderfeature exception for AVP command (#36824): @potiuk
• check transform job status before deferring SageMakerTransformOperator (#36680): @Lee-W
• check ProcessingJobStatus status before deferring SageMakerProcessingOperator (#36658): @Lee-W
• check job_status before BatchOperator execute in deferrable mode (#36523): @Lee-W
• Update the redshift hostname check to avoid possible bugs (#36703): @hussein-awala
• Refresh credentials in AwsEcsExecutor (#36179): @syedahsn
• Fix on docstring for apply_wildcard parameter in S3ListOperator (#36679): @sasidharan-rathinam
• Use base aws classes in AWS DMS Operators/Sensors (#36772): @Taragolis
• Use base aws classes in AWS Redshift Data API Operators (#36764): @Taragolis
• Use base aws classes in Amazon EventBridge Operators (#36765): @Taragolis
• Use base aws classes in Amazon QuickSight Operators/Sensors (#36776): @Taragolis
• Use base aws classes in AWS Datasync Operators (#36766): @Taragolis
• Use base aws classes in Amazon DynamoDB Sensors (#36770): @Taragolis
• Use base aws classes in AWS CloudFormation Operators/Sensors (#36771): @Taragolis
• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk
• Bump min version of amazon-provider related dependencies (#36660): @potiuk

Provider apache.beam: 5.6.0rc2

• Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697): @potiuk

Provider apache.druid: 3.8.0rc2

• Provide the logger_name param in providers hooks in order to override the logger name (#36675): @hussein-awala
• Fix successful Apache Druid task submissions reported as failed (#36813): @listik

Provider apache.hive: 6.4.2rc1

• Fix assignment of template field in __init__ in hive-stats (#36905): @romsharon98
• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk

Provider apache.spark: 4.7.1rc1

• Bump min version for grpcio-status in spark provider (#36662): @potiuk

Provider atlassian.jira: 2.5.1rc1

• Use lax atlassian-python-api limitation (#36841): @Taragolis
• Limit atlassian-python-api to <3.41.6 (#36815): @Taragolis

Provider celery: 3.5.2rc2

• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

Provider cncf.kubernetes: 7.14.0rc2

• Add SparkKubernetesOperator crd implementation (#22253): @hamedhsn
• Template field support for configmaps in the KubernetesPodOperator (#36922): @vizeit
• Create a generic callbacks class for KubernetesPodOperator (#35714): @hussein-awala
• avoid retrying after KubernetesPodOperator has been marked as failed (#36749): @Lee-W
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis
• Increase tenacity wait in read_pod_logs (#36955): @o-nikolas
• Fix dynamic configmaps as environment variable in KubernetesPodOperator (#37001): @vizeit
• Change field type for kube_config in kubernetes cluster connection (#36752): @arjunanan6
• Changing wording in docstring for CNCF provider (#36547): @amoghrajesh
• Add support of Pendulum 3 (#36281): @Taragolis

Provider cohere: 1.1.2rc1

• Bump Cohere min version to limit backtracking on upgrade (#36666): @potiuk

Provider common.sql: 1.10.1rc1

• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk

Provider databricks: 6.1.0rc2

• [FEAT] adds repair run functionality for databricks (#36601): @gaurav7261
• Fix databricks_sql hook query failing on empty result for return_tuple (#36827): @potiuk
• Rename columns to valid namedtuple attributes + ensure Row.fields are retrieved as tuple (#36949): @Joffreybvn
• check status before DatabricksSubmitRunOperator & DatabricksSubmitRunOperator executes in deferrable mode (#36862): @vatsrahul1001

Provider dbt.cloud: 3.6.0rc2

• OpenLineage: Add dag_id when generating run_id for task instance. (#36659): @kacpermuda
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

Provider discord: 3.6.0rc2

• Add docs for discord webhook connection (#36720): @shohamy7

Provider elasticsearch: 5.3.2rc1

• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

Provider exasol: 4.4.2rc2

• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk

Provider google: 10.14.0rc2

• Add templated fields to BigQueryToSqlBaseOperator from BigQueryToPostgresOperator (#36663): @romsharon98
• Added Check for Cancel Workflow Invocation and added new Query Workflow Invocation operator (#36351): @varuntwr
• Implement Google Analytics Admin (GA4) operators (#36276): @moiseenkov
• Add operator to collect diagnostic tarball for Dataproc cluster (#36899): @Flacode
• Add scopes into a GCP token (#36974): @moiseenkov
• feat: full support for google credentials in gcloud-aio clients (#36849): @m1racoli
• Fix assignment of template field in __init__ in auto_ml.py (#36934): @romsharon98
• fix: respect connection ID and impersonation in GKEStartPodOperator (#36861): @m1racoli
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis
• Fix deprecations into the GCP Dataproc links (#36834): @Taragolis
• Fix assignment of template field in __init__ in SFTPToGCSOperator (#36603): @romsharon98
• Check cluster state before defer Dataproc operators to trigger (#36892): @Lee-W
• prevent templated field logic checks in operators __init__ in dataproc (#36489): @romsharon98
• Preserve ASCII control characters directly through the BigQuery load API (#36533): @romsharon98
• Change default parquet_row_group_size in BaseSQLToGCSOperator (#36817): @renzepost
• Fix google operators handling of impersonation chain (#36903): @m1racoli
• style(providers/google): improve BigQueryInsertJobOperator type hinting (#36894): @Lee-W
• Deprecate AutoMLTrainModelOperator for Vision and Video (#36473): @VladaZakharova
• Remove backward compatibility check for KubernetesPodOperator module (#36724): @hussein-awala
• Remove backward compatibility check for KubernetesPodTrigger module from google provider (#36721): @hussein-awala
• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk
• prevent templated field logic checks in operators __init__ in BigQueryToPostgresOperator operator (#36491): @romsharon98
• reword GoogleBaseHookAsync as GoogleBaseAsyncHook in docstring (#36946): @Lee-W

Provider hashicorp: 3.6.2rc1

• Explicitly passing raise_on_deleted_version=True to read_secret_version in Hashicorp operator (#36532): @romsharon98

Provider http: 4.9.0rc2

• Allow default requests parameters like proxy to be defined in extra options field of a Airflow HTTP Connection (#36733): @dabla
• Add deferrable implementation in HTTPSensor (#36904): @vatsrahul1001
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

Provider mongo: 3.6.0rc2

• Switch to mongo_conn_id argument into the MongoHook constructor (#36896): @Taragolis

Provider mysql: 5.5.2rc2

• Bump min version of mysql-connector-python (#36668): @potiuk

Provider odbc: 4.4.1rc2

• Fix ODBC Hook sql select return empty table (#36709): @roel-w
• Rename columns to valid namedtuple attributes + ensure Row.fields are retrieved as tuple (#36949): @Joffreybvn

Provider openlineage: 1.5.0rc2

• OpenLineage: Add dag_id when generating run_id for task instance. (#36659): @kacpermuda
• Prepare docs 2nd wave of Providers January 2024 (#36945): @eladkal

Provider pagerduty: 3.6.1rc1

• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis
• Revert "Provide the logger_name param in providers hooks in order to override the logger name (#36675)" (#37015): @Taragolis
• Prepare docs 2nd wave of Providers January 2024 (#36945): @eladkal
• Provide the logger_name param in providers hooks in order to override the logger name (#36675): @hussein-awala

Provider papermill: 3.6.1rc2

• Fix assignment of template field in __init__ in papermill.py (#36530): @romsharon98
• Bump min provider version for papermill (#36669): @potiuk

Provider pinecone: 1.1.2rc1

• Limit pinecone-client to <3.0 (#36818): @Taragolis

Provider presto: 5.4.1rc2

• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk

Provider salesforce: 5.6.2rc1

• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk

Provider slack: 8.6.0rc2

• Optionally use client.files_upload_v2 in Slack Provider (#36757): @Taragolis
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

Provider snowflake: 5.3.0rc2

• Add openlineage support for CopyFromExternalStageToSnowflakeOperator (#36535): @kacpermuda
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis
• check query status before SnowflakeSqlApiOperator execute in deferrable mode (#36850): @vatsrahul1001

Provider tableau: 4.4.1rc1

• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

Provider telegram: 4.3.1rc1

• Bump min telegram-bot version (#36670): @potiuk

Provider trino: 5.6.1rc2

• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk

Provider weaviate: 1.3.1rc2

• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis
• Fix assignment of template field in __init__ in weaviate (#36908): @romsharon98

Provider yandex: 3.8.0rc2

• feat: add Yandex Cloud Lockbox secrets backend (#36449): @ImpressionableRaccoon
• Fix stacklevel in warnings.warn into the providers (#36831): @Taragolis

All users involved in the PRs:
@romsharon98 @kacpermuda @dabla @renzepost @vizeit @varuntwr @AchimGaedkeLynker @hamedhsn @Taragolis @Lee-W @ferruzzi @sasidharan-rathinam @hussein-awala @chrishronek @syedahsn @vatsrahul1001 @arjunanan

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[m1racoli]

Tested #36861, connection ID and impersonation chain are now properly passed. ✅

[arjunanan6]

Test #36752 and it works as expected.

[Lee-W]

Verified #36892, #36946 and #36894. will continue work on the rest

[potiuk]

All my changes work ! 🎉

[Taragolis]

Validate changes in

• Fix stacklevel in warnings.warn into the providers #36831
• Fix deprecations into the GCP Dataproc links #36834
• Limit pinecone-client to <3.0 #36818
• Use lax atlassian-python-api limitation #36841
• Limit atlassian-python-api to <3.41.6 #36815
• Switch to mongo_conn_id argument into the MongoHook constructor #36896

I plan to check later other my changes

[romsharon98]

Validate changes in:
#36911
#36905
#36663
#36934
#36603
#36489
#36533
#36491
#36532
#36530
#36908

[VladaZakharova]

Hi!
#36473 works as well, thank you :)

[moiseenkov]

#36276 works as expected

[AchimGaedkeLynker]

#36828 works as expected

[Lee-W]

Tested #36586, #36578, #36550, #36680, #36658, #36749

[renzepost]

#36817 works as expected