Status of testing Providers that were prepared on May 19, 2023

[eladkal]

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.1.0rc2

• DynamoDBToS3Operator - Add a feature to export the table to a point in time. (#31142): @utkarsharma2
• Add deferrable param in SageMakerTransformOperator (#31063): @pankajkoti
• Add deferrable param in SageMakerTrainingOperator (#31042): @pankajkoti
• Add deferrable param in SageMakerProcessingOperator (#31062): @pankajkoti
• Add IAM authentication to Amazon Redshift Connection by AWS Connection (#28187): @IAL32
• StepFunctionStartExecutionOperator: get logs in case of failure (#31072): @eladkal
• Add on_kill to EMR Serverless Job Operator (#31169): @dacort
• Add Deferrable Mode for EC2StateSensor (#31130): @syedahsn
• Loop through paginated response to check for cluster id (#29732): @nsAstro
• Add template field to S3ToRedshiftOperator (#30781): @alextgu
• Add extras links to some more EMR Operators and Sensors (#31032): @ferruzzi
• Add retries to S3 delete_bucket (#31192): @ferruzzi
• Add tags param in RedshiftCreateClusterSnapshotOperator (#31006): @pankajastro
• improve/fix glue job logs printing (#30886): @vandonr-amz
• Import aiobotocore only if deferrable is true (#31094): @pankajastro
• Update return types of get_key methods on S3Hook (#30923): @jonshea
• Support shareIdentifier in BatchOperator (#30829): @phanikumv
• BaseAWS - Override client when resource_type is user to get custom waiters (#30897): @utkarsharma2
• Add future-compatible mongo Hook typing (#31289): @potiuk
• Handle temporary credentials when resource_type is used to get custom waiters (#31333): @vincbeck

Provider apache.beam: 5.1.0rc2

• Update SDKs for google provider package (#30067): @lwyszomi

Provider apache.hdfs: 4.0.0rc2

• Remove snakebite-py3 based HDFS hooks and sensors (#31262): @potiuk

Provider apache.hive: 6.1.0rc2

• Update return types of get_key methods on S3Hook (#30923): @jonshea

Provider apache.pinot: 4.1.0rc2

• Add guide for Apache Pinot operators (#28653): @shahar1

Provider cncf.kubernetes: 7.0.0rc2

• Remove deprecated features from KubernetesHook (#31402): @dstandish
• K8s hook should still work with missing default conn (#31187): @dstandish
• Add protocol to define methods relied upon by KubernetesPodOperator (#31298): @dstandish
• fix kubernetes task decorator pickle error (#31110): @JCoder01
• Empty xcom result file log message more specific (#31228): @dstandish
• Add options to KubernetesPodOperator (#30992): @RachitSharma2001
• add missing read for K8S config file from conn in deferred KubernetesPodOperator (#29498): @hussein-awala

Provider common.sql: 1.5.0rc2

• Add conditional output processing in SQL operators (#31136): @potiuk
• Remove noisy log from SQL table check (#31037): @jon-evergreen

Provider databricks: 4.2.0rc2

• Add conditional output processing in SQL operators (#31136): @potiuk
• Add cancel all runs functionality to Databricks hook (#31038): @phanikumv
• Add retry param in databrics async operator (#30744): @pankajastro
• Add repair job functionality to databricks hook (#30786): @phanikumv
• Add DatabricksPartitionSensor (#30980): @harishkesavarao
• Deprecate Databricks Deferrable Operator (#30761): @pankajastro

Provider dbt.cloud: 3.2.0rc2

• Optimize deferrable mode execution in DbtCloudJobRunSensor (#30968): @phanikumv
• Optimize deferrable execution mode for DbtCloudRunJobOperator (#31188): @phanikumv

Provider elasticsearch: 4.5.0rc2

• Upper-bind elasticearch integration (#31255): @potiuk

Provider google: 10.1.0rc2

• Add deferrable mode to DataprocInstantiateInlineWorkflowTemplateOperator (#30878): @bkossakowska
• Add deferrable mode to GCSObjectUpdateSensor (#30579): @phanikumv
• Add protocol to define methods relied upon by KubernetesPodOperator (#31298): @dstandish
• Add BigQueryToPostgresOperator (#30658): @jbbqqf
• DataflowTemplatedJobStartOperator fix overwriting of location with default value, when a region is provided. (#31082): @VVildVVolf
• Optimize deferrable mode execution in GCSObjectsWithPrefixExistenceSensor (#30939): @phanikumv
• Add deferrable mode to GCSObjectsWithPrefixExistenceSensor (#30618): @phanikumv
• Refactor GCS list hook to support multiple prefixes (#30815): @shahar1
• Fix removed delegate_to parameter in deferrable GCS sensor (#30810): @potiuk
• Add use_legacy_sql param to BigQueryGetDataOperator (#31190): @shahar1
• Add as_dict param to BigQueryGetDataOperator (#30887): @shahar1
• Add flag apply_gcs_prefix to S3ToGCSOperator (#31127): @moiseenkov
• Add priority parameter to BigQueryHook (#30655): @ying-w
• implement gcs_schema_object for BigQueryCreateExternalTableOperator (#30961): @attilaszombati
• Optimize deferrable mode execution in bigquery sensors (#30946): @phanikumv
• Optimize deferrable mode execution in GCSObjectUpdateSensor (#30920): @phanikumv
• Optimize deferrable mode in GCSObjectExistenceSensor (#30901): @phanikumv
• CreateBatchPredictionJobOperator Add batch_size param for Vertex AI BatchPredictionJob objects (#31118): @VVildVVolf
• GKEPodHook needs to have all methods KPO calls (#31266): @dstandish
• Add CloudBuild build id log (#30516): @tnk-ysk
• Switch default Google ads to v13 (#31382): @ahidalgob
• Switch to google ads v13 (#31369): @potiuk
• Update SDKs for google provider package (#30067): @lwyszomi

Provider microsoft.azure: 6.1.0rc2

• Add deferrable mode to WasbPrefixSensor (#30252): @phanikumv
• Optimize deferrable execution mode AzureDataFactoryPipelineRunStatusSensor (#30983): @phanikumv
• Optimize deferrable execution for AzureDataFactoryRunPipelineOperator (#31214): @phanikumv

Provider mongo: 3.2.0rc2

• Remove the upper version bound on pymongo (#31189): @jonshea
• Add future-compatible mongo Hook typing (#31289): @potiuk

Provider neo4j: 3.3.0rc2

• Removed Mandatory Encryption in Neo4jHook (#30418): @eldar-eln-bigabid

Provider oracle: 3.7.0rc2

• Add docs for Oracle operators (#30979): @vchiapaikeo
• Fix deprecated import in Oracle example (#31166): @potiuk

Provider pagerduty: 3.2.0rc2

• Add test_connection method to PagerdutyHook (#31344): @utkarsharma2
• Add test_connection method to PagerdutyEventsHook (#31346): @utkarsharma2
• Add Change event V2 API support for pagerduty (#31222): @utkarsharma2
• Fixed typo in PagerdutyEventsHook (#31209): @utkarsharma2

Provider redis: 3.2.0rc2

• Upgrade redis (#31279): @Owen-CH-Leung

Provider slack: 7.3.0rc2

• Add Documentation for notification feature extension (#29191): @ephraimbuddy
• Standardize Slack Notifier (#31244): @eladkal

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@ying-w @alextgu @RachitSharma2001 @hussein-awala @jonshea @vincbeck @dacort @VVildVVolf @pankajastro @shahar1 @potiuk @vandonr-amz @ferruzzi @eladkal @utkarsharma2 @harishkesavarao @bkossakowska
@dstandish @jon-evergreen @tnk-ysk @IAL32 @nsAstro @Owen-CH-Leung @eldar-eln-bigabid @ephraimbuddy @moiseenkov @jbbqqf @attilaszombati @ahidalgob @vchiapaikeo @JCoder01 @syedahsn @pankajkoti @lwyszomi
@phanikumv

[dacort]

Confirmed #31169

[shahar1]

Provider apache.pinot: 4.1.0rc1

• Add guide for Apache Pinot operators (#28653):

Provider google: 10.1.0rc1

• Refactor GCS list hook to support multiple prefixes (#30815):
• Add use_legacy_sql param to BigQueryGetDataOperator (#31190):
• Add as_dict param to BigQueryGetDataOperator (#30887):

[raphaelauv]

Provider cncf.kubernetes: 6.2.0rc1
add missing read for K8S config file from conn in deferred KubernetesPodOperator
✔️ #29498:

[harishkesavarao]

Provider databricks: 4.2.0rc1

• Add DatabricksPartitionSensor #30980

[ying-w]

Confirmed #30655 by verifying priority on ui when running the following dag

from pendulum import datetime
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
from airflow.decorators import task, dag

default_args = {
    "owner": "example",
    "retries": 0,
}

@dag(
    description="tmp",
    start_date=datetime(2023, 5, 1),
    schedule=None,
)
def testing():
    SQLExecuteQueryOperator(
        task_id="test",
        conn_id="bigquery",
        sql="SELECT 2",
        hook_params={
            "use_legacy_sql": False,
            "location": "us",
            "priority": "BATCH",  # this is the new part
            "api_resource_configs": {"query": {"useQueryCache": False}},
        },
    )

testing()

[phanikumv]

#30829: works good. Tested it by passing a shareIdentifier while submitting the Batch job

[2023-05-17, 08:57:54 UTC] {batch_client.py:328} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) check status (RUNNABLE) in ['RUNNING', 'SUCCEEDED', 'FAILED']
[2023-05-17, 08:57:54 UTC] {batch_client.py:343} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) status check (7 of 4200) in the next 7.67 seconds
[2023-05-17, 08:58:03 UTC] {batch_client.py:328} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) check status (STARTING) in ['RUNNING', 'SUCCEEDED', 'FAILED']
[2023-05-17, 08:58:03 UTC] {batch_client.py:343} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) status check (8 of 4200) in the next 8.68 seconds
[2023-05-17, 08:58:12 UTC] {batch_client.py:328} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) check status (SUCCEEDED) in ['RUNNING', 'SUCCEEDED', 'FAILED']
[2023-05-17, 08:58:21 UTC] {batch_client.py:328} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) check status (SUCCEEDED) in ['SUCCEEDED', 'FAILED']
[2023-05-17, 08:58:21 UTC] {batch_client.py:269} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) has completed
[2023-05-17, 08:58:21 UTC] {batch.py:310} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) CloudWatch Events details found. Links to logs:
[2023-05-17, 08:58:21 UTC] {batch.py:313} INFO - 
[2023-05-17, 08:58:21 UTC] {batch_client.py:245} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) succeeded: {'jobArn': 'arn:aws:batch:us-east-2:396807896376:job/733b153e-fda5-4f3c-a29b-95eb8c22a6d4', 'jobName': 'test_***_job', 'jobId': '733b153e-fda5-4f3c-a29b-95eb8c22a6d4', 'jobQueue': 'arn:aws:batch:us-east-2:396807896376:job-queue/providers_team_job_queue', 'status': 'SUCCEEDED', 'shareIdentifier': 'providers', 'schedulingPriority': 1, 'attempts': [{'container': {'containerInstanceArn': 'arn:aws:ecs:us-east-2:396807896376:container-instance/AWSBatch-providers_team_compute_env-0f056134-f57a-3a1f-9cbb-8d84bdd0f935/bcefcb90921e460085877d330bc5f495', 'taskArn': 'arn:aws:ecs:us-east-2:396807896376:task/AWSBatch-providers_team_compute_env-0f056134-f57a-3a1f-9cbb-8d84bdd0f935/0c4cdbb8cbfa4605aa1ad82e28384f87', 'exitCode': 0, 'logStreamName': 'providers_team_job_defn/default/0c4cdbb8cbfa4605aa1ad82e28384f87', 'networkInterfaces': []}, 'startedAt': 1684313885354, 'stoppedAt': 1684313885432, 'statusReason': 'Essential container in task exited'}], 'statusReason': 'Essential container in task exited', 'createdAt': 1684313824707, 'startedAt': 1684313885354, 'stoppedAt': 1684313885432, 'dependsOn': [], 'jobDefinition': 'arn:aws:batch:us-east-2:396807896376:job-definition/providers_team_job_defn:2', 'parameters': {}, 'container': {'image': 'public.ecr.aws/amazonlinux/amazonlinux:latest', 'command': ['echo', 'hello world'], 'volumes': [], 'environment': [], 'mountPoints': [], 'ulimits': [], 'exitCode': 0, 'containerInstanceArn': 'arn:aws:ecs:us-east-2:396807896376:container-instance/AWSBatch-providers_team_compute_env-0f056134-f57a-3a1f-9cbb-8d84bdd0f935/bcefcb90921e460085877d330bc5f495', 'taskArn': 'arn:aws:ecs:us-east-2:396807896376:task/AWSBatch-providers_team_compute_env-0f056134-f57a-3a1f-9cbb-8d84bdd0f935/0c4cdbb8cbfa4605aa1ad82e28384f87', 'logStreamName': 'providers_team_job_defn/default/0c4cdbb8cbfa4605aa1ad82e28384f87', 'networkInterfaces': [], 'resourceRequirements': [{'value': '1', 'type': 'VCPU'}, {'value': '2048', 'type': 'MEMORY'}], 'secrets': []}, 'tags': {}, 'propagateTags': False, 'platformCapabilities': ['EC2'], 'eksAttempts': []}
[2023-05-17, 08:58:21 UTC] {batch.py:329} INFO - AWS Batch job (733b153e-fda5-4f3c-a29b-95eb8c22a6d4) succeeded

[tnk-ysk]

I tested #30516, it works as expected

[phanikumv]

#30968 works as expected

[2023-05-17, 10:13:51 UTC] {dbt.py:488} INFO - Getting the status of job run 151181329.
[2023-05-17, 10:13:51 UTC] {base.py:73} INFO - Using connection ID 'dbt_test_conn' for task execution.
[2023-05-17, 10:13:52 UTC] {dbt.py:493} INFO - Current status of job run 151181329: SUCCESS
[2023-05-17, 10:13:52 UTC] {taskinstance.py:1345} INFO - Marking task as SUCCESS. dag_id=example_dbt_cloud, task_id=job_run_sensor_async, execution_date=20230517T101254, start_date=20230517T101351, end_date=20230517T101352
[2023-05-17, 10:13:52 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 0

[phanikumv]

#30579, #30939, #30618, #30920, #30901 work as expected

[potiuk]

Checked and marked all mine as ✅ (except #31080 - which looks good but I have no databricks account/config to check it with) . Looks good. @Stormhand I would appreciate if you could check if #31080 is solved for you - you'd need to install https://pypi.org/project/apache-airflow-providers-databricks/4.2.0rc1 (it should pull common-sql 1.5.0rc1 automatically).

[pankajkoti]

Confirmed #31063, #31042, and #31062 (all of them are for Provider amazon: 8.1.0rc1) are included in the RC and they ran fine on my example DAG. Thank you for the efforts!

[utkarsharma2]

Tested - DynamoDBToS3Operator - Add a feature to export the table to a point in time. (#31142) <#31142> With below dag: from datetime import datetime from airflow import DAG from airflow.providers.amazon.aws.transfers.dynamodb_to_s3 import DynamoDBToS3Operator with DAG( dag_id='example_export_dynamodb', schedule_interval=None, start_date=datetime(2021, 1, 1), tags=['example'], catchup=False, ) as dag: dynamodb_to_s3_operator = DynamoDBToS3Operator( task_id="dynamodb_to_s3", dynamodb_table_name="test", s3_bucket_name="tmp9", file_size=4000, export_time=datetime.now(), s3_key_prefix="test1" ) Thanks, Utkarsh Sharma

…

On Thu, May 18, 2023 at 11:13 AM Pankaj Koti ***@***.***> wrote: Confirmed #31063 <#31063>, #31042 <#31042>, and #31062 <#31062> (all of them are for Provider amazon: 8.1.0rc1 <https://pypi.org/project/apache-airflow-providers-amazon/8.1.0rc1>) are included in the RC and they ran fine on my example DAG. Thank you for the efforts! — Reply to this email directly, view it on GitHub <#31322 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDLAHKV4MSGUBN2S2WFJZ3XGWZHDANCNFSM6AAAAAAYD5TKEE> . You are receiving this because you were mentioned.Message ID: ***@***.***>