Status of testing Providers that were prepared on August 19, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.28.0rc1

• Add incremental export and cross account export functionality in DynamoDBToS3Operator (#41304): @Ghoul-SSZ
  Linked issues:
  • Linked Issue #40737: @Kuhlmann-Itagyba-bah
• EKS Overrides for AWS Batch submit_job (#40718): @ssilb4
• Fix AwsTaskLogFetcher missing logs (#41515): @vincbeck
  Linked issues:
  • Linked Issue #40875: @yaningz
• Issue-41243 Unpin the moto dependency (#41256): @vikramaditya91
  Linked issues:
  • Linked Issue #41243: @potiuk
• Fix RedshiftDataOperator not running in deferred mode when it should (#41206): @borismo
• Partial fix for example_dynamodb_to_s3.py (#41517): @ferruzzi
• Remove deprecated code is AWS provider (#41407): @vincbeck
  Linked issues:
  • Linked Issue #41396: @eladkal
• Limit moto temporarily - 5.0.12 is breaking our tests (#41244): @potiuk

Provider apache.spark: 4.10.0rc1

• fix: resolve AirflowProviderDeprecationWarning in spark provider (#41358): @phi-friday
  Linked issues:

Provider celery: 3.8.0rc1

• Remove deprecated SubDags (#41390): @kaxil

Provider cncf.kubernetes: 8.4.0rc1

• Remove deprecated SubDags (#41390): @kaxil
• spark kubernetes operator arguments description reordering (#41372): @dirrao

Provider common.sql: 1.16.0rc1

• fix: rm deprecated import in common.sql (#41461): @phi-friday
  Linked issues:
  • Linked Issue #41460: @phi-friday

Provider docker: 3.13.0rc1

• feat(docker): Replace use_dill with serializer (#41356): @phi-friday

Provider elasticsearch: 5.5.0rc1

• add execute_sql API to fix ElasticSearchSQLHook (#41537): @Owen-CH-Leung
  Linked issues:
  • Linked Issue #41486: @Pooort

Provider fab: 1.3.0rc1

• Feature: Allow set Dag Run resource into Dag Level permission (#40703): @joaopamaral
• Remove deprecated SubDags (#41390): @kaxil

Provider google: 10.22.0rc1

• Add CloudRunServiceHook and CloudRunCreateServiceOperator (#40008): @jx2lee
  Linked issues:
  • Linked Issue #38760: @ophie200
• add missing sync_hook_class to CloudDataTransferServiceAsyncHook (#41417): @Lee-W
• Refactor DataprocCreateBatchOperator and Dataproc system tests (#41527): @moiseenkov
• Upgrade package gcloud-aio-auth>=5.2.0 (#41262): @moiseenkov

Provider microsoft.azure: 10.4.0rc1

• Microsoft Power BI operator to refresh the dataset (#40356): @ambika-garg
• Export Azure Container Instance log messages to XCOM (#41142): @perry2of5

Provider microsoft.mssql: 3.9.0rc1

• Add methodtools as dependency to mssql provider (#41392): @potiuk
  Linked issues:
  • Linked Issue #41330: @rawwar

Provider openlineage: 1.11.0rc1

• feat: [openlineage] add debug facet to all events (#41217): @kacpermuda
• feat: add fileloc to DAG info in AirflowRunFacet (#41311): @kacpermuda
• chore: remove openlineage client deprecated from_environment() method (#41310): @kacpermuda
• fix: get task dependencies without serializing task tree to string (#41494): @mobuchowski
• fix: return empty data instead of None when OpenLineage on_start method is missing (#41268): @kacpermuda

Provider papermill: 3.8.0rc1

• restore python 3.12 support for papermill (#41548): @morokosi

Provider snowflake: 5.7.0rc1

• Fix: Pass hook parameters to SnowflakeSqlApiHook and prep them for AP… (#41150): @BTeclaw
  Linked issues:
  • Linked Issue #39622: @alexcsolomon1

Provider yandex: 3.12.0rc1

• providers/yandex: fix typing (#40997): @got686-yandex

Provider ydb: 1.3.0rc1

• ydb provider: add database to table name in bulk upsert, use bulk upsert in system test (#41303): @uzhastik

All users involved in the PRs:
@morokosi @mobuchowski @borismo @dirrao @phi-friday @Ghoul-SSZ @joaopamaral @BTeclaw @potiuk @vikramaditya91 @uzhastik @ambika-garg @Owen-CH-Leung @moiseenkov @ssilb4 @got686-yandex @kaxil @vincbeck @le

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[moiseenkov]

Hi,
#41527, #41262 work as expected

[sc250072]

Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?

https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpypi.org%2Fproject%2Fapache-airflow-providers-teradata%2F2.6.0rc1%2F&data=05%7C02%7CSATISH.CHINTHANIPPU%40teradata.com%7Cc798de64676b411f9b7a08dcc0279df7%7C9151cbaafc6b4f4889bb8c4a71982138%7C0%7C0%7C638596523614516170%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=uL1wmpzD4EDpAJZPPvDPVbqC%2BbuDSIhwniZPqKyMQ3M%3D&reserved=0

[phi-friday]

1. feat(docker): Replace use_dill with serializer #41356 works fine.
2. fix: resolve AirflowProviderDeprecationWarning in spark provider #41358 I didn't use spark so I can't check, but given that _sql is simply an alias for sql, it should be fine.
3. fix: rm deprecated import in common.sql #41461 I couldn't find a good way to verify common.sql, but since we've only removed the part that generates a more detailed error message, it should be fine.

[potiuk]

Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?

According to our rules - periodically we bump all providers min-airflow version. https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow -> then we relaease all providers with min-airflow version bumped - we also remove all pre-min-airflow backports, this allows to keep airflow providers free from back-compatibility issues.

[potiuk]

Checked that all my changes are in.

[jx2lee]

@eladkal
#40008 works fine! (unittest & example run)

[Owen-CH-Leung]

@eladkal the ElasticSearchSQLHook is now working as expected.

[vikramaditya91]

#41256: @vikramaditya91

Works fine

[kacpermuda]

#41494 tried to fix the OOM error in the scheduler that the OpenLineage can cause when generating a dag_tree from a huge DAG (related issue: #41505). It works but we've just got information about another production case where the scheduler went OOM with another complex DAG. There is a fix prepared in #41587 that will remove the dag_tree entirely so there will no more errors like this. I'd like to request an rc2 for OpenLineage provider (@eladkal) that will include that fix, as it is a bug that can cause some problems in bigger deployments.

[joaopamaral]

Tested #40703 with both access_control formats and it's working fine:

[eladkal]

@kacpermuda I will exclude openlineage from this release