Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Mar 31, 2023

We are temporarily switching to vendored-in google ads library
in order to handle the clash of two issues:

  • Google ads switching off Google Ads v11 API
  • Google provider clinging on to old version of google-ads client
    due to protobuf <4 requirement from other google clients

Strategic solution being worked out in #30067

Based on #30399 (so only last commit counts)


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk requested a review from kaxil as a code owner March 31, 2023 18:43
@boring-cyborg boring-cyborg bot added area:dev-tools area:providers provider:google Google (including GCP) related issues labels Mar 31, 2023
@potiuk potiuk force-pushed the use-vendored-in-google-ads branch from 762a8fd to 7961795 Compare March 31, 2023 18:53
@potiuk
Copy link
Member Author

potiuk commented Mar 31, 2023

cc: @lwyszomi

BTW. vendoring-in google-ads immmediately frees a lot of other dependencies in our constraints

< # This constraints file was automatically generated on 2023-03-31T06:33:43Z
---
> # This constraints file was automatically generated on 2023-03-31T19:06:44Z
43a44
> Shapely==1.8.5.post1
188,189c189
< google-ads==18.0.0
< google-api-core==2.8.2
---
> google-api-core==2.11.0
194,204c194,204
< google-cloud-aiplatform==1.16.1
< google-cloud-appengine-logging==1.1.3
< google-cloud-audit-log==0.2.4
< google-cloud-automl==2.8.0
< google-cloud-bigquery-datatransfer==3.7.0
< google-cloud-bigquery-storage==2.14.1
< google-cloud-bigquery==2.34.4
< google-cloud-bigtable==2.11.1
< google-cloud-build==3.9.0
< google-cloud-compute==0.7.0
< google-cloud-container==2.11.1
---
> google-cloud-aiplatform==1.23.0
> google-cloud-appengine-logging==1.3.0
> google-cloud-audit-log==0.2.5
> google-cloud-automl==2.11.1
> google-cloud-bigquery-datatransfer==3.11.1
> google-cloud-bigquery-storage==2.19.1
> google-cloud-bigquery==3.9.0
> google-cloud-bigtable==2.17.0
> google-cloud-build==3.14.0
> google-cloud-compute==1.11.0
> google-cloud-container==2.18.0
206,213c206,213
< google-cloud-datacatalog==3.9.0
< google-cloud-dataflow-client==0.5.4
< google-cloud-dataform==0.2.0
< google-cloud-dataplex==1.1.0
< google-cloud-dataproc-metastore==1.6.0
< google-cloud-dataproc==5.0.0
< google-cloud-dlp==3.8.0
< google-cloud-kms==2.12.0
---
> google-cloud-datacatalog==3.12.0
> google-cloud-dataflow-client==0.8.3
> google-cloud-dataform==0.5.1
> google-cloud-dataplex==1.4.3
> google-cloud-dataproc-metastore==1.10.1
> google-cloud-dataproc==5.4.1
> google-cloud-dlp==3.12.1
> google-cloud-kms==2.16.1
215,222c215,222
< google-cloud-logging==3.2.1
< google-cloud-memcache==1.4.1
< google-cloud-monitoring==2.11.0
< google-cloud-orchestration-airflow==1.4.1
< google-cloud-os-login==2.7.1
< google-cloud-pubsub==2.13.5
< google-cloud-redis==2.9.0
< google-cloud-resource-manager==1.6.0
---
> google-cloud-logging==3.5.0
> google-cloud-memcache==1.7.1
> google-cloud-monitoring==2.14.2
> google-cloud-orchestration-airflow==1.7.1
> google-cloud-os-login==2.9.1
> google-cloud-pubsub==2.15.2
> google-cloud-redis==2.12.1
> google-cloud-resource-manager==1.9.1
227c227
< google-cloud-tasks==2.10.1
---
> google-cloud-tasks==2.13.1
232c232
< google-cloud-workflows==1.7.1
---
> google-cloud-workflows==1.10.1
235c235
< googleapis-common-protos==1.56.4
---
> googleapis-common-protos==1.59.0
239c239
< grpc-google-iam-v1==0.12.4
---
> grpc-google-iam-v1==0.12.6
330c330
< neo4j==5.6.0
---
> neo4j==5.7.0
349c349
< oracledb==1.2.2
---
> oracledb==1.3.0
355c355
< pandas-gbq==0.17.9
---
> pandas-gbq==0.19.1
383,384c383,384
< proto-plus==1.19.6
< protobuf==3.20.0
---
> proto-plus==1.22.2
> protobuf==3.20.3
420c420
< pytest-instafail==0.4.2
---
> pytest-instafail==0.5.0

Let's see if the tests pass.,

@potiuk
Copy link
Member Author

potiuk commented Mar 31, 2023

I might want to add some limits back to limit the risk (seems that some static checks are failing with the new versions, so we might want to limit them until #30067 is completed.

@potiuk
Copy link
Member Author

potiuk commented Mar 31, 2023

Yep. Tests are failing so I will likely add the limits back.

@potiuk potiuk force-pushed the use-vendored-in-google-ads branch from 7961795 to fe19710 Compare March 31, 2023 19:45
@potiuk
Copy link
Member Author

potiuk commented Mar 31, 2023

Actually - it's not that bad - just one test really faiked and rest was one search/replace missing I think. Let's see.

@potiuk potiuk force-pushed the use-vendored-in-google-ads branch 3 times, most recently from 422ab33 to 46e2714 Compare April 1, 2023 07:36
@potiuk potiuk changed the title Switch to using vendored-in google ads. Bump dependencies of google provider after ads removal Apr 1, 2023
@potiuk potiuk marked this pull request as draft April 1, 2023 08:01
@potiuk potiuk force-pushed the use-vendored-in-google-ads branch from 46e2714 to bbe84b6 Compare April 1, 2023 08:04
We are temporarily switching to vendored-in google ads library
in order to handle the clash of two issues:

* Google ads switching off Google Ads v11 API
* Google provider clinging on to old version of google-ads client
  due to protobuf <4 requirement from other google clients

Strategic solution being worked out in apache#30067
@potiuk potiuk force-pushed the use-vendored-in-google-ads branch from bbe84b6 to 648830d Compare April 1, 2023 08:45
@potiuk
Copy link
Member Author

potiuk commented Apr 2, 2023

Closing this one with preference of implementing the right changes as part of #30067

@potiuk potiuk closed this Apr 2, 2023
@lwyszomi
Copy link
Contributor

lwyszomi commented Apr 3, 2023

cc: @lwyszomi

BTW. vendoring-in google-ads immmediately frees a lot of other dependencies in our constraints

< # This constraints file was automatically generated on 2023-03-31T06:33:43Z
---
> # This constraints file was automatically generated on 2023-03-31T19:06:44Z
43a44
> Shapely==1.8.5.post1
188,189c189
< google-ads==18.0.0
< google-api-core==2.8.2
---
> google-api-core==2.11.0
194,204c194,204
< google-cloud-aiplatform==1.16.1
< google-cloud-appengine-logging==1.1.3
< google-cloud-audit-log==0.2.4
< google-cloud-automl==2.8.0
< google-cloud-bigquery-datatransfer==3.7.0
< google-cloud-bigquery-storage==2.14.1
< google-cloud-bigquery==2.34.4
< google-cloud-bigtable==2.11.1
< google-cloud-build==3.9.0
< google-cloud-compute==0.7.0
< google-cloud-container==2.11.1
---
> google-cloud-aiplatform==1.23.0
> google-cloud-appengine-logging==1.3.0
> google-cloud-audit-log==0.2.5
> google-cloud-automl==2.11.1
> google-cloud-bigquery-datatransfer==3.11.1
> google-cloud-bigquery-storage==2.19.1
> google-cloud-bigquery==3.9.0
> google-cloud-bigtable==2.17.0
> google-cloud-build==3.14.0
> google-cloud-compute==1.11.0
> google-cloud-container==2.18.0
206,213c206,213
< google-cloud-datacatalog==3.9.0
< google-cloud-dataflow-client==0.5.4
< google-cloud-dataform==0.2.0
< google-cloud-dataplex==1.1.0
< google-cloud-dataproc-metastore==1.6.0
< google-cloud-dataproc==5.0.0
< google-cloud-dlp==3.8.0
< google-cloud-kms==2.12.0
---
> google-cloud-datacatalog==3.12.0
> google-cloud-dataflow-client==0.8.3
> google-cloud-dataform==0.5.1
> google-cloud-dataplex==1.4.3
> google-cloud-dataproc-metastore==1.10.1
> google-cloud-dataproc==5.4.1
> google-cloud-dlp==3.12.1
> google-cloud-kms==2.16.1
215,222c215,222
< google-cloud-logging==3.2.1
< google-cloud-memcache==1.4.1
< google-cloud-monitoring==2.11.0
< google-cloud-orchestration-airflow==1.4.1
< google-cloud-os-login==2.7.1
< google-cloud-pubsub==2.13.5
< google-cloud-redis==2.9.0
< google-cloud-resource-manager==1.6.0
---
> google-cloud-logging==3.5.0
> google-cloud-memcache==1.7.1
> google-cloud-monitoring==2.14.2
> google-cloud-orchestration-airflow==1.7.1
> google-cloud-os-login==2.9.1
> google-cloud-pubsub==2.15.2
> google-cloud-redis==2.12.1
> google-cloud-resource-manager==1.9.1
227c227
< google-cloud-tasks==2.10.1
---
> google-cloud-tasks==2.13.1
232c232
< google-cloud-workflows==1.7.1
---
> google-cloud-workflows==1.10.1
235c235
< googleapis-common-protos==1.56.4
---
> googleapis-common-protos==1.59.0
239c239
< grpc-google-iam-v1==0.12.4
---
> grpc-google-iam-v1==0.12.6
330c330
< neo4j==5.6.0
---
> neo4j==5.7.0
349c349
< oracledb==1.2.2
---
> oracledb==1.3.0
355c355
< pandas-gbq==0.17.9
---
> pandas-gbq==0.19.1
383,384c383,384
< proto-plus==1.19.6
< protobuf==3.20.0
---
> proto-plus==1.22.2
> protobuf==3.20.3
420c420
< pytest-instafail==0.4.2
---
> pytest-instafail==0.5.0

Let's see if the tests pass.,

yes, it will upgrade other SDK, because google-ads forced us to for example install protobuf<4. When you removed it from providers.yaml we don't have that restriction and SDK can be installed in latests version without conflicts.

@potiuk
Copy link
Member Author

potiuk commented Apr 3, 2023

yes, it will upgrade other SDK, because google-ads forced us to for example install protobuf<4. When you removed it from providers.yaml we don't have that restriction and SDK can be installed in latests version without conflicts.

It's not even protobuf (yet). Those are other limits (I think limitation google ads put on google-api-core mainly) that holds other libraries back. I decided to keep the limitations back because of potentialy small breaking changes (I had to fix a few tests and even make a few changes to make the tests pass and I did not feel comfortable doing that at the same time, especiallly as I know you are fixing those in #30067 so it's better to release all those there.

For now. I've added those limits here:

https://github.com/apache/airflow/blob/main/airflow/providers/google/provider.yaml#L80

  # needed by vendored-in google-ads
  - googleapis-common-protos<2.0.0,>=1.5.8
  - google-api-core==2.8.2
  - google-auth-oauthlib<1.0.0,>=0.3.0
  - grpcio<2.0.0,>=1.38.1
  - grpcio-status<2.0.0,>=1.38.1
  - PyYAML<7.0,>=5.1
  - proto-plus==1.19.6
  - protobuf!=3.18.*,!=3.19.*,<=3.20.0,>=3.12.0

So just a watchout you will have to remove those when you work on #30067

@potiuk potiuk deleted the use-vendored-in-google-ads branch June 4, 2023 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants