Skip to content

Bump google-cloud-spanner version for py containers #23480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tvalentyn merged 6 commits into from
Oct 5, 2022
Merged

Bump google-cloud-spanner version for py containers #23480

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c370fe9
Bump google-cloud-spanner version for py containers
Abacn Oct 4, 2022
a5da891
Pin spanner client version to 3.x
Abacn Oct 4, 2022
e701608
regenerate base_image_requirements
Abacn Oct 4, 2022
71fb187
Remove grpcio-gcp dependency
Abacn Oct 4, 2022
a3158ea
[Test only] only run spanner test for postcommit
Abacn Oct 5, 2022
2cd1c31
Revert "[Test only] only run spanner test for postcommit"
Abacn Oct 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
* Decreased TextSource CPU utilization by 2.3x (Java) ([#23193](https://github.com/apache/beam/issues/23193)).
* Fixed bug when using SpannerIO with RuntimeValueProvider options (Java) ([#22146](https://github.com/apache/beam/issues/22146)).
* Fixed issue for unicode rendering on WriteToBigQuery ([#10785](https://github.com/apache/beam/issues/10785))
* Bumped google-cloud-spanner dependency version to 3.x for Python SDK ([#21198](https://github.com/apache/beam/issues/21198)).

## New Features / Improvements

Expand All @@ -78,6 +79,8 @@
## Bugfixes

* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
* Fixed Python cross-language JDBC IO Connector cannot read or write rows containing Numeric/Decimal type values ([#19817](https://github.com/apache/beam/issues/19817)).

## Known Issues

* ([#X](https://github.com/apache/beam/issues/X)).
Expand Down
24 changes: 12 additions & 12 deletions sdks/python/container/py37/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ beautifulsoup4==4.11.1
bs4==0.0.1
CacheControl==0.12.11
cachetools==4.2.4
certifi==2022.9.14
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
Expand All @@ -44,17 +44,17 @@ execnet==1.9.0
fastavro==1.6.1
fasteners==0.18
firebase-admin==5.2.0
flatbuffers==2.0.7
flatbuffers==22.9.24
freezegun==1.2.2
future==0.18.2
gast==0.4.0
google-api-core==2.8.2
google-api-python-client==2.62.0
google-api-python-client==2.64.0
google-apitools==0.5.31
google-auth==2.11.1
google-auth==2.12.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-bigquery==3.3.2
google-cloud-bigquery==3.3.3
google-cloud-bigquery-storage==2.13.2
google-cloud-bigtable==1.7.2
google-cloud-core==2.3.2
Expand All @@ -66,26 +66,25 @@ google-cloud-profiler==3.1.0
google-cloud-pubsub==2.13.7
google-cloud-pubsublite==1.5.0
google-cloud-recommendations-ai==0.7.1
google-cloud-spanner==1.19.3
google-cloud-spanner==3.22.0
Copy link
Contributor

@tvalentyn tvalentyn Oct 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests fail with:
09:43:29 google-cloud-spanner 3.22.0 requires sqlparse, which is not installed.

these files need to be regenerated (see header at the top of the file).

Also, I think it would be safer to set the upper bound >=3.0.0 in setup.py.

google-cloud-storage==2.5.0
google-cloud-videointelligence==1.16.3
google-cloud-vision==1.0.2
google-crc32c==1.5.0
google-pasta==0.2.0
google-python-cloud-debugger==3.1
google-resumable-media==2.3.3
google-resumable-media==2.4.0
googleapis-common-protos==1.56.4
greenlet==1.1.3
grpc-google-iam-v1==0.12.4
grpcio==1.49.1
grpcio-gcp==0.2.2
grpcio-status==1.48.2
guppy3==3.1.2
h5py==3.7.0
hdfs==2.7.0
httplib2==0.20.4
idna==3.4
importlib-metadata==4.12.0
importlib-metadata==5.0.0
iniconfig==1.1.1
joblib==1.2.0
keras==2.10.0
Expand All @@ -111,7 +110,7 @@ parameterized==0.8.1
pbr==5.10.0
pluggy==1.0.0
proto-plus==1.22.1
protobuf==3.19.5
protobuf==3.19.6
psycopg2-binary==2.9.3
py==1.11.0
pyarrow==9.0.0
Expand All @@ -129,7 +128,7 @@ pytest-timeout==2.1.0
pytest-xdist==2.5.0
python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2022.2.1
pytz==2022.4
PyYAML==6.0
regex==2022.9.13
requests==2.28.1
Expand All @@ -141,8 +140,9 @@ scipy==1.7.3
six==1.16.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.41
sqlparse==0.4.3
tenacity==5.1.5
tensorboard==2.10.0
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.0
Expand Down
24 changes: 12 additions & 12 deletions sdks/python/container/py38/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ beautifulsoup4==4.11.1
bs4==0.0.1
CacheControl==0.12.11
cachetools==4.2.4
certifi==2022.9.14
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
Expand All @@ -44,17 +44,17 @@ execnet==1.9.0
fastavro==1.6.1
fasteners==0.18
firebase-admin==5.2.0
flatbuffers==2.0.7
flatbuffers==22.9.24
freezegun==1.2.2
future==0.18.2
gast==0.4.0
google-api-core==2.8.2
google-api-python-client==2.62.0
google-api-python-client==2.64.0
google-apitools==0.5.31
google-auth==2.11.1
google-auth==2.12.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-bigquery==3.3.2
google-cloud-bigquery==3.3.3
google-cloud-bigquery-storage==2.13.2
google-cloud-bigtable==1.7.2
google-cloud-core==2.3.2
Expand All @@ -66,26 +66,25 @@ google-cloud-profiler==3.1.0
google-cloud-pubsub==2.13.7
google-cloud-pubsublite==1.5.0
google-cloud-recommendations-ai==0.7.1
google-cloud-spanner==1.19.3
google-cloud-spanner==3.22.0
google-cloud-storage==2.5.0
google-cloud-videointelligence==1.16.3
google-cloud-vision==1.0.2
google-crc32c==1.5.0
google-pasta==0.2.0
google-python-cloud-debugger==3.1
google-resumable-media==2.3.3
google-resumable-media==2.4.0
googleapis-common-protos==1.56.4
greenlet==1.1.3
grpc-google-iam-v1==0.12.4
grpcio==1.49.1
grpcio-gcp==0.2.2
grpcio-status==1.48.2
guppy3==3.1.2
h5py==3.7.0
hdfs==2.7.0
httplib2==0.20.4
idna==3.4
importlib-metadata==4.12.0
importlib-metadata==5.0.0
iniconfig==1.1.1
joblib==1.2.0
keras==2.10.0
Expand All @@ -111,7 +110,7 @@ parameterized==0.8.1
pbr==5.10.0
pluggy==1.0.0
proto-plus==1.22.1
protobuf==3.19.5
protobuf==3.19.6
psycopg2-binary==2.9.3
py==1.11.0
pyarrow==9.0.0
Expand All @@ -129,7 +128,7 @@ pytest-timeout==2.1.0
pytest-xdist==2.5.0
python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2022.2.1
pytz==2022.4
PyYAML==6.0
regex==2022.9.13
requests==2.28.1
Expand All @@ -141,8 +140,9 @@ scipy==1.9.1
six==1.16.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.41
sqlparse==0.4.3
tenacity==5.1.5
tensorboard==2.10.0
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.0
Expand Down
24 changes: 12 additions & 12 deletions sdks/python/container/py39/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ beautifulsoup4==4.11.1
bs4==0.0.1
CacheControl==0.12.11
cachetools==4.2.4
certifi==2022.9.14
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
Expand All @@ -44,17 +44,17 @@ execnet==1.9.0
fastavro==1.6.1
fasteners==0.18
firebase-admin==5.2.0
flatbuffers==2.0.7
flatbuffers==22.9.24
freezegun==1.2.2
future==0.18.2
gast==0.4.0
google-api-core==2.8.2
google-api-python-client==2.62.0
google-api-python-client==2.64.0
google-apitools==0.5.31
google-auth==2.11.1
google-auth==2.12.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-bigquery==3.3.2
google-cloud-bigquery==3.3.3
google-cloud-bigquery-storage==2.13.2
google-cloud-bigtable==1.7.2
google-cloud-core==2.3.2
Expand All @@ -66,26 +66,25 @@ google-cloud-profiler==3.1.0
google-cloud-pubsub==2.13.7
google-cloud-pubsublite==1.5.0
google-cloud-recommendations-ai==0.7.1
google-cloud-spanner==1.19.3
google-cloud-spanner==3.22.0
google-cloud-storage==2.5.0
google-cloud-videointelligence==1.16.3
google-cloud-vision==1.0.2
google-crc32c==1.5.0
google-pasta==0.2.0
google-python-cloud-debugger==3.1
google-resumable-media==2.3.3
google-resumable-media==2.4.0
googleapis-common-protos==1.56.4
greenlet==1.1.3
grpc-google-iam-v1==0.12.4
grpcio==1.49.1
grpcio-gcp==0.2.2
grpcio-status==1.48.2
guppy3==3.1.2
h5py==3.7.0
hdfs==2.7.0
httplib2==0.20.4
idna==3.4
importlib-metadata==4.12.0
importlib-metadata==5.0.0
iniconfig==1.1.1
joblib==1.2.0
keras==2.10.0
Expand All @@ -111,7 +110,7 @@ parameterized==0.8.1
pbr==5.10.0
pluggy==1.0.0
proto-plus==1.22.1
protobuf==3.19.5
protobuf==3.19.6
psycopg2-binary==2.9.3
py==1.11.0
pyarrow==9.0.0
Expand All @@ -129,7 +128,7 @@ pytest-timeout==2.1.0
pytest-xdist==2.5.0
python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2022.2.1
pytz==2022.4
PyYAML==6.0
regex==2022.9.13
requests==2.28.1
Expand All @@ -141,8 +140,9 @@ scipy==1.9.1
six==1.16.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.41
sqlparse==0.4.3
tenacity==5.1.5
tensorboard==2.10.0
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.0
Expand Down
2 changes: 1 addition & 1 deletion sdks/python/container/run_generate_requirements.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
# 2) A list of dependencies from base_image_requirements_manual.txt, which we
# maintain manually.

# It is recommended to run this script via gralde commands such as:
# It is recommended to run this script via gradle commands such as:
# ./gradlew :sdks:python:container:generatePythonRequirementsAll
# ./gradlew :sdks:python:container:py38:generatePythonRequirements

Expand Down
7 changes: 1 addition & 6 deletions sdks/python/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,9 +285,6 @@ def get_portability_package_data():
'gcp': [
'cachetools>=3.1.0,<5',
'google-apitools>=0.5.31,<0.5.32',
# Transitive dep. Required for google-cloud-spanner v1.
# See: https://github.com/apache/beam/issues/22454
'google-api-core!=2.8.2,<3',
# NOTE: Maintainers, please do not require google-auth>=2.x.x
# Until this issue is closed
# https://github.com/googleapis/google-cloud-python/issues/10566
Expand All @@ -301,9 +298,7 @@ def get_portability_package_data():
'google-cloud-bigquery-storage>=2.6.3,<2.14',
'google-cloud-core>=0.28.1,<3',
'google-cloud-bigtable>=0.31.1,<2',
# google-cloud-spanner 2.x causes dependency parsing backoff
'google-cloud-spanner>=1.13.0,!=2,<4',
'grpcio-gcp>=0.2.2,<1',
'google-cloud-spanner>=3.0.0,<4',
# GCP Packages required by ML functionality
'google-cloud-dlp>=3.0.0,<4',
'google-cloud-language>=1.3.0,<2',
Expand Down