Skip to content

Conversation

@Abacn
Copy link
Contributor

@Abacn Abacn commented Oct 4, 2022

Follow up of #21198

Following Python37 PostCommit integration tests on Dataflow fails after #23365.

apache_beam.io.gcp.experimental.spannerio_write_it_test.SpannerWriteIntegrationTest.test_spanner_update
apache_beam.io.gcp.experimental.spannerio_write_it_test.SpannerWriteIntegrationTest.test_write_batches

They were not detected by the time of merging because the same tests on direct runner succeeded, and the test suite experienced timeout and did not report dataflow results, until recently (Oct 3) the test suite somehow no longer timeout.

The fail on Dataflow runner is due to mismatch of the version of cloud-spanner between pipeline construction and worker container, then no-such-module exception thrown at coder_impl at pipeline execution.

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@Abacn
Copy link
Contributor Author

Abacn commented Oct 4, 2022

Run Python 3.7 PostCommit

@Abacn
Copy link
Contributor Author

Abacn commented Oct 4, 2022

R: @tvalentyn

@Abacn
Copy link
Contributor Author

Abacn commented Oct 4, 2022

Trigger phrase seems not working

@github-actions
Copy link
Contributor

github-actions bot commented Oct 4, 2022

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@codecov
Copy link

codecov bot commented Oct 4, 2022

Codecov Report

Merging #23480 (2cd1c31) into master (72237d6) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #23480      +/-   ##
==========================================
- Coverage   73.45%   73.44%   -0.01%     
==========================================
  Files         718      718              
  Lines       95878    95876       -2     
==========================================
- Hits        70425    70420       -5     
- Misses      24142    24145       +3     
  Partials     1311     1311              
Flag Coverage Δ
python 83.19% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...python/apache_beam/runners/worker/worker_status.py 74.66% <0.00%> (-2.00%) ⬇️
sdks/python/apache_beam/io/localfilesystem.py 90.97% <0.00%> (-0.76%) ⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py 88.62% <0.00%> (-0.48%) ⬇️
sdks/python/apache_beam/ml/inference/base.py 95.65% <0.00%> (-0.19%) ⬇️
...thon/apache_beam/ml/inference/pytorch_inference.py 0.00% <0.00%> (ø)
...hon/apache_beam/ml/inference/tensorrt_inference.py 0.00% <0.00%> (ø)
...hon/apache_beam/runners/worker/bundle_processor.py 93.42% <0.00%> (+0.12%) ⬆️
...thon/apache_beam/ml/inference/sklearn_inference.py 95.77% <0.00%> (+0.32%) ⬆️
...che_beam/runners/interactive/interactive_runner.py 91.77% <0.00%> (+1.26%) ⬆️
.../python/apache_beam/testing/test_stream_service.py 92.85% <0.00%> (+4.76%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Abacn Abacn force-pushed the pycontainerspanner branch from 4b7f358 to c370fe9 Compare October 4, 2022 16:31
google-cloud-pubsublite==1.5.0
google-cloud-recommendations-ai==0.7.1
google-cloud-spanner==1.19.3
google-cloud-spanner==3.22.0
Copy link
Contributor

@tvalentyn tvalentyn Oct 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests fail with:
09:43:29 google-cloud-spanner 3.22.0 requires sqlparse, which is not installed.

these files need to be regenerated (see header at the top of the file).

Also, I think it would be safer to set the upper bound >=3.0.0 in setup.py.

@Abacn
Copy link
Contributor Author

Abacn commented Oct 4, 2022

Having issues on running ./gradlew :sdks:python:container:py38:generatePythonRequirements, get error

ERROR: Could not find a version that satisfies the requirement google-python-cloud-debugger (from versions: none)
ERROR: No matching distribution found for google-python-cloud-debugger

Seems like it only supports linux. Needs a linux machine to run the script.

Also, found google-python-cloud-debugger is set to shutdown by May 31,2023: https://cloud.google.com/debugger/docs/release-notes

entered #23487

# google-cloud-spanner 2.x causes dependency parsing backoff
'google-cloud-spanner>=1.13.0,!=2,<4',
'google-cloud-spanner>=3.0.0,<4',
'grpcio-gcp>=0.2.2,<1',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also a spanner v1 dep.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, introduced in #9606. Will remove

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reran generate requirements

@Abacn
Copy link
Contributor Author

Abacn commented Oct 4, 2022

Run Python_PVR_Flink PreCommit

@Abacn
Copy link
Contributor Author

Abacn commented Oct 4, 2022

Run Python 3.7 PostCommit

@Abacn
Copy link
Contributor Author

Abacn commented Oct 5, 2022

postcommit test run out of quota again: "Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/38753 instances, 2/1210 CPUs, 25/344105 disk GB, 0/758 SSD disk GB, 1/271 instance groups, 1/274 managed instance groups, 1/468 instance templates, 1/0 in-use IP addresses."

@Abacn
Copy link
Contributor Author

Abacn commented Oct 5, 2022

Run Python 3.7 PostCommit

@github-actions github-actions bot added the build label Oct 5, 2022
@Abacn
Copy link
Contributor Author

Abacn commented Oct 5, 2022

PostCommit with only SpannerIO tests passed: https://ci-beam.apache.org/job/beam_PostCommit_Python37_PR/462/testReport/

test_spanner_error | 11 min | Fixed
test_spanner_error | 5 sec | Passed
test_spanner_update | 8 min 45 sec | Fixed
test_spanner_update | 7.4 sec | Passed
test_write_batches | 7 sec | Passed
test_write_batches | 11 min | Fixed

@github-actions github-actions bot removed the build label Oct 5, 2022
@Abacn
Copy link
Contributor Author

Abacn commented Oct 5, 2022

Run Python_PVR_Flink PreCommit

@tvalentyn tvalentyn merged commit b9aa159 into apache:master Oct 5, 2022
@Abacn Abacn deleted the pycontainerspanner branch October 5, 2022 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants