[BEAM-12735] Adding Python XLang examples to the RC validation script #15307

ihji · 2021-08-10T18:30:59Z

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

ihji · 2021-08-10T18:32:07Z

R: @chamikaramj

codecov · 2021-08-10T18:48:33Z

Codecov Report

Merging #15307 (8c6fafd) into master (2fd9875) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #15307      +/-   ##
==========================================
- Coverage   83.81%   83.79%   -0.03%     
==========================================
  Files         441      441              
  Lines       59745    59801      +56     
==========================================
+ Hits        50075    50109      +34     
- Misses       9670     9692      +22

Impacted Files	Coverage Δ
sdks/python/apache_beam/utils/interactive_utils.py	`87.80% <0.00%> (-7.32%)`	⬇️
...ks/python/apache_beam/runners/worker/data_plane.py	`87.70% <0.00%> (-2.90%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (-2.24%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`90.33% <0.00%> (-0.38%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`88.85% <0.00%> (-0.16%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.51% <0.00%> (-0.13%)`	⬇️
sdks/python/apache_beam/io/avroio.py	`60.60% <0.00%> (ø)`
sdks/python/apache_beam/io/textio.py	`97.07% <0.00%> (ø)`
sdks/python/apache_beam/io/tfrecordio.py	`93.39% <0.00%> (ø)`
sdks/python/apache_beam/transforms/ptransform.py	`93.54% <0.00%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2fd9875...8c6fafd. Read the comment docs.

chamikaramj

Thanks.

chamikaramj · 2021-08-15T16:50:09Z

release/src/main/scripts/run_rc_validation.sh

  echo "You don't have gnome-terminal installed."
-  if [[ "$INSTALL_GNOME_TERMINAL" != true ]]; then
-    sudo apt-get upgrade
+  if [[ "$INSTALL_GNOME_TERMINAL" = true ]]; then


The condition here was inverted. Did we have a bug before ?

I believe so. The software needs to be installed when the variable is true.

chamikaramj · 2021-08-15T16:52:29Z

release/src/main/scripts/run_rc_validation.sh

+  if [[ "$INSTALL_KUBECTL" = true ]]; then
+    sudo apt-get install kubectl
+  else
+    echo "kubectl is not installed. Validation on Python cross-language Kafka taxi will be skipped."


Should this be a failure for validation instead of skipping ?

Looks like we already exit the program but only the printed message is misleading. Update the messages.

chamikaramj · 2021-08-15T17:40:03Z

release/src/main/scripts/run_rc_validation.sh

+  CLUSTER_NAME=xlang-kafka-cluster-$RANDOM
+  if [[ "$python_xlang_kafka_taxi_dataflow" = true ]]; then
+    gcloud container clusters create --project=${USER_GCP_PROJECT} --region=${USER_GCP_REGION} --no-enable-ip-alias $CLUSTER_NAME
+    kubectl apply -R -f ${LOCAL_BEAM_DIR}/.test-infra/kubernetes/kafka-cluster


Did you confirm that works. I think we Beam Kafka IT is currently failing due to a port issue when starting up this cluster: https://issues.apache.org/jira/browse/BEAM-9482

It worked with clouddfe project. I think there was no other program on clouddfe project using the same port assigned to k8s Kafka cluster.

We should get it to a state where the release manager can consistently run the test using the default (apache-beam-testing) project. (I'm not sure if you'll actually hit https://issues.apache.org/jira/browse/BEAM-9482 or not).

Yes, but it's a separate work. We need to update k8s configs for dynamically assigning the ports.

chamikaramj · 2021-08-15T17:41:18Z

release/src/main/scripts/run_rc_validation.sh

+      --runner DataflowRunner \
+      --num_workers 5 \
+      --temp_location=${USER_GCS_BUCKET}/temp/ \
+      --experiments=use_runner_v2 \


You should not need to manually specify this experiment for Beam 2.32.0 and later.

chamikaramj · 2021-08-15T17:42:01Z

release/src/main/scripts/run_rc_validation.sh

+      --runner DataflowRunner \
+      --num_workers 5 \
+      --temp_location=${USER_GCS_BUCKET}/temp/ \
+      --experiments=use_runner_v2 \


Ditto regarding experiment.

chamikaramj · 2021-08-15T17:43:01Z

release/src/main/scripts/run_rc_validation.sh

+      echo "* How to verify results:"
+      echo "* 1. Goto your Dataflow job console and check whether there is any error."
+      echo "* 2. Check whether your ${SQL_TAXI_SUBSCRIPTION} subscription has data below:"
+      # run twice since the first execution would return 0 messages


Any idea why ?

No idea. I found that sometimes gcloud pubsub pull command just returns empty result (mostly when the first pull command after the subscription creation). Supposedly, this on-screen outputs only provide the hint to the release manager that any data exists in the sink. Visiting the web console might be needed if the hint doesn't help.

chamikaramj · 2021-08-15T17:44:03Z

release/src/main/scripts/run_rc_validation.sh

+      sleep 10m
+      echo "* How to verify results:"
+      echo "* 1. Goto your Dataflow job console and check whether there is any error."
+      echo "* 2. Check whether ${KAFKA_TAXI_DF_DATASET}.xlang_kafka_taxi has data, retrieving BigQuery data as below: "


Will it be possible to run a 'grep' to confirm that the output is not empty ?

I don't think it would be reliably possible since the data is constantly changing. Manual review is still important not only for this tests but also other existing validations.

Data changes but can we just verify that the output is not empty ? Based on my observation there's always some output data for these pipelines after few minutes.

chamikaramj · 2021-08-18T18:02:59Z

Thanks. LGTM.

chamikaramj · 2021-08-18T18:03:27Z

Retest this please

ihji · 2021-08-18T20:08:22Z

Run Python_PVR_Flink PreCommit

ihji · 2021-08-18T20:08:56Z

Run Java PreCommit

[BEAM-12735] Adding Python XLang examples to the RC validation script

de65e36

chamikaramj reviewed Aug 15, 2021

View reviewed changes

ihji added 2 commits August 16, 2021 10:52

update

d59fdd0

check xlang test outputs

8c6fafd

ihji merged commit 7587508 into apache:master Aug 18, 2021

[BEAM-12735] Adding Python XLang examples to the RC validation script #15307

[BEAM-12735] Adding Python XLang examples to the RC validation script #15307

Uh oh!

Conversation

ihji commented Aug 10, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

ihji commented Aug 10, 2021

Uh oh!

codecov bot commented Aug 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj commented Aug 18, 2021

Uh oh!

chamikaramj commented Aug 18, 2021

Uh oh!

ihji commented Aug 18, 2021

Uh oh!

ihji commented Aug 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Aug 10, 2021 •

edited

Loading