[BEAM-13008] Create gradle tasks for the Beam Playground #15761

damondouglas · 2021-10-20T17:24:50Z

This PR creates gradle tasks for the Beam Playground project as well as a bundled playgroundPrecommit task. Future work needs to configure the precommit process to install necessary dependencies to properly run the playgroundPrecommit task. Note that when running ./gradlew playgroundPrecommit in the current state will show multiple issues and errors in the playground code.

I'd like to request the following reviewers or whomever they delegate.
R: @pabloem
R: @tvalentyn

./gradlew playground:tasks

./gradlew playground:backend:tasks

./gradlew playground:frontend:tasks

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-10-20T17:48:36Z

Codecov Report

Merging #15761 (b0ff33f) into master (a05aa45) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #15761      +/-   ##
==========================================
- Coverage   83.54%   83.52%   -0.02%     
==========================================
  Files         445      445              
  Lines       61185    61185              
==========================================
- Hits        51114    51107       -7     
- Misses      10071    10078       +7

Impacted Files	Coverage Δ
sdks/python/apache_beam/utils/interactive_utils.py	`92.68% <0.00%> (-2.44%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (-2.24%)`	⬇️
sdks/python/apache_beam/io/source_test_utils.py	`88.47% <0.00%> (-1.39%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`89.25% <0.00%> (-0.17%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.64% <0.00%> (+0.12%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a05aa45...b0ff33f. Read the comment docs.

settings.gradle.kts

build.gradle.kts

pabloem · 2021-10-20T18:43:57Z

have you ran this locally? can you share the output?

damondouglas · 2021-10-20T19:28:23Z

(cc: @pabloem )
The following is the output running:

./gradlew playgroundPrecommit

@ilya-kozyrev would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle. Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the go dependency references.

> Task :playground:backend:tidy FAILED
go: finding module for package beam.apache.org/playground/backend/internal/environment
beam.apache.org/playground/backend/cmd/server imports
        beam.apache.org/playground/backend/internal/environment: cannot find module providing package beam.apache.org/playground/backend/internal/environment: unrecognized import path "beam.apache.org/playground/backend/internal/environment": reading https://beam.apache.org/playground/backend/internal/environment?go-get=1: 404 Not Found

@ElessarST would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle.

For the mockito related errors, I realize there is this SOF post about commiting/notcommiting mockito generated code. I personally am in favor of commiting but I'll leave it to your judgement to figure out a way to make the ./gradlew playgroundPrecommit pass either through another gradle task that would generate these dependencies.
Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the dart dependency references.

> Task :playground:frontend:analyze FAILED

  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:21:8 - Target of URI doesn't exist: 'package:playground/generated/playground.pbgrpc.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:31:8 - Target of URI doesn't exist: 'code_repository_test.mocks.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:54:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:87:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:121:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function

13 issues found.

damondouglas · 2021-10-20T20:02:01Z

build.gradle.kts

-    "playground/frontend/assets/**/*.svg",
-    "playground/frontend/assets/**/*.png",
-    "playground/frontend/assets/**/*.jpg",
+    "playground/frontend/**/*.svg",


@ElessarST @ilya-kozyrev (cc: @pabloem ) what is the license for these SVGs?

pabloem · 2021-10-20T21:48:57Z

LGTM. This is wonderful.

pabloem · 2021-10-20T21:49:04Z

Run Java PreCommit

ElessarST · 2021-10-21T09:39:09Z

@damondouglas I created PR with frontend changes damondouglas#2

ilya-kozyrev · 2021-10-21T13:30:27Z

(cc: @pabloem ) The following is the output running:

./gradlew playgroundPrecommit

@ilya-kozyrev would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle. Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the go dependency references.

> Task :playground:backend:tidy FAILED
go: finding module for package beam.apache.org/playground/backend/internal/environment
beam.apache.org/playground/backend/cmd/server imports
        beam.apache.org/playground/backend/internal/environment: cannot find module providing package beam.apache.org/playground/backend/internal/environment: unrecognized import path "beam.apache.org/playground/backend/internal/environment": reading https://beam.apache.org/playground/backend/internal/environment?go-get=1: 404 Not Found

@ElessarST would you mind to fix the following and make a PR to https://github.com/damondouglas/beam/tree/BEAM-13008-playground-build-gradle.

For the mockito related errors, I realize there is this SOF post about commiting/notcommiting mockito generated code. I personally am in favor of commiting but I'll leave it to your judgement to figure out a way to make the ./gradlew playgroundPrecommit pass either through another gradle task that would generate these dependencies.
Also you'll want to confirm that the ./gradlew playground:generateProto task either generates in the right place or change the dart dependency references.

> Task :playground:frontend:analyze FAILED

  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:21:8 - Target of URI doesn't exist: 'package:playground/generated/playground.pbgrpc.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:103:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:105:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:107:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
  error - lib/modules/editor/repository/code_repository/code_client/grpc_code_client.dart:109:12 - Case expressions must be constant. - non_constant_case_expression
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:31:8 - Target of URI doesn't exist: 'code_repository_test.mocks.dart'. Try creating the file referenced by the URI, or Try using a URI for a file that does exist. - uri_does_not_exist
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:54:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:87:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function
  error - test/modules/editor/repository/code_repository/code_repository_test.dart:121:22 - The function 'MockCodeClient' isn't defined. Try importing the library that defines 'MockCodeClient', correcting the name to the name of an existing function, or defining a function named 'MockCodeClient'. - undefined_function

13 issues found.

@damondouglas Regarding this error

> Task :playground:backend:tidy FAILED
go: finding module for package beam.apache.org/playground/backend/internal/environment
beam.apache.org/playground/backend/cmd/server imports
        beam.apache.org/playground/backend/internal/environment: cannot find module providing package beam.apache.org/playground/backend/internal/environment: unrecognized import path "beam.apache.org/playground/backend/internal/environment": reading https://beam.apache.org/playground/backend/internal/environment?go-get=1: 404 Not Found

It happened because we are still waiting for #15654 to get merged. Can we merge Environments first of all to resolve it?

damondouglas · 2021-10-21T16:09:10Z

@pabloem

./gradlew playgroundPrecommit

has the following output:

BUILD SUCCESSFUL in 21s
5 actionable tasks: 5 executed

pabloem · 2021-10-21T16:16:55Z

LGTM. Merging.

… for the Beam Playground * Implement Beam Playground gradle tasks * Fix trailing whitespace * Remove format from precommit * Fix order of includes and add descriptions * [BEAM-13008] add generated files and fix analysis issues (apache#2) * Reconcile errors with playgroundPrecommit output Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>

damondouglas added 2 commits October 20, 2021 17:17

Implement Beam Playground gradle tasks

efd1ff4

Fix trailing whitespace

b940ca0

pabloem reviewed Oct 20, 2021

View reviewed changes

settings.gradle.kts Outdated Show resolved Hide resolved

build.gradle.kts Outdated Show resolved Hide resolved

damondouglas added 2 commits October 20, 2021 19:36

Remove format from precommit

f7fc854

Fix order of includes and add descriptions

b8d9cd2

damondouglas commented Oct 20, 2021

View reviewed changes

tvalentyn approved these changes Oct 21, 2021

View reviewed changes

ElessarST and others added 2 commits October 21, 2021 07:46

[BEAM-13008] add generated files and fix analysis issues (#2)

9e240cf

Reconcile errors with playgroundPrecommit output

b0ff33f

pabloem merged commit dc0548e into apache:master Oct 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-13008] Create gradle tasks for the Beam Playground #15761

[BEAM-13008] Create gradle tasks for the Beam Playground #15761

Uh oh!

damondouglas commented Oct 20, 2021 •

edited

Loading

Uh oh!

codecov bot commented Oct 20, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

damondouglas commented Oct 20, 2021 •

edited

Loading

Uh oh!

damondouglas Oct 20, 2021

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

ElessarST commented Oct 21, 2021

Uh oh!

ilya-kozyrev commented Oct 21, 2021

Uh oh!

damondouglas commented Oct 21, 2021

Uh oh!

pabloem commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[BEAM-13008] Create gradle tasks for the Beam Playground #15761

[BEAM-13008] Create gradle tasks for the Beam Playground #15761

Uh oh!

Conversation

damondouglas commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov bot commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

damondouglas commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damondouglas Oct 20, 2021

Choose a reason for hiding this comment

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

pabloem commented Oct 20, 2021

Uh oh!

ElessarST commented Oct 21, 2021

Uh oh!

ilya-kozyrev commented Oct 21, 2021

Uh oh!

damondouglas commented Oct 21, 2021

Uh oh!

pabloem commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

damondouglas commented Oct 20, 2021 •

edited

Loading

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Oct 20, 2021 •

edited

Loading

damondouglas commented Oct 20, 2021 •

edited

Loading