Extract Go and Python Beam symbols for Playground #23378

alexeyinkin · 2022-09-27T09:11:03Z

This PR:

Adds a Gradle task to do all post-checkout tasks.
Adds a Python script to extract Beam symbols from Python SDK.
Adds a Gradle task to call that script and to place the symbols so that the frontend app can use them.
Updates a frontend Dockerfile to do the same.
Updates the README explaining those changes.

The symbols are not yet used by the app after loading. The app will start using them when the Akvelon editor replaces the old editor. Still this PR is needed for easier testing of the Akvelon editor before the switch and to simplify. It also simplifies post-checkout tasks.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

playground/frontend/playground_components/lib/src/services/symbols/loaders/yaml.dart

playground/frontend/playground_components/lib/src/services/symbols/symbols_notifier.dart

playground/frontend/playground_components/lib/src/playground_components.dart

playground/frontend/playground_components/lib/src/services/symbols/loaders/yaml.dart

playground/frontend/playground_components/test/src/services/symbols/symbols_notifier_test.dart

playground/frontend/playground_components/test/src/services/symbols/loaders/yaml_test.dart

alexeyinkin · 2022-09-29T08:44:20Z

R: @damondouglas

github-actions · 2022-09-29T08:45:04Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

damondouglas · 2022-10-03T21:23:09Z

playground/frontend/README.md

-cd ..
-flutter pub get
-flutter pub run build_runner build
+$ ./gradlew :playground:frontend:configure


Can we remove the $? It makes it more convenient to copy/paste for users consuming the README instructions. There are many others below as well.

I tried this command in two different environments and received two different errors. In the first environment, I cloned and checked out the branch in the Google Cloud shell. This led to several incremental retries with the message showing below.

Attempting `./gradlew :playground:frontend:configure` leads to `pub get failed (server unavailable) -- attempting retry 7 in 64 seconds...`

However, when I navigated to the playground/frontend directory and ran pub get, I didn't have any problems except for this message:

Git error. Command: `git clone --mirror https://github.com/BertrandBev/code_field.git /home/<user>/.pub-cache/git/cache/code_field-4fdb625753a3dad07e371ca34045120ddd9a3f88 stderr: Cloning into bare repository '/home/<user>/.pub-cache/git/cache/code_field-4fdb625753a3dad07e371ca34045120ddd9a3f88'...

So I attempted a second environment which was on my own linux machine.

Running the command in this context, yielded the following error.

FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':playground:frontend:flutterClean'. > A problem occurred starting process 'command 'flutter''

I ran the command again as ./gradlew :playground:frontend:configure --stacktrace and it yielded:

java.io.IOException: Cannot run program "flutter" (in directory "/playground/frontend"): error=2, No such file or directory at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25) ... 6 more Caused by: java.io.IOException: error=2, No such file or directory

However, running flutter --version, I see:

Flutter 3.3.2 • channel stable • https://github.com/flutter/flutter.git Framework • revision e3c29ec00c (3 weeks ago) • 2022-09-14 08:46:55 -0500 Engine • revision a4ff2c53d8 Tools • Dart 2.18.1 • DevTools 2.15.0

With Cloud Shell, I narrowed down the problem and filed an issue:
https://issuetracker.google.com/issues/251039262

Can you please check if git clone --mirror also fails for you? If so, please upvote the issue and maybe provide additional details there (project ID?).

As for the local machine, I cannot reproduce this, and I cannot really see how is Flutter in $PATH for the terminal but not in $PATH for Gradle ran from that same terminal. Can you show the output of these?

cat /etc/*-release

which flutter

echo $PATH

./gradlew :playground:frontend:printPath (a new task I just added)

Running the above commands will show I tested on Linux Debian and that flutter is in my path. I don't want to delay this PR further and thank you for this information. However, running ./gradlew :playground:frontend:configure resulted in a Module not found yaml error which I resolved by installing the missing python library. Now ./gradlew :playground:frontend:configure --stacktrace reveals this error:

> Task :playground:frontend:playground_components:extractBeamSymbolsPython FAILED Traceback (most recent call last): File "playground/frontend/playground_components/tools/extract_symbols_python/extract_symbols_python.py", line 109, in <module> class_names = get_dir_symbols_recursive(args.dir) File "playground/frontend/playground_components/tools/extract_symbols_python/extract_symbols_python.py", line 100, in get_dir_symbols_recursive class_names.update(get_file_symbols(file_path)) File "playground/frontend/playground_components/tools/extract_symbols_python/extract_symbols_python.py", line 66, in get_file_symbols [target] = member_node.targets ValueError: too many values to unpack (expected 1)

damondouglas · 2022-10-03T22:04:06Z

playground/frontend/README.md

+### Docker

-`$ flutter pub run build_runner build`
+To run the frontend app with Docker, run this in the frontend directory:


Is Docker the only way to run this application? Is there any way that we can include instructions for developers who want to contribute, but who are prevented for various reasons from using Docker?

May we consider adding instructions for these developers, including where it is expected to execute flutter pub run build_runner build such as in the playground_components folder? I'm assuming such command runs in the context of the gradle command to setup the project, however, please see the errors I encountered when executing this setup command.

Docker is not required. The recommended way to run this locally is still flutter run.
I updated the README stating the cases when one might prefer Docker.

damondouglas · 2022-10-03T22:12:24Z

...ground/frontend/playground_components/tools/extract_symbols_python/extract_python_symbols.py

+from typing import List
+
+
+def should_include_class_node(class_node: ast.ClassDef) -> bool:


Is there a plan for unit tests for this method as well as the other methods?

Done. I made a test for the entire command in Dart.

First I thought about testing Python code in Python, but it has a drawback. We will have 3 more scripts for this job in other languages, very much different inside but with an identical API. In Dart we can make 4 nearly identical tests instead of 4 very different native tests on those languages. So the Dart solution is easier to maintain.

…#23304)

apache#23304)

github-actions · 2022-10-13T12:13:56Z

Reminder, please take a look at this pr: @Abacn

damondouglas

@pabloem I wasn't able to fully review this PR but didn't want to hold it back.

Pros:

flutter code is readable and flutter run results in a working frontend
flutter test passes

Opportunities:

Not clear whether https://github.com/apache/beam/blob/master/playground/frontend/README.md?plain=1#L32-L37 will remain after this PR will merge and personally I think it helps to have these instructions
./gradlew :playground:frontend:configure kept throwing errors and I can't rule out whether it is my execution environment

damondouglas · 2022-10-14T16:14:22Z

playground/frontend/README.md

-cd playground_components
-flutter pub get
-flutter pub run build_runner build
-cd ..
-flutter pub get
-flutter pub run build_runner build


Does this PR still intend on deleting https://github.com/apache/beam/blob/master/playground/frontend/README.md?plain=1#L32-L37?

I deleted these lines because with the introduction of symbols extraction the building got more complicated. That is why :playground:frontend:configure was introduced.

With the recent idea of keeping generated files under version control, :playground:frontend:configure may remain as simple as the above 6 commands. This means it can hardly break, so I guess it should be safe to delete them from the REAMDE. Otherwise we will have two locations to maintain the build commands:

:playground:frontend:configure Gradle task.

README.

Yet the failure of Gradle on your machine with Flutter in path is a concern.

Do you want to keep those commands as a backup?

alexeyinkin · 2022-10-14T17:49:51Z

It's OK to hold this PR until everything builds on any machine. This does not block us because meanwhile we are busy on the editor.

We expect to migrate to Akvelon editor next week. This means we will no longer have a git dependency which means Google Cloud Shell will start working.

Meanwhile I will fix the Python script using your feedback and list pyyaml in the README.

github-actions · 2022-10-19T12:56:37Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2022-10-27T12:14:26Z

Reminder, please take a look at this pr: @damccorm

damccorm · 2022-10-27T13:01:06Z

stop reviewer notifications

github-actions · 2022-10-27T13:02:51Z

Stopping reviewer notifications for this pull request: requested by reviewer

pabloem · 2022-11-08T17:52:40Z

can you rebase this PR please @alexeyinkin ?

# Conflicts: # .gitignore # playground/frontend/build.gradle # playground/frontend/playground_components/build.gradle.kts

alexeyinkin · 2022-11-09T21:05:23Z

@pabloem I merged master.

pabloem · 2022-11-10T20:25:14Z

oh I realize this was not lgtmd? @damondouglas should I review this? merge? waqit for your review?

alexeyinkin · 2022-11-11T03:49:31Z

@damondouglas now that we use the editor from pub.dev instead of git, we do not hit the bug of Google Cloud Shell, and it works all the way from ./gradlew :frontend:playground:configure to web preview.

alexeyinkin · 2022-11-11T14:08:48Z

playground/frontend/playground_components/pubspec.yaml

    - assets/translations/en.yaml

 flutter_gen:
-  output: lib/src/generated/


**/generated/* is git-ignored on the repository level, so we evade that.

alexeyinkin · 2022-11-11T14:16:42Z

Meanwhile I also did

[Task]: Version-control all generated Playground files #23534
in this PR so as not to be blocked by this.

* Extract Beam symbols for Go SDK (apache#23917) * Fix after review, add tests (apache#23917) * Fix tests, lazy symbols loading on SDK switch (apache#23917)

# Conflicts: # learning/tour-of-beam/frontend/pubspec.lock # playground/frontend/playground_components/lib/src/controllers/playground_controller.dart # playground/frontend/playground_components/lib/src/controllers/snippet_editing_controller.dart

damondouglas

@alexeyinkin Would the following remain an issue if we merged this PR?

1. I noticed that the text for some of the buttons/dropdowns were misaligned:

and

2. Possibly related to 1. There appears to be a size overflow error.

…23304)

alexeyinkin · 2022-11-18T02:12:15Z

@damondouglas how did you build and run it to get that? My tests:

1. Cloud Shell

cp playground/frontend/lib/config.example.dart playground/frontend/lib/config.g.dart
cd playground/frontend/
docker build -t playground-frontend .
docker run -p 8080:8080 playground-frontend

And then Web Preview:

Flutter 3.3.2, Chrome.

2. Local debug with Android Studio

cp playground/frontend/lib/config.example.dart playground/frontend/lib/config.g.dart

Flutter 3.3.7, Chrome.

damondouglas · 2022-11-18T15:54:45Z

Good thank you. I executed the commands locally in the playground/frontend/README on a Linux Debian machine. I'm not remoting into the device currently to inform the flutter version but updated prior to execution. I'd say then this PR is good to go.

damondouglas

@pabloem LGTM

…3304)

pabloem · 2022-11-23T01:10:44Z

lgtm thanks

alexeyinkin added 5 commits September 27, 2022 10:56

Extract and load Beam symbols (apache#23304)

1ce69af

Move a file (apache#23304)

f6fc091

Move a file (apache#23304)

aefff80

Update Playground frontend README (apache#23304)

ec00f47

Update Playground fronten pubspec.lock (apache#23304)

aa4c857

github-actions bot added the build label Sep 27, 2022

Add the licence (apache#23304)

90745a3

Malarg suggested changes Sep 27, 2022

View reviewed changes

alexeyinkin added 2 commits September 28, 2022 14:11

Fix after review (apache#23304)

cb92462

Add tests for SymbolsNotifier and YamlSymbolsLoader (apache#23304)

0fc90c0

alexeyinkin requested a review from Malarg September 28, 2022 14:34

Malarg suggested changes Sep 29, 2022

View reviewed changes

Fix after review (apache#23304)

2ee38a5

alexeyinkin requested a review from Malarg September 29, 2022 07:38

Malarg approved these changes Sep 29, 2022

View reviewed changes

alexeyinkin marked this pull request as ready for review September 29, 2022 08:43

github-actions bot added the Next Action: Reviewers label Sep 29, 2022

alexeyinkin mentioned this pull request Oct 3, 2022

[Tour of Beam][Frontend] Content Tree and SDK models (#23316) #23417

Merged

4 tasks

damondouglas requested changes Oct 3, 2022

View reviewed changes

alexeyinkin added 2 commits October 5, 2022 08:57

Update README, move symbols from python.yaml to python.g.yaml (apache…

6816b10

…#23304)

Test extract_symbols_python.py, exclude generated files from rat check (

5a352cd

apache#23304)

github-actions bot added the slow-review label Oct 13, 2022

damondouglas reviewed Oct 14, 2022

View reviewed changes

github-actions bot removed the slow-review label Oct 19, 2022

github-actions bot added the slow-review label Oct 27, 2022

github-actions bot removed the slow-review label Oct 27, 2022

alexeyinkin added 2 commits November 9, 2022 18:23

Merge branch 'master' into pg_23304_beam-symbols

a65a067

# Conflicts: # .gitignore # playground/frontend/build.gradle # playground/frontend/playground_components/build.gradle.kts

Pass extracted symbols to the editor (apache#23304)

c825b78

Version-control generated files of Playground Frontend (apache#23534)

92e1b22

github-actions bot added the learning label Nov 11, 2022

alexeyinkin commented Nov 11, 2022

View reviewed changes

Minor fixes (apache#23534)

d64e54c

alexeyinkin requested a review from damondouglas November 11, 2022 14:17

alexeyinkin and others added 5 commits November 16, 2022 19:10

Extract Beam symbols for Go SDK (apache#23917) (#326)

4099de5

* Extract Beam symbols for Go SDK (apache#23917) * Fix after review, add tests (apache#23917) * Fix tests, lazy symbols loading on SDK switch (apache#23917)

Add newly generated files, add config.example.dart (apache#23304)

619b3f6

Fix (apache#23304)

9f7c02c

gofmt extract_symbols_go.go (apache#23304)

688781c

damondouglas reviewed Nov 17, 2022

View reviewed changes

Fix pythonG and goG falling out of generated asset constants (apache#…

c469e92

…23304)

damondouglas approved these changes Nov 18, 2022

View reviewed changes

alexeyinkin changed the title ~~Extract Python Beam symbols for Playground~~ Extract Go and Python Beam symbols for Playground Nov 21, 2022

Update flutter_code_editor to v0.1.9, pale read-only blocks (apache#2…

624292a

…3304)

pabloem merged commit 730b391 into apache:master Nov 23, 2022

		from typing import List


		def should_include_class_node(class_node: ast.ClassDef) -> bool:

Extract Go and Python Beam symbols for Playground #23378

Extract Go and Python Beam symbols for Playground #23378

Uh oh!

Conversation

alexeyinkin commented Sep 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexeyinkin commented Sep 29, 2022

Uh oh!

github-actions bot commented Sep 29, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damondouglas Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 13, 2022

Uh oh!

damondouglas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexeyinkin commented Oct 14, 2022

Uh oh!

github-actions bot commented Oct 19, 2022

Uh oh!

github-actions bot commented Oct 27, 2022

Uh oh!

damccorm commented Oct 27, 2022

Uh oh!

github-actions bot commented Oct 27, 2022

Uh oh!

pabloem commented Nov 8, 2022

Uh oh!

alexeyinkin commented Nov 9, 2022

Uh oh!

pabloem commented Nov 10, 2022

Uh oh!

alexeyinkin commented Nov 11, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexeyinkin commented Nov 11, 2022

Uh oh!

damondouglas left a comment

Choose a reason for hiding this comment

1. I noticed that the text for some of the buttons/dropdowns were misaligned:

alexeyinkin commented Sep 27, 2022 •

edited

Loading

damondouglas Oct 3, 2022 •

edited

Loading