Add YAML Editor and Visualization Panel

[Chenzo1001]

This PR introduces a YAML editor and visualization tool that allows users to directly edit, validate, and explore YAML-based pipeline configurations within the platform, under issue #35772. The editor is built on Monaco, with features like font size control, word wrap, dry-run mode, error console, and the ability to upload/download YAML files.

On top of that, the PR adds a visualization panel using React Flow with auto-layout (Dagre). YAML transforms are represented as nodes with emojis, animated edges, and direct links to the corresponding Apache Beam YAML documentation. This significantly improves the usability of YAML workflows by providing both immediate feedback and intuitive visual representation.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @Chenzo1001, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the JupyterLab side panel for Apache Beam users by introducing a comprehensive YAML editor and an interactive pipeline visualization tool. The new editor, built on Monaco, provides a rich editing experience with features like syntax highlighting, real-time validation, and file management. Complementing this, a React Flow-based visualization panel dynamically renders YAML-defined pipelines as interactive graphs, complete with auto-layout and visual cues. This integration aims to streamline the process of creating, validating, and understanding Apache Beam YAML configurations, offering immediate visual feedback and improving overall developer productivity.

Highlights

• YAML Editor Integration: A new Monaco-based YAML editor is integrated into the JupyterLab side panel, offering features such as adjustable font size, word wrap, dry-run mode, an error console, and the ability to upload and download YAML files.
• Pipeline Visualization: A visualization panel powered by React Flow is introduced, which graphically represents YAML transforms as interactive nodes with emojis and animated edges, utilizing Dagre for automatic layout.
• Real-time Validation and Feedback: The system provides real-time validation and immediate feedback on YAML syntax and pipeline configuration errors, enhancing the development experience.
• Enhanced Usability: The combination of an interactive editor and an intuitive visual representation significantly improves the usability and understanding of YAML-based Apache Beam pipeline configurations.
• New Python Utility for YAML Processing: A new Python utility (yaml_parse_utils.py) is added to handle the core logic of parsing Beam YAML, validating the pipeline structure, and converting it into a format suitable for graph visualization.
• Frontend Dependency Additions: Key frontend dependencies including @monaco-editor/react, @xyflow/react, dagre, lodash, and react-split have been added to support the new editor and visualization functionalities.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[github-actions]

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[liferoad]

Overall, looks good. Left few comments. Will ask Gemini to do another review. Please also fix the failed workflows.

[liferoad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant new feature: a YAML editor and visualization panel. The frontend work with React, Monaco, and React Flow is extensive and well-structured. The backend Python script for parsing the YAML is a good start but has some critical flaws. My review focuses on improving the robustness and completeness of the implementation. Key issues include a potential crash in the Python parser for empty pipelines, incorrect graph generation logic that only supports linear pipelines, unsafe data passing from the frontend to the Python kernel, and some incomplete UI functionality (the properties panel is read-only). I've also pointed out several minor issues like leftover console logs and type mismatches. Addressing these points will make the new feature more reliable and fully functional.

[liferoad]

Please check the failed workflows. @Chenzo1001

[Chenzo1001]

@liferoad All checks are passed except a few exceptions.

[liferoad]

--- /runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/apache_beam_jupyterlab_sidepanel/yaml_parse_utils.py:before	2025-09-05 16:23:14.785409
+++ /runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/apache_beam_jupyterlab_sidepanel/yaml_parse_utils.py:after	2025-09-05 16:30:48.032341
@@ -10,14 +10,18 @@
 #  License for the specific language governing permissions and limitations under
 #  the License.
 
+import dataclasses
 import json
-from typing import Dict, List, TypedDict, Any
-import dataclasses
 from dataclasses import dataclass
+from typing import Any
+from typing import Dict
+from typing import List
+from typing import TypedDict
 
+import yaml
+
+import apache_beam as beam
 from apache_beam.yaml.main import build_pipeline_components_from_yaml
-import apache_beam as beam
-import yaml
 
 # ======================== Type Definitions ========================
 
lint: exit 1 (128.59 seconds) /runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python> time /runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/scripts/run_pylint.sh pid=1410
lint: commands_post[0]> bash /runner/_work/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
  lint: FAIL code 1 (453.80=setup[323.50]+cmd[0.01,0.24,0.87,0.03,0.52,128.59,0.03] seconds)
  evaluation failed :( (453.98 seconds)

[liferoad]

> Task :validateChanges FAILED
Found template start at line 19
Found template end at line 60
Template section: lines 19 to 60
Found unreleased section at line 62: # [2.69.0] - Unreleased
Starting validation from line 63
Checking line 66: * New highly anticipated feature X added to Python SDK ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Checking line 67: * New highly anticipated feature Y added to Java SDK ([#Y](https://github.com/apache/beam/issues/Y)).
  No bracketed language reference found
  Has issue link: true
Checking line 68: * [Python] Add YAML Editor and Visualization Panel ([#35947](https://github.com/apache/beam/pull/35947)).
  Found language reference: [Python] at position 2
  Is part of link: false
  Adding error: Line 68: Language references should use parentheses () instead of brackets []: * [Python] Add YAML Editor and Visualization Panel ([#35947](https://github.com/apache/beam/pull/35947)).
  Has issue link: false
  Adding error: Line 68: Missing or malformed issue link. Each entry should end with ([#X](https://github.com/apache/beam/issues/X)): * [Python] Add YAML Editor and Visualization Panel ([#35947](https://github.com/apache/beam/pull/35947)).
Checking line 72: * Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Checking line 76: * X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Checking line 80: * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Checking line 84: * X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Checking line 88: * Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Checking line 92: * ([#X](https://github.com/apache/beam/issues/X)).
  No bracketed language reference found
  Has issue link: true
Found 2 errors

[Incubating] Problems report is available at: file:///runner/_work/beam/beam/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

[liferoad]

will merge after all the workflows are green.

[Chenzo1001]

@liferoad All workflows are passed.