[Failing Test]: JS2PY not supporting Python 3.12

[tvalentyn]

What happened?

Beam YAML added added a dependency on js2py library:

beam/sdks/python/apache_beam/yaml/yaml_mapping.py

Line 34 in eb5c73d

import js2py

It currently doesn't work for Python 3.12:

• Python 3.11: RuntimeError: Your python version made changes to the bytecode PiotrDabkowski/Js2Py#282 (comment)
• KeyError in python3.12.0 PiotrDabkowski/Js2Py#317

This means that YAML functionality might not be available in Python 3.12 until the issue is resolved. It seems that there has been no releases to Js2Py since 2022.

How critical is this dependency for Beam YAML? @Polber would you have time to investigate whether this dependency is still required and/or work with Js2Py maintainers on adding Python 3.12 support?

Thanks!

To repro, we need to make some patches to Beam repo like ones in #30828, create a py3.12 envrionment, run:

(py312) :python$ pytest apache_beam/yaml/programming_guide_test.py
=========================================================== test session starts ============================================================
platform linux -- Python 3.12.2, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/valentyn/projects/beam/beam/beam/sdks/python
configfile: pytest.ini
plugins: xdist-3.5.0, requests-mock-1.12.1, timeout-2.3.1, hypothesis-6.100.0
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 0 items / 1 error                                                                                                                

================================================================== ERRORS ==================================================================
_______________________________________ ERROR collecting apache_beam/yaml/programming_guide_test.py ________________________________________
/home/valentyn/.pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:995: in exec_module
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
apache_beam/yaml/__init__.py:18: in <module>
    from apache_beam.yaml.yaml_transform import *
apache_beam/yaml/yaml_transform.py:39: in <module>
    from apache_beam.yaml.yaml_combine import normalize_combine
apache_beam/yaml/yaml_combine.py:33: in <module>
    from apache_beam.yaml import yaml_mapping
apache_beam/yaml/yaml_mapping.py:34: in <module>
    import js2py
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/__init__.py:72: in <module>
    from .base import PyJsException
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/base.py:2965: in <module>
    @Js
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/base.py:165: in Js
    return PyJsFunction(val, FunctionPrototype)
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/base.py:1377: in __init__
    cand = fix_js_args(func)
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:27: in fix_js_args
    code = append_arguments(six.get_function_code(func), ('this', 'arguments'))
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:121: in append_arguments
    arg = name_translations[inst.arg]
E   KeyError: 3
============================================================= warnings summary =============================================================
apache_beam/__init__.py:74
  /home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/__init__.py:74: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.12. You may encounter bugs or missing features.
    warnings.warn(

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

../../../../../../.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:141
  /home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:141: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    code_obj.co_firstlineno, code_obj.co_lnotab,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= short test summary info ==========================================================
ERROR apache_beam/yaml/programming_guide_test.py - KeyError: 3

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Polber]

There are actually some known performance issues with that library that were recently discovered after running some load tests. I was planning on revisiting and possibly using a different library/solution for JavaScript UDF's. It was safeguarded by marking as "experimental" so I see no reason in disabling the feature for the time being.

[tvalentyn]

I see no reason in disabling the feature for the time being

Ok. we could not install the dependency for certain Python versions by adding a version guard, and change the YAML functionality to support Javascript UDF if the necessary libraries are imported.

[Polber]

That makes sense to me. I added that logic in: #30843