[SPARK-43348][PYTHON] Support `Python 3.8` in PyPy3 #41024

dongjoon-hyun · 2023-05-02T23:53:32Z

What changes were proposed in this pull request?

This PR aims two goals.

Make PySpark support Python 3.8+ with PyPy3
Upgrade PyPy3 to Python 3.8 in our GitHub Action Infra Image to enable test coverage

Note that there was one failure at test_create_dataframe_from_pandas_with_day_time_interval test case. This PR skips the test case and SPARK-43354 will recover it after further investigation.

Why are the changes needed?

Previously, PySpark fails at PyPy3 Python 3.8 environment.

pypy3 version is: Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:13)
[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
Starting test(pypy3): pyspark.sql.tests.pandas.test_pandas_cogrouped_map (temp output: /__w/spark/spark/python/target/f1cacde7-d369-48cf-a8ea-724c42872020/pypy3__pyspark.sql.tests.pandas.test_pandas_cogrouped_map__rxih6dqu.log)
Traceback (most recent call last):
  File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/__w/spark/spark/python/pyspark/__init__.py", line 59, in <module>
    from pyspark.rdd import RDD, RDDBarrier
  File "/__w/spark/spark/python/pyspark/rdd.py", line 54, in <module>
    from pyspark.java_gateway import local_connect_and_auth
  File "/__w/spark/spark/python/pyspark/java_gateway.py", line 32, in <module>
    from pyspark.serializers import read_int, write_with_length, UTF8Deserializer
  File "/__w/spark/spark/python/pyspark/serializers.py", line 69, in <module>
    from pyspark import cloudpickle
  File "/__w/spark/spark/python/pyspark/cloudpickle/__init__.py", line 1, in <module>
    from pyspark.cloudpickle.cloudpickle import *  # noqa
  File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle.py", line 56, in <module>
    from .compat import pickle
  File "/__w/spark/spark/python/pyspark/cloudpickle/compat.py", line 13, in <module>
    from _pickle import Pickler  # noqa: F401
ModuleNotFoundError: No module named '_pickle'

To support Python 3.8 in PyPy3.

From PyPy3.8, _pickle is removed.
- import Pickler class directly from the pickle module cloudpipe/cloudpickle#458
We need this change.
- compat: Import Pickler from "pickle" instead of "_pickle" cloudpipe/cloudpickle#469

Does this PR introduce any user-facing change?

This is an additional support.

How was this patch tested?

Pass the CIs.

dongjoon-hyun · 2023-05-03T01:41:23Z

Thank you, @HyukjinKwon .

dongjoon-hyun · 2023-05-03T03:45:29Z

Thank you, @Yikun .

dongjoon-hyun · 2023-05-03T04:30:21Z

python/pyspark/cloudpickle/compat.py

-        from pickle import _Pickler as Pickler  # noqa: F401
-else:
-    import pickle  # noqa: F401
-    from _pickle import Pickler  # noqa: F401


From PyPy Python3.8, _pickle is removed.

import Pickler class directly from the pickle module cloudpipe/cloudpickle#458 (comment)

dongjoon-hyun · 2023-05-03T04:31:18Z

python/pyspark/cloudpickle/compat.py

-    import pickle  # noqa: F401
-    from _pickle import Pickler  # noqa: F401
+import pickle  # noqa: F401
+from pickle import Pickler  # noqa: F401


This is the same with the upstream.

compat: Import Pickler from "pickle" instead of "_pickle" cloudpipe/cloudpickle#469

dongjoon-hyun · 2023-05-03T04:47:55Z

Could you review once more, @HyukjinKwon and @Yikun .
I found that we didn't support Python 3.8+ in PyPy3 environment.
I converted this PR from INFRA to PYTHON module in both the JIRA and here.

dongjoon-hyun · 2023-05-03T08:10:59Z

Thank you, @HyukjinKwon !

dongjoon-hyun · 2023-05-03T14:01:51Z

Merged to master. Thank you all!
The last commit was disabling a single test case.

…st only with PyPy 3.8 ### What changes were proposed in this pull request? This PR is a followup of #41024 that skips the test only with PyPy 3.8. ### Why are the changes needed? To narrow the scope of testing skipped. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? CI in this PR should verify the change. Closes #41085 from HyukjinKwon/SPARK-43354-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

### What changes were proposed in this pull request? This PR aims two goals. 1. Make PySpark support Python 3.8+ with PyPy3 2. Upgrade PyPy3 to Python 3.8 in our GitHub Action Infra Image to enable test coverage Note that there was one failure at `test_create_dataframe_from_pandas_with_day_time_interval` test case. This PR skips the test case and SPARK-43354 will recover it after further investigation. ### Why are the changes needed? Previously, PySpark fails at PyPy3 `Python 3.8` environment. ``` pypy3 version is: Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:13) [PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] Starting test(pypy3): pyspark.sql.tests.pandas.test_pandas_cogrouped_map (temp output: /__w/spark/spark/python/target/f1cacde7-d369-48cf-a8ea-724c42872020/pypy3__pyspark.sql.tests.pandas.test_pandas_cogrouped_map__rxih6dqu.log) Traceback (most recent call last): File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 188, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 111, in _get_module_details __import__(pkg_name) File "/__w/spark/spark/python/pyspark/__init__.py", line 59, in <module> from pyspark.rdd import RDD, RDDBarrier File "/__w/spark/spark/python/pyspark/rdd.py", line 54, in <module> from pyspark.java_gateway import local_connect_and_auth File "/__w/spark/spark/python/pyspark/java_gateway.py", line 32, in <module> from pyspark.serializers import read_int, write_with_length, UTF8Deserializer File "/__w/spark/spark/python/pyspark/serializers.py", line 69, in <module> from pyspark import cloudpickle File "/__w/spark/spark/python/pyspark/cloudpickle/__init__.py", line 1, in <module> from pyspark.cloudpickle.cloudpickle import * # noqa File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle.py", line 56, in <module> from .compat import pickle File "/__w/spark/spark/python/pyspark/cloudpickle/compat.py", line 13, in <module> from _pickle import Pickler # noqa: F401 ModuleNotFoundError: No module named '_pickle' ``` To support Python 3.8 in PyPy3. - From PyPy3.8, `_pickle` is removed. - cloudpipe/cloudpickle#458 - We need this change. - cloudpipe/cloudpickle#469 ### Does this PR introduce _any_ user-facing change? This is an additional support. ### How was this patch tested? Pass the CIs. Closes apache#41024 from dongjoon-hyun/SPARK-43348. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…st only with PyPy 3.8 ### What changes were proposed in this pull request? This PR is a followup of apache#41024 that skips the test only with PyPy 3.8. ### Why are the changes needed? To narrow the scope of testing skipped. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? CI in this PR should verify the change. Closes apache#41085 from HyukjinKwon/SPARK-43354-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

[SPARK-43348][INFRA] Upgrade PyPy to Python 3.8

8dfee0a

github-actions bot added the BUILD label May 2, 2023

dongjoon-hyun changed the title ~~[SPARK-43348][INFRA] Upgrade PyPy to Python 3.8~~ [SPARK-43348][INFRA] Upgrade PyPy to Python 3.8 in GitHub Action Infra Image May 2, 2023

HyukjinKwon approved these changes May 3, 2023

View reviewed changes

HyukjinKwon requested a review from Yikun May 3, 2023 00:57

Yikun approved these changes May 3, 2023

View reviewed changes

Fix

d0eea7b

github-actions bot added CORE PYTHON labels May 3, 2023

dongjoon-hyun commented May 3, 2023

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-43348][INFRA] Upgrade PyPy to Python 3.8 in GitHub Action Infra Image~~ [SPARK-43348][INFRA] Support Python 3.8 in PyPy3 and test in GitHub Action Infra Image May 3, 2023

dongjoon-hyun changed the title ~~[SPARK-43348][INFRA] Support Python 3.8 in PyPy3 and test in GitHub Action Infra Image~~ [SPARK-43348][INFRA] Support Python 3.8 in PyPy3 and test in GitHub Action May 3, 2023

dongjoon-hyun changed the title ~~[SPARK-43348][INFRA] Support Python 3.8 in PyPy3 and test in GitHub Action~~ [SPARK-43348][PYTHOIN] Support Python 3.8 in PyPy3 and test in GitHub Action May 3, 2023

dongjoon-hyun changed the title ~~[SPARK-43348][PYTHOIN] Support Python 3.8 in PyPy3 and test in GitHub Action~~ [SPARK-43348][PYTHOIN] Support Python 3.8 in PyPy3 May 3, 2023

dongjoon-hyun changed the title ~~[SPARK-43348][PYTHOIN] Support Python 3.8 in PyPy3~~ [SPARK-43348][PYTHON] Support Python 3.8 in PyPy3 May 3, 2023

dongjoon-hyun mentioned this pull request May 3, 2023

[SPARK-43347][PYTHON] Remove Python 3.7 Support #41023

Closed

HyukjinKwon approved these changes May 3, 2023

View reviewed changes

Create SPARK-43354

7f96222

github-actions bot added the SQL label May 3, 2023

dongjoon-hyun closed this in 90a3a7e May 3, 2023

dongjoon-hyun deleted the SPARK-43348 branch May 3, 2023 14:01

HyukjinKwon mentioned this pull request May 8, 2023

[SPARK-43348][PYTHON][TESTS][FOLLOW-UP] Skip the day-time-interval test only with PyPy 3.8 #41085

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-43348][PYTHON] Support `Python 3.8` in PyPy3 #41024

[SPARK-43348][PYTHON] Support `Python 3.8` in PyPy3 #41024

Uh oh!

dongjoon-hyun commented May 2, 2023 •

edited

Loading

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

dongjoon-hyun May 3, 2023 •

edited

Loading

Uh oh!

dongjoon-hyun May 3, 2023 •

edited

Loading

Uh oh!

dongjoon-hyun commented May 3, 2023 •

edited

Loading

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-43348][PYTHON] Support Python 3.8 in PyPy3 #41024

[SPARK-43348][PYTHON] Support Python 3.8 in PyPy3 #41024

Uh oh!

Conversation

dongjoon-hyun commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

dongjoon-hyun May 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

dongjoon-hyun commented May 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-43348][PYTHON] Support `Python 3.8` in PyPy3 #41024

[SPARK-43348][PYTHON] Support `Python 3.8` in PyPy3 #41024

dongjoon-hyun commented May 2, 2023 •

edited

Loading

dongjoon-hyun May 3, 2023 •

edited

Loading

dongjoon-hyun May 3, 2023 •

edited

Loading

dongjoon-hyun commented May 3, 2023 •

edited

Loading