[ci/docker] Move CI + Docker to Python 3.8 [build_base] by krfricke · Pull Request #35040 · ray-project/ray

krfricke · 2023-05-04T13:56:08Z

Why are these changes needed?

Implementing #34863, this PR updates our CI and Docker images to use Python 3.8 as the default Python version.

It also updates CUDA version numbers and removes stale CUDA images.

The compatibility test becomes and explicit Python 3.7 compatibility test. Currently the tests we are running are (with some filters):

python/ray/tests/horovod
python/ray/tests/lightgbm
python/ray/tests/xgboost
python/ray/tests/ray_lightning
python/ray/tests/ml_py37_compat (keras + torch)
python/ray/air
python/ray/train (AIR tests)
python/ray/data (AIR tests)

We may want to add more tests here from other libraries, e.g. rllib (cc @sven1977 @kouroshHakha) and Serve (cc @iycheng @edoakes)

Technical details

Due to a bug in libstdc++ (e.g. tracked here: #33858), the Ray processes sometimes error out. It looks like this occurred rarely in Python 3.7, but after the switch to Python 3.8 we see it more often in some of our tests.

Note that this is an ephemeral failure and not due to specific tests. For instance, we saw test_trial_runner_3.py fail regularly, but it was always different tests. It seemed that just by the sheer amount of tests, and the quick succession of starting/stopping Ray clusters, one of these tests was bound to fail.

For those tests, we introduced an intermediate solution using pytest reruns, namely the --reruns 3 option. This specifies reruns on the subtest level, as opposed to bazels test-suite level. The reruns option was able to make these flaky tests green. However, this undermines our flaky test tracking, as actually flaky tests will be detected less often. After moving to Ubuntu 22 and a new stdlibc++, we should remove these subtest-reruns.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Kai Fricke <kai@anyscale.com>

ollie-iterators · 2023-05-08T14:00:20Z

The link error should be fixed by merging in recent changes from the main branch

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke · 2023-06-13T13:17:11Z

Failing tests are also failing on master

aslonnie

I do not think I am the owner of any of these changes, and you probably do not need my approval. I am generally confused about why many of the changes are required for python version change. I feel that they at least need more explanations to justify.

for example, if we need to change some of the example code when upgrade from python 3.7 to 3.8. should we worry that the examples do not work on python 3.9, 3.10, which we claim to support and have container images that are released already?

I think most of the changes are not really due to python interpreter upgrade, but mostly due to library version changes of ray's dependencies. because the dependency version resolving for each python version will be different for each library (due to the python_version constraints in requirements.txt files), how do we know if ray works on python 3.9 / 3.10 ?

aslonnie · 2023-06-13T13:39:27Z

 ENV PATH "/home/ray/anaconda3/bin:$PATH"
 ARG DEBIAN_FRONTEND=noninteractive
-ARG PYTHON_VERSION=3.7.16
+ARG PYTHON_VERSION=3.8.16


I think they have 3.8.17 already

aslonnie · 2023-06-13T14:16:00Z

    frame = pd.DataFrame()
    for result in results:
-        frame = frame.append(result.metrics_dataframe, ignore_index=True)
+        frame = pd.concat([frame, result.metrics_dataframe], ignore_index=True)


what change caused this change?

df.append has been deprecated and removed in e.g. pandas 1.5.x, which will be installed in some combinations of our requirements (1.3.5 is the last py3.7 release and still supported the deprecated API)

aslonnie · 2023-06-13T14:16:26Z

+modin==0.12.1; python_version < '3.8'
+modin==0.12.1; python_version >= '3.8'


these are the same versions? why need the version split?

See https://github.com/ray-project/ray/pull/35040/files#r1220435414 - we will remove the version pin in a follow-up (which will also upgrade pandas). This is to make sure that the py 3.7 version pin remains in that follow-up.

aslonnie · 2023-06-13T14:33:29Z

 # Requires decord which is unavailable for arm64
 gluoncv==0.10.5.post0; platform_machine != "arm64"
 gpy==1.10.0
+gpytorch==1.8.1


why is gpytorch added?

IIRC newer versions (which are available for py 3.8) would lead some tests to fail

bveeramani

LGTM! If you could add a comment about the Modin dependency issue, that'd be awesome.

Signed-off-by: Kai Fricke <kai@anyscale.com>

richardliaw

overall looks good

amogkam

Can we also update the corresponding documentation (installing ray guide) and the dockerhub info that the default python version is now 3.8 and not 3.7

…35040) Implementing ray-project#34863, this PR updates our CI and Docker images to use Python 3.8 as the default Python version. It also updates CUDA version numbers and removes stale CUDA images. The compatibility test becomes and explicit Python 3.7 compatibility test. Currently the tests we are running are (with some filters): - python/ray/tests/horovod - python/ray/tests/lightgbm - python/ray/tests/xgboost - python/ray/tests/ray_lightning - python/ray/tests/ml_py37_compat (keras + torch) - python/ray/air - python/ray/train (AIR tests) - python/ray/data (AIR tests) Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Kai Fricke added 4 commits May 5, 2023 11:18

[ci/docker] Move CI + Docker to Python 3.8 [build_base]

1900cad

Signed-off-by: Kai Fricke <kai@anyscale.com>

cuda file versions

71b3237

Signed-off-by: Kai Fricke <kai@anyscale.com>

miniconda [build_base]

fde1bbd

Signed-off-by: Kai Fricke <kai@anyscale.com>

pin dependencies, solve rllib example errors

f277786

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke force-pushed the ci/python-3-8 branch from 9507a73 to f277786 Compare May 5, 2023 10:28

Kai Fricke added 8 commits May 5, 2023 14:40

ax

5a26526

Signed-off-by: Kai Fricke <kai@anyscale.com>

Merge remote-tracking branch 'upstream/master' into ci/python-3-8

c8f000a

dask

9ae4b7c

Signed-off-by: Kai Fricke <kai@anyscale.com>

revert dask, gpytorch [build_base]

d353824

Signed-off-by: Kai Fricke <kai@anyscale.com>

scipy

f85c390

Signed-off-by: Kai Fricke <kai@anyscale.com>

Move py_36_compat to py_37

c0aed91

Signed-off-by: Kai Fricke <kai@anyscale.com>

Only build ray-ml image for specific Pythons

e53d8e9

Signed-off-by: Kai Fricke <kai@anyscale.com>

Upper bound Py3.7 compat tests

f65ebcc

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke marked this pull request as ready for review May 8, 2023 08:01

krfricke requested review from a team, ArturNiederfahrenhorst, alanwguo, amogkam, avnishn, gjoliver, gvspraveen, kouroshHakha, maxpumperla, smorad and sven1977 as code owners May 8, 2023 08:01

keras legacy optimizer

001d7b4

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label May 8, 2023

Windows + Mac

936c9f5

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke removed the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label May 8, 2023

krfricke requested review from bveeramani, c21, raulchen, scottjlee and scv119 as code owners June 12, 2023 19:32

Kai Fricke added 5 commits June 12, 2023 22:36

Merge remote-tracking branch 'upstream/master' into ci/python-3-8

b05cd12

ax [build_base]

7f881e8

Signed-off-by: Kai Fricke <kai@anyscale.com>

Merge remote-tracking branch 'upstream/master' into ci/python-3-8

2cddcca

skip more ax tests [build_base]

04d1e97

Signed-off-by: Kai Fricke <kai@anyscale.com>

skip more ax tests [build_base]

fb62c19

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jun 13, 2023

aslonnie self-requested a review June 13, 2023 13:37

aslonnie reviewed Jun 13, 2023

View reviewed changes

edoakes approved these changes Jun 13, 2023

View reviewed changes

scv119 approved these changes Jun 13, 2023

View reviewed changes

bveeramani approved these changes Jun 14, 2023

View reviewed changes

Kai Fricke added 5 commits June 14, 2023 08:49

modin, mlflow [build_base]

8012c36

Signed-off-by: Kai Fricke <kai@anyscale.com>

modin, mlflow, keras [build_base]

22ff476

Signed-off-by: Kai Fricke <kai@anyscale.com>

More suggestions [build_base]

7c75d84

Signed-off-by: Kai Fricke <kai@anyscale.com>

Merge remote-tracking branch 'upstream/master' into ci/python-3-8

0f7dba5

ax-platform + gpytorch [build_base]

549329e

Signed-off-by: Kai Fricke <kai@anyscale.com>

richardliaw approved these changes Jun 14, 2023

View reviewed changes

amogkam reviewed Jun 14, 2023

View reviewed changes

alanwguo approved these changes Jun 14, 2023

View reviewed changes

pcmoritz approved these changes Jun 14, 2023

View reviewed changes

krfricke merged commit 7d3bc5e into ray-project:master Jun 15, 2023

krfricke deleted the ci/python-3-8 branch June 15, 2023 07:18

akshay-anyscale mentioned this pull request Jul 21, 2023

Add service deployment instructions to stable diffusion template #37645

Closed

8 tasks

		modin==0.12.1; python_version < '3.8'
		modin==0.12.1; python_version >= '3.8'

Conversation

krfricke commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Technical details

Related issue number

Checks

Uh oh!

ollie-iterators commented May 8, 2023

Uh oh!

krfricke commented Jun 13, 2023

Uh oh!

aslonnie left a comment

Choose a reason for hiding this comment

Uh oh!

aslonnie Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aslonnie Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

krfricke Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

aslonnie Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

krfricke Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aslonnie Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

krfricke Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bveeramani left a comment

Choose a reason for hiding this comment

Uh oh!

richardliaw left a comment

Choose a reason for hiding this comment

Uh oh!

amogkam left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

krfricke commented May 4, 2023 •

edited

Loading