Call TriggerGlobalGC when the plasma store is full

[edoakes]

Why are these changes needed?

If the plasma store is full, it may be due to cyclic references to ObjectIDs in python somewhere on the cluster. This change causes workers to trigger a global GC in an attempt to garbage collect such references and free up space before giving up when the plasma store is full.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.

[edoakes]

Dependent on #7328.

[AmplabJenkins]

Can one of the admins verify this patch?

[ericl]

Should we call this once or multiple times? I guess it's possible you might need a series of gc calls, though I'm not sure about the scenario.

[edoakes]

Let's call it multiple times for now to cover all scenarios. This is throttled anyways to once per heartbeat interval across the cluster.

[ericl]

Can we add an end to end Python test for this, similar to the one in the ref count tests?

[ericl]

Oh nevermind, I see it now.

[ericl]

Here's another test you can add that introduces cyclic refs in both the worker and driver:

import ray
import gc
import random
import time
import json
import numpy as np


ray.init(num_cpus=1, object_store_memory=2000 * 1024 * 1024)

def block():
    return np.zeros(100 * 1024 * 1024, dtype=np.uint8)


class CyclicRef:
    def __init__(self, v):
        self.value = v
        self.cyclic_ref = self


refs = [
    CyclicRef(ray.put(block())) for _ in range(10)
]


@ray.remote
def f(x):
    y = CyclicRef(x)
#    gc.collect()
    return x


for j in range(1000):
    i = random.randint(0, len(refs) - 1)
    res = f.remote(refs[i].value)
    refs[i] = CyclicRef(res)
    ray.get(res)
    print("processing", j)
#    gc.collect()

Couple other suggestions:

• only print "gc collect freed ...." in raylet.pyx if num freed refs > 0, to avoid spam
• only print the object store memory message if the all retries failed

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22458/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22456/
Test PASSed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22467/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22460/
Test FAILed.