__del__ destructor for Universes?

[mnmelo]

The title says it all. @richardjgowers mentioned it'd be interesting to implement, but currently that leads to mem leaks as the garbage collector can't clean up the Universe.

See the discussion on #293.

If Universe has a del I end up with two uncollectables: the Universe itself and the trajectory (which also has a del). I tried to find any sort of circular reference between the two, but found nothing.
Is it worth pursuing this?

[richardjgowers]

So adding the del destructor does create a memory leak.. :)

[mnmelo]

So, I think I was able to finally track this down. I always thought it was a problem of having the Reader also having a __del__, because that's apparently the easiest way to have uncollectable circular references. But the snippet below shows I was wrong:

import gc
class Universe():
     def __init__(self):
         self.trajectory = Reader()
     def __del__(self):
         print "Universe is dying"

class Reader():
    def __init__(self):
        self.somevar = 3
    def __del__(self):
        print "Trajectory is dying"

u = Universe()
u = None
gc.collect() # This is empty, meaning all got properly collected

Now let's add some more realism to our toy Universe:

import gc
class Universe():
     def __init__(self):
         self.trajectory = Reader()
         self.atom = Atom(self) # Having this or the following lines breaks garbage collection
         self.universe = self
     def __del__(self):
         print "Universe is dying"

class Reader():
    def __init__(self):
        self.somevar = 3
    def __del__(self):
        print "Trajectory is dying"

class Atom():
    def __init__(self, parent):
        self.universe = parent

u = Universe()
u = None
gc.collect()
gc.garbage # [<__main__.Universe instance at 0x1004a91b8>, <__main__.Reader instance at 0x1004a9200>]

As far as I know there's no way to clean this automatically. A user would not only have to set u.universe = None but also do so for each Atom.universe, AtomGroup.universe, Residue.universe etc. listed under the Universe object.
Hopeless?

[richardjgowers]

https://docs.python.org/2/library/weakref.html

If we made all the references to Universe (from Atom etc) a weakref.ref, it might fix everything?

Can you try?

class Atom():
   self.universe = weakref.ref(parent)

[mnmelo]

Ah, was just writing to propose that!
The downside: if the Universe goes out-of-scope, all the referrers will become hollow. I don't know how much of a use-case problem that is.

I'll try it out and let you know.
EDIT: I got the idea from the nice article at http://eli.thegreenplace.net/2009/06/12/safely-using-destructors-in-python/

[richardjgowers]

import gc
import weakref

class Universe():
     def __init__(self):
         self.trajectory = Reader()
         self.atom = Atom(self) # Having this or the following lines breaks garbage collection
         self.universe = weakref.ref(self)
     def __del__(self):
         print "Universe is dying"

class Reader():
    def __init__(self):
        self.somevar = 3
    def __del__(self):
        print "Trajectory is dying"

class Atom():
    def __init__(self, parent):
         self.universe = weakref.ref(parent)

Leads to

In [1]: u = Universe()

In [2]: del u
Universe is dying
Trajectory is dying

In [3]: gc.collect()
Out[3]: 0

In [4]: gc.garbage
Out[4]: []

So possibly fixed?

I think deleting Universe should break everything that refers to Universe... makes sense to me

[mnmelo]

Well, but now what happens is that if you let the Universe go out-of-scope but still have AtomGroups lying around, the Universe is still alive and you can still use those AtomGroups:

def first10(top,traj):
    u = Universe(top,traj)
    return u.atoms[:10]
ag = first10("top", "traj")
ag.positions  #This works now, but wouldn't anymore with the weakrefs

I see no problem with breaking this. It'll have to be well-documented. What's your take?

[richardjgowers]

Ah, I didn't think of scopes like that.... I don't think AtomGroups or any other products of Universe are designed to live standalone though, so I think the above code should break.

I guess the next step is to implement this onto the real code and see which tests break