Adding auxiliary information to trajectory

[richardjgowers]

Original started on discussion board

It would be nice if we could add arbitrary data to run alongside a trajectory, so..

u = mda.Universe('this', 'that')
u.trajectory.add_auxiliary('some.file')

for ts in u.trajectory:
    print ts.stuff_from_file

I think the best route might be adding in the hooks after a Timestep has been read, so

def _read_next_timestep(self):
    ts = self._read_timestep()
    for aux in self._auxiliaries:
        aux.read_next(ts)
    return ts

So each aux object is like a mini Reader object, which augments a trajectory Reader.

In the longer term, it would be cool if u.trajectory.add_auxiliary(PullForce('pull.xvg')), then made AtomGroup.pullforce a viable attribute, this should be possible with the new dynamic classes + transplant system.

[jbarnoud]

From what I understand of your examples, you see the auxiliaries as objects that implement a read_next method. Could they just be regular iterators?

Also, how do you see the assignment of the auxiliaries to an AtomGroup? In the case of pull.xvg as described in the mailing list (as an output of Gromacs for a simulation with pulling), we only have access to a time serie of pulling force. Should we inspect the TPR to know what atoms are pulled?

[jbarnoud]

From what I understand of your examples, you see the auxiliaries as objects that implement a read_next method. Could they just be regular iterators?

Answering to myself. It has to be a custom object, and a simple iterator won't do.

Indeed, we will have to handle the random seeking of the trajectory, so we will not only have to read values from the auxiliaries sequentially. Also, we will have to handle alignement of the auxiliaries: an auxiliary may be save with a different frequency as the coordinates.

[richardjgowers]

What I was thinking is that if the ts is passed in, then either the frame or time information from this would let the aux figure out what to return. Sort of, here's a Timestep that says it's from t=10ps, what would you like to add to this?

Ah yeah, sparse attributes aren't really handled currently. We'd need something like how pandas does missing information.

And yes, meshing different frequencies is a lovely headache for someone.

[jbarnoud]

What I was thinking is that if the ts is passed in, then either the frame or time information from this would let the aux figure out what to return. Sort of, here's a Timestep that says it's from t=10ps, what would you like to add to this?

OK. So this is very similar to something I had in mind for coordinate transformations. I'll tidy up my proposal so it can be discussed in parallel, then.

[kain88-de]

@richardjgowers I thought the add_auxillary method could just manipulate the __dict__ of the Reader class directly.

class Reader(object):
    def add_auxillary(self, name, val):
        self.__dict__[name] = val

read = Reader()
read.add_auxillary('pulling_force', np.arange(5))
read.pulling_force

This makes it very easy to add new values. We woudn't need to write special classes that can be added as auxillaries as in your last example. This way the information is easily accesible like velocities and positions. I could imagine that val is either a filename or a class that handles the access for xvg files for csv file objects. (actually add_auxillary could figure that out based on fileending as well and then use the classes itself for covenience).

[richardjgowers]

@kain88-de yeah hacking it into the namespace is cool. But there still needs to be some sort updating of the auxiliary data every time a frame is read?

[kain88-de]

But there still needs to be some sort updating of the auxiliary data every time a frame is read?

Yes. I've just given a simple example of the __dict__ change that I thought of using. But the class for updating could be very generic or a collection of generic ones. For one the add_auxillary method should have a simple algorithm to chose the best class and also accept a class as parameter.
I would like that the following list of auxillary data is possible

reader.add_auxillary('pulling_force', ndarray)
reader.add_auxillary('pulling_force', dataframe)
reader.add_auxillary('pulling_force', 'pull.xvg')
reader.add_auxillary('pulling_force', 'energy.edr')
reader.add_auxillary('pulling_force', 'some-variable.csv')

Of course each needs to be handled specially. Maybe with a class. And we need to do more then just add to the __dict__ method. The good thing about this is we can easily extend the TimeStep object with more information.

[jbarnoud]

I would prefer the auxiliary data to be in a separate namespace. I think it would be cleaner to call ts.aux.pullforce rather than ts.pullforce. The TimeStep name space can get crowded and messy otherwise. Also, having a aux namespace make clear that the data we access comes from somewhere else.

@kain88-de I am not sure I see how what you suggest differs from @richardjgowers initial proposal. Could you elaborate?

[kain88-de]

@kain88-de I am not sure I see how what you suggest differs from

It doesn't differ that much. Rather extends and generalizes it. With @richardjgowers example we would need to write a new class for every new auxillary that we want to add. While I propose that we simple write classes to read the different file formats, this is because xvg and others can actually contain a lot of different observables, this leaves the user more freedom.

# Richard
u.trajectory.add_auxiliary(PullForce('pull.xvg'))
# Me
u.trajectory.add_auxiliary('pulling_force', XVGReader('pull.xvg'))
## And an alias
u.trajectory.add_pulling('pull.xvg')

[jbarnoud]

On 21/03/16 23:02, kain88-de wrote:

@kain88-de <https://github.com/kain88-de> I am not sure I see how
what you suggest differs from

It doesn't differ that much. Rather extends and generalizes it. With
@richardjgowers https://github.com/richardjgowers example we would
need to write a new class for every new auxillary that we want to add.
While I propose that we simple write classes to read the different
file formats, this is because xvg and others can actually contain a
lot of different observables, this leaves the user more freedom.

Richard

u.trajectory.add_auxiliary(PullForce('pull.xvg'))

Me

u.trajectory.add_auxiliary('pulling_force', XVGReader('pull.xvg'))

And an alias

u.trajectory.add_pulling('pull.xvg')

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#785 (comment)

I understand. Then I like yours better. Anyway, in your example,
XVGReader will have to inherit from a Auxiliary class that know how
to handle the jumps in the trajectory, and the data alignment when the
auxiliary data is not saved at the same frequency than the trajectory.

[kain88-de]

XVGReader will have to inherit from a Auxiliary class that know how to handle the jumps in the trajectory,

yes that we the headache @richardjgowers talked about would only need to be solved once.

[orbeckst]

Is this what @fiona-naughton is going to do for GSoC? If so, we should restart the discussion here, especially with Fiona included.
I think the outcome of this discussion should be the definition of a preliminary API; it might change during testing but there should be something concrete to work towards to,