refer to instruments from another instrument

[peendebak]

When trying to write a virtual gates instrument I encountered the following issue. First I create two drivers:

server_name='test'
ivvi1 = qcodes.instrument_drivers.QuTech.IVVI.IVVI(name='ivvi1', server_name=server_name, adress='ASRL2')
ivvi2 = qcodes.instrument_drivers.QuTech.IVVI.IVVI(name='ivvi2', server_name=server_name, adress='ASRL3')

Then I create a virtual instrument mapping gates to the combination of the two:

gate_map = {
    'L': (0, 8),  'M': (0, 9),  'R': (1, 11), ...
}
gates = virtual_gates(name='gates', gate_map=gate_map, server_name=server_name, instruments=[ivvi1, ivvi2])

The virtual_gates object (a subclass of qcodes.Instrument) automatically creates parameters with the names from gate_map and creates get and set functions mapping to the IVVI drivers. This works for server_name=None, but fails otherwise.

The problem is that when virtual_gates is created on a server the argument instruments=[ivvi1, ivvi2 is being pickled, which is now allowed.

How to solve this issue?

[AdriaanRol]

@peendebak , I had the same issue when using my meta-instruments. The solution is to have the virtual gates instrument live on a separate instrument server. This way the instruments it get's handed (upon instantiating the different name instrument server) are already pickled representations of the actual instruments in which case it should work fine.

[damazter]

I think it could be helpful as well to have a possibility to have all these instruments (ivvi1, ivvi2 and virtual_gates) living on the same server. It should be possible to circumvent the pickeling if the object live on the same server. This would then in turn enable unpickeled access to the driver which could be useful in some circumstances (I am thinking here about theoretical simulations, but I think experimental counterparts should exist as well).

So although your solution should work @AdriaanRol , I personally think this feature should be added as well @peendebak

[peendebak]

I will use the suggestion by @AdriaanRol for now. I will leave the issue open until we have a proper solution.

[AdriaanRol]

I would very much like the implementation @damazter suggests.

[alexcjohnson]

Interesting... the most generic solution I see to this would be to write our own pickler, that when it encounters a RemoteInstrument or RemoteComponent, looks to see if it's on the same server we're putting the new instrument on, and reconstructs it as a local reference on the other end. Sounds like a bit of a project, but should be doable.

[peendebak]

@AdriaanRol @alexcjohnson @giulioungaretti

So far I have worked around this issue by make a virtual instrument with server_name=None. I want to use multiprocessing, so this does not work for me any more. The suggestion by @AdriaanRol to place the virtual instrument on another server doesn't work for me. An example script is included below

#%% Test virtual gates
import qcodes
from qcodes import Instrument
import logging
from functools import partial

import multiprocessing
if __name__=='__main__':
    try:
        #multiprocessing.set_start_method('spawn')
        pass
    except:
        pass

class VirtualIVVI(Instrument):

    def __init__(self, name, **kwargs):
        super().__init__(name, **kwargs)
        self.x=0
        self.add_parameter('x',  get_cmd=self.getx, set_cmd=self.setx)

    def getx(self):
        return self.x

    def setx(self, val):
        self.x=val


class virtual_gates(Instrument):

    def __init__(self, name, instruments, **kwargs):
        super().__init__(name, **kwargs)
        self._instrument_list = instruments

        for gate in range(len(self._instrument_list)):
            self.add_parameter('c%d' % gate,
                           get_cmd=partial(self._get, gate=gate),
                           set_cmd=partial(self._set, gate=gate) )

    def _get(self, gate):
        logging.debug('virtual_gates._get: %s'  %  (gate, ) )
        return self._instrument_list[gate].get('x')

    def _set(self, value, gate):
        logging.debug('virtual_gates._set: gate %s, value %s' % (gate, value))
        i = self._instrument_list[gate]
        i.set('x', value)

#%% Make instruments

if __name__=='__main__':

    server_name='test3'

    ivvi1 = VirtualIVVI(name='one', server_name=server_name)
    ivvi2 = VirtualIVVI(name='two', server_name=server_name)

#%% Good
if __name__=='__main__':
    v = virtual_gates(name='virtual', server_name=None, instruments=[ivvi1, ivvi2])
    v.c1.set(23)
    print(v.c1.get())


#%% Not working
if __name__=='__main__':
    v = virtual_gates(name='virtual', server_name='testv', instruments=[ivvi1, ivvi2])
    v.c1.set(24)
    print(v.c1.get())

#%% Not working
if __name__=='__main__':
    v = virtual_gates(name='virtual', server_name=server_name, instruments=[ivvi1, ivvi2])
    v.c1.get()
    v.c1.set(25)

[alexcjohnson]

EDIT: more complete version of this in the following comment.

@peendebak here's what I think should work right now:

class virtual_gates(Instrument):
    # instruments will be a list of RemoteInstrument objects, which can be
    # given to a server on creation but not later on, so it needs to be
    # listed in shared_kwargs
    shared_kwargs = ['instruments']

    # everything else in this class is the same as you had it
    def __init__(self, name, instruments=(), **kwargs):
        super().__init__(name, **kwargs)
        self._instrument_list = instruments

        for gate in range(len(self._instrument_list)):
            self.add_parameter('c%d' % gate,
                           get_cmd=partial(self._get, gate=gate),
                           set_cmd=partial(self._set, gate=gate) )

    def _get(self, gate):
        logging.debug('virtual_gates._get: %s'  %  (gate, ) )
        return self._instrument_list[gate].get('x')

    def _set(self, value, gate):
        logging.debug('virtual_gates._set: gate %s, value %s' % (gate, value))
        i = self._instrument_list[gate]
        i.set('x', value)

if __name__=='__main__':

    lowlevel_server_name='server1'

    ivvi1 = VirtualIVVI(name='one', server_name=lowlevel_server_name)
    ivvi2 = VirtualIVVI(name='two', server_name=lowlevel_server_name)

    # meta instrument needs to go on a different server from the
    # low-level instruments it references
    meta_server_name='server2'

    v = virtual_gates(name='virtual', server_name=meta_server_name, instruments=[ivvi1, ivvi2])
    v.c1.get()
    v.c1.set(25)

Does that work for you? I know it's awkward and hard to explain... I guess that will be part of the coming multiprocessing cleanup.

[alexcjohnson]

A self-contained version of the above, that I actually tested :)

@peendebak if you get this to work, and it explains things enough to adapt to your real case, then one of us should flesh it out into a real tutorial on this.

# %% Test virtual gates
from qcodes import Instrument
import logging
from functools import partial

import multiprocessing


class VirtualIVVI(Instrument):

    def __init__(self, name, **kwargs):
        super().__init__(name, **kwargs)
        self.x = 0
        self.add_parameter('x',  get_cmd=self.getx, set_cmd=self.setx)

    def getx(self):
        return self.x

    def setx(self, val):
        self.x = val


class virtual_gates(Instrument):
    # instruments will be a list of RemoteInstrument objects, which can be
    # given to a server on creation but not later on, so it needs to be
    # listed in shared_kwargs
    shared_kwargs = ['instruments']

    # everything else in this class is the same as you had it
    def __init__(self, name, instruments=(), **kwargs):
        super().__init__(name, **kwargs)
        self._instrument_list = instruments

        for gate in range(len(self._instrument_list)):
            self.add_parameter('c%d' % gate,
                               get_cmd=partial(self._get, gate=gate),
                               set_cmd=partial(self._set, gate=gate))

    def _get(self, gate):
        logging.debug('virtual_gates._get: %s' % (gate,))
        return self._instrument_list[gate].get('x')

    def _set(self, value, gate):
        logging.debug('virtual_gates._set: gate %s, value %s' % (gate, value))
        i = self._instrument_list[gate]
        i.set('x', value)

if __name__ == '__main__':

    multiprocessing.set_start_method('spawn')

    lowlevel_server_name = 'server1'

    ivvi1 = VirtualIVVI(name='one', server_name=lowlevel_server_name)
    ivvi2 = VirtualIVVI(name='two', server_name=lowlevel_server_name)

    # meta instrument needs to go on a different server from the
    # low-level instruments it references
    meta_server_name = 'server2'

    v = virtual_gates(name='virtual', server_name=meta_server_name,
                      instruments=[ivvi1, ivvi2])

    print('initial value of v.c1:', v.c1.get())
    v.c1.set(25)
    print('set v.c1 to 25 and it and ivvi2.x have that value:',
          v.c1.get(), ivvi2.x.get())

    ivvi1.x.set(13)
    print('set ivvi1.x to 13 and v.c0 has that value:', v.c0.get())

[giulioungaretti]

Made into cookbook/tutorial. @peendebak if it works for your thing, write it here as well.