Instrument server timeout check causes unnecessary errors

[AdriaanRol]

When setting a dac voltage or acquiring data the time between a function call and the response getting back is typically larger than the default timeout (digging around in the code tells me this is 2s but I could not find where this is set).

I have had the error below when 1 setting a voltage on the IVVI (directly related to the delay in setting it slowly over a large range) and 2 when acquiring data with the CBox (homebuilt FPGA). Now that I think of it, this may also be the error I am seeing in #119 when loading in sequences in the AWG (typically takes somewhere between 3s and 1min depending on the sequence).

I understand the need for a timeout on a server call, however currently it is raising exceptions where it should not. I am again reverting back to using server_name=None to circumvent the error.

Solutions I can think of, 1. longer default timeout, pro easy to implement, con not general to say a measurement that takes very long (several mins is not uncommon when averaging a lot). 2, have no timeout and rely on the underlying VISA timeouts.

Both are not very good but to me this is a serious issue as it breaks the way I work. I also think that whatever solution we implement should not be something I do when instantiating the instrument but something general that the user (other than me) does not have to think about.

---------------------------------------------------------------------------
Empty                                     Traceback (most recent call last)
<ipython-input-30-3a8352c28258> in <module>()
----> 1 IVVI.dac1.set(-40)

d:\githubrepos\qcodes\qcodes\instrument\remote.py in set(self, value)
    128         # to 'write' below. how do we decide, and how do we let the user
    129         # do it?
--> 130         self._instrument.connection.ask('set', self.name, value)
    131 
    132     # manually copy over validate and __getitem__ so they execute locally

d:\githubrepos\qcodes\qcodes\instrument\server.py in ask(self, func_name, *args, **kwargs)
    134         Query the server copy of this instrument, expecting a response
    135         '''
--> 136         return self.manager.ask('ask', self.id, func_name, args, kwargs)
    137 
    138     def write(self, func_name, *args, **kwargs):

d:\githubrepos\qcodes\qcodes\utils\multiprocessing.py in ask(self, timeout, *query)
    332                 if self._error_queue.empty():
    333                     # only raise if we're not about to find a deeper error
--> 334                     raise e
    335             self._check_for_errors(self._expect_error)
    336 

d:\githubrepos\qcodes\qcodes\utils\multiprocessing.py in ask(self, timeout, *query)
    324 
    325             try:
--> 326                 res = self._check_response(timeout)
    327 
    328                 while not self._response_queue.empty():

d:\githubrepos\qcodes\qcodes\utils\multiprocessing.py in _check_response(self, timeout)
    301 
    302     def _check_response(self, timeout):
--> 303         res = self._response_queue.get(timeout=timeout)
    304         if res == SERVER_ERR:
    305             self._expect_error = True

C:\Anaconda3\lib\multiprocessing\queues.py in get(self, block, timeout)
    103                     timeout = deadline - time.time()
    104                     if timeout < 0 or not self._poll(timeout):
--> 105                         raise Empty
    106                 elif not self._poll():
    107                     raise Empty

Empty: 

[AdriaanRol]

While doing a night run that does simple 2D scans of some parameters my measurement crashed.
The error I get is the nondescriptive Empty that the instrument server seems to generate, from the line number I deduce that this is the same timeout that I have created this issue for.

I relate this directly to having my instruments on the instrument server, I verified that the instruments involved in the loop are still running and still responsive, indicating to me that there was no "hard" exception. I can execute the same cell and everything runs fine.

As the server only returns Empty, it is very hard to debug. As it is a timing related issue, I expect that this specific example will be hard to reproduce.

I add this to the issue as the above example is a specific problem with loading AWG-sequences or setting parameters that have sweep delay built in. This is an instance where it crashed my night run after running for several hours.

@alexcjohnson , I am worried about this and this is certainly something that will makes me want to avoid the instrument server.

D:\GitHubRepos\PycQED_py3\modules\measurement\measurement_control.py in measurement_function(self, x)
    217             raise ValueError('size of x "%s" not equal to # sweep functions' % x)
    218         for i, sweep_function in enumerate(self.sweep_functions[::-1]):
--> 219             sweep_function.set_parameter(x[::-1][i])
    220             # x[::-1] changes the order in which the parameters are set, so
    221             # it is first the outer sweep point and then the inner.This

d:\githubrepos\qcodes\qcodes\instrument\remote.py in set(self, value)
    128         # to 'write' below. how do we decide, and how do we let the user
    129         # do it?
--> 130         self._instrument.connection.ask('set', self.name, value)
    131 
    132     # manually copy over validate and __getitem__ so they execute locally

d:\githubrepos\qcodes\qcodes\instrument\server.py in ask(self, func_name, *args, **kwargs)
    134         Query the server copy of this instrument, expecting a response
    135         '''
--> 136         return self.manager.ask('ask', self.id, func_name, args, kwargs)
    137 
    138     def write(self, func_name, *args, **kwargs):

d:\githubrepos\qcodes\qcodes\utils\multiprocessing.py in ask(self, timeout, *query)
    332                 if self._error_queue.empty():
    333                     # only raise if we're not about to find a deeper error
--> 334                     raise e
    335             self._check_for_errors(self._expect_error)
    336 

d:\githubrepos\qcodes\qcodes\utils\multiprocessing.py in ask(self, timeout, *query)
    324 
    325             try:
--> 326                 res = self._check_response(timeout)
    327 
    328                 while not self._response_queue.empty():

d:\githubrepos\qcodes\qcodes\utils\multiprocessing.py in _check_response(self, timeout)
    301 
    302     def _check_response(self, timeout):
--> 303         res = self._response_queue.get(timeout=timeout)
    304         if res == SERVER_ERR:
    305             self._expect_error = True

C:\Anaconda3\lib\multiprocessing\queues.py in get(self, block, timeout)
    103                     timeout = deadline - time.time()
    104                     if timeout < 0 or not self._poll(timeout):
--> 105                         raise Empty
    106                 elif not self._poll():
    107                     raise Empty

Empty: 

[alexcjohnson]

@AdriaanRol yikes, this is no good... Thinking about it more, we may actually be fine with no timeouts at all. My only hesitation about this is making sure we have a bulletproof distinction between queries that expect a response (ask) and those that do not (write). But if we make a mistake like that it will already break the measurement, it will just do so by raising an error within the timeout (and we haven't seen that failure mode, as far as I know) rather than potentially blocking forever. So I think I'll take the timeout out completely for now, then when I get a chance look at the server communication architecture and make sure it's as hard as possible to mix up ask and write.

[alexcjohnson]

so I don't forget why I'm leaving this issue open:

• simplify more by getting rid of the error_queue completely and just code responses and errors differently in the response_queue
• unify the way all servers receive and respond to queries so the ask / write distinction is really robust