Leaking sockets?

[jenshnielsen]

Steps to reproduce

1. Run code for some time on windows including possible shutdown and restart of notebook and instrument communication.

Expected behaviour

Things should keep working

Actual behaviour

Notebook fails with socket related connection issues from Tornado and ZMQ.

      File "c:\users\triton2acq\anaconda3\envs\qcodes-master\lib\site-packages\j
upyter_client\multikernelmanager.py", line 33, in wrapped
        r = method(*args, **kwargs)
      File "c:\users\triton2acq\anaconda3\envs\qcodes-master\lib\site-packages\j
upyter_client\ioloop\manager.py", line 33, in wrapped
        socket = f(self, *args, **kwargs)
      File "c:\users\triton2acq\anaconda3\envs\qcodes-master\lib\site-packages\j
upyter_client\connect.py", line 492, in connect_shell
        return self._create_connected_socket('shell', identity=identity)
      File "c:\users\triton2acq\anaconda3\envs\qcodes-master\lib\site-packages\j
upyter_client\connect.py", line 476, in _create_connected_socket
        sock = self.context.socket(socket_type)
      File "c:\users\triton2acq\anaconda3\envs\qcodes-master\lib\site-packages\z
mq\sugar\context.py", line 146, in socket
        s = self._socket_class(self, socket_type, **kwargs)
      File "zmq\backend\cython\socket.pyx", line 285, in zmq.backend.cython.sock
et.Socket.__cinit__ (zmq\backend\cython\socket.c:3861)
    zmq.error.ZMQError: No buffer space available

    
 OSError: [WinError 10055] An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Regular internet (browsing etc) is flaky too

We suspect that sockets are leaked and not cleaned up. This may be in the VISA driver of
network instruments such as ZNMB20 VNA

System

operating system
Windows
qcodes branch
master
qcodes commit
?

[jenshnielsen]

One way of reproducing this it to use both PYQTGraph and matplotlib using the qt5 backend.
This is probably because two different qt event loops are started.

Doing:

%matplotlib qt5

import matplotlib.pyplot as plt
import qcodes as qc

from toymodel import AModel
model = AModel()

Seems to be sufficient to get

[I 11:07:21.917 NotebookApp] Saving file at /Tutorial-Copy1.ipynb
Traceback (most recent call last):
  File "/Users/jhn/.pyenv/versions/3.5.3/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/jhn/.pyenv/versions/3.5.3/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jhn/.pyenv/versions/3.5.3/envs/qcodes-3.5.3/lib/python3.5/site-packages/ipykernel/__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "/Users/jhn/.pyenv/versions/3.5.3/envs/qcodes-3.5.3/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-120>", line 2, in initialize
  File "/Users/jhn/.pyenv/versions/3.5.3/envs/qcodes-3.5.3/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhn/.pyenv/versions/3.5.3/envs/qcodes-3.5.3/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 445, in initialize
    self.init_sockets()
  File "/Users/jhn/.pyenv/versions/3.5.3/envs/qcodes-3.5.3/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 235, in init_sockets
    self.shell_port = self._bind_socket(self.shell_socket, self.shell_port)
  File "/Users/jhn/.pyenv/versions/3.5.3/envs/qcodes-3.5.3/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 177, in _bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5653)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
zmq.error.ZMQError: Address already in use

Not sure why it is needed to create a Mock Model?

[jenshnielsen]

A bit more investigation. This can be shortened to

%gui qt5
from toymodel import AModel
model = AModel()

The Mock model starts a server process so using the multiprocessing so thats probably where the issue is

[jenshnielsen]

The output from netstat -tnl contains a number of localhost sockets that I believe
are coming from the Jupyter ZMQ internal communications.

tcp4       0      0  127.0.0.1.49646        127.0.0.1.49663        ESTABLISHED
tcp4       0      0  127.0.0.1.49663        127.0.0.1.49646        ESTABLISHED
tcp4       0      0  127.0.0.1.49645        127.0.0.1.49662        ESTABLISHED
tcp4       0      0  127.0.0.1.49662        127.0.0.1.49645        ESTABLISHED
tcp4       0      0  127.0.0.1.49644        127.0.0.1.49661        ESTABLISHED
tcp4       0      0  127.0.0.1.49661        127.0.0.1.49644        ESTABLISHED
tcp4       0      0  127.0.0.1.49647        127.0.0.1.49659        ESTABLISHED
tcp4       0      0  127.0.0.1.49659        127.0.0.1.49647        ESTABLISHED
tcp6       0      0  ::1.8888                                      ::1.49650                                     ESTABLISHED
tcp6       0      0  ::1.49650                                     ::1.8888                                      ESTABLISHED
tcp6       0      0  ::1.8888                                      ::1.49640                                     ESTABLISHED
tcp6       0      0  ::1.49640                                     ::1.8888                                      ESTABLISHED

Shutting down the notebook after a crash ends up in something like:

tcp4      31      0  172.20.2.104.57751     162.125.65.3.443       CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50668        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50639        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50636        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50543        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50493        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50481        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50416        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50386        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50350        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50330        CLOSE_WAIT
tcp4      10      0  127.0.0.1.49723        127.0.0.1.50123        CLOSE_WAIT

These sockets never seems to be fully closed

[jenshnielsen]

This is probably caused by a combination of the fork and the fact that the qt eventloop sets up additional sockets

https://github.com/ipython/ipykernel/blob/master/ipykernel/eventloops.py#L24

def _notify_stream_qt(kernel, stream):

    from IPython.external.qt_for_kernel import QtCore

    if _use_appnope() and kernel._darwin_app_nap:
        from appnope import nope_scope as context
    else:
        from contextlib import contextmanager
        @contextmanager
        def context():
            yield

    def process_stream_events():
        while stream.getsockopt(zmq.EVENTS) & zmq.POLLIN:
            with context():
                kernel.do_one_iteration()

    fd = stream.getsockopt(zmq.FD)
    notifier = QtCore.QSocketNotifier(fd, QtCore.QSocketNotifier.Read, kernel.app)
    notifier.activated.connect(process_stream_events)

[jenshnielsen]

So I suggest, Rewrite mock instruments to not depend on Model. This has the added benefit of making the tutorial simpler.

Add a warning to the QCodes process to ensure that we catch any unintended use of multiprocessing remaining (there shouldn't be any thats not behind the config option)

[jenshnielsen]

It's also possible to reproduce the issue without qcodes or QT.

Simply start a long running background process and try restarting the notebook before joining it.

from multiprocessing import Process

def f(name):
    import time
    while 1:
        time.sleep(1)

    
p = Process(target=f, args=('bob',))
p.start()
#p.join()

[giulioungaretti]

@jenshnielsen closing this as we at least removed multiprocessing.

[jenshnielsen]

Agreed. For the record there may still be issues with the pyqtgraph remote proxy thread. So if we see something like this again, that would be where I would look