Logging

[MerlinSmiles]

Has anyone here experience on logging in packages like qcodes? Including multiprocessing?

I see some logging is spread out in some of the files, one resource I read is this http://pythonsweetness.tumblr.com/post/67394619015/use-of-logging-package-from-within-a-library
Which has some interesting comments on not using the root logger from within a package.

My intention last night was to route all logging from within qcodes to a logging handler defined in the notebook. In some parts it worked fine, and in others it failed completely.

I.e. I wanted those negative delays to log into that logging handler, currently they written to the process stdout or sterr and the starting process reads those in, basically removing a lot of logging information that comes with it.

Does anyone know how to organize this logging functionality so that I can capture all logging from qcodes?

[MerlinSmiles]

Hm, am I the only one interested in this?

I figured out a basic package wide logging implementation which works nice, apart from the multiprocessing part. So basically it is useless right now.
There is a logging branch up there where you can peak in if you like.

For the multiprocessing one basically needs to send a logging_queue to the processes and the processes should have a QueueHandler where they push out the log messages, on the other hand the package logger can receive them.
Here is a great resource with up to date examples.
I however fail on implementing that into the subclassed multiprocessing. So i'm kind of giving up on this now.

[guenp]

@MerlinSmiles @alexcjohnson I think logging is an important milestone. It's very useful for debugging and so on. Perhaps we can create a logging handler for the status widget?

[MerlinSmiles]

@guenp I totally agree that this should come soon. As you know, I am working on this logging stuff, and to me it looks promising.

My plan is to have a totally independent logging server, right now I use ZMQ to pass logging records and it works well.

We need this to be an independen python kernel so it doesn't get killed upon restarting the notebook which is especially happening during debugging, and things like that. Also for multiprocessing this is necessary, there logging handlers are not enough.

For the status: The basics are ready, and work. There is need for some polishing, documentation, and some Qt beauty as well as ease of use. The logging branch is not up to date...

[MerlinSmiles]

I just realize that we should add a simple user logging version, where the user can log things along with the experiment. Its basically also just a logger but that saves logs to the experiment folder.

qc.log('Just collected the mixture because its fun')

[peendebak]

@MerlinSmiles To log messages from the objects (e.g. instruments) in other processes I started using
multiprocessing_logging.

By adding the following two lines to the qcodes/__init__.py I get all the message from the logging framework in the main process.

# redirect all logging to the parent process
import multiprocessing_logging
multiprocessing_logging.install_mp_handler()

I also tried to redirect pythons print function to the loggers in subprocesses, but this I could not get working.

[MerlinSmiles]

@peendebak
Interesting! Yesterday I was reading about this one: https://github.com/humangeo/multilog ...

I think print should not be used in any instrument, actually anywhere in a module...

[peendebak]

@MerlinSmiles We can prevent the qcodes drivers from using print, but still other libs (e.g. pyvisa) will use print statements. So redirecting all output from a subprocess still seems something we should do.