Performance using ZMQ

[mfitzp]

I've recently upgraded to the dev-0.4 version (from this repo) and am experiencing poorer performance on variable I/O compared to previous versions.

For example running the following in a Python cell creates a variable called e at 2.5MB:

e = np.random.rand(32768,10)
mb = e.nbytes / 1024**2.
mb

Passing this into a Matlab cell and processing it as follows takes ~ 1m40s:

%%matlab -i e -o f
f=mean(e);

This is a lot slower than the previous implementation (I can't benchmark until I find where I installed it from!)

Is it possible to select the protocol/communication method when initialising the extension? Thanks

[cekees]

I have had some issues with performance when moving from the new zmq
communication as well, but I haven't been in a situation where I had time
to benchmark it. I suspect it is because numpy arrays are getting converted
to lists and then serialized to JSON. I suspect you can write a numpy
array to hdf5 from python and read it back into matlab faster than the
current approach to serializing large arrays. There's also a big memory
hit when you go from a numpy array of doubles to a generic list. -Chris

On Wed, Jun 11, 2014 at 10:32 AM, Martin Fitzpatrick <
notifications@github.com> wrote:

I've recently upgraded to the dev-0.4 version (from this repo) and am
experiencing poorer performance on variable I/O compared to previous
versions.

For example running the following in a Python cell creates a variable
called e at 2.5MB:

e = np.random.rand(32768,10)
mb = e.nbytes / 1024**2.
mb

Passing this into a Matlab cell and processing it as follows takes ~ 1m40s:

%%matlab -i e -o f
f=mean(e);

This is a lot slower than the previous implementation (I can't benchmark
until I find where I installed it from!)

Is it possible to select the protocol/communication method when
initialising the extension? Thanks

—
Reply to this email directly or view it on GitHub
#72.

[arokem]

Thanks for reporting, and thanks @cekees for the suggestions. It is not currently possible to switch between communication protocols, but we need to do some thinking about this.

I can confirm that this takes an annoyingly long time with the current implementation (54.6 sec on my laptop). This is not substantially different in the release 0.3 (50.6 sec), which also used the zmq communication model. In 0.2, which was the last version to use the webserver, things are indeed much zippier: 2.45 sec.

This is very confusing to me, because we did test performance quite extensively before doing this move. I'll need to go back and revisit our notes from that. For the time being, one simple way to revert back is to check out the 0.2 tag and install that.

[blink1073]

I'd say this is fixed on the current master thanks to base64 encoding of arrays:

In [15]: e = np.random.rand(32768, 10)

In [16]: %time m.mean(e);
CPU times: user 36.9 ms, sys: 10.4 ms, total: 47.4 ms
Wall time: 607 ms
Out[16]: 
{'content': {'datadir': '/tmp/MatlabData/', 'figures': [], 'stdout': ''},
 'result': array([[ 0.49988451,  0.50171806,  0.49738513,  0.49900619,  0.49779332,
          0.50118379,  0.50171328,  0.50040391,  0.50208133,  0.50055755]]),
 'success': True}