Efficient Array Passing

[blink1073]

Bottom line: array passing is 10X faster without using temporary files.
Uses base64 encoding to pass arrays between Matlab and Python using a similar method to complex. For Python, numeric-type arrays are converted to float64 and then passed as a dictionary. For Matlab, numeric arrays are converted to double and sent back as a structure.
Also handles complex type arrays. Side benefit: arrays are now returned as ndarrays.

With Base64:
In [3]: %time mlab.set_variable('ha', np.linspace(1.1, 2, 10000).reshape(100, 100))
CPU times: user 4.31 ms, sys: 1.61 ms, total: 5.92 ms
Wall time: 163 ms
Out[3]: {'success': 'true', 'result': 1, 'message': 'Successfully completed request'}

In [4]: %time b = mlab.get_variable('ha')
CPU times: user 3.93 ms, sys: 1.09 ms, total: 5.02 ms
Wall time: 66.8 ms

In [5]: b
Out[5]: b
array([[ 1.1       ,  1.10009001,  1.10018002, ...,  1.10873087,
         1.10882088,  1.10891089],
       [ 1.1090009 ,  1.10909091,  1.10918092, ...,  1.11773177,
         1.11782178,  1.11791179],
       [ 1.1180018 ,  1.11809181,  1.11818182, ...,  1.12673267,
         1.12682268,  1.12691269],
       ..., 
       [ 1.97308731,  1.97317732,  1.97326733, ...,  1.98181818,
         1.98190819,  1.9819982 ],
       [ 1.98208821,  1.98217822,  1.98226823, ...,  1.99081908,
         1.99090909,  1.9909991 ],
       [ 1.99108911,  1.99117912,  1.99126913, ...,  1.99981998,
         1.99990999,  2.        ]])

On master branch:
In [5]: %time mlab.set_variable('ha', np.linspace(1.1, 2, 10000).reshape(100, 100))
CPU times: user 16.7 ms, sys: 1.08 ms, total: 17.7 ms
Wall time: 1.19 s
Out[5]: {'result': 1, 'message': 'Successfully completed request', 'success': 'true'}

In [6]: %time b = mlab.get_variable('ha')
CPU times: user 5.33 ms, sys: 1.2 ms, total: 6.54 ms
Wall time: 821 ms

In [7]: type(b)
Out[7]: list

[blink1073]

I'm still cleaning this up/getting tests to pass.

[blink1073]

All tests pass!

[blink1073]

I'd prefer this to the create-a-file approach, since it is all in memory, and does not require you to be on the same machine to share the data.

[blink1073]

I'm working on support for complex arrays, don't merge yet.

[blink1073]

All set!

[arokem]

Sounds great. I will not have time to look at it before Monday, but I will
do it then.

On Saturday, January 17, 2015, Steven Silvester notifications@github.com
wrote:

All set!

—
Reply to this email directly or view it on GitHub
#103 (comment)
.

[blink1073]

This was also an exercise in grokking the code base. FYI, my Matlab license expires in ~40 days.

[arokem]

Do you think Peter is OK with us using his code? Did the original have a license attached to it?

[blink1073]

http://www.fourmilab.ch/webtools/base64/

"This software is in the public domain. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, without any conditions or restrictions. This software is provided “as is” without express or implied warranty. "

[blink1073]

Go for it.