MXNet high api

[ilkarman]

Redo MXNet high_api example using .fit() to match Tensorflow and CNTK.

[ThomasDelteil]

@ilkarman FYI MXNet has released 1.1.0 with several speed improvements.
There is also the MXNet/Gluon API that lets you write imperative code

[ilkarman]

@ThomasDelteil Thanks! I also see MXNet now supports NCCL instead of 'device' for kvstore multi-gpu, so will be really cool to test. Gluon is interesting but it seems marginally slower than MXNet? https://github.com/ilkarman/DeepLearningFrameworks/blob/master/notebooks/Gluon_CNN.ipynb

[ThomasDelteil]

Yes it is a bit slower because the network is not compiled and optimized but you can call '.hybridize()' and you lose the advantage of being able to debug in depth but the computation graph is then cached and you should see the performance improvements

[ThomasDelteil]

On my instance, running the Gluon notebook I get:
Wall time: 51.9 s
When using a HybridSequential block and .hybridize():
Wall time: 45.9 s

OS:  linux
Python:  3.6.3 |Anaconda custom (64-bit)| (default, Oct 13 2017, 12:02:49) 
[GCC 7.2.0]
MXNet:  1.1.0
Numpy:  1.13.3
GPU:  ['Tesla V100-SXM2-16GB']
CUDA Version 9.0.176
CuDNN Version  7.0.5

[ThomasDelteil]

Also, MXNet backend is enqueuing operators and tries to smartly parallelize the execution to avoid starving off your GPU.
When computing the loss on every batch you are forcing data to be copied to the CPU, because you are adding the training loss as a scalar. The GPU wait for this data to be copied over, you are essentially having an implicit nd.waitall().

If you rewrite your loop to have the loss added on the GPU directly and only retrieved at the end of the epoch:

%%time
# Main training loop: 62s
sym.hybridize()
for j in range(EPOCHS):
    train_loss = nd.zeros(1, ctx=ctx)
    for data, target in yield_mb(x_train, y_train, BATCHSIZE, shuffle=True):
        # Get samples
        data = nd.array(data).as_in_context(ctx)
        target = nd.array(target).as_in_context(ctx)
        with autograd.record():
            # Forwards
            output = sym(data)
            # Loss
            loss = criterion(output, target)
        # Back-prop
        loss.backward()
        trainer.step(data.shape[0])
        train_loss += nd.sum(loss)
    # Log
    # Wait for the operations of the current epoch to complete    
    nd.waitall()
    print('Epoch %3d: loss: %5.4f'%(j, train_loss.asscalar()/len(x_train)))

You get a massive performance boost:
You reach Wall time: 39.4 s

[ThomasDelteil]

I will issue a PR if you want @ilkarman, I am thinking that might change a bit the benchmarks 😄

[ilkarman]

Thanks a lot Thomas that's awesome! @miguelgfierro I'll give these a go. Curious how it compares to the highest-level .fit() (I assume same time). We recently got voltas so can run on the P100 and V100

[ThomasDelteil]

Have you guys had time to check the notebooks on your P100 DLVM already? @miguelgfierro @ilkarman

[ilkarman]

@ThomasDelteil I ran the gluon CNN on the P100 with old CUDA 8 (and old gluon) so it was faster but not by that much. Sorry I was distracted with Chainer multi-gpu today. I'll try to update early tomorrow morning. Hopefully once DSVM is updated we will re-run with faster Gluon to match your speed

[ilkarman]

@ThomasDelteil I have finished updating my branch here - https://github.com/ilkarman/DeepLearningFrameworks/tree/mxnetdev

If I pull this into master it will lose you as contributor, so if you want you can pull this branch into your fork instead of master and issue a PR which I will pull into master, preserving you as contributor. Can do same for the multiGPU examples, unless you suggest otherwise?

[ilkarman]

Also if you fancy being added as a collaborator to make future PRs easier?

[ThomasDelteil]

@ilkarman thanks, I'll be very happy to be a collaborator 😃
I'll open a PR with your branch today