Implement 3D deconvolution (cuDNN)

[leezu]

This pull request adds support for 3D deconvolution when compiled with cudnn and run on GPU. It does not break the current 2D Deconvolution non-cudnn code and simply throws an error if the non-cudnn code is called with a 3D kernel.

As the non-gpu version does not support 3D deconvolution, this pull request does not add test cases to compare that their output is similar. Instead I manually compared the output to the PyTorch 3D Deconvolution implementation.

I have seen that the cudnn convolution code uses DMLC_DECLARE_FIELD(workspace).set_default(1024).set_range(0, 8192). Shall the cudnn deconvolution code be adapted to use a default of 1024 for as well? Currently the default is 512.

The following code can be used to compare with PyTorch:

import mxnet as mx
import numpy as np
import torch

TEST = "3D_TM"

if TEST == "3D_TM":
    DATA_SHAPE = (1, 1, 3, 3, 3)

    weights = np.random.normal(scale=0.3, size=(1, 1, 3, 3, 3))
    weights[0][0][0][0] = 1
    data = np.random.normal(size=DATA_SHAPE)

    ### pytorch
    t_c = torch.nn.ConvTranspose3d(1, 1, 3, bias=None)

    t_weights = torch.from_numpy(weights)
    t_c.weight.data = t_weights

    t_data = torch.from_numpy(data)
    t_v_data = torch.autograd.Variable(t_data)

    t_v_out = t_c.forward(t_v_data)
    t_out = t_v_out.data.numpy()

    ### mxnet
    ctx = mx.gpu(0)
    rand = mx.sym.Variable('rand')
    d = mx.sym.Deconvolution(rand, name='d', kernel=(3, 3, 3), num_filter=1)

    modG = mx.mod.Module(
        symbol=d, data_names=('rand', ), label_names=None, context=ctx)
    modG.bind(data_shapes=[("rand", DATA_SHAPE)])

    m_weights = mx.ndarray.array(weights)
    modG.init_params(arg_params = {'d_weight' : m_weights})

    m_data = mx.ndarray.array(data)

    batch = mx.io.DataBatch([m_data], None)
    modG.forward(batch)

    m_out = modG.get_outputs()[0].asnumpy()


elif TEST == "2D_TMM":
    DATA_SHAPE = (1, 1, 3, 3)

    weights = np.random.normal(scale=0.3, size=(1, 1, 1, 1))
    weights[0][0][0][0] = 1
    data = np.random.normal(size=DATA_SHAPE)

    ### pytorch
    t_c = torch.nn.ConvTranspose2d(1, 1, 1, bias=None)

    t_weights = torch.from_numpy(weights)
    t_c.weight.data = t_weights

    t_data = torch.from_numpy(data)
    t_v_data = torch.autograd.Variable(t_data)

    t_v_out = t_c.forward(t_v_data)
    t_out = t_v_out.data.numpy()

    ### mxnet gpu
    ctx = mx.gpu(0)
    rand = mx.sym.Variable('rand')
    d = mx.sym.Deconvolution(rand, name='d', kernel=(1, 1), num_filter=1)

    modG = mx.mod.Module(
        symbol=d, data_names=('rand', ), label_names=None, context=ctx)
    modG.bind(data_shapes=[("rand", DATA_SHAPE)])

    m_weights = mx.ndarray.array(weights)
    modG.init_params(arg_params={'d_weight' : m_weights})

    m_data = mx.ndarray.array(data)

    batch = mx.io.DataBatch([m_data], None)
    modG.forward(batch)

    m_out = modG.get_outputs()[0].asnumpy()

    ### mxnet cpu
    ctx = mx.cpu()
    modG_cpu = mx.mod.Module(
        symbol=d, data_names=('rand', ), label_names=None, context=ctx)
    modG_cpu.bind(data_shapes=[("rand", DATA_SHAPE)])
    modG_cpu.init_params(mx.init.Normal(0.02))

    modG_cpu.set_params(arg_params={'d_weight' : m_weights}, aux_params={})

    modG_cpu.forward(batch)

    m_c_out = modG_cpu.get_outputs()[0].asnumpy()

[piiswrong]

please fix gpu test

[leezu]

Tests are passing now.

The upsampling operator initialized the target_shape to (0,0), which is not meaningful. While this worked for the old Deconvolution code (as shape (0,0) was the default target_shape), this pull request adapted the default target_shape to (), which is more meaningful given that shapes of different dimension can be supplied and aligns with the way the convolution code handles the defaults.

[sxjscience]

I decide to merge this in.

[leezu]

I rebased on current master, which restarted the tests. They finished running now, so the status is again green.

[sxjscience]

Why not use grad.dptr_ directly?

[leezu]

It would be out of scope.

[piiswrong]

I just noticed 2d deconv isn't tested. Could you add tests?

[leezu]

What kind of test do you have in mind? Currently there is:

• https://github.com/dmlc/mxnet/blob/master/tests/python/unittest/test_operator.py#L607
• https://github.com/dmlc/mxnet/blob/master/tests/python/unittest/test_operator.py#L649
• https://github.com/dmlc/mxnet/blob/master/tests/python/unittest/test_operator.py#L705
• https://github.com/dmlc/mxnet/blob/master/tests/python/unittest/test_operator.py#L718
• https://github.com/dmlc/mxnet/blob/master/tests/python/gpu/test_operator_gpu.py#L362

Where the latter does a pair test between the cpu and gpu version of 2d deconv.

Would pair tests against other libraries (that could be installed via pip) be desirable?