General support of OPs for second-order gradient

[lightingghost]

As I saw mxnet has autograd package to support high order gradients, I tried to implement wgan-gp with mxnet. But I got an error of

mxnet.base.MXNetError: [08:32:17] C:\projects\mxnet-distro-win\mxnet-build\nnvm\src\pass\gradient.cc:187: Operator _backward_Convolution is non-differentiable because it didn't register FGradient attribute.

It seems convolution operator still does not support higher order gradients?

[sxjscience]

Supporting higher order gradients is on the agenda. Refer to the discussion in #5699. Need to ping Eric @piiswrong

[sxjscience]

I find the issue is similar as #9979 . Let's move the discussion there.

[lightingghost]

@sxjscience I think #9979 is a question rather than a feature request. In his case, he should use autograd.grad and set create_graph=True to solve the problem. However, in this case, it seems second order derivative is not implemented for some operators?

[piiswrong]

2nd order gradient is only implemented for a few operators like * and exp.

[aodhan-domhnaill]

I looked at the code and it seems like the problem is that the current implementation is kind of only doing symbolic math down one level. For example, sin,

// sin                                                                                                                                                                                                      
MXNET_OPERATOR_REGISTER_UNARY_WITH_RSP(sin, cpu, mshadow_op::sin)
MXNET_ADD_SPARSE_OP_ALIAS(sin)
.describe(R"code(Computes the element-wise sine of the input array.                                                                                                                                         
                                                                                                                                                                                                            
The input should be in radians (:math:`2\pi` rad equals 360 degrees).                                                                                                                                       
                                                                                                                                                                                                            
.. math::                                                                                                                                                                                                   
   sin([0, \pi/4, \pi/2]) = [0, 0.707, 1]                                                                                                                                                                   
                                                                                                                                                                                                            
The storage type of ``sin`` output depends upon the input storage type:                                                                                                                                     
                                                                                                                                                                                                            
   - sin(default) = default                                                                                                                                                                                 
   - sin(row_sparse) = row_sparse                                                                                                                                                                           
                                                                                                                                                                                                            
)code" ADD_FILELINE)
.set_attr<nnvm::FGradient>("FGradient", ElemwiseGradUseIn{ "_backward_sin" });

MXNET_OPERATOR_REGISTER_BINARY_WITH_SPARSE_CPU_DR(_backward_sin, unary_bwd<mshadow_op::sin_grad>);

The fact that it names it _backward_sin instead of using cos is probably why this can't chain the differentiation down.

[sxjscience]

@lightingghost Borrow the discussion in #9979 here

import mxnet.ndarray as nd
from mxnet import autograd

x = nd.array([3.0])
x.attach_grad()
with autograd.record():
    y = x**2
    y_grad = autograd.grad(y, x, create_graph=True, retain_graph=True)[0]
    z = y_grad ** 2
z.backward()
print(z.grad)

MXNetError: [12:44:29] src/pass/gradient.cc:187: Operator _backward_power_scalar is non-differentiable because it didn't register FGradient attribute.

[sxjscience]

@aidan-plenert-macdonald That's correct, we need to check the backward pass of these operators and register the gradient.

[sxjscience]

We need to investigate the current OPs and check whether their backward OPs have registered the gradient. Also, we need to add a new testing utility to help check the correctness of our implementation.

[aodhan-domhnaill]

@sxjscience Would it not be better to register the backward OPs as combinations of the forward ones so the gradients can be computed endlessly? Basically not limiting the functionality to 2nd order, but the option of going to nth order

[sxjscience]

@aidan-plenert-macdonald Yes, we should do that.