fix operators to support large arrays.

[zheng-da]

We're working on a model that requires very large NDArrays. For example, we want to create an NDArray as follows:

arr = mx.nd.random.normal(shape=(50000000, 100))

The current implementation doesn't fail with an error, but it doesn't generate a matrix correctly (it only fills the rows at the beginning).

mx.nd.zeros also fails.

It's unclear what operators support and which operators don't.

[frankfliu]

@mxnet-label-bot [Bug, Operator]

[wkcn]

The type of iterator is int in MXNet kernel.
So it will fail when the number of iteration is greater than 2^31 - 1, (namely 2147483647).
50000000 * 100 > 2^31 - 1

Please see the code:
CPU:
https://github.com/apache/incubator-mxnet/blob/master/src/operator/mxnet_op.h#L506

GPU:
https://github.com/apache/incubator-mxnet/blob/master/src/operator/mxnet_op.h#L627

[apeforest]

Will take a look

[apeforest]

@wkcn is right. We need to chang int to index_t. I am busy with other tasks now and can only come to this in one week. Let me know if it requires an immediate fix.

[apeforest]

JIRA task created: https://issues.apache.org/jira/browse/MXNET-1185

[wkcn]

@apeforest
will it drop the performance down If changing the type of iteration to int64_t?
In PyTorch, the type of iteration is a template type.

[zheng-da]

i'm fixing some of the operators. but we need a systematic fix. The problem is everywhere. i'll provide a temp fix for some of the operators.

[zheng-da]

@wkcn in cpu, it shouldn't be a problem. I heard concerns on GPUs. Potentially, we can use int64_t for CPU and int for GPU.

[wkcn]

@zheng-da
Maybe we can try to use int32_t for small for-loop, and int64_t for large for-loop.

[zheng-da]

@wkcn My concern is that this modification makes the code complex. As for using different int types for CPU and GPU, it's relatively easier. We can use the template argument to easily achieve it.

[zheng-da]

@pengzhao-intel what is the performance difference between int32 and int64 in intel CPUs?

[zheng-da]

@apeforest I have fixed some of the operators, including all random generators, zeros, ones, full, arange, gather_nd.
zheng-da@2c3d9a3
But we need to do more to fix the rest of the operators.