bug in distributed async training using DCASGD (at least)

[feevos]

Description

Dear all, following the tutorial on distributed training (which runs perfectly in dist_sync mode) I tried to modify it and run in asynchronous mode using DCASGD. The code crashes giving a lot of error messages. The code does not crash when I use other optimizers (SGD,Adam) however, the model does not seem to train ((accuracy remains constant up to 5 epochs, and very low, for Adam (lr = 0.001, it is 0), for SGD(m=0.8, lr = 0.1) it is 0.1 (weird that it has same value with lr). I don't know if the problem is located only in DCASGD or in kv_store (see also #11910 and #11855).

Environment info (Required)

----------Python Info----------
Version : 3.6.5
Compiler : GCC 7.2.0
Build : ('default', 'Apr 29 2018 16:14:56')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 10.0.1
Directory : /data/dia021/Software/anaconda3/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version : 1.3.0
Directory : /data/dia021/Software/mxnet
Commit Hash : 8917318
----------System Info----------
Platform : Linux-4.4.114-94.11-default-x86_64-with-SuSE-12-x86_64
system : Linux
node : bracewell-login
release : 4.4.114-94.11-default
version : #1 SMP Thu Feb 1 19:28:26 UTC 2018 (4309ff9)
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-27
Off-line CPU(s) list: 28-55
Thread(s) per core: 1
Core(s) per socket: 14
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
Stepping: 1
CPU MHz: 2599.993
BogoMIPS: 5199.98
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb invpcid_single pln pts dtherm intel_pt spec_ctrl retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx xsaveopt cqm_llc cqm_occup_llc
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0028 sec, LOAD: 1.0938 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.9857 sec, LOAD: 0.6167 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 1.1474 sec, LOAD: 1.7171 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.3098 sec, LOAD: 1.5781 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0472 sec, LOAD: 1.3546 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0127 sec, LOAD: 0.0574 sec.

Package used (Python):

Error Message:

Error message is produced ONLY with DCASGD optimizer.

2018-08-01 17:36:59,801 INFO rsync /home/dia021/Projects/isprs_potsdam/distributed/ -> b039:/home/dia021/Projects/isprs_potsdam/distributed
2018-08-01 17:37:00,154 INFO rsync /home/dia021/Projects/isprs_potsdam/distributed/ -> b040:/home/dia021/Projects/isprs_potsdam/distributed
[17:37:10] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[17:37:10] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__

  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
Traceback (most recent call last):
  File "cifar10_dist.py", line 174, in <module>
    train_batch(batch, ctx, net, trainer)
  File "cifar10_dist.py", line 164, in train_batch
    trainer.step(this_batch_size)
  File "/data/dia021/Software/mxnet/gluon/trainer.py", line 243, in step
    self._update(ignore_stale_grad)
  File "/data/dia021/Software/mxnet/gluon/trainer.py", line 336, in _update
    upd(i, grad, arr)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "cifar10_dist.py", line 174, in <module>
    train_batch(batch, ctx, net, trainer)
  File "cifar10_dist.py", line 164, in train_batch
    trainer.step(this_batch_size)
  File "/data/dia021/Software/mxnet/gluon/trainer.py", line 243, in step
    self._update(ignore_stale_grad)
  File "/data/dia021/Software/mxnet/gluon/trainer.py", line 336, in _update
    upd(i, grad, arr)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.

  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/data/dia021/Software/mxnet/kvstore.py", line 83, in updater_handle
    updater(key, lhs, rhs)
  File "/data/dia021/Software/mxnet/optimizer.py", line 1483, in __call__
    self.optimizer.update_multi_precision(index, weight, grad, self.states[index])
  File "/data/dia021/Software/mxnet/optimizer.py", line 274, in update_multi_precision
    self.update(index, weight, grad, state)
  File "/data/dia021/Software/mxnet/optimizer.py", line 894, in update
    if mom:
  File "/data/dia021/Software/mxnet/ndarray/ndarray.py", line 352, in __bool__
    raise ValueError("The truth value of an NDArray with multiple elements " \
ValueError: The truth value of an NDArray with multiple elements is ambiguous.

Minimum reproducible example

I am operating in an HPC environment, that uses SLURM manager. The script to launch the job is:

#!/bin/bash -l

#SBATCH --job-name="DSTR"
#SBATCH -q express
#SBATCH -t 02:30:30
#SBATCH --nodes=2
#SBATCH --cpus-per-task=28
#SBATCH --gres=gpu:4
#SBATCH --mem=128gb


./get_nodes_names.sh > workers_ip.txt

python /data/dia021/Software/mxnet/tools/launch.py  -n $(wc -l < workers_ip.txt) -s  $(wc -l < workers_ip.txt) -H workers_ip.txt   --sync-dst-dir /home/dia021/Projects/isprs_potsdam/distributed   --launcher ssh "python cifar10_dist.py"

the executable get_nodes_names.sh is a small script that prints the IP addresses (or node names) in workers_ip.txt file. With the same script I run also dist_sync and goes great.

The executable I am running is cifar10_dist.py modified from it's original version in lines 30 and 102, this is the complete executable:

#!/usr/bin/env python

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

from __future__ import print_function
import random, sys

import mxnet as mx
from mxnet import autograd, gluon, kv, nd
from mxnet.gluon.model_zoo import vision

import numpy as np

# @@@@@@@@@@@@ MODIFICATION HERE @@@@@@@@@@@@@@@@@
# Create a distributed key-value store
store = kv.create('dist_async')
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

# Clasify the images into one of the 10 digits
num_outputs = 10

# 64 images in a batch
batch_size_per_gpu = 64
# How many epochs to run the training
epochs = 2

# How many GPUs per machine
gpus_per_machine = 4
# Effective batch size across all GPUs
batch_size = batch_size_per_gpu * gpus_per_machine

# Create the context (a list of all GPUs to be used for training)
ctx = [mx.gpu(i) for i in range(gpus_per_machine)]

# Convert to float 32
# Having channel as the first dimension makes computation more efficient. Hence the (2,0,1) transpose.
# Dividing by 255 normalizes the input between 0 and 1
def transform(data, label):
    return nd.transpose(data.astype(np.float32), (2,0,1))/255, label.astype(np.float32)

class SplitSampler(gluon.data.sampler.Sampler):
    """ Split the dataset into `num_parts` parts and sample from the part with index `part_index`

    Parameters
    ----------
    length: int
      Number of examples in the dataset
    num_parts: int
      Partition the data into multiple parts
    part_index: int
      The index of the part to read from
    """
    def __init__(self, length, num_parts=1, part_index=0):
        # Compute the length of each partition
        self.part_len = length // num_parts
        # Compute the start index for this partition
        self.start = self.part_len * part_index
        # Compute the end index for this partition
        self.end = self.start + self.part_len

    def __iter__(self):
        # Extract examples between `start` and `end`, shuffle and return them.
        indices = list(range(self.start, self.end))
        random.shuffle(indices)
        return iter(indices)

    def __len__(self):
        return self.part_len

# Load the training data
train_data = gluon.data.DataLoader(gluon.data.vision.CIFAR10(train=True, transform=transform),
                                      batch_size,
                                      sampler=SplitSampler(50000, store.num_workers, store.rank))

# Load the test data 
test_data = gluon.data.DataLoader(gluon.data.vision.CIFAR10(train=False, transform=transform),
                                     batch_size, shuffle=False)

# Use ResNet from model zoo
net = vision.resnet18_v1()

# Initialize the parameters with Xavier initializer
net.collect_params().initialize(mx.init.Xavier(), ctx=ctx)

# SoftmaxCrossEntropy is the most common choice of loss function for multiclass classification
softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss()

# @@@@@@@@@@@@ MODIFICATION HERE @@@@@@@@@@@@@@@@@
# Create a distributed key-value store
# This is for distributed ASYCHRONOYS training 
opt = mx.optimizer.DCASGD(momentum=0.8, learning_rate=0.01)
store.set_optimizer(opt)l
trainer = gluon.Trainer(net.collect_params(), optimizer=opt, kvstore=store, update_on_kvstore=True) 
# @@@@@@@@@@@@ MODIFICATION HERE @@@@@@@@@@@@@@@@@
# Create a distributed key-value store


# Evaluate accuracy of the given network using the given data
def evaluate_accuracy(data_iterator, net):

    acc = mx.metric.Accuracy()

    # Iterate through data and label
    for i, (data, label) in enumerate(data_iterator):

        # Get the data and label into the GPU
        data = data.as_in_context(ctx[0])
        label = label.as_in_context(ctx[0])

        # Get network's output which is a probability distribution
        # Apply argmax on the probability distribution to get network's classification.
        output = net(data)
        predictions = nd.argmax(output, axis=1)

        # Give network's prediction and the correct label to update the metric
        acc.update(preds=predictions, labels=label)

    # Return the accuracy
    return acc.get()[1]

# We'll use cross entropy loss since we are doing multiclass classification
loss = gluon.loss.SoftmaxCrossEntropyLoss()

# Run one forward and backward pass on multiple GPUs
def forward_backward(net, data, label):

    # Ask autograd to remember the forward pass
    with autograd.record():
        # Compute the loss on all GPUs
        losses = [loss(net(X), Y) for X, Y in zip(data, label)]

    # Run the backward pass (calculate gradients) on all GPUs
    for l in losses:
        l.backward()

# Train a batch using multiple GPUs
def train_batch(batch, ctx, net, trainer):

    # Split and load data into multiple GPUs
    data = batch[0]
    data = gluon.utils.split_and_load(data, ctx)

    # Split and load label into multiple GPUs
    label = batch[1]
    label = gluon.utils.split_and_load(label, ctx)

    # Run the forward and backward pass
    forward_backward(net, data, label)

    # Update the parameters
    this_batch_size = batch[0].shape[0]
    trainer.step(this_batch_size)

# Run as many epochs as required
for epoch in range(epochs):

    # Iterate through batches and run training using multiple GPUs
    batch_num = 1
    for batch in train_data:

        # Train the batch using multiple GPUs
        train_batch(batch, ctx, net, trainer)

        batch_num += 1

    # Print test accuracy after every epoch
    test_accuracy = evaluate_accuracy(test_data, net)
    print("Epoch %d: Test_acc %f" % (epoch, test_accuracy))
    sys.stdout.flush()

(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)

Steps to reproduce

Under a slurm environment, run the bash script using sbatc script.sh

What have you tried to solve it?

Tried different optimizers, and it seems the problem persists only on DCASGD however it may be related to kvstore (because async doesn't train).

[Roshrini]

Thanks for reporting this issue.
@nswamy Can you please add labels: Bug, Distributed, Optimizer