CTC ERROR WITH CUDA ILLEGAL MEMORY ACCESS ERROR

[jiarenyf]

For bugs or installation issues, please provide the following information.
The more information you provide, the more likely people will be able to help you.

Environment info

Operating System: Ubuntu14.04

Compiler: install by pip

Package used (Python/R/Scala/Julia): python

MXNet version:0.11

Error Message:

Please paste the full error message, including stack trace.

  File "/usr/local/lib/python2.7/dist-packages/mxnet/module/module.py", line 680, in update_metric
[21:29:57] /home/travis/build/dmlc/mxnet-distro/mxnet-build/dmlc-core/include/dmlc/logging.h:304: [21:29:57] src/engine/./threaded_engine.h:329: [21:29:57] /home/travis/build/dmlc/mxnet-distro/mxnet-build/mshadow/mshadow/./stream_gpu-inl.h:55: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x19fd3c) [0x7f6953c81d3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xcf7eb8) [0x7f69547d9eb8]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd4a300) [0x7f695482c300]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd32dfc) [0x7f6954814dfc]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35a78) [0x7f6954817a78]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35c75) [0x7f6954817c75]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd33b1b) [0x7f6954815b1b]
[bt] (7) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f69420bca60]
[bt] (8) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f6963d65184]
[bt] (9) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6963a91ffd]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 8 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x19fd3c) [0x7f6953c81d3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd33094) [0x7f6954815094]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35a78) [0x7f6954817a78]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35c75) [0x7f6954817c75]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd33b1b) [0x7f6954815b1b]
[bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f69420bca60]
[bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f6963d65184]
[bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6963a91ffd]

terminate called after throwing an instance of 'dmlc::Error'
  what():  [21:29:57] src/engine/./threaded_engine.h:329: [21:29:57] /home/travis/build/dmlc/mxnet-distro/mxnet-build/mshadow/mshadow/./stream_gpu-inl.h:55: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x19fd3c) [0x7f6953c81d3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xcf7eb8) [0x7f69547d9eb8]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd4a300) [0x7f695482c300]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd32dfc) [0x7f6954814dfc]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35a78) [0x7f6954817a78]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35c75) [0x7f6954817c75]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd33b1b) [0x7f6954815b1b]
[bt] (7) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f69420bca60]
[bt] (8) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f6963d65184]
[bt] (9) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6963a91ffd]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 8 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x19fd3c) [0x7f6953c81d3c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd33094) [0x7f6954815094]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35a78) [0x7f6954817a78]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd35c75) [0x7f6954817c75]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0xd33b1b) [0x7f6954815b1b]
[bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f69420bca60]
[bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f6963d65184]
[bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6963a91ffd]

Minimum reproducible example

if you are using your own code, please provide a short script that reproduces the error.

Thank you very mush if any one can solve my problem ...

[piiswrong]

@szha

[szha]

@jiarenyf Which version are you using? 0.11.0 or 0.11.1? What's the output of

python -c 'import os, mxnet; print open(os.path.join(os.path.dirname(mxnet.__file__), "COMMIT_HASH")).readline().strip()'

[jiarenyf]

@szha
I install 2 versions of mxnet on different pc:
1, mxnet-cu80==0.10.0.post2 and "d220e192897da763ed8e8a6135f9e8b24cc1a2ce"
2, mxnet-cu80==0.11.0rc3 and "53274b4a2b0d73f3fbdb10cfb5f9ed0c8263fda7"
And both have the problem mention above ...

[szha]

Do you mind sharing a small chunk of data so that I can run the program and debug?

[szha]

@jiarenyf unfortunately I wasn't able to reproduce the problem. I'm using an EC2 p2 host which has K80 GPUs. I was able to run 100 epochs without problem. Also, I'm running from the current head of mxnet.

INFO:root:Epoch[100] Time cost=2.508 ...

Could it be hardware/driver issue?

[jiarenyf]

@szha
I found that when adding forceResize in the imageIter-kwargs, the problem solved ...
And I wonder why the imageIter center-crop the image when the img-size is not the same with the data-shape, but not given a choice to force-reshape it ...

[szha]

Thanks. Good to know.

[jiarenyf]

@szha
I found that the problem is not because of the data, but maybe because of the local variables in python functions that are collected by the gc, such that the cuda access illegal memory.
And I add "gc.disable()" at the beginning of each epoch, and then add "gc.enable()" at the end of each epoch, the problem never occurs at least for 5 days .
I think maybe this could help you to debug ...

[szha]

@jiarenyf thanks. Does it still happen if you use the latest nightly build?

[jiarenyf]

@szha
Yes. I have updated to the version 0.11.0 by "pip install mxnet-cu80==0.11.0".

[szha]

@jiarenyf the nightly build is on 0.11.1. You can get it by using pip install -U --pre mxnet-cu80

[jiarenyf]

@szha
I found another reason of that problem.

As in "https://github.com/jiarenyf/mxWrapper/blob/02fd9b0fcd37f7224648efad651a6f83a1f06d78/mxHelper/mxData.py#L158", the labels is initialize to empty, while in "https://github.com/jiarenyf/mxWrapper/blob/02fd9b0fcd37f7224648efad651a6f83a1f06d78/mxHelper/model/model.py#L98", it directly uses batch.label without considering the batch.pad.
So if not meets "data size % batch size ==0", the error occurs when accessing empty labels.

And the data set I offered you to debug has 800 images, and it meets: "800 % 100 == 0" (100 is batch size). So the problem never occurs on your workplace.

I found this problem because I happen to change the train set size to 111 and the test size to 33, where "111+33 % 100 != 0".
And after I add "https://github.com/jiarenyf/mxWrapper/blob/02fd9b0fcd37f7224648efad651a6f83a1f06d78/mxHelper/mxData.py#L169" (remove empty), and change "https://github.com/jiarenyf/mxWrapper/blob/02fd9b0fcd37f7224648efad651a6f83a1f06d78/mxHelper/mxData.py#L170" (set "pad" to 0), then the problem does not occur.

I wonder is there some error or bug in my implementation: "https://github.com/jiarenyf/mxWrapper/blob/master/mxHelper/model/model.py" ?