Version >=0.13.1 doesn't work with digits-server

[lukeyeager]

I’m going crazy trying to track down this bug. I’ve been trying to find something to go on, but I don’t have much. I can’t find any error messages anywhere.

I’ve been noticing that the DIGITS “production” server (digits-server) crashes sometimes, but the “development” server (digits-devserver) never does.

Caffe v0.12.2 is fine. So is v0.13.0. But v0.13.1 and v0.13.2 crash the production server. There’s something in these changes that doesn’t play nice with the production server.

Version cuDNN   CNMeM   digits-devserver        digits-server   Crash time      Last message
0.12.2  v2              OK                      OK
0.13.0  v3              OK                      OK
0.13.0                  OK                      OK
0.13.1  v3      1.0.0   OK                      CRASH           net.forward()   None
0.13.1  v3              OK                      CRASH           caffe.Net()     cudnn_conv_layer.cpp:256] Reallocating workspace storage: 100
0.13.1          1.0.0   OK                      CRASH           net.forward()   None
0.13.1                  OK                      CRASH           net.forward()   None

The production server uses gunicorn for the webserver framework, and the development server uses Flask. I’m looking into what the differences could be (path setup, memory usage, environment variables, etc.) but I haven’t come up with anything so far. Any ideas about what I should look for?

Things I’ve investigated:

• Make vs. CMake
  • makes no difference (no pun intended)
• Out-of-memory
  • I’m using LeNet on a 6GB card. Should be no problem.
  • Plus, there’s no out-of-memory errors
• Timeout
  • Nope. When it works, this finishes in ~0.002 seconds. And when it fails, it fails pretty much instantly, too.

/cc @slayton @drnikolaev

[lukeyeager]

Steps to repro:

1. Build NVcaffe 0.13.1 (any configuration of CUDA/cuDNN/CNMeM should do)
2. Start DIGITS with ./digits-server
3. Test an image on a trained model

Can anyone else verify that they're seeing the same issue?

[gheinrich]

I have the same problem on the sqrl machine:

• caffe v0.13.2 with CuDNN and CNMEM
• CuDNN v3
• CUDA 7.5
• DIGITS version efb31e5cc
• 1 Tesla K40c + 2 Tesla K40m

WARNING: Logging before InitGoogleLogging() is written to STDERR
F0929 00:49:19.445307 22468 math_functions.hpp:176] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
2015-09-29 00:49:20 [22513] [INFO] Booting worker with pid: 22513
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0929 00:49:21.574678 22513 math_functions.hpp:176] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
2015-09-29 00:49:22 [22520] [INFO] Booting worker with pid: 22520

When using the dev-server it is working fine.

[slayton58]

cudaError_t = 2 resolves to cudaErrorMemoryAllocation (driver_types.h in CUDA toolkit), so that could be a start.

You could try adding a cudaDeviceSynchronize() call in the CUDA_CHECK macro (include/caffe/util/device_alternate.hpp) before CHECK_EQ is called -- this should ensure that when the error occurs you should get more accurate reporting of where it occurred

[lukeyeager]

Fixed with v0.14