The GIL isn't released when called from Python

[seanbell]

Are there any plans to allow for releasing the Python GIL? When using Python data layers, this allows for multithreading inside the data layer.

A PR for this feature is on the Berkeley caffe project: BVLC#2230

[drnikolaev]

Hi @seanbell . Well, https://github.com/NVIDIA/caffe/blob/caffe-0.16/include/caffe/layers/python_layer.hpp#L23
Are there any specific failures?

[seanbell]

Hey @drnikolaev -- I searched the repo but somehow didn't see that. I'll try it out and let you know if I have any issues. Thanks!!

[RSly]

hi @seanbell did you have any progress in multi-threaded python data layer?
I am interested to know what the steps are to achieve this?

maybe you have a public code to share with us?
thanks

[seanbell]

Hi @drnikolaev -- I don't think that this actually allows for multithreading in Python. In particular, when you call functions like net.forward() or solver.step() from Python (in python/caffe/_caffe.cpp), the C++ backend code should release the GIL and later re-acquire it inside the Python layer just before jumping into Python. While it seems to be doing something related to the GIL inside the Python layer, it's not releasing the GIL in python/caffe/_caffe.cpp.

I've been trying to figure out why my code is running so much slower on this version (compared to my own fork that incorporates BVLC#2230), and I think it's the lack of GIL release.

[drnikolaev]

Got it, thanks. Little bit busy these days but I'll have a look. Thank you.

[drnikolaev]

BTW, @seanbell - may I have some good code snippet and what perf do you expect from it? Would help me as a reference.

[seanbell]

Sure, I'll work on getting something together soon.

The main thing that I expect to be able to do is run a background thread inside a Python data layer (e.g. a prefetching thread) that runs in parallel with the GPU to get the next minibatch ready, communicating with a queue.

[seanbell]

@drnikolaev If I make a PR against some branch, what should I use?

[drnikolaev]

@seanbell please use https://github.com/NVIDIA/caffe/tree/caffe-0.16

[seanbell]

Hi @drnikolaev -- here is the patch that I used on a different private branch:
seanbell-GIL-based-on-BVLC-2230.patch.txt

This was applied to an earlier version of nv-caffe (a version also called 0.16, but when it was still experimental, which later got overwritten by a new 0.16). It won't work as-is, but should give a sense of the main changes.

Main notes:

• _caffe.cpp: release the GIL inside each of the main Python function calls (look for ReleaseGIL). This is the important part that I think is missing from the current v0.16 code.
• PythonLayer: acquire the GIL before calling any Python functions (look for AcquireGIL). This is now needed given the above GIL release.
• GetPythonLayer: check to make sure that the layer name is valid upfront. Not really needed, but nice to have.
• PYTHON_LAYER_ERROR: improved Python error macro from [Python] PythonLayer takes parameters by string and better exception handling BVLC/caffe#2001.
• This code was based on [Python] release/acquire Global Interpreter Lock (GIL)  BVLC/caffe#2230 (as it appeared many months ago)

Regarding performance, without the ability to run a background Python thread in parallel, the GPU must wait for the entire minibatch to be prepared, which results in ~1.35-1.5x slower training for me.

[seanbell]

@drnikolaev let me know how I can help. This is a major bottleneck for us.