add option for lmdb

[mavenlin]

This is to replace PR #386.

[shelhamer]

Thanks for the rebase and making this optional! However, it doesn't build. Perhaps a missing include?

src/caffe/layers/data_layer.cpp:206:24: error: use of undeclared identifier 'MDB_NOTLS'
        MDB_RDONLY|MDB_NOTLS, 0664), MDB_SUCCESS) << "mdb_env_open failed";

[mavenlin]

@shelhamer I did include lmdb.h in the data_layer.hpp file.
Strange it does not complain about lmdb.h not found but instead MDB_NOTLS undeclared.
It builds on my machine though.

[sergeyk]

Yeah I'm trying to figure out what's going on with the homebrew lmdb. Its header file does not include MDB_NOTLS flag, but all other ones I can find on the internet do. Trying to build lmdb from source right now

[shelhamer]

Alright, this works fine on Ubuntu 12.04, although I had to install from source since there isn't a package:

git clone git://gitorious.org/mdb/mdb.git
cd mdb/libraries/liblmdb
make && sudo make install

[shelhamer]

@sergeyk if you get lmdb to install right on osx please PR it to homebrew.

[sergeyk]

Okay, I got lmdb to work correctly with this brew recipe: https://gist.github.com/sergeyk/bb8274c218ada01d1fea

@shelhamer could you check that this works for you?

[sergeyk]

@mavenlin Great PR!

One last thing: could you include a test for the lmdb functionality? It should go into test_data_layer.cpp; you basically need to add a FillLMDB method, analogous to FillLevelDB, and then include some cases.

Bonus contributor points if you also adjust the docs/installation.md to note the new dependency on lmdb

[shelhamer]

@sergeyk your formula works for me, and clones from the same repo I built on linux. I dropped my homebrew fork to silence some notifications, could you PR it?

[sergeyk]

@shelhamer yup

[sergeyk]

@mavenlin As soon as you include a test for the lmdb functionality (see my comment above), we will merge. I like LMDB.

[mavenlin]

Hi @sergeyk, I've been busy with my project these days, I'll get back asap.

[sergeyk]

NIPS time, we're in the same boat. Thanks!

[mavenlin]

@sergeyk I've updated the tests, basically just copied from test_data_layer, changed FillLevelDB to FillLMDB.

[shelhamer]

lmdb now properly installs by homebrew as of Homebrew/legacy-homebrew@182654e

[jeffdonahue]

@sergeyk fyi I did some testing and cleanup of this last night, looks good to merge to me.

[jeffdonahue]

(Though ideally we'd have an option in convert_imageset.cpp to output to LMDB, which I think @mavenlin said he was working on.)

[mavenlin]

Tools added, if anything missing, please let me know.

[sergeyk]

Looks great, thanks for the great PR @mavenlin! I'm a new fan of lmdb -- seems like the best way to have multiple processes reading the same large dataset.

[kloudkl]

I rebased #478 and found that the steps to install lmdb were not added in the documentation. @sergeyk, could you fix it?

[sguada]

@sergeyk @mavenlin, now we need to add one more requirement to the installation.

Maybe we should define a way to make new requirements optional.

[mavenlin]

@sguada @kloudkl @sergeyk
I think there are several ways:

1. create git submodules, put lmdb as a submodule of caffe, so that it will be automatically checkout and made during make of caffe. Other dependencies managed by git can also be add as submodules.
2. create a plugin system for caffe, so that database backends can be added without changing the main code of caffe. The plugin system can extend to other components of caffe, for example, layers.
3. define a macro, so that lmdb backend will not be made if the user's system does not support.
4. create a ppa for lmdb, so that it can be install the same way as all the other components through apt-get.

[sergeyk]

(3) is easiest, (2) is best. @mavenlin want to take a stab at (3)?

[mavenlin]

@sergeyk no problems

[kloudkl]

Likewise, OpenCL is optional in #415. It's up to the users to enable it explicitly.

[sguada]

@mavenlin could you also add the instructions to install lmdb to the documentation?

[hyc]

LMDB is in Ubuntu repos already... https://launchpad.net/ubuntu/+source/lmdb

[mavenlin]

@hyc but in trusty tahr only?
My group member and I tried trusty tahr, but it turns out to be very slow running caffe.
It might be due to the driver, we tried a whole bunch of drivers, but none work out.

[hyc]

What kernel is in trusty tahr? We've hit a lot of performance issues in recent kernels. http://www.openldap.org/lists/openldap-technical/201312/msg00133.html
http://www.openldap.org/lists/openldap-devel/201309/msg00008.html

[mavenlin]

@hyc We also tried downgrading the kernel to older ones, but failed. We didn't explore more on this but just downgrade the whole system to 13.10.

[mavenlin]

@hyc Oh, I didn't mean that lmdb is slowed down on 14.04, either with leveldb and lmdb the performance got very slow. I think it is cuda, as previous issues pointed out that some versions of nvidia drivers cause caffe to be slow. Only a limited number of nvidia drivers can be installed on 14.04, older versions usually fail the installation.

[shelhamer]

@mavenlin the installation docs still need updating to include lmdb for at least Ubuntu 12.04 and OS X since these are the official platforms (and ideally more recent Ubuntu versions too). Thanks.