Grow LMDB map size incrementally

[lukeyeager]

Several people have proposed decreasing the default LMDB map size from 1TB down to something smaller (#3644, #3728). The large default causes problems on ARM and Windows, and makes Valgrind unhappy (#2404).

@hyc, a maintainer (the author?) of LMDB, suggested catching MDB_MAP_FULL errors and growing the map size as needed.

you can just grow the map whenever you hit a MDB_MAP_FULL error
NVIDIA/DIGITS#206 (comment)

This was implemented with only a few lines of code in DIGITS:
https://github.com/NVIDIA/DIGITS/blob/v3.0.0/tools/create_db.py#L636-L660

2016-02-25 12:48:41 [DEBUG] Processed 200/60000
2016-02-25 12:48:42 [DEBUG] Doubling LMDB map size to 20MB ...
2016-02-25 12:48:43 [DEBUG] Processed 500/60000
2016-02-25 12:48:44 [DEBUG] Doubling LMDB map size to 40MB ...
2016-02-25 12:48:45 [DEBUG] Processed 800/60000
2016-02-25 12:48:48 [DEBUG] Processed 1200/60000
2016-02-25 12:48:48 [DEBUG] Doubling LMDB map size to 80MB ...
2016-02-25 12:48:50 [DEBUG] Processed 1600/60000
2016-02-25 12:48:52 [DEBUG] Processed 1800/60000
2016-02-25 12:48:54 [DEBUG] Processed 2200/60000
2016-02-25 12:48:56 [DEBUG] Doubling LMDB map size to 160MB ...
2016-02-25 12:48:56 [DEBUG] Processed 2600/60000
...

I submit that the same should be done for Caffe rather than simply changing the default map size.

[shelhamer]

Sounds like a reasonable approach to me -- PRs welcome.

[hyc]

re: ARM, the MDB_VL32 feature in LMDB is working well for 32bit platforms (including ARM and Win32) and large mapsizes aren't an issue with them. Naturally there has never been an issue with ARM64 or Win64.

[lukeyeager]

Naturally there has never been an issue with ARM64 or Win64.

To be clear, Win32 fails to create a 1TB file because it won't allow files greater than 4GB, while Win64 technically doesn't crash but still creates a 1TB file? That still sounds like a problem to me. Or am I missing something?

Either way, the proposed solution is still the right idea - correct?

[hyc]

Win32 will certainly create files larger than 4GB if you ask it. E.g. most DVDs are larger than 4GB... The MDB_VL32 patches allows 32bit systems to operate on 64bit DBs. LMDB/lmdb@29ebd02

As for 1TB files, with the patches for ITS#8324 LMDB will no longer preallocate the entire file on Windows; instead it will grow incrementally.

The proposed solution works of course. But in general it's not necessary. (Large map sizes are still a problem for valgrind of course.)

[lukeyeager]

Thanks for the clarifications, @hyc. If I'm understanding things correctly (and I'm probably still not),

• Without LMDB/lmdb@fb5a768, Windows will allocate 1TB files on disk
• Without LMDB/lmdb@29ebd02, 32-bit systems are unable to create a 1TB map

That's great that these patches are in the master branch, but neither are present with version 0.9.10, which comes with the Ubuntu 14.04 package.

I'd vote for making Caffe work with old versions of LMDB rather than asking people to run bleeding-edge code for a dependency.

[hyc]

Yes, I think you've got it correct.

LMDB 0.9.10 is ancient, released Nov 2013. LMDB 0.9.18 was released a few weeks ago.
But OK, do whatever makes sense for your user population...