Set map_size for LMDB

[lukeyeager]

@crohkohl, @danst18, I'm breaking the discussion in #203 out into a new issue.

Here's the situation as I understand it. Please correct me if any of this is wrong.

map_size	Linux	OSX & Windows
lower than size of dataset	LMDB runs out of memory	?
higher than system memory	No problem	LMDB can't allocate enough memory

On Linux, you can just set it as high as you like and never see a problem. But that strategy blows up on other platforms.

Should [map_size] be made configurable?
#203 (comment)

This is a sufficient but lazy solution. I would like to understand whether this can be avoided programmatically somehow before making a decision. My googling skills are failing me.

[crohkohl]

okay, my googling let to this issue: jnwatson/py-lmdb#85
It seams that the possible patch leads to a significant loss of speed.

However, I think a practical solution would be to parse the input data file in create_db.py and compute the total required size and give it a 50 mb buffer on top.

[lukeyeager]

Thanks for pointing out that issue on py-lmdb, that provides some good context.

I think a practical solution would be to parse the input data file in create_db.py and compute the total required size and give it a 50 mb buffer on top.

How will you know how much data is required for each image / data point? Remember they may be encoded as PNGs or JPEGs, and they may be different resolutions as well.

[crohkohl]

How will you know how much data is required for each image / data point? Remember they may be encoded as PNGs or JPEGs, and they may be different resolutions as well.

I think it is save to assume the non-encoded size as an upper bound which is NumBytes=NumImages*shape[0]*shape[1]*shape[2]*sizeof(datatype)*sizeof(label) bytes. It should be sufficiently large and much smaller than 1 TB. We can also use min(NumBytes,1TB) if we want to stick with the 1 TB.

[hyc]

If all you need is to do a bulk import, and never have to worry about adding new records later, you can just grow the file incrementally. Or you can use the windows sparse file patch and take the 10-20% performance hit.

[lukeyeager]

@hyc, thanks for joining the discussion!

The situation is actually the opposite. I'm only creating the databases in this context, and I don't need to read from them again - that's done with a different program (Caffe). So the memory-mapping isn't really helping at all. Can I open a write-only database in such a way that I don't need to provide a map_size at all?

[hyc]

No, there's no mode like that. But if you're creating a DB and know that no other thread will try to access it concurrently, you can just grow the map whenever you hit a MDB_MAP_FULL error. See the postfix LMDB driver for an example http://fossies.org/dox/postfix-2.11.6/slmdb_8c_source.html

[lukeyeager]

just grow the map whenever you hit a MDB_MAP_FULL error

That could work. Thanks, I'll look into that!

[lukeyeager]

Frustratingly, py-lmdb doesn't expose any way to access the map_size directly.
https://github.com/dw/py-lmdb/blob/py-lmdb_0.86/lmdb/cpython.c#L1756-L1774

So it seems I have to close and re-open the environment. In my quick little test, this didn't seem to affect performance. I'm not sure what the implications are for a big dataset. Example changes here:

lukeyeager@83e65f7

@crohkohl, can you try this out and see what happens on Windows? I'll try it with a big dataset on my machine, too.

[hyc]

You can't use the env_info method?

[crohkohl]

@lukeyeager

can you try this out and see what happens on Windows? I'll try it with a big dataset on my machine, too.

In principle it works. However, I seem to get random crashes - could it be that the resizing is not thread-safe?

[lukeyeager]

You can't use the env_info method?

It looks like that method is only for querying information.

[lukeyeager]

I seem to get random crashes - could it be that the resizing is not thread-safe?

Thanks for the testing! That's certainly possible. See what happens if you set write_threads to 1.
https://github.com/lukeyeager/DIGITS/blob/83e65f7946/tools/create_db.py#L147

It seems to work on my machine, but race conditions like this can be tricky.

[dw]

@lukeyeager A .set_mapsize method for py-lmdb is easily implemented, I'm exploring this at present.