NestedVector: exceeds max size error

[jedfrechette]

While using the prebuilt binary for version 18.60 I've been running in to errors like the following with some point clouds:

[ERROR] C:\Research\PoissonRecon\PoissonRecon\Src\nestedVector.h (Line 229)
              PoissonRecon::NestedVector<int,1,20>::resize
              Resize size exceeds max size, considering increasing nesting:
1844674407156206222 > 1099511627776

In the particular case I'm looking at now this error repeats 3 times with the number to the left of the > increasing by a small amount (a few hundred units) each time before failing completely. Older versions, from before NestedVector were implemented, are able to run successfully using the same parameters on this data set.

[mkazhdan]

How big is the point-set and what depth are you using?

[mkazhdan]

If you can compile the code, the fix is to change line 54 in PreProcessor.h from:
#define NESTED_VECTOR_LEVELS 1
to:
#define NESTED_VECTOR_LEVELS 2

[jedfrechette]

I've got a few data sets with similar properties that I'm working with at the moment, some of which fail and some of which work. Here are 2 examples, both of which are being reconstructed at a depth of 14

Data set #1 has 120 million points and a physical bounding box of 101x112x12 and fails with the nested vector error.
Data set #2 has 130 million points and a physical bounding box of 101x134x7 and reconstructs without an error.

It should be noted that neither of these data sets uniformly fill the bounding box and instead consist of a strip of points roughly 50 units wide oriented diagonally from the bottom left to top right corner.

[mkazhdan]

Thanks. Out of curiosity, did the problem resolve when you recompiled with two levels of nesting? (If you are having trouble compiling, I can also post a second executable using two-level-nesting.)

[jedfrechette]

I don't have a dev environment setup at the moment so if you are able to post a binary that would help me be able to test sooner.

BTW, I started going back through the previous binaries that are available and 13.80 is the last one that is able to reconstruct my data set # 1. Versions 13.99 to 18.04 fail with a Segmentation fault and then 18.05 fails with the nested vector error.

[mkazhdan]

Apologies. I was not able to find the datasets and so was not able to test on them.
But I did post a new .zip file containing PoissonRecon.BigData.exe which uses two nesting levels.
It's a little slower and uses more memory, but hopefully should do the trick.

[jedfrechette]

The 18.60 BigData version does the trick, but at the expense of memory as you say.

For my test data set # 2 (I can't share these data sets publicly), that increase was about 50%, with the 1 level nesting 18.60 binary using a maximum of 42.6 GB and the 2 level BigData binary requiring 66.1 GB.

[mkazhdan]

I see. I conservatively also increased the number of bits required for storage. If the memory usage is an issue, I can take that back down. It should reduce the memory, though the increase in running time that comes with using two-level nesting would remain.

[mkazhdan]

Just posted an updated PoissonRecon.TwoLevel.exe

[jedfrechette]

I’ll be out of the office for the next couple of days but am happy to run more tests when I get back, if that would be helpful.

At least, for my use case, the increased memory usage and run time isn’t an issue. Memory usage is manageable and I'd rather have the process take a little longer and complete rather than fail. That being said, memory isn't always so freely available so would it be possible to estimate when the 2nd nesting level may be required and only enable it in those scenarios?

I confess I haven't dug into the code, and at this point I don't have an intuitive sense of which variables are most important here, so any guidance on what tests would be most helpful would be appreciated.