Blobs in reconstruction from iPhone depths

[eduardohenriquearnold]

Hi,

I am using the olohuone/iphone/short_capture capture to reconstruct a mesh using TSDF integration with the provided colmap poses in transformations_colmap.json and iPhone depth maps. When I sample a pointcloud from the TSDF I notice there are many blobs in the middle of the room, indicating that either there are bad poses or bad depth maps. See attached the TSDF point cloud (yellow) overlaid on GT pointcloud (color).

I have also tried using the transformations.json containing Polycam poses/intrinsics and found the same issue. I also attempted to remove depths greater than 8m to no avail. There's still the question of filtering depth maps by confidence, pending #9 , but I'm also wondering if there are other artifacts in the depth maps - e.g. how were they re-scaled from their native resolution (256, 192)?

Have you reconstructed a mesh from iPhone depths before and, if so, did you observe any of these issues before?

[maturk]

Those streaks / wrong points you see around object edges are common for the iPhone depths and it is due to them being quite inaccurate precisely at object boundaries / edges. So backprojecting the edges (I guess it still seems to occur even in the TSDF) results in the edges of objects being quite inconsistent.

What has worked well for me in the past is manually filtering edges from the raw iPhone depthmaps before any meshing / TSDF / backprojection operations. You can do this via a "dilated" canny-edge detection method or otherwise.

[eduardohenriquearnold]

I don't think we're talking about the same issue. The blobs I'm talking about are not at the boundaries of objects, they are floating mid-air, see below:

I believe these are bad depths observed from sensors far on the other side of the room. I tried ignoring depths greater than 5m as a proxy for confidence and that has removed the majority of these blobs.

In other words, having the iphone lidar confidence maps (#9) would be really useful!

[maturk]

Hmmm that doesn't look right to me at all, and from my experience the iPhone depths are not that bad when simply back-projecting naively. Using the transformations_colmap.json poses is correct, but the results do not look right. For reference, here is a code snippet that creates a TSDF volume from the iphone depths and then saves a pointcloud from the TSDF. Are you certain there are no bugs, like using the wrong intrinsics or perhaps using the wrong pose format. Note the cheeky np.linalg.inv(pose) and OPENGL_TO_OPENCV conversion in the code-snippet above.

Disclaimer, it has been a very long time since I have myself messed with the mushroom dataset or the above code.

I haven't checked the olohuone scene but here is a screen grab of the vr_room using the above code.

[eduardohenriquearnold]

Thanks for the reply.

This is quite a large scene, the iphone depth range is supposedly accurate for < 5m (Polycam ref), so when you're 15m away from the wall, the depth error can be really high.

Thanks for the reference code, I have used it and obtained the same results. Do note that the code is clipping the depths at 4m (https://github.com/maturk/dn-splatter/blob/97588b4290128ce7ba6fdbfaac3020b42b17de4c/dn_splatter/data/mushroom_utils/pointcloud_utils.py#L157), so you won't see the artifacts I'm mentioning unless you increase that threshold.