Use depth view in object detection #83
Conversation
|
I want to test something before I submit. |
src/camera/CameraUtils.ts
Outdated
| const depthUV = transformRgbToDepthUv(rgbUv, xrDeviceCamera); | ||
| const depthUV = transformRgbToDepthUv( | ||
| rgbUv, | ||
| viewProjectionMatrix.clone().invert(), |
There was a problem hiding this comment.
We're currently inverting viewProjectionMatrix both times we use it. For a minor optimization, we could move the line that creates viewProjectionMatrixInverse up and use that so we only invert once.
| this.depthProjectionMatrices[viewId].copy(camera.projectionMatrix); | ||
| this.depthViewMatrices[viewId].copy(camera.matrixWorldInverse); | ||
| } | ||
| this.depthViewProjectionMatrices[viewId].multiplyMatrices( |
There was a problem hiding this comment.
One thing to double-check during testing: the depth view-projection matrix currently seems to built as view * projection. Given how it's later passed around and applied as a clip-from-world transform, I wonder if it may need to be the other way around (projection * view)?
There was a problem hiding this comment.
Yes good catch. Thanks.
dli7319
force-pushed
the
depth_view_object_detection
branch
3 times, most recently
from
1bdf8e1 to
4283965
Compare
|
@nsalminen @ruofeidu Can you review this again? I made tons of changes. You can now change the simulator depth fov. |
dli7319
force-pushed
the
depth_view_object_detection
branch
from
4283965 to
a9ba290
Compare
ruofeidu
left a comment
ruofeidu
left a comment
There was a problem hiding this comment.
LGTM, thanks Nels for the catch!
3 participants
Setting matchDepthView to false allows us to use depth without incurring the performance penalty of Chrome reprojecting the depth. Now, we reproject the depth samples when running object detection which should be much faster.
Tested on Moohan only.