Distance to camera and surface normal at each pixel

[benjamin-heasly]

Issue by benjamin-heasly
Friday Nov 07, 2014 at 14:53 GMT
Originally opened as https://github.com/DavidBrainard/RenderToolboxDevelop/issues/14

---

For each Ward Land rendering, we would like to read out additional facts about the object underneath each pixel:

• the surface distance to the camera
• the surface normal vector

Mitsuba might have a depth map plugin already. It also might have documentation about writing plugins for outputting data like depth maps. We should check.

[benjamin-heasly]

Comment by benjamin-heasly
Friday Nov 14, 2014 at 22:26 GMT

---

The documentation for Mitsuba 0.5 contains a detailed developer documentation that shows how to write a custom integrator which reports the distance to the camera from each visible scene surface. It also gives thorough explanations of the code. Nice.

To get surface distances to camera, I think we could implement this integrator almost exactly as documented. One change we might make would be to output the distance to the camera in raw scene units, instead of normalizing distances into the range [0 1]. Depending of the film type we write to, raw distances might require special encoding for large numbers.

To get the surface normal vector, we might be able to implement a very similar integrator. But we would need to figure out where the surface normal vector is stored and how to "dig it out" when we need it. This would require spending some time getting into the Mitsuba code.

[benjamin-heasly]

Comment by benjamin-heasly
Friday Dec 05, 2014 at 22:40 GMT

---

I built a spectral version of Mitsuba 0.5 and played around with the Mitsuba "field integrator". This integrator actually does a lot of what we want, right out of the box.

One really nice thing about the field integrator is it collects all of our data and factoids in one rendering pass. For example, it can produce one big multi-channel openexr image which contains "distance-to-camera", "albedo", and the usual 31 spectral slices.

Distance-To-Camera
I was able to make a distance-to-camera image pretty easily by modifying an existing WardLand Mitsuba scene and pulling out one slice by hand. It would take a bit of thinking and wiring to integrate this into the rest of our Render Toolbox and Virtual Scenes code.

For example, we probably want a nice syntax and automatic scene file generation for this kind of factoid output, which would be new.

Also, we'd need to make sure this kind of output doesn't mess up the processing of regular spectral output. In the case of the big openexr file, the expected channels are all there, but the names are changed to avoid naming collisions.

Surface Normals
I also tried getting out the object surface normal at each pixel. But this was not so easy. I get a disappointing error:

"The field integrator implementation requires renderings to be done in RGB when extracting positional data or surface normals / UV coordinates."

So to get surface normals I think we'd have to do one of two things: either keep two builds of Mitsuba side by side, one RGB and one spectral, or modify Mitsuba to overcome this "implementation requirement" somehow.