`orbitControl()` Compatibility with `imageLight()`

[perminder-17]

Increasing Access

The current implementation of orbitControl() does not seamlessly work with the imageLight() function, resulting in unexpected behavior. When using a 360-degree image and adding a sphere with imageLight(), the use of orbitControl() does not provide the expected interaction. It appears as though the sphere is static, while only the image responds to orbitControl().

However when we use texture() we can achieve the possible result but without other functions such as shininess().

20240102222110.mp4

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature enhancement details

Expected Behavior:
The orbitControl() should seamlessly interact with the imageLight() function. When a sphere with an imageLight() is present, the orbitControl should behave as if the image is fully embedded in the sphere. This enhancement would significantly improve user experience, especially when working with 360-degree images.

After the implementation it should work similar to this.

treejs.feature.360.mp4

[perminder-17]

When we simply rotate the sphere it still shows inconsistency.

20240102223928.mp4

It would be immensely valuable if the WEBGL stewards or you, @davepagurek , could assist us in pinpointing the location for fixing it. Seeking for your approach how we could implement it.

[davepagurek]

There are two things going on in the threejs recording:

1. The reflections on objects take the camera angle into account
2. The background image also rotates with the scene

To accomplish (1), we have to apply the rotation of the camera to the normal vector that we pass in here to look up a coordinate on the light texture:

p5.js/src/webgl/shaders/lighting.glsl

Line 113 in fe33439

vec2 newTexCoor = mapTextureToNormal( worldNormal );

To do that, we have to add a new uniform to the shaders, similar to uNormalMatrix, which is the inverse transpose of the model view matrix:

p5.js/src/webgl/p5.Shader.js

Lines 343 to 344 in fe33439

	this._renderer.uNMatrix.inverseTranspose(this._renderer.uMVMatrix);
	this.setUniform('uNormalMatrix', this._renderer.uNMatrix.mat3);

The normal matrix is used to update normals based on the current model's transform, which is a product of the camera transform and all the other transformations applied to the current object. For this, we only want the camera transform, so instead of doing the inverse transpose on uMVMatrix, we need to do this on the current camera's matrix. In a shader, multiplying a normal by that will apply the camera rotation.

For (2), the easy but imperfect way is to draw a giant textured sphere around everything, and then call cleatDepth() afterwards so it doesn't accidentally clip everything. It's imperfect though because it would still apply any position changes from the camera---ideally, the sphere is so large that position changes make no difference. The best way to do this might be to make a new default shader for sphere mapped backgrounds that dimilarly gets its color from mapTextureToNormal, but where the initial normal comes from the angle to the camera (normalize(worldPosition)) multiplied by the camera normal matrix.

[inaridarkfox4231]

up is fixed to (0,1,0), but this may be the cause. Maybe not. It's just a guess.
here

  vec3 irradiance = vec3(0.0);

  vec3 up = vec3(0.0, 1.0, 0.0);

  vec3 right = normalize(cross(up, normal));
  up = normalize(cross(normal, right));

I think it is useless to change the specifications of orbitControl() no matter how much you change it. People who want to use it normally will be in trouble. That's the wrong idea.

What is introduced there is Sphere Environment Mapping, right? In my opinion, the specification of that is completely different from imageLight(), but I can't understand why you had to bring it up.

It's essentially just an application of texture mapping. Since it uses normals to determine the texture function arguments, it is completely different from imageLight(), and extremely lightweight. It works fine on my mobile phone as well.

If you want to implement sphere environment mapping, shouldn't you change the title or create a new issue to explain the path to implementation? imageLight() is no longer relevant.

It seems that the image used for imageLight() can be a general image and does not need to be a spherical map. It's something completely different feature.
refer this

[davepagurek]

imageLight() is pretty similar to sphere mapped normals---thats what's going on for most of it. The additional thing that imageLight does is it precalculates and caches some blurred versions of the texture the first time you use it, and then uses those in the sphere mapped texture lookup based on the shininess of the material rather than using the texture directly.

I may have misinterpreted the inconsistency @perminder-17 mentioned. I was thinking you meant how in the threejs recording, the reflections change as you change the camera angle. Other lights in p5 do this, but the imageLight reflections currently don't, so that's potentially something we might want to change for the sake of consistency.

But what @inaridarkfox4231 brought up, which is a good point, is that you can use non sphere map images as a light source. This looks ok at a fixed camera angle, but non sphere map images would have a visible seam if the reflections changed with the camera. I'm not sure how distracting that would be, it's potentially a reason to not be consistent with the camera affecting the light positions.

[inaridarkfox4231]

To me, the Three.js video introduced above looks like just a sphere mapping. I can only see it that way.
It's not lighting, it's just drawing using a sphere map. It's just a variant of texture mapping.

Was imageLight() just a variant of texture mapping? It doesn't really seem that way though. If so, there is no need for complicated calculations. Just give us more options for how to process textures inside the shader and you're done.

There is no need to prepare a process to process the texture material in advance (I can't understand this process, sorry). Just calculate a variable that reads the texture from the transformed normal vector.

However, the current p5.js does not support cubemaps, so we may need to prepare to support it.

That video is cube mapping, right? I have the same one too. It is very similar to what was on the webgpu reference site.
env mapping test

2024-01-06.00-45-45.mp4

cubemap: https://github.com/webgpu/webgpu-samples/tree/main/public/assets/img/cubemap

If this video looks rich, it's not because of the lighting. It's simply because it uses a beautiful cube map. You're probably misunderstanding something.

In the first place, as I wrote above, imageLight() requires too much calculation and my mobile phone crashes. On the other hand, sphere mapping is just a variant of texture drawing, so it is lightweight. It works fine.

Will imageLight() be able to perform cube mapping in the future? If that happens, that would be amazing. Wonderful. I'm rooting for you. that's all.

[davepagurek]

@perminder-17 it would maybe be helpful to clarify what aspect of the threejs video you were referring to in your issue so that we can continue talking about what addressing it might involve.

The rest of this comment is just adding some context on what imageLight is doing and future directions one could take it.

`imageLight()` explanation

The preprocessing is to compute a few different sphere map textures at varying levels of roughness so that shininess() affects the reflected values:

Screen.Recording.2024-01-05.at.2.12.08.PM.mov

The other extra bit that happens that is different from pure reflection mapping is that it combined two roughness levels for diffuse and specular light, to parallel the lighting pipeline for things like point lights.

Will imageLight() be able to perform cube mapping in the future? If that happens, that would be amazing. Wonderful. I'm rooting for you. that's all.

A simplified cube mapping experience would be a good feature, and would be able to use a large chunk of the lighting code that imageLight uses! As for supporting cube maps as opposed to equirectangular textures, some support is starting to be added in #6665.

imageLight() requires too much calculation and my mobile phone crashes.

#6613 was also merged recently to make progress on this. I haven't debugged too much further, but we're open to optimizing the process more.

[perminder-17]

I apologize for the confusion @davepagurek . While I was exploring its functionality in three.js, I discovered a feature similar to orbitcontrol. Essentially, when you rotate the image, the reflections on the sphere dynamically adjust. My objective with this issue is precisely that—I aim to have the reflections on the sphere change in sync with the rotation of the image.

Basically I need the reflections on objects take the camera angle into account.

Currently it's something like

Not.working.orbitControl.mp4

But when the reflections on object take camera angle into account

orbit.control.mp4

orbit.control.shininess.mp4

We can take a look and adjust the camera more correctly. Maybe adding a 360-degree Image can help us achieve something similar to three.js

@davepagurek I am unsure if this seems to you a bug or not. If not I will close the issue as you tell. I am not sure about box() and torus(). Sphere() is the one I am working on.

[davepagurek]

got it! In that case, I believe this section of an earlier comment is the bit that's relevant:

To accomplish (1), we have to apply the rotation of the camera to the normal vector that we pass in here to look up a coordinate on the light texture:

p5.js/src/webgl/shaders/lighting.glsl

Line 113 in fe33439

vec2 newTexCoor = mapTextureToNormal( worldNormal );

To do that, we have to add a new uniform to the shaders, similar to uNormalMatrix, which is the inverse transpose of the model view matrix:

p5.js/src/webgl/p5.Shader.js

Lines 343 to 344 in fe33439

	this._renderer.uNMatrix.inverseTranspose(this._renderer.uMVMatrix);
	this.setUniform('uNormalMatrix', this._renderer.uNMatrix.mat3);

The normal matrix is used to update normals based on the current model's transform, which is a product of the camera transform and all the other transformations applied to the current object. For this, we only want the camera transform, so instead of doing the inverse transpose on uMVMatrix, we need to do this on the current camera's matrix. In a shader, multiplying a normal by that will apply the camera rotation.

Is that similar to the technique you were using to generate the second and third video? And in those videos, is there anything remaining that you think you would need to fix to get the desired behaviour you're after?

[perminder-17]

got it! In that case, I believe this section of an earlier comment is the bit that's relevant:

To accomplish (1), we have to apply the rotation of the camera to the normal vector that we pass in here to look up a coordinate on the light texture:

p5.js/src/webgl/shaders/lighting.glsl

Line 113 in fe33439

vec2 newTexCoor = mapTextureToNormal( worldNormal );

To do that, we have to add a new uniform to the shaders, similar to uNormalMatrix, which is the inverse transpose of the model view matrix:

p5.js/src/webgl/p5.Shader.js

Lines 343 to 344 in fe33439

	this._renderer.uNMatrix.inverseTranspose(this._renderer.uMVMatrix);
	this.setUniform('uNormalMatrix', this._renderer.uNMatrix.mat3);

The normal matrix is used to update normals based on the current model's transform, which is a product of the camera transform and all the other transformations applied to the current object. For this, we only want the camera transform, so instead of doing the inverse transpose on uMVMatrix, we need to do this on the current camera's matrix. In a shader, multiplying a normal by that will apply the camera rotation.

Is that similar to the technique you were using to generate the second and third video? And in those videos, is there anything remaining that you think you would need to fix to get the desired behaviour you're after?

Yess... I am going with the same technique. Do you think I should raise a PR? Does the videos looks good to you?

[davepagurek]

Yess... I am going with the same technique. Do you think I should raise a PR? Does the videos looks good to you?

I think it's looking close to what I would expect, although it would be helpful to see how it looks with an equirectangular input image so we can make sure the seams and stretching we see is just due to the input image.