Added a Method (`panorama(img)`) which adds a sphereMapped Background. by perminder-17 · Pull Request #6808 · processing/p5.js

perminder-17 · 2024-02-13T12:40:36Z

Resolves #6752

Changes:

I have developed a function that integrates a sphere-mapped background into the scene. The fragment shader is responsible for accurately rendering the reflections of the background.

PR-360.1.mp4

Screenshots of the change:

PR Checklist

npm run lint passes
Inline documentation is included / updated
Unit tests are included / updated

perminder-17 · 2024-02-13T12:44:51Z

The PR is ready for your review, @davepagurek. Please share your thoughts on it. Additionally, I'm unsure whether to store the sphereMapping file separately or use it directly within a function.

Also calling @haroon10725 as you were intrested in solving this. Please share your thoughts on this one.

davepagurek

Nice work on this so far!

davepagurek · 2024-02-13T12:55:57Z

src/webgl/p5.RendererGL.js

  }

+  _getSphereMapping(img) {
+    this.sphereMapping = this._pInst.createFilterShader(


It looks like this is creating a new shader every frame. Maybe you can wrap lines 1689-1691 call in if (!this.sphereMapping) { ... }? I noticed the frame rate seems to drop in your video when you start using the sphere map background, I'm hoping it's just because of this and it'll run faster afterwards.

davepagurek · 2024-02-13T12:58:08Z

src/webgl/shaders/sphereMapping.frag

@@ -0,0 +1,32 @@
+#version 300 es


For maximum compatibility, do you think we can do this one in GL ES 100? So, taking out the #version line, and using attribute instead of in, using the spacial variable gl_FragColor instead of defining an out fragColor, and using texture2D() instead of texture()?

Oh.. you mean varying instead of in? I guess attribute is supported in vertex shaders only?

oops yes you're right, that should be varying

davepagurek · 2024-02-13T12:59:25Z

src/webgl/p5.RendererGL.js

    properties.quadraticAttenuation = this.quadraticAttenuation;

    properties._enableLighting = this._enableLighting;
+    properties.sphereMapping = this.sphereMapping;


I think the shader doesn't need to be pushed/popped since we can just create it once and then reuse it with different images, so we can probably take this line out

davepagurek · 2024-02-13T13:00:55Z

src/webgl/p5.RendererGL.js

        this._pInst.noStroke();
        this._pInst.blendMode(constants.BLEND);
        this._pInst.shader(this.filterShader);
+        this.filterShader.setUniform('uNewNormalMatrix', this.uNMatrix.mat3);


Since this only is used in the sphere mapping shader, can we do this and the inverseTranspose line in getSphereMapping where we're setting another uniform already?

davepagurek · 2024-02-13T13:04:06Z

src/webgl/shaders/sphereMapping.frag

+out vec4 fragColor;
+
+void main() {
+  vec4 viewModelPosition  = uModelViewMatrix  * vec4(faPosition, 1.0);


I think uModelViewMatrix here is going to be made for the filter shader's rectangle, so I'm not sure it's what we want for doing reflections. Does it work if you set vViewPosition to all zeros, since we're assuming the position of the camera is negligible, assuming the texture is on an infinitely large sphere? And if so, can we just do n = normalize(vGlovalNormal.xyz) directly without doing a reflect?

davepagurek · 2024-02-13T13:06:34Z

src/webgl/material.js

+  out vec3 vNormal;
+  out vec3 faPosition;
+  out vec2 vTexCoord;
+  out vec3 fvNormal;


I think we don't use these in the shader, we can probably take them out from here

davepagurek · 2024-02-13T13:06:49Z

src/webgl/shaders/sphereMapping.frag

+  vec2 suv;
+  suv.y = 0.5 + 0.5 * n.y;
+  suv.x = atan(n.z, n.x) / (2.0 * PI) + 0.5;
+  newTexColor = texture(uSampler, suv.xy);


perminder-17 · 2024-02-13T16:03:46Z

I think uModelViewMatrix here is going to be made for the filter shader's rectangle, so I'm not sure it's what we want for doing reflections. Does it work if you set vViewPosition to all zeros, since we're assuming the position of the camera is negligible, assuming the texture is on an infinitely large sphere? And if so, can we just do n = normalize(vGlovalNormal.xyz) directly without doing a reflect?

You're right about that, your logic seems correct. However, despite following this approach, we aren't achieving the desired outcome. Take a look at my code for reference.

#define PI 3.141592

precision highp float;

uniform sampler2D uSampler;
uniform mat3 uNewNormalMatrix;
uniform mat3 uCameraRotation;
uniform mat4 uNewModelViewMatrix;
uniform mat4 uModelViewMatrix;

varying vec2 vTexCoord;
varying vec3 faNormal;
varying vec3 faPosition;

void main() {
 // vec4 viewModelPosition  = uModelViewMatrix  * vec4(faPosition, 1.0);
  vec3 vViewPosition  = vec3(0.0);
  vec4 newTexColor = texture2D(uSampler, vTexCoord);
  vec3 vGlobalNormal = uNewNormalMatrix  * faNormal ;
  vec3 n = normalize(vGlobalNormal.xyz);
  vec2 suv;
  suv.y = 0.5 + 0.5 * n.y;
  suv.x = atan(n.z, n.x) / (2.0 * PI) + 0.5;
  newTexColor = texture2D(uSampler, suv.xy);
  vec4 baseColor = newTexColor;
  gl_FragColor = baseColor;
}

we are getting something like:-

for.review.mp4

davepagurek · 2024-02-14T16:35:53Z

Good point, if we take out that matrix, we need to account for the different perspective from each pixel in a different way. Using uMVMatrix uses the filter camera's perspective to do that, where we ideally would want the active camera from before the filter shader is called. Maybe the thing to do is just to add new uniforms for the camera's field of view and the camera's aspect ratio? Then in the fragment shader, the initial normal before applying the normal rotation matrix would be something like:

start as vec3(0., 0., 1.)
rotate about the X axis by the angle mix(-fovy/2., fovy/2., vTexCoord.y)
rotate about the Y axis by the angle mix(-fovx/2., fox/2., vTexCoord.x) where fovx = fovy * aspect

perminder-17 · 2024-02-15T08:55:42Z

Good point, if we take out that matrix, we need to account for the different perspective from each pixel in a different way. Using uMVMatrix uses the filter camera's perspective to do that, where we ideally would want the active camera from before the filter shader is called. Maybe the thing to do is just to add new uniforms for the camera's field of view and the camera's aspect ratio? Then in the fragment shader, the initial normal before applying the normal rotation matrix would be something like:

start as vec3(0., 0., 1.)

rotate about the X axis by the angle mix(-fovy/2., fovy/2., vTexCoord.y)

rotate about the Y axis by the angle mix(-fovx/2., fox/2., vTexCoord.x) where fovx = fovy * aspect

Thanks...Previously, I was confused about the rotation matrix. I believe I understand your point now and have made the necessary updates. The code is still functioning as it was before. Let me know if any thing still needs to be changed. I'll look into the minor cleanups, indentations, examples once the code looks good to you.

davepagurek

Thanks for making those updates! I was testing it out here: https://editor.p5js.org/davepagurek/sketches/iOfzR8r7g

It looks like the filter shader is clipping the box in the sketch at some angles. Maybe we need a clearDepth() in there? (Should that be happening for all filter shaders?)

I also notice what seems to be a tilt happening sometimes you rotate the view. Do you know if this happens because of the new way we're calculating the initial normal, or did this happen before too? It could be that it's something about how we're applying the normal matrix, not sure yet.

Screen.Recording.2024-02-15.at.8.01.37.PM.mov

perminder-17 · 2024-02-16T12:39:12Z

I also notice what seems to be a tilt happening sometimes you rotate the view. Do you know if this happens because of the new way we're calculating the initial normal, or did this happen before too? It could be that it's something about how we're applying the normal matrix, not sure yet.

Ah, I just overlooked this one. I see where the issue lies. It's not with the new Normal or old Normal matrix, but rather with how we're calculating it. Specifically, we're performing an inverse transpose of the uMVMatrix. The problem arises when we rotate the scene by 180 degrees; this alters the orientation of the normals. For instance, if we drag the scene downwards by clicking the left button and pulling down, after a 180-degree rotation, dragging the cursor downwards causes the scene to move upwards. The orientations is getting changed, so in between when you try to drag the scene downward, it's tilting funny - like half of it's going up and half of it's going down!

The result after fixing this have a look:-

https://editor.p5js.org/aman12345/sketches/AwsE_YTLa

davepagurek · 2024-02-20T17:55:34Z

The rotation of the background looks good now! Although I noticed in the sketch that the tilt behaviour we saw in the background before now seems to be present in the reflections in imageLight: https://editor.p5js.org/davepagurek/sketches/HjrSsZN2n

Screen.Recording.2024-02-20.at.12.22.25.PM.mov

This doesn't seem to be the case using the 1.9.1 beta build though: https://editor.p5js.org/davepagurek/sketches/edkYxkJAa

I'm not sure what might have changed to cause that yet, do you have any ideas?

davepagurek · 2024-02-20T17:56:24Z

src/webgl/light.js

+ */
+p5.prototype.panorama = function (img) {
+  this.filter(this._renderer._getSphereMapping(img));
+  this.clearDepth();


Do you think we should do this for all filter shaders instead of just this one?

You're right. I was thinking we had addressed all the issues stemming from the filterShaders. However, I just noticed that the box clipping persists when we use other shaders too. Thanks for catching that.

perminder-17 · 2024-02-20T18:13:17Z

The rotation of the background looks good now! Although I noticed in the sketch that the tilt behaviour we saw in the background before now seems to be present in the reflections in imageLight: https://editor.p5js.org/davepagurek/sketches/HjrSsZN2n

Screen.Recording.2024-02-20.at.12.22.25.PM.mov
This doesn't seem to be the case using the 1.9.1 beta build though: https://editor.p5js.org/davepagurek/sketches/edkYxkJAa

I'm not sure what might have changed to cause that yet, do you have any ideas?

Oh, no, this isn't a bug. I apologize. I was actually working on fixing the code above and testing it with my current camera matrix. So, I just reverted those changes. In this pull request, it's functioning somewhat like this:
https://editor.p5js.org/aman12345/sketches/TAWBmKR-5

davepagurek · 2024-02-20T19:18:33Z

Ah ok makes sense, thanks for clarifying!

davepagurek · 2024-02-20T18:25:34Z

src/webgl/light.js

+ * this method, users can obtain a complete 360-degree view
+ * of a scene.
+ *
+ * Using Panorama is straightforward. Similar to calling a


Rather than describing what users can do, we can address this directly to users like instructions, e.g. "Similar to calling background(color), call panorama(img) before drawing your scene to create a 360-degree background out of your image." I think the paragraph above could be rephrased similarly.

Also, we generally stay away from calling anything simple or straightforward so that people don't feel bad if they struggle using something.

This section where we describe it as being similar to background is actually maybe a good thing to start this documentation with instead of ending it with it, what do you think?

davepagurek · 2024-02-20T19:12:39Z

src/webgl/light.js

+ * `background(color)`, users only need to call the `panorama(img)`, and
+ * beneath it, anything created will form a 360-degree scene.
+ * To enable 360-degree viewing, it is essential to invoke
+ * `orbitControl()`; otherwise, the method will not function as intended.


Maybe instead of saying it's essential, since it still works if you programmatically animate a camera, we can say to either use orbitControl or try changing the orientation of the camera to see different parts of the background.

davepagurek · 2024-02-20T19:15:56Z

src/webgl/light.js

 };

+/**
+ * Creates a Panorama with given image.


We can probably take this line out if we move that last paragraph up to the top, it serves as a pretty good intro.

davepagurek · 2024-02-20T19:18:19Z

src/webgl/light.js

+ * Creates a Panorama with given image.
+ *
+ *
+ * `panorama(img)` is a method designed to transform a standard


Maybe we could try to explain what kind of images this is intended to be used with? Possibly mentioning things like maps as a common example of a 360 degree view mapped to a rectangular image, and maybe how HDRIs are also in this format

perminder-17 · 2024-02-20T20:42:15Z

Please let me know if we can enhance the readability and user-friendliness of this document.

davepagurek

Just a few more comments on the documentation and then I think we're good to go!

src/webgl/light.js

davepagurek · 2024-02-22T21:38:23Z

src/webgl/light.js

 };

+/**
+ * The `panorama(img)` method adeptly transforms images such as


How do you feel about this as an update to the wording for this paragraph? I tried to simplify the language a little, and describe more what the content of the image should be.

* The `panorama(img)` method transforms images containing * 360-degree content, such as maps or HDRIs, into immersive * 3D backgrounds that surround your scene. This is similar to calling * `background(color)`; call `panorama(img)` before drawing your * scene to create a 360-degree background from your image. It * operates on the concept of sphere mapping, where the image is * mapped onto an infinitely large sphere based on the angles of the * camera.

Sounds good!

Co-authored-by: Dave Pagurek <dave@davepagurek.com>

perminder-17 · 2024-02-24T17:50:40Z

Sorry for the delay working on this. I have made the changes.

davepagurek

Thanks for your continued work on this, I think this is good to go now!

perminder-17 · 2024-02-27T16:41:43Z

Thanks for your continued work on this, I think this is good to go now!

Thanks. Excited to see this feature how it goes.....and really thanks for your constant review on this. Love to contribute more like this❤

Sphere-Mapping

9d2914b

prettier fix

997db50

davepagurek reviewed Feb 13, 2024

View reviewed changes

perminder-17 added 3 commits February 13, 2024 21:19

fixes

71dc723

fixes

692ea24

fixes

71a32d5

perminder-17 added 3 commits February 15, 2024 14:15

fixes

9db3b2e

fixes

bc9558c

fixes

2c61205

davepagurek reviewed Feb 16, 2024

View reviewed changes

perminder-17 added 3 commits February 16, 2024 22:11

fixes

88d251c

fixes

242aedd

fixes

ae6e257

davepagurek reviewed Feb 20, 2024

View reviewed changes

perminder-17 added 2 commits February 21, 2024 00:28

removed clearDepth for panorma

3c491c4

clearDepth-all-filterShader

30844cf

davepagurek reviewed Feb 20, 2024

View reviewed changes

docs

0117abd

perminder-17 mentioned this pull request Feb 22, 2024

1.9.1 Release #6822

Closed

3 tasks

davepagurek reviewed Feb 22, 2024

View reviewed changes

docs

47e9a6f

Co-authored-by: Dave Pagurek <dave@davepagurek.com>

perminder-17 and others added 2 commits February 24, 2024 23:18

docs

9741865

Co-authored-by: Dave Pagurek <dave@davepagurek.com>

docs

a063911

davepagurek approved these changes Feb 27, 2024

View reviewed changes

davepagurek merged commit d7b5d58 into processing:main Feb 27, 2024

perminder-17 deleted the sphere-Mapping branch February 27, 2024 16:59

Uh oh!

Comments

Conversation

perminder-17 commented Feb 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Uh oh!

perminder-17 commented Feb 13, 2024

Uh oh!

davepagurek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perminder-17 Feb 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perminder-17 commented Feb 13, 2024

Uh oh!

davepagurek commented Feb 14, 2024

Uh oh!

perminder-17 commented Feb 15, 2024

Uh oh!

davepagurek left a comment

Choose a reason for hiding this comment

Uh oh!

perminder-17 commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davepagurek commented Feb 20, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perminder-17 commented Feb 20, 2024

Uh oh!

davepagurek commented Feb 20, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perminder-17 commented Feb 20, 2024

Uh oh!

davepagurek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

perminder-17 commented Feb 24, 2024

Uh oh!

davepagurek left a comment

Choose a reason for hiding this comment

Uh oh!

perminder-17 commented Feb 13, 2024 •

edited

Loading

perminder-17 Feb 13, 2024 •

edited

Loading

perminder-17 commented Feb 16, 2024 •

edited

Loading

perminder-17 commented Feb 27, 2024 •

edited

Loading