How to jointly optimize the pose?

[Seasandwpy]

Hi,
I tried to optimize the camera pose jointly with shape and texture codes, where I set the azimuth and elevation as 0 and distance as 0.5 in the beginning, and add the parameters in the optimizer. However the result is still blurry after 500 iterations, I would like to ask if this is normal or I miss something in the steps?

[wbjang]

Hello @Seasandwpy ,

From my experience, the pose is optimized first then shape/texture latent vectors are optimized later according to the roughly estimated pose. If the network cannot find the right pose in the first few iterations, please try with other hyper-parameters.

For distance, ShapeNet-SRN Car has near = 0.8 and far = 1.8, I would suggest starting from 1.3. For elevation, it is better to start from 0.5 so that the camera is not on the surface.

If you train ShapeNet-SRN Cars and apply for other datasets, scaling also matters. To me, the second car seems a bit larger than the training set.

Hope this helps.

Wonbong

[Kulbear]

Hi Wonbong,

In the paper, you mentioned "We minimize the photometric loss (5) jointly with respect to shape and texture codes and camera parameters (fixing the decoder parameters Θ)". And according to your previous reply "From my experience, the pose is optimized first then shape/texture latent vectors are optimized later according to the roughly estimated pose."

I wonder is this my misunderstanding that the above two statements are opposite?
At test time, your shape code and your appearance code are unknown, even with a frozen network, how can you find a camera pose given this condition?

[wbjang]

Hello @Kulbear

Shape/Texture code, as well as camera pose, are optimized simultaneously. What I meant in the previous reply was that even though all three are optimized simultaneously, the model found the camera pose first and then optimize shape/texture codes later.

The frozen network works as a prior so that the model finds the camera pose and shape/texture codes accordingly.

In the failure cases, the model is stuck on the bad(local) camera pose (the model cannot move out of the local optima), and shape/texture codes are updated based on the bad(local) camera pose.

Cheers,
Wonbong

[chengzhag]

Hi @Seasandwpy, @Kulbear,

How are your attempts to reproduce the pose estimation going? I would appreciate it if you could share your own implementation.

Hi @wbjang,

this work is really impressive, thank you for sharing your code. It would be great if you could also publish the part that optimizes code and pose simultaneously.