Question about depth prediction training

[kang-1-2-3]

Hello, thank you for your excellent work.

I am trying to fine-tune the checkpoint on the nuScenes dataset, following the depth loss implementation in the VGGT repository for depth and camera transformation prediction:

https://github.com/facebookresearch/vggt/blob/main/training/loss.py

However, I observed that the predicted 'loss_conf_depth' becomes very large for some scenes (while loss_reg_depth and loss_grad_depth seems normal), as shown in the figure. Did you encounter a similar issue when training the model? Do you have any suggestions for stabilizing the depth confidence during fine-tuning?

[LinZhuoChen]

Hi~ Thank you for your attention.
I am not familiar with nuScenes dataset. Are the depth maps annotation sparse or dense ? How do you process the invalid depth part?

I think the downward trend of depth conf looks fairly normal. In some scenarios, depth conf does spike to relatively large values, but this typically stabilizes during subsequent finetuning.

[kang-1-2-3]

Thank you for your prompt reply.

The nuScenes dataset consists of short image sequences. In my setup, I generate depth annotations by projecting sparse LiDAR points onto the images. I keep only depth points within the range of 3–50 m, and then normalize the depths using the anchor frames.

When the confidence becomes larger than 5, the depth confidence loss increases significantly. However, I am not sure whether this is expected behavior or whether there might be a mistake in my training code.

May I ask whether you have any plans to release the training code? That would be very helpful for checking whether my implementation is aligned with yours.