Kernel becomes Non- Positive Semi Definite during training

[Schobs]

I am using the CIGP for regression given extracted features from a CNN. The NLL becomes Nan after converging to ~670. The repo to reproduce this is given:

https://github.com/Schobs/Mini-GP

run python landmark_cigp.py

[WjianFan]

1. I used to meet this problem,you can try to change the activation function from sigmiod to Relu(Relu6).
2. If the problem is caused by the gradient exploding, you can try to use nn.torch.nn.utils.clip_grad_norm_ to slove the problem.
3. Another reason maybe be the nll loss is too big, such as 1 billon , you can try nll = nll /1000 to decrease the nll.
4. The nll become nan is because the sigma become no positive definite, you can try plus a small noise to sigma.I used to add a epson * a diagonal matri which all the element is one. The size of epson is from the choice of yourself. I used to choose 1e-5 or 1-e-4

[WjianFan]

I am sorry to reply you again after a long time. I use the CIGP these days because I use this model currently. I meet this problem too. My model is introduce the neural network to the kernel function. Because of the complication of neural function, the process of training occurred the problem that the value of it becomes nan value after times of training. I find it a good solution that you can reset the initial value of the parameter of CIGP such as the lengthscale and the beta and the scale. You can try to set them 1 2 3 4and so on. I believe it a very useful way to slove it. Hope the suggestion is useful to you. Best wish!