Using another number of channels for the MSI does not produce good results.

[ggous]

Hello and thanks for the code!

I am trying to run the code using my data.

msi_np: (128, 128, 9)
hsi_np: (32, 32, 172)
GT: (128, 128, 172)
R: (9, 172)

I am using factor = 4.

Now, If I run the code, at the beginning the psnr increases a little and the mse decreases. But, after that , I have very very small psnr increases and mse decreases. So, the result is not good.

Starting optimization with ADAM
Iteration 00000  PSNR_HR 14.642  SSIM: 0.007  RMSE: 0.370  MSE_gap 0.3936448
Iteration 00100  PSNR_HR 21.290  SSIM: 0.459  RMSE: 0.172  MSE_gap -0.0005656
Iteration 00200  PSNR_HR 21.959  SSIM: 0.503  RMSE: 0.159  MSE_gap -0.0001561
Iteration 00300  PSNR_HR 22.270  SSIM: 0.524  RMSE: 0.154  MSE_gap -0.0000897
Iteration 00400  PSNR_HR 22.442  SSIM: 0.539  RMSE: 0.151  MSE_gap -0.0000407
Iteration 00500  PSNR_HR 22.534  SSIM: 0.549  RMSE: 0.149  MSE_gap -0.0000331
Iteration 00600  PSNR_HR 22.587  SSIM: 0.554  RMSE: 0.148  MSE_gap -0.0000049
Iteration 00700  PSNR_HR 22.632  SSIM: 0.561  RMSE: 0.148  MSE_gap -0.0000417
Iteration 00800  PSNR_HR 22.661  SSIM: 0.566  RMSE: 0.147  MSE_gap 0.0000083
Iteration 00900  PSNR_HR 22.676  SSIM: 0.568  RMSE: 0.147  MSE_gap 0.0000062
Iteration 01000  PSNR_HR 22.689  SSIM: 0.571  RMSE: 0.147  MSE_gap 0.0000040

Trying to figure out what is happening, I saw that you run your code with 3 channels instead of 9 that I am using.
So, I tried to use only my first 3 channels instead of whole 9 and it works!
Psnr increases rapidly and mse decreases rapidly.

Starting optimization with ADAM
Iteration 00000  PSNR_HR 14.462  SSIM: -0.008  RMSE: 0.378  MSE_gap 0.2317122
Iteration 00100  PSNR_HR 30.097  SSIM: 0.750  RMSE: 0.061  MSE_gap -0.0001769
Iteration 00200  PSNR_HR 31.852  SSIM: 0.785  RMSE: 0.050  MSE_gap -0.0000441
Iteration 00300  PSNR_HR 33.112  SSIM: 0.805  RMSE: 0.044  MSE_gap -0.0000508
Iteration 00400  PSNR_HR 33.890  SSIM: 0.826  RMSE: 0.040  MSE_gap 0.0000725
Iteration 00500  PSNR_HR 34.117  SSIM: 0.836  RMSE: 0.039  MSE_gap -0.0000005
Iteration 00600  PSNR_HR 34.926  SSIM: 0.849  RMSE: 0.036  MSE_gap 0.0000029
Iteration 00700  PSNR_HR 35.062  SSIM: 0.853  RMSE: 0.035  MSE_gap 0.0000663
Iteration 00800  PSNR_HR 35.074  SSIM: 0.855  RMSE: 0.035  MSE_gap 0.0001004
Iteration 00900  PSNR_HR 35.494  SSIM: 0.862  RMSE: 0.033  MSE_gap 0.0000339
Iteration 01000  PSNR_HR 35.588  SSIM: 0.864  RMSE: 0.033  MSE_gap -0.0000163
Iteration 01000  PSNR_HR 35.179  SSIM: 0.859  RMSE: 0.035  MSE_gap 0.0000062

The thing is now, how I can use the whole 9 channels and why this problem exists.

The only point where you use the number of channels in your code, is in the gdd class, at layer:

self.guide_enc = nn.Sequential(
                    conv(3, num_channels_down, filter_size_down, bias=need_bias, pad=pad),
                    bn(num_channels_down),
                    act(act_fun))

There, you use `3` as the first argument. I am using `9` in order for my problem to run. But, as I said it doesn't produce good results.

Thanks!

UPDATE:

In the line:

total_loss = mse(out_LR, hsi_torch) * param_balance + mse(out_rgb_torch, msi_torch)

inside the def closure, if I ommit the mse(out_rgb_torch, msi_torch), it works fine!

total_loss = mse(out_LR, hsi_torch) * param_balance

But, the final image misses the color spectrum..

So, it seems it does not have problem with the 9 and 3 channels but something in the out_rgb_torch calculation?