Hi @fsalv

[bibuzz]

Hi @bibuzz!

Any help would be appreciated

Please clone the repository again, there was an issue in how our code was managing an even number of temporal frames. Now it should work with your 4 channels. Anyway, note that out Temporal Reduction Block works better with an odd number (minimum of 3), so you should consider adding a temporal step to your dataset.

#-------------
MEAN = 7433.6436 # mean of the proba-v dataset
STD = 2353.0723 # std of proba-v the dataset

Since I am using sentinel images and not of prob-v dataset, then the MEAN and STD will be different ? How do I calculate them?

Those two values are used in the normalization and denormalization layers and they were simply computed on all the training images of the dataset. Just compute mean and standard deviation of your images pixel values.

Originally posted by @fsalv in #7 (comment)

[bibuzz]

Hello Sir,

As you said I have added odd temporal LRs (now 5 LRs) for training and calculated the mean and STD (average of all the HRs and LRs used for training)

But then I get an issue in the build RAMs network step

build rams network

rams_network = RAMS(scale=SCALE, filters=FILTERS,
kernel_size=KERNEL_SIZE, channels=CHANNELS, r=R, N=N)

Error

InvalidArgumentError Traceback (most recent call last)
~\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\framework\ops.py in _create_c_op(graph, node_def, inputs, control_inputs, op_def)
1879 try:
-> 1880 c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
1881 except errors.InvalidArgumentError as e:

InvalidArgumentError: Dimension size must be evenly divisible by 9 but is 4 for '{{node lambda_729/DepthToSpace}} = DepthToSpaceT=DT_FLOAT, block_size=3, data_format="NHWC"' with input shapes: [?,?,?,4].

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
in
1 # build rams network
----> 2 rams_network = RAMS(scale=SCALE, filters=FILTERS,
3 kernel_size=KERNEL_SIZE, channels=CHANNELS, r=R, N=N)

D:\SuperResolution\RAMS\utils\network.py in RAMS(scale, filters, kernel_size, channels, r, N)
146 x = conv3d_weightnorm(scale ** 2, (3,3,3), padding='valid')(x)
147 x = layers.Lambda(lambda x: x[...,0,:])(x)
--> 148 x = layers.Lambda(lambda x: tf.nn.depth_to_space(x, 3))(x)
149
150

This is the values for network

#-------------

Network Settings

#-------------
FILTERS = 48 # features map in the network
KERNEL_SIZE = 3 # convolutional kernel size dimension (either 3D and 2D)
CHANNELS = 5 # number of temporal steps
R = 8 # attention compression
N = 12 # number of residual feature attention blocks
lr = 1e-4 # learning rate (Nadam optimizer)
BATCH_SIZE = 48 # batch size
EPOCHS_N = 120 # number of epochs

Is there anything that I am missing?

[fsalv]

depth_to_space operation had hardcoded scale 3, but in your case it's 2. I have updated network.py, now it should allow any scale.

[bibuzz]

Hello Sir,

Thankyou for your quick response, I had actually tried that, made it into 2(my scale), but ended up with another error.
I have attached the entire error stack

Error.txt

TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.

During handling of the above exception, another exception occurred:
ValueError: Dimensions must be equal, but are 9 and 5 for '{{node tf.operators.add_29/AddV2}} = AddV2[T=DT_FLOAT](Placeholder, Placeholder_1)' with input shapes: [?,?,?,9], [?,?,?,5].

I tried changing line 152 to filters instead of 9,but the same type error and the value error has changed
152 x_global_res = RTAB(x_global_res, filters, kernel_size, r)

ValueError: Dimensions must be equal, but are 48 and 5 for '{{node tf.operators.add_14/AddV2}} = AddV2[T=DT_FLOAT](Placeholder, Placeholder_1)' with input shapes: [?,?,?,48], [?,?,?,5].

I tried changed it to my temporal value 5
x_global_res = RTAB(x_global_res, 5 , kernel_size, r) ,
but then kernel crashed

[bibuzz]

Hello Sir

Thankyou for quick response, I had tried changing it to my scale 2 before but there was another error and it got solved
by changing the network values (changed R=4 and N=8 values),
Trying to get it to train now by removing the mask code in training.py and loss.py

Just a question regarding MEAN and STD for clarity sir,

Its not the MEAN and STD of the original png image before tiling. The MEAN and STD of each tiled png image that is put in train/NIR/ folder only (calculated using QGIS gdalinfo -stats) is added and then average found

[ imgset0000 (HR_img1Mean + LR_img2Mean + LR_img3Mean +LR_img4Mean + LR_img5Mean) +
imagset0001 (HR_img1Mean + LR_img2Mean + LR_img3Mean +LR_img4Mean + LR_img5Mean) ] / 10
10 being total number of images

This I took as the MEAN value in network.py. Is this correct ? Similarly calculated for STD

[bibuzz]

I ran the train, and tested too but the result was distorted. I am not sure how to not include the masking ( as I am not including the SM.png and QM.png) in the loss.py.
I removed the SSIM all together as not sure how to alter it.

Original code

l1_loss = (1.0/total_pixels_masked)*tf.reduce_sum(
tf.abs(
tf.subtract(cropped_labels_masked,
corrected_cropped_predictions)
), axis=[1, 2]
)
Altered code

l1_loss = tf.reduce_sum(
tf.abs(
cropped_predictions
), axis=[1, 2]
)

Output validation image

loss.py

loss.txt

training.py

training.txt

The loss displayed is too large and PSNR in negative
Epoch 1/100
96/64 [=============================================] - 4s 45ms/step - Loss: 17301928.0000 - PSNR: -9.7797 - Val Loss: 0.0000e+00 - Val PSNR: 0.0000e+00

Any clue would of help

[fsalv]

No, your altered loss is not right. You have completely removed the subtraction with the ground truth, so it can't work. You can hardcode the masks as a matrix of all ones, or you should change the code to avoid the multiplications by cropped_y_mask . (Note also that QM are only used in the preprocessing stage, while the masks used for loss and metric computations are the SM).