The experimental results on ImageNet dataset

[creaitr]

As written in the README, the results on ImageNet are not good like the paper.
Can you tell me how different the accuracy results are?

[hustzxd]

The accuracy of ResNet18-w4a4 is about 0.60. I think there may be some problems in the code or the hyperparameters used in the experiments is not suitable. Unluckily, I haven't tried comprehensive experiments with different hyperparameters (Maybe in the future). If you have better ideas or results, welcome to discuss it with me.

[creaitr]

According to the Eq. (5) in the LSQ paper, the derivative of v is limited by the range.
Thus, I think the backward function can be reimplemented as follow:

grad_weight = indicate_middle * grad_weight
return grad_weight, grad_alpha, None, None, None

In my case, I got about 70% on ResNet-18-w4a4. However, it is still lower than the FP model.

[hustzxd]

Thank you for pointing out the problem, I will retrain the model.
By the way, I think 70% is excellent for ResNet18-w4a4. I will share it with you if I have better results.

[creaitr]

Thank you!
Actually, I used tensorflow & tensorpack frameworks.
I also wonder about the results of pytorch.

[KwangHoonAn]

Hi, In your code I think grad_scale factor s*gradScaleFactor is missing even in forward function in your custom autograd function
Is there any specific reason?

[hustzxd]

Thank for your question.
In the paper

s = gradscale(s, gradScaleFactor)

Function 1 gradscale(x, scale):
# x: Input tensor 
# scale: Scale gradient by this 
yOut = x 
yGrad = x × scale 
y = detach(yOut - yGrad) + yGrad # Return yOut in forward, pass gradient to yGrad in backward 
return y

In the lsq.py

def round_pass(x):
    y = x.round()
    y_grad = x
    return y.detach() - y_grad.detach() + y_grad

alpha = grad_scale(self.alpha, self.alpha_scale)

or

class FunLSQ(torch.autograd.Function):
     forward():
           # we use alpha
     backward():
           # firstly compute grad_alpha
           grad_alpha = grad_alpha / g

The two implementations are both ok in my opinion.
In section 2.2 of the paper, the author considers that step size should update neither fast nor slow, preferably similar to weight update. In fact, the gradient of step size is too large. As a result, we change the grad_alpha smaller after we compute grad_alpha.

[sijeh]

Thank you for pointing out the problem, I will retrain the model.
By the way, I think 70% is excellent for ResNet18-w4a4. I will share it with you if I have better results.

Hi,hustzxd
Is the accuracy improved based on resnet18_W4_A4 after changing the code？

[michaelklachko]

Has anyone reproduced 3 bit accuracy from the paper? We already know that 4 bits should have no accuracy drop (multiple papers), but 3 bit is more challenging.

[phuocphn]

Hi,hustzxd
Could you share the implementation of Resnet-18 that you used for evaluating ?. In the LSQ paper, the authors mentioned that the first and last layer will be quantized to 8-bit, but there are some confusions when quantizing the first layer, such as the normalized input have both negative and positive values. I have tried to use your implementation of ActLSQ in the beginning ( input --> ActLSQ --> conv1) and put ActLSQ after each RELU activation, but the accuracy dropped significantly.

In the w8a8 case, I get the accuracy less than the original full-precision model that is used to initialize the quantized model.
Thanks in advance.

[hustzxd]

Thanks for your questions.
In the w4a4 case, I don't quantize the first and last layers.

In the w8a8 case, I quantize the data into INT8 and activations after ReLU into UINT8
input --> ActLSQ(signed=True, nbits=8) --> conv1--> BN --> ReLU --> ActLSQ(signed=False, nbits=8)

I will update the results of ResNet18 soon.

[phuocphn]

Hi hustxzd,

Thank you for your valuable feedbacks. I trained a new model with your suggestions to see any differences. Thanks to you, the result I got is higher than the previous runs, but the accuracy in the case of w8a8 and w4a4 is not matched or exceeded the full-precision model accuracy. ( I used the full-precision standard Resnet-18 model with the accuracy of 70.94% to initialize the quantized models, I don't know whether this caused the low accuracy in the quantized models or not)

Therefore, if it possible, I hope that you can share the code for training and other hyper-parameters you have used in your experiments for references.

This is my Resnet-18 implementation based on your suggestion: https://gist.github.com/phuocphn/6ef18c2c3a895d4545951c447f370f28

Thank you so much.

[hustzxd]

The hyper-parameters are shared in the README.md
And maybe you can try to use weight decay for weight only.

def add_weight_decay(model, weight_decay, skip_keys):
    decay, no_decay = [], []
    for name, param in model.named_parameters():
        if not param.requires_grad:
            continue  # frozen weights
        added = False
        for skip_key in skip_keys:
            if skip_key in name:
                no_decay.append(param)
                added = True
                break
        if not added:
            decay.append(param)
    return [{'params': no_decay, 'weight_decay': 0.}, {'params': decay, 'weight_decay': weight_decay}]

criterion = nn.CrossEntropyLoss().cuda(args.gpu)
params = add_weight_decay(model, weight_decay=args.weight_decay, skip_keys=['alpha'])
optimizer = torch.optim.SGD(params, args.lr,
                                momentum=args.momentum,
                                weight_decay=args.weight_decay)

[phuocphn]

The hyper-parameters are shared in the README.md
And maybe you can try to use weight decay for weight only.

def add_weight_decay(model, weight_decay, skip_keys):
    decay, no_decay = [], []
    for name, param in model.named_parameters():
        if not param.requires_grad:
            continue  # frozen weights
        added = False
        for skip_key in skip_keys:
            if skip_key in name:
                no_decay.append(param)
                added = True
                break
        if not added:
            decay.append(param)
    return [{'params': no_decay, 'weight_decay': 0.}, {'params': decay, 'weight_decay': weight_decay}]

criterion = nn.CrossEntropyLoss().cuda(args.gpu)
params = add_weight_decay(model, weight_decay=args.weight_decay, skip_keys=['alpha'])
optimizer = torch.optim.SGD(params, args.lr,
                                momentum=args.momentum,
                                weight_decay=args.weight_decay)

Thank you very much for your helps. I have tried to use your provided hyper-parameters and didn't use weight decay for step size parameters, but the the result doesn't improve too much (I use a model with the accuracy of 70.2 to initialize the quantized models, but I got the final accuracy of 69.7x in case of w8a8 ). I have some questions regarding to this and I really need your helps again. Thank you so much

1. In the w8a8 case, how many additional epoch of training you need to get the result of (69.68 88.92 √ )?.
   In my case, I trained only 1 epoch as the authors of LSQ said, but the accuracy is not close to the full-precision model enough.
   Do you need to perform RELU calibration step as mentioned in the paper [Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference] ?
2. Regards to layer ordering, I wonder about whether you put the ActLSQ layers before or after pooling layers.?
   Currently, I am using this order (conv1--> bn --> relu --> actlsq(signed=False, nbits=8) --> maxpool) and ( basic_resnet_block_layer_4 --> avgpool --> actlsq(signed=False, nbits=8) --> fc) and it doesn't work well.
3. Regards to loading the full-precision checkpoint, I wonder did you tried to load the optimizer's state dict of the previous run or just only the weights of the model?
   It seems to me that when I try to resume training from a checkpoint, the accuracy drop a lot (even when I train again with the full-precision model) and it needs 1 - 2 epochs to catch up the accuracy of the previous run.

Thank you

[qiulinzhang]

Hi hustxzd,

Thank you for your valuable feedbacks. I trained a new model with your suggestions to see any differences. Thanks to you, the result I got is higher than the previous runs, but the accuracy in the case of w8a8 and w4a4 is not matched or exceeded the full-precision model accuracy. ( I used the full-precision standard Resnet-18 model with the accuracy of 70.94% to initialize the quantized models, I don't know whether this caused the low accuracy in the quantized models or not)

Therefore, if it possible, I hope that you can share the code for training and other hyper-parameters you have used in your experiments for references.

This is my Resnet-18 implementation based on your suggestion: https://gist.github.com/phuocphn/6ef18c2c3a895d4545951c447f370f28

Thank you so much.

Hello, the official pytorch offered pretrained-model for resnet18 is 69.7, where your "Resnet-18 model with the accuracy of 70.94% " comes from? whether it is publicly accessible?

[hustzxd]

Maybe you can find a better pretrained-model here. (https://pypi.org/project/pytorchcv/)