InCorrect LoRALayer initialization of lora_A

[Icontofig]

Hello. I'm troubling with the LoRALayer Defination in src/t5_prompt.py.

I followed your modify of modeling_t5.py（t5_prompt.py） to modify a Bart. But During Training I got the grad of LoRALayer all zero. I checked the parameters of the LoRALayer and got that both lora_A and lora_B is all zero.

Things are solved after I modified the defination of lora_A as following:

std_dev = 1 / torch.sqrt(torch.tensor(r).float())#after modified  
self.lora_A = nn.Parameter(torch.randn((in_features, r)) * std_dev, requires_grad = True) #after modified  
# self.lora_A = nn.Parameter(torch.zeros((in_features, r)), requires_grad = True) # before modified  

So I'm not sure that if you have uploaded an incorrent version of src/t5_prompt.py.