Issues using deepspeed transformer kernel

[sriramiscool]

Hello,
I have been training models using deepspeed to train bert models.
The code I use is adapted from bing bert example.
I use --deepspeed_transformer_kernel=True when training and after training load the same model and perform prediction.

I am facing 4 issues when performing prediction using trained model:

Some minor changes made to the model and forward function:
def forward(self, input_ids, valid_length=None, masked_lm_labels=None, embedding_mode=True):

1. When I load a deepspeed model without kernel into a model with transformer kernel (I verified that the weights loaded are exact by double transformation) the embedding generated by model with transformer kernel and without transformer kernels are very different.

2. When using model with transformer kernel the embeddings generated by 2 inputs, input_ids1.shape[1] = 16 and input_ids2.shape[1] = 128, and valid_length = [10], and input_ids1[:, :10] == input_ids2[:, :10] the embeddings generated are not the same they may vary at the first decimal itself.

3. For some awkward number for input_ids.shape[1] like 9: the embedding generated are all NANs.

4. When input_ids.shape[1] < 5 : the program crashes with cuda error (does not display the error most of the time).

NOTE:
These issues do not exist if I use a model without deepspeed kernel.

Please let me know if this is expected and if so why and if not how can it be potentially fixed. Thanks

---

Below is code snippet (not complete) and the output:

def main():

start = time.time()
args = construct_arguments()
#INITIALIZE MODEL WITH DEEPSPEED TRANFORMER KERNEL
model_with_ds_tr_kernel, _ = prepare_model_optimizer(args)
args.saved_embedding_path = os.path.join(args.output_dir, "embeddings",
        args.embedding_type)
os.makedirs(args.saved_embedding_path, exist_ok=True)
args.logger.info(f"Created directory to save embeddings: {args.saved_embedding_path}")

if None in [args.load_training_checkpoint, args.load_checkpoint_id]:
    args.logger.info("load_training_checkpoint and load_checkpoint_id is missing.")
    sys.exit()
#LOAD A SAVED MODEL
global_step, global_data_samples, last_global_step_from_restore = load_checkpoint(args, model_with_ds_tr_kernel)

#LOAD SAME MODEL BUT WITHOUT TRANSFORMER KERNEL
from m5_transformers.models.bert import BertForPreTrainingPreLN
model_no_ds_tr_kernel = BertForPreTrainingPreLN.from_pretrained("../conversion/dsmNoTK_500M_1024_spm")
model_no_ds_tr_kernel.to("cuda")

model_no_ds_tr_kernel.eval()
model_with_ds_tr_kernel.eval()

print("-----------------------------------ISSUE 1-------------------------------------------------")
print("EMBEDDINGS DIFFERENT FOR MODEL WITH AND WITHOUT TRANSFORMER KERNEL")
seq_len = 7
ipid1 = torch.arange(512).view([1,512])
ipid1 += 1000
ipid1[:,seq_len:] = 3
ipid1 = ipid1.to("cuda")
valid_len = torch.tensor([seq_len], dtype=torch.long)
valid_len = valid_len.to("cuda")
with torch.no_grad():
    s1,p1 = model_with_ds_tr_kernel.network(ipid1, valid_len, embedding_mode=True)
    s2,p2 = model_no_ds_tr_kernel(ipid1, valid_len, embedding_mode=True)
s1 = s1.cpu().numpy()[:,:seq_len]
s2 = s2.cpu().numpy()[:,:seq_len]
print("EMBEDDINGS GENERATED SAME: {}".format(np.allclose(s1,s2, atol=1e-4)))
print("EMBEDDINGS ABSOLUTE MEAN ERROR: {}".format(np.abs(s1-s2).mean()))


print("-----------------------------------ISSUE 2-------------------------------------------------")
print("EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: 2 INPUTS DIFFERENT SIZE BATCH BUT SAME DATA")
ipid2 = torch.arange(16).view([1,16])
ipid2 += 1000
ipid2[:,seq_len:] = 3
ipid2 = ipid2.to("cuda")
valid_len = torch.tensor([seq_len], dtype=torch.long)
valid_len = valid_len.to("cuda")
with torch.no_grad():
    s1,p1 = model_with_ds_tr_kernel.network(ipid1, valid_len, embedding_mode=True)
    s2,p2 = model_with_ds_tr_kernel.network(ipid2, valid_len, embedding_mode=True)
s1 = s1.cpu().numpy()[:,:seq_len]
s2 = s2.cpu().numpy()[:,:seq_len]
print("EMBEDDINGS GENERATED SAME: {}".format(np.allclose(s1,s2, atol=1e-4)))
print("EMBEDDINGS ABSOLUTE MEAN ERROR: {}".format(np.abs(s1-s2).mean()))


print("-----------------------------------ISSUE 3-------------------------------------------------")
print("EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: INPUTS WITH ODD BATCH LENGTH")
ipid1 = torch.arange(9).view([1,9])
ipid1 += 1000
ipid1[:,seq_len:] = 3
ipid1 = ipid1.to("cuda")
valid_len = torch.tensor([seq_len], dtype=torch.long)
valid_len = valid_len.to("cuda")
with torch.no_grad():
    s1,p1 = model_with_ds_tr_kernel.network(ipid1, valid_len, embedding_mode=True)
s1 = s1.cpu().numpy()[:,:seq_len]
print("EMBEDDINGS GENERATED : {}".format(s1))


print("-----------------------------------ISSUE 4-------------------------------------------------")
print("EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: INPUTS WITH VERY SMALL BATCH LENGTH")
seq_len = 2
ipid1 = torch.arange(3).view([1,3])
ipid1 += 1000
ipid1[:,seq_len:] = 3
ipid1 = ipid1.to("cuda")
valid_len = torch.tensor([seq_len], dtype=torch.long)
valid_len = valid_len.to("cuda")
print("SYSTEM HAS CRASHED IF NOTHING IS PRINTED AFTER THIS...")
with torch.no_grad():
    s1,p1 = model_with_ds_tr_kernel.network(ipid1, valid_len, embedding_mode=True)
s1 = s1.cpu().numpy()[:,:seq_len]
print("EMBEDDINGS GENERATED : {}".format(s1))

elapsed = time.time() - start
logger = args.logger
logger.info(f"Elapsed time: {elapsed} seconds")

if name == "main":
main()

OUTPUT:
-----------------------------------ISSUE 1-------------------------------------------------
EMBEDDINGS DIFFERENT FOR MODEL WITH AND WITHOUT TRANSFORMER KERNEL
EMBEDDINGS GENERATED SAME: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.9891793131828308
-----------------------------------ISSUE 2-------------------------------------------------
EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: 2 INPUTS DIFFERENT SIZE BATCH BUT SAME DATA
EMBEDDINGS GENERATED SAME: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.010894775390625
-----------------------------------ISSUE 3-------------------------------------------------
EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: INPUTS WITH ODD BATCH LENGTH
EMBEDDINGS GENERATED : [[[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]
...
[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]]]
-----------------------------------ISSUE 4-------------------------------------------------
EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: INPUTS WITH VERY SMALL BATCH LENGTH
SYSTEM HAS CRASHED IF NOTHING IS PRINTED AFTER THIS...

[RezaYazdaniAminabadi]

Hi @sriramiscool

Thanks for pointing out these issues.
We are aware of some of these issues and are constantly working on fixing them asap. For the last two issues you mentioned, I pushed a PR, #587. Could you please check it the issues are resolved?
I also think this PR would solve your first issue. But, please let me know if it does not and I further investigate this.
I am not sure if I understand the second issue. Are you saying that the data only exists in [:, :10] and the rest is zero? Can you please explain this further?

Thanks.
Reza

[szhengac]

@RezaYazdaniAminabadi Yes. We mean the results become very different when we add more paddings. We expected a small difference due to masked softmax, but the difference is actually non-negligible.

[RezaYazdaniAminabadi]

Can you try something here? Pad the mask to be a high negative value (like -10000) for the part of the input that is padded, and let's see if the results are still too much different. This will reduce the effect of this padded part when going through the softmax by making them a high negative value, which result in near-zero softmax output.

[RezaYazdaniAminabadi]

Also, let's use random values on the padded input instead of zero (this should not impact the result).
Thanks,
Reza

[szhengac]

Can you try something here? Pad the mask to be a high negative value (like -10000) for the part of the input that is padded, and let's see if the results are still too much different. This will reduce the effect of this padded part when going through the softmax by making them a high negative value, which result in near-zero softmax output.

We did this in the model.

[szhengac]

We converted 1 to 0 and 0 to -10000 in the mask in the model.

[RezaYazdaniAminabadi]

I see, thanks for pointing this.
Would you mind double-checking this with the new PR (#587)?
For the padding, I have seen similar issue and there seems to be a problem when all the padded data are zeros or ones (similar). I could solve that by changing the padding data to random values. I wonder if the same behavior exists in your testing.
Looking forward to seeing your result.
Thanks,
Reza

[sriramiscool]

Thank you for pointing to the PR, we will try it out and let you know soon. And we did try random padding tokens as well, the result is still the same. Thank you, Sriram

…

On Tue, Dec 8, 2020, 12:18 AM Reza Yazdani ***@***.***> wrote: I see, thanks for pointing this. Would you mind double-checking this with the new PR (#587 <#587>)? For the padding, I have seen similar issue and there seems to be a problem when all the padded data are zeros or ones (similar). I could solve that by changing the padding data to random values. I wonder if the same behavior exists in your testing. Looking forward to seeing your result. Thanks, Reza — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#589 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADUIPRG6YGATAITI6SELVLLSTXONHANCNFSM4URJ5UBA> .

[sriramiscool]

Hello,
I tried the PR.
Looks like it fixed the Issue 3 and Issue 4 but the Issue 1 and 2 still seem exist.

Below is the output for the same code as earlier using the PR you pointed me to. Please let me know what can be done next.
Thanks.

-----------------------------------ISSUE 1-------------------------------------------------
EMBEDDINGS DIFFERENT FOR MODEL WITH AND WITHOUT TRANSFORMER KERNEL
EMBEDDINGS GENERATED SAME: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.9900880455970764
-----------------------------------ISSUE 2-------------------------------------------------
EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: 2 INPUTS DIFFERENT SIZE BATCH BUT SAME DATA
EMBEDDINGS GENERATED SAME: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0124969482421875
-----------------------------------ISSUE 3-------------------------------------------------
EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: INPUTS WITH ODD BATCH LENGTH
EMBEDDINGS GENERATED : [[[ -3.377 -1.132 3.041 ... -3.508 -3.05 -4.23 ]
[ -4.402 1.008 1.122 ... -1.098 -0.9053 -1.595 ]
[ -2.691 2.217 1.026 ... 7.35 4.28 -3.467 ]
...
[ -0.654 -1.821 2.586 ... 3.568 6.945 -0.9917]
[ -2.203 -3.877 -4.36 ... -19.33 -0.8467 -0.373 ]
[ 5.758 -4.156 5.67 ... -6.32 -12.195 0.2412]]]
-----------------------------------ISSUE 4-------------------------------------------------
EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: INPUTS WITH VERY SMALL BATCH LENGTH
SYSTEM HAS CRASHED IF NOTHING IS PRINTED AFTER THIS...
EMBEDDINGS GENERATED : [[[-2.16 -1.155 3.951 ... -3.059 -2.559 -2.781 ]
[-2.053 1.162 2.062 ... -1.6045 -2.066 -1.168 ]]]

[sriramiscool]

Also found some interesting pattern with issue 2 actually.
By varying lengths we observe that is input1.shape[1] = x and
input1.shape[1]-(16-input1.shape[1]%16) <= input2.shape[1] <= input1.shape[1]+(input1.shape[1]%16)
Then the embeddings are the same with no diff.
Further, for input2.shape[1] \in range (129, 160) or range(257, 336) the different in embeddings is super large and for others is the absolute mean difference is in 0.01 range.

Below is the code snippet and result.
THanks

print("-----------------------------------ISSUE 2-------------------------------------------------")
print("EMBEDDINGS FOR MODEL WITH TRANSFORMER KERNEL: 2 INPUTS DIFFERENT SIZE BATCH BUT SAME DATA")
seq_len = 9
ipid2 = torch.arange(128).view([1,128])
ipid2 += 1000
ipid2[:,seq_len:] = torch.randint(10,32000,(128-seq_len,))
ipid2 = ipid2.to("cuda")
for cnrt in range(9,512):
    ipid1 = torch.arange(cnrt).view([1,cnrt])
    ipid1 += 1000
    ipid1[:,seq_len:] = torch.randint(10,32000,(cnrt-seq_len,))
    ipid1 = ipid1.to("cuda")
    valid_len = torch.tensor([seq_len], dtype=torch.long)
    valid_len = valid_len.to("cuda")
    with torch.no_grad():
        s1,p1 = model_with_ds_tr_kernel.network(ipid1, valid_len, embedding_mode=True)
        s2,p2 = model_with_ds_tr_kernel.network(ipid2, valid_len, embedding_mode=True)
    s1 = s1.cpu().numpy()[:,:seq_len]
    s2 = s2.cpu().numpy()[:,:seq_len]
    print("EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and {}: {}".format(cnrt, np.allclose(s1,s2, atol=1e-4)))
    print("EMBEDDINGS ABSOLUTE MEAN ERROR: {}".format(np.abs(s1-s2).mean()))

OUTPUT:
.......
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01132965087890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 108: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01132965087890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 109: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01132965087890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 110: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01132965087890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 111: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01132965087890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 112: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01132965087890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 113: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 114: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 115: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 116: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 117: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 118: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 119: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 120: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 121: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 122: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 123: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 124: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 125: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 126: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 127: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 128: True
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 129: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 6.85546875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 130: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.1875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 131: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.359375
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 132: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 6.7890625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 133: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.8828125
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 134: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.36328125
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 135: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.67578125
..........
..........
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 158: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 6.34375
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 159: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 5.78515625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 160: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 4.57421875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 161: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 162: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 163: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 164: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 165: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 166: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 167: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 168: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 169: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0118865966796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 170: False
............
............
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 253: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.011138916015625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 254: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.011138916015625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 255: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.011138916015625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 256: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.011138916015625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 257: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 6.61328125
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 258: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.0859375
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 259: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.296875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 260: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.16796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 261: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.06640625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 262: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.5625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 263: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.78515625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 264: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.13671875
....................
....................
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 332: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.33203125
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 333: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.0546875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 334: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 6.6328125
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 335: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.046875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 336: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 4.5546875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 337: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01085662841796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 338: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01085662841796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 339: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01085662841796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 340: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01085662841796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 341: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01085662841796875
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 342: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.01085662841796875
............................
............................
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 511: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 0.0116729736328125

[RezaYazdaniAminabadi]

Hi @sriramiscool

Thanks for checking this. And this behavior is for sure strange.
I have updated the branch more today. Could you give it another try?
Also, if you can let me know of the model configuration that you are testing and what sequence-length you are using and which part is padded, I can solve this faster.
Thanks,
Reza

[sriramiscool]

Sure, I can do that. Thanks.

And other details:
Model max sequence length: 512
At the time of input in the code I pasted earlier:
seq_len = 9
ipid2 = torch.arange(128).view([1,128])
input 2 array length is 128 but the valid sequence length is 9, rest of the tokens are random tokens (as you suggested earlier).
input 1 array length I varried from 9 to 512.

The configurations used:
DEEPSPEED CONFIG:
{
"train_batch_size": 1024,
"train_micro_batch_size_per_gpu": 32,
"steps_per_print": 1000,
"prescale_gradients": false,
"zero_optimization": {
"stage": 2
},
"zero_allow_untested_optimizer": true,
"optimizer": {
"type": "LANS",
"params": {
"lr": 0.0001,
"weight_decay": 0.01,
"bias_correction": true,
"eps": 0.000001
}
},
"gradient_clipping": 1,
"wall_clock_breakdown": false,
"fp16": {
"enabled": true,
"loss_scale": 0,
"initial_scale_power": 20,
"loss_scale_window": 1000
}
}

model config:
{
"name": "bert_345M_1024",
"bert_token_file": "bert_asin.model",
"bert_model_config": {
"vocab_size_or_config_json_file": 32003,
"hidden_size": 1024,
"num_hidden_layers": 24,
"num_attention_heads": 16,
"intermediate_size": 4096,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"attention_probs_dropout_prob": 0.1,
"max_position_embeddings": 512,
"initializer_range": 0.02
},
"data": {
"mixed_seq_datasets": {
"512": {
...
}
}
},
"mixed_seq_training": {
"128": {
"warmup_proportion": 0.05,
"const_proportion": 0.25,
"learning_rate": 0.003,
"num_workers": 2,
"total_training_steps": 112500
},
"512": {
"warmup_proportion": 0.025,
"const_proportion": 0.125,
"learning_rate": 0.003,
"num_workers": 2,
"total_training_steps": 12500
}
}
}

Other command line parameters:
CUDA_VISIBLE_DEVICES=0 deepspeed deepspeed_predict_test.py
--cf model_config.json
--max_seq_length 512
--output_dir $OUTPUT_DIR
--deepspeed
--deepspeed_transformer_kernel
--job_name $JOB_NAME
--deepspeed_config deepspeed_config.json
--load_training_checkpoint chkpt_folder
--load_checkpoint_id model1
--attention_dropout_checkpoint
--log_file ${JOB_NAME}.log

[sriramiscool]

Hello,
I tried the latest commit and looks like the super large difference I was seeing like these:
"
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 262: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.5625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 263: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.78515625
EMBEDDINGS GENERATED SAME FOR PADDED SEQ LENS 128 and 264: False
EMBEDDINGS ABSOLUTE MEAN ERROR: 7.13671875
"

are gone. And all differences are of order 0.01. And the issue 1 still persists.

THanks

[RezaYazdaniAminabadi]

Okay, cool. So, there is only one issue remaining. I will work on it and let you know.

Thanks
Reza