CUDA: Check failed: e == cudaSuccess: misaligned address with 3-layer BERT pretraining

[szhengac]

When I pretrained a 3-layer BERT model using GluonNLP 0.10 on one p3.24dn instance with 32GB GPU memory, I received CUDA: Check failed: e == cudaSuccess: misaligned address. With batch size 128 in total, it uses 11GB GPU memory and no error occurs. But when I slightly increased the total batch size to 176 or double it to 256, I received the error. I have cherry-picked #17767.

@sxjscience you may want to try the setting in numpy version.

[szha]

What's the mxnet version/commit?

[szhengac]

i have tried 5b50600 and 1.8.0. I got the error when vocab size is 52000, and this does not happen when vocab size is 32000. So I guess embedding layer may have something to do with it.

[sxjscience]

Let me check it later today. It only happens in fp16?

[szhengac]

i only tried fp16

[sxjscience]

Would you try again with export MXNET_SAFE_ACCUMULATION=0? So we can check if it's caused by https://github.com/apache/incubator-mxnet/pull/18385/files

[szhengac]

5b50600 does not include that pr

[sxjscience]

So the error still appear on the master even if https://github.com/apache/incubator-mxnet/pull/18385/files is included?

[szhengac]

i didn't use 2.0 as it is not compatible as 0.10.

[szhengac]

but 1.8.0 contains #18385

[szhengac]

when i skip the attention layer and only keep the embedding layer and final dense layer, i received Segmentation fault with batch size 256. batch size 128 is fine.