distilbert: fix creation of sinusoidal embeddings

[stefan-it]

Hi,

similar issue as reported by @stas00 with BART, see #8226.

The creation of sinusoidal embeddings is currently not working on PyTorch 1.8+.

It fails with:

  File "/mnt/europeana-bert/flair/flair/embeddings/token.py", line 820, in __init__
    self.model = AutoModel.from_pretrained(model, config=config, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/modeling_auto.py", line 728, in from_pretrained
    return MODEL_MAPPING[type(config)].from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1034, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 419, in __init__
    self.embeddings = Embeddings(config)  # Embeddings
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 88, in __init__
    create_sinusoidal_embeddings(
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 76, in create_sinusoidal_embeddings
    out[:, 0::2] = torch.FloatTensor(np.sin(position_enc[:, 0::2]))
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

I've seen this problem when trying to train a model in Flair with DistilBERT as feature-based embeddings, as well as when training a DistilBERT model from scratch using the official example.

It can be reproduced in a nvcr.io/nvidia/pytorch:20.12-py3 container, that comes with PyTorch 1.8.

[VictorSanh]

lgtm! thanks @stefan-it