[WIP] seq2seq example

[mgoldey]

This PR presents an example seq2seq use case and bug fixes necessary for this to execute with reasonable accuracy.

The utils_seq2seq.py file defines the data format for training data, and the run_seq2seq.py file takes training, development, and test data and produces a model. The README.md discusses how to execute this toy problem. The specific toy problem in use here is formatting a date string to the American style, which is a trivial example. On my local setup using GPUs, this example executes within 5 minutes. Production models should include more learnings.

I welcome feedback about how to strengthen performance here and the best route to increase testing.

This relies on a few bug fixes which have been incorporated in this branch

• Without a fix for AttributeError: 'Model2Model' object has no attribute 'prepare_model_kwargs' in 2.5.1 #3038, PreTrainedEncoderDecoder won't instantiate at all.
• Without a fix for update the config.is_decoder=True before initialize the decoder #2435, BERT models fail completely on this use case as the BERT decoder isn't instantiated correctly without CrossAttention in that case.
• I strongly suspect that the input to the decoder in the PreTrainedEncoderDecoder class is incorrect as present in the code base, and commit 9fcf73a has a proposed fix. It doesn't make sense to have the expected token ids as input to the decoder when the decoder needs to learn how to decode from the embeddings. Incomplete understanding - will fix

[mgoldey]

As a non-blocking question, I do note that a lot of the examples use argparse to parse comparatively long lists of arguments. I've maintained the extant style in this PR to avoid causing noise and confusion Would it be acceptable if I broke with this style to use a JSON file to store all the arguments for an experiment?

[patrickvonplaten]

Hi @mgoldey, sorry for only responding now. Thanks a lot for adding a seq2seq example :-) I will take a look early next week and maybe we can have a quick chat how to merge this PR and #3383.

[mgoldey]

That sounds good. I'm still tweaking things on my end for accuracy and improved logic as I get more familiar with the code base here. I'll see if I can rebase of #3383 by then, depending on my other workload. Feel free to reach out via google hangouts if you're comfortable.

[dbaxter240]

Hi, I've been using your example to guide my own setup of a seq2seq model (thank you!) Is this line in your pull request a typo? Think it's just supposed to be cls_token, sep_token, pad_token = ( ...

[dbaxter240]

I believe this is supposed to be

elif model_type in ['gpt2']

instead of "not in"

[patrickvonplaten]

Sorry, to answer only now! I'll will soon add a Encoder-Decoder google colab that shows how to use seq2seq

[mgoldey]

Thanks - fine to close. We've moved forward without using seq2seq due to poor overall accuracy with the scale of data in place.