How to use Longformer for summarization

[chetanambi]

Hi - Do you have a sample code on how to use Longformer for summarization tasks.

[christianversloot]

Perhaps this will help: https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16

[patil-suraj]

LongformerEncoderDecoder will be added to the lib once #9278 is merged! It can be used for summarization or any other seq2seq task.

[chetanambi]

@patil-suraj, @christianversloot I tried this model longformer2roberta. It is actually giving better summary than Pegasus (reddit-tifu). I will be waiting for LongformerEncoderDecoder to be added to the library.

I just have one question - how many input tokens longformer2roberta model supports? I believe it's 2048. Could you please confirm,

[patil-suraj]

longformer2roberta should support 4096 tokens.

And LED is now on master!

[chetanambi]

@patil-suraj Awesome! I am trying LED but getting below error. Could you please take a look?

from transformers import LEDForConditionalGeneration, LEDTokenizer
model_name = 'allenai/led-base-16384'
tokenizer = LEDTokenizer.from_pretrained(model_name) 
model = LEDTokenizer.from_pretrained(model_name)
batch = tokenizer.prepare_seq2seq_batch(article, truncation=True, padding='longest', return_tensors="pt").to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)

---

AttributeError Traceback (most recent call last)
in ()
5 model = LEDTokenizer.from_pretrained(model_name)
6 batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest', return_tensors="pt").to(torch_device)
----> 7 translated = model.generate(**batch)
8 tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)

AttributeError: 'LEDTokenizer' object has no attribute 'generate'

[patil-suraj]

model = LEDTokenizer.from_pretrained(model_name)

here you are assigning tokenizer as the model, it should be

model = LEDForConditionalGeneration.from_pretrained(model_name)

[chetanambi]

@patil-suraj Thank you! My bad. I have corrected the typo. Below code seems to be working fine.

from transformers import LEDForConditionalGeneration, LEDTokenizer
model_name = 'allenai/led-base-16384'
tokenizer = LEDTokenizer.from_pretrained(model_name) 
model = LEDForConditionalGeneration.from_pretrained(model_name)
input_ids = tokenizer(src_text, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)
output = tokenizer.decode(output_ids[0], skip_special_tokens=True)

I am getting very short summary of just 20 tokens from the above code. So I was looking for the default values for below parameters for LED model. I could not find it in config.json file. For the longformer2roberta model I found these values in config.json. Could you please let me know where can I find these values for LED model.

• num_beams
• no_repeat_ngram_size,
• early_stopping,
• length_penalty,
• min_length,
• max_length

[chetanambi]

@patil-suraj Any inputs/suggestions here?

[yww211]

Hi,
I have a question about the LEDForConditionalGeneration forward args.
The decoder_input_ids has a comment that decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) – Provide for translation and summarization training. By default, the model will create this tensor by shifting the input_ids to the right, following the paper..
Form the forward method in LEDForConditionalGeneration, i can see that when not assigning the decoder_input_ids in the forward method of LEDForConditionalGeneration object , the decoder_input_ids will be generated by shifting the labels value one token to right in the forward method.

So my question is if i want to explictly pass the decoder_input_ids to the forward method, do i need to explictly shift it one token as the code shows before the forward pass?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.