[Manuscript] TermDiffuSum: A Term-guided Diffusion Model for Extractive Summarization of Legal Documents

• 🎈 Manuscript is under review since September 15, 2024.

• 💻 Github Link.

• 📖 Paper Link.

• 📁 Datasets Link.

• Currently provided is partial code; the complete code and dataset are being further organized, and we will update it subsequently.

Abstract: Legal document summarization (LDS) task aims to extract essential information from legal documents automatically. It is a critical task in legal artificial intelligence (LegalAI) and has attracted wide attention in recent years. Recently, diffusion models have been explored for extractive summarization and have demonstrated remarkable capabilities. However, limitations remain in fully understanding the legal terminology, which is abundant in legal documents. To address the limitation, this paper presents a novel term-guided diffusion extractive model for legal document summarization, named TermDiffuSum, which leverages legal terminology within the diffusion model for extractive summarization of legal documents. To enhance the model's understanding of legal terminology, it introduces a noise-weighting schedule that allocates more attention to sentences containing a higher concentration of legal terms during the diffusion process. In addition, TermDiffuSum incorporates a re-ranking loss function to refine the model's selection of more relevant summaries by leveraging the relationship between the candidate summaries generated by the diffusion process and the reference summaries. Experimental results on a self-constructed legal summarization dataset reveal that TermDiffuSum outperforms existing diffusion-based summarization models, achieving improvements of 3.10 in ROUGE-1, 2.84 in ROUGE-2, and 2.89 in ROUGE-L. To further validate the generalizability of TermDiffuSum, we conduct experiments on three public datasets from news and social media domains, with results affirming the effectiveness of our approach.

Train & Test

Train

# train on LegalAFSum
python run_train_sum.py --diff_steps 500 --model_arch transformer_sum --lr 1e-5 --seed 101 --noise_schedule legal_noise --in_channel 128 --modality roc --submit no --padding_mode pad --app "--predict_xstart True --training_mode e2e --roc_train legal " --notes legal --bsz 32 --epochs 10 --save_name _legal_ 

# train for CNN/DM
python run_train_sum.py --diff_steps 500 --model_arch transformer_sum --lr 1e-5 --seed 101 --noise_schedule legal_noise --in_channel 128 --modality roc --submit no --padding_mode pad --app "--predict_xstart True --training_mode e2e --roc_train cnndm " --notes cnndm --bsz 32 --epochs 10 --save_name _cnndm_

# train for Reddit
python run_train_sum.py --diff_steps 500 --model_arch transformer_sum --lr 1e-5 --seed 101 --noise_schedule legal_noise --in_channel 128 --modality roc --submit no --padding_mode pad --app "--predict_xstart True --training_mode e2e --roc_train reddit " --notes reddit --bsz 32 --epochs 10 --save_name _reddit_

# train for XSum
python run_train_sum.py --diff_steps 500 --model_arch transformer_sum --lr 1e-5 --seed 101 --noise_schedule legal_noise --in_channel 128 --modality roc --submit no --padding_mode pad --app "--predict_xstart True --training_mode e2e --roc_train xsum " --notes xsum --bsz 32 --epochs 10 --save_name _xsum_

Test

python batch_decode_sum.py [checkpoint_path] -1.0 ema

All of our code will be open-sourced here upon the acceptance of the manuscript.

Citation

If you find our work inspiring or use our codebase in your research, please consider giving a star ⭐ and a citation.