Getting Started

1. Installation

conda create --name summary python=3.9
conda activate summary
pip install -r requirement.txt

Prepare pretrain model weights

Code prefixed with "LLama" are designed to work with the LLaMA 2 7B model weights. Similarly, code prefixed with "llama3" are compatible with the LLaMA 3 8B and LLaMA 3.1 8B model weights. You can obtain the official LLaMA consolidated format weights(Instruct version) by downloading them from the official Meta AI website or the Hugging Face model hub.

2. Data Processing

1. Download datasets from their respective official repositories:

   • SQuALITY
   • CovidET
   • QMSum

2. Preprocess the datasets using the provided Jupyter notebook: data_process.ipynb.

3. Training, Inference, and Evaluation

To train, run inference, and evaluate the model, execute the following script:

bash exps/finetuning_*_generate_evaluate.sh

For multi-reference Rouge scores and Bert-score evaluations on the SQuALITY dataset, use the notebook multi_reference_evaluation_SQuAlITY.ipynb.

Results

• output/gpt-4o: Contains the results generated by GPT-4o (2024-08-06).
• output/LLaMA3-lora-hyper: Contains the results of IDEAL_lora (based on LLaMA3.1-8B).
• output/GPTRank: Contains the results of IDEAL_LoRA and Socratic on the Squality dataset (aligned with the original test set order), as well as the comparison results of the two methods' summaries using GPTRank.

Acknowledgment

Our project is developed based on the following repositories:

• LLaMA-Adapter
• llama-models