[Enhancement]create huggingface_gptneox_convert.py

[zhang-ge-hao]

Let the gptneox HF model file can convert to FT model.

Mainly based on tabby's script, thanks to @wsxiaoys.

But original script got error using tensor parallel.(link)

I fixed the problem and hope to open this script for everyone to use.

[zhang-ge-hao]

Examples:

Get HF model files:

git lfs clone https://huggingface.co/TabbyML/NeoX-70M
git lfs clone https://huggingface.co/TabbyML/NeoX-1.3B

Convert to the 1-gpu model files:

python ../examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py -i ../models/gptneox/model/NeoX-70M -o ../models/gptneox/c-model/NeoX-70M -i_g 1 -m_n gptneox
python ../examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py -i ../models/gptneox/model/NeoX-1.3B -o ../models/gptneox/c-model/NeoX-1.3B -i_g 1 -m_n gptneox

Convert to the 2-gpu model files(to use tensor parallel):

python ../examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py -i ../models/gptneox/model/NeoX-70M -o ../models/gptneox/c-model/NeoX-70M -i_g 2 -m_n gptneox
python ../examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py -i ../models/gptneox/model/NeoX-1.3B -o ../models/gptneox/c-model/NeoX-1.3B -i_g 2 -m_n gptneox

Run and validate 1-GPU model files:

python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-70M/1-gpu --tokenizer_path ../models/gptneox/model/NeoX-70M --sample_input_file gptneox_input
python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-1.3B/1-gpu --tokenizer_path ../models/gptneox/model/NeoX-1.3B --sample_input_file gptneox_input

Run and validate tensor parallel model files:

mpirun -n 2 --allow-run-as-root python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-70M/2-gpu --tokenizer_path ../models/gptneox/model/NeoX-70M --sample_input_file gptneox_input --tensor_para_size 2
mpirun -n 2 --allow-run-as-root python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-1.3B/2-gpu --tokenizer_path ../models/gptneox/model/NeoX-70M --sample_input_file gptneox_input --tensor_para_size 2

Results:

root@f0305219ab2b:/workspace/FasterTransformer/build# python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-70M/1-gpu --tokenizer_path ../models/gptneox/model/NeoX-70M --sample_input_file gptneox_input

=============== Arguments ===============
output_len: 32
beam_width: 1
top_k: 1
top_p: 0.0
temperature: 1.0
len_penalty: 0.0
beam_search_diversity_rate: 0.0
tensor_para_size: 1
pipeline_para_size: 1
ckpt_path: ../models/gptneox/c-model/NeoX-70M/1-gpu
tokenizer_path: ../models/gptneox/model/NeoX-70M
lib_path: ./lib/libth_transformer.so
sample_input_file: gptneox_input
max_batch_size: 8
repetition_penalty: 1.0
max_seq_len: 1024
inference_data_type: fp16
time: False
enable_random_seed: False
=========================================

[INFO] batch size: 2
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[INFO] batch 0, beam 0:
[Context]
Hello,

[Output]


I'm a bit confused about the name of the game. I'm a bit confused about the name of the game. I'm a bit confused about<|endoftext|><|endoftext|>

[INFO] batch 1, beam 0:
[Context]
Gama start,

[Output]
 and then the next time you see the same thing, you'll see the same thing again.

I'm not sure if I'm going to be able

root@f0305219ab2b:/workspace/FasterTransformer/build# python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-1.3B/1-gpu --tokenizer_path ../models/gptneox/model/NeoX-1.3B --sample_input_file gptneox_input

=============== Arguments ===============
output_len: 32
beam_width: 1
top_k: 1
top_p: 0.0
temperature: 1.0
len_penalty: 0.0
beam_search_diversity_rate: 0.0
tensor_para_size: 1
pipeline_para_size: 1
ckpt_path: ../models/gptneox/c-model/NeoX-1.3B/1-gpu
tokenizer_path: ../models/gptneox/model/NeoX-1.3B
lib_path: ./lib/libth_transformer.so
sample_input_file: gptneox_input
max_batch_size: 8
repetition_penalty: 1.0
max_seq_len: 1024
inference_data_type: fp16
time: False
enable_random_seed: False
=========================================

[INFO] batch size: 2
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[INFO] batch 0, beam 0:
[Context]
Hello,

[Output]
 I'm a newbie. I'm trying to install Ubuntu on my laptop. I have a Dell Inspiron 1525 with Windows 7. I have a USB<|endoftext|><|endoftext|>

[INFO] batch 1, beam 0:
[Context]
Gama start,

[Output]
 and the first of the two-day event.

The first day of the Gama start was a bit of a letdown. The first few laps

root@f0305219ab2b:/workspace/FasterTransformer/build# mpirun -n 2 --allow-run-as-root python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-70M/2-gpu --tokenizer_path ../models/gptneox/model/NeoX-70M --sample_input_file gptneox_input --tensor_para_size 2

=============== Arguments ===============
output_len: 32
beam_width: 1
top_k: 1
top_p: 0.0
temperature: 1.0
len_penalty: 0.0
beam_search_diversity_rate: 0.0
tensor_para_size: 2
pipeline_para_size: 1
ckpt_path: ../models/gptneox/c-model/NeoX-70M/2-gpu
tokenizer_path: ../models/gptneox/model/NeoX-70M
lib_path: ./lib/libth_transformer.so
sample_input_file: gptneox_input

=============== Arguments ===============
output_len: 32
beam_width: 1
top_k: 1
top_p: 0.0
temperature: 1.0
len_penalty: 0.0
beam_search_diversity_rate: 0.0
max_batch_size: 8
repetition_penalty: 1.0
max_seq_len: 1024
inference_data_type: fp16
time: False
enable_random_seed: False
=========================================

tensor_para_size: 2
pipeline_para_size: 1
ckpt_path: ../models/gptneox/c-model/NeoX-70M/2-gpu
tokenizer_path: ../models/gptneox/model/NeoX-70M
lib_path: ./lib/libth_transformer.so
sample_input_file: gptneox_input
max_batch_size: 8
repetition_penalty: 1.0
max_seq_len: 1024
inference_data_type: fp16
time: False
enable_random_seed: False
=========================================

[INFO] batch size: 2
[INFO] batch size: 2
[INFO] WARNING: Have initialized the process group
[INFO] WARNING: Have initialized the process group
[WARNING] gemm_config.in is not found; using default GEMM algo
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][INFO] NCCL initialized rank=1 world_size=2 tensor_para=NcclParam[rank=1, world_size=2, nccl_comm=0x55c57ea19a60] pipeline_para=NcclParam[rank=0, world_size=1, nccl_comm=0x55c57d9102b0]
[FT][INFO] NCCL initialized rank=0 world_size=2 tensor_para=NcclParam[rank=0, world_size=2, nccl_comm=0x555d83d01d00] pipeline_para=NcclParam[rank=0, world_size=1, nccl_comm=0x555d83145c90]
[INFO] batch 0, beam 0:
[Context]
Hello,

[Output]


I'm a bit confused about the name of the game. I'm a bit confused about the name of the game. I'm a bit confused about<|endoftext|><|endoftext|>

[INFO] batch 1, beam 0:
[Context]
Gama start,

[Output]
 and then the next time you see the same thing, you'll see the same thing again.

I'm not sure if I'm going to be able

root@f0305219ab2b:/workspace/FasterTransformer/build# mpirun -n 2 --allow-run-as-root python ../examples/pytorch/gptneox/gptneox_example.py --ckpt_path ../models/gptneox/c-model/NeoX-1.3B/2-gpu --tokenizer_path ../models/gptneox/model/NeoX-70M --sample_input_file gptneox_input --tensor_para_size 2

=============== Arguments ===============
output_len: 32
beam_width: 1
top_k: 1
top_p: 0.0
temperature: 1.0
len_penalty: 0.0
beam_search_diversity_rate: 0.0
tensor_para_size: 2
pipeline_para_size: 1
ckpt_path: ../models/gptneox/c-model/NeoX-1.3B/2-gpu
tokenizer_path: ../models/gptneox/model/NeoX-70M
lib_path: ./lib/libth_transformer.so
sample_input_file: gptneox_input
max_batch_size: 8
repetition_penalty: 1.0
max_seq_len: 1024
inference_data_type: fp16
time: False
enable_random_seed: False
=========================================


=============== Arguments ===============
output_len: 32
beam_width: 1
top_k: 1
top_p: 0.0
temperature: 1.0
len_penalty: 0.0
beam_search_diversity_rate: 0.0
tensor_para_size: 2
pipeline_para_size: 1
ckpt_path: ../models/gptneox/c-model/NeoX-1.3B/2-gpu
tokenizer_path: ../models/gptneox/model/NeoX-70M
lib_path: ./lib/libth_transformer.so
sample_input_file: gptneox_input
max_batch_size: 8
repetition_penalty: 1.0
max_seq_len: 1024
inference_data_type: fp16
time: False
enable_random_seed: False
=========================================

[INFO] batch size: 2
[INFO] batch size: 2
[INFO] WARNING: Have initialized the process group
[INFO] WARNING: Have initialized the process group
[WARNING] gemm_config.in is not found; using default GEMM algo
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][INFO] NCCL initialized rank=0 world_size=2 tensor_para=NcclParam[rank=0, world_size=2, nccl_comm=0x5616bfcc8ac0] pipeline_para=NcclParam[rank=0, world_size=1, nccl_comm=0x5616c1590d90]
[FT][INFO] NCCL initialized rank=1 world_size=2 tensor_para=NcclParam[rank=1, world_size=2, nccl_comm=0x5611d61198b0] pipeline_para=NcclParam[rank=0, world_size=1, nccl_comm=0x5611d5963340]
[INFO] batch 0, beam 0:
[Context]
Hello,

[Output]
 I'm a newbie. I'm trying to install Ubuntu on my laptop. I have a Dell Inspiron 1525 with Windows 7. I have a USB<|endoftext|><|endoftext|>

[INFO] batch 1, beam 0:
[Context]
Gama start,

[Output]
 and the first of the two-day event.

The first day of the Gama start was a bit of a letdown. The first few laps

[zhang-ge-hao]

@byshiue

Hi, looking forward to you taking a look.

I add several cases and hope to make it an easy-to-validate PR. 🤗🤗🤗

[wsxiaoys]

Great work! If you’re interested, welcome to also send out a PR to tabby as well

[byshiue]

Thank you for the great work. Can you help adding some examples into the gptneox_guide.md?

[zhang-ge-hao]

Thank you for the great work. Can you help adding some examples into the gptneox_guide.md?

@byshiue Updated.

[cksac]

will this works for https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b?, which seems same format as gptneox

[zhang-ge-hao]

will this works for https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b?, which seems same format as gptneox

if the model can be loaded by GPTNeoXForCausalLM, the script can theoretically convert it.

if not, I will consider adding this support when I have time.

[zhang-ge-hao]

@byshiue

Hi, looking forward to you taking a look, again. 🤗🤗🤗

[zhang-ge-hao]

@byshiue

If there is anything else I need to change, I will update it as soon as possible.

[byshiue]

@byshiue

If there is anything else I need to change, I will update it as soon as possible.

Thank you for the work. We are waiting the internal unit tests and it looks well now.

[hmzo]

@AkiyamaYummy

FasterTransformer/examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py

Line 40 in c6e8f60

val = (val / factor) if factor > 1 else val

hello, thank you for the great work! I have a question about this convert script: why do we need scale bias here when tp > 1?

[zhang-ge-hao]

@AkiyamaYummy

FasterTransformer/examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py

Line 40 in c6e8f60

val = (val / factor) if factor > 1 else val

hello, thank you for the great work! I have a question about this convert script: why do we need scale bias here when tp > 1?

FT will respectfully add the multi bias in multi GPUs when tp > 1, and all-reduce them afterward.

If not scale the biases, it's equal to they will be added multi times.