w/o model-parallel usability numbers reproduce

[g-karthik]

I've been using DeepSpeed successfully with my large model train jobs. But this blog post says ZeRO-1 and ZeRO-2 power up to 6B and 13B param training respectively w/o model-parallelism.

Where exactly is the code that validates/enables others to reproduce this claim? I want to see what model was used, what was the batch size, what was the max sequence length, etc. for this claim.

[samyam]

@g-karthik, sorry for the late response. In case this is still relevant to you, these are for GPT-2 style language model, even though you should be able to run any model of similar sizes using ZeRO-1 and ZeRO-2. The exact model configuration, batch size, GPU count etc are as follows:

The sequence length is 1K. You can find this table in the appendix of our archive paper (https://arxiv.org/pdf/1910.02054.pdf). You can find more details in the paper itself.

To reproduce the results, you can use our Megatron-LM tutorial (https://www.deepspeed.ai/tutorials/megatron/). Once you have Megatron-LM running with DeepSpeed, you can set the model parallelism degree to 1 to disable model parallelism, use model configurations and GPU count shown in the table above and enable ZeRO-2 in deepspeed config file.

Please note that to run a 13B parameter model with ZeRO-2, you would need around 128 GPUs with 32 GB memory each to share the optimizer and gradient states. But you should be able to fit a 5-8B parameter model with as little as 16 GPUs.

[g-karthik]

Hi @samyam thanks for the detailed info!

I did try to use the tutorial you linked earlier, and it seemed to work in some cases and failed in others.

Back then, I faced this error in those other cases (pulled from my logs): FileExistsError: [Errno 17] File exists: '/path/to/DeepSpeedExamples/Megatron-LM/data/webtext/openwebtext.lazy'

Note that I did not create a symbolic link to the data as mentioned in the tutorial you linked - I put the actual data (openwebtext.json) there.

Do you know why this is happening?

UPDATE:

I tried running it again now, I got the same FileExistsError (multiple rows showing that in the logs) but after the error, it shows:

FileExistsError: [Errno 17] File exists: '/path/to/DeepSpeedExamples/Megatron-LM/data/webtext/openwebtext.lazy'

> padded vocab (size: 50257) with 175 dummy tokens (new size: 50432)
> found end-of-document token: 50256
building GPT2 model ...
 > number of parameters on model parallel rank 1: 329328640
 > number of parameters on model parallel rank 0: 329328640

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "pretrain_gpt2.py", line 707, in <module>
  File "pretrain_gpt2.py", line 707, in <module>
  File "pretrain_gpt2.py", line 707, in <module>
    main()
    main()
    main()
  File "pretrain_gpt2.py", line 652, in main
  File "pretrain_gpt2.py", line 652, in main
  File "pretrain_gpt2.py", line 652, in main
    args.eod_token = get_train_val_test_data(args)
    args.eod_token = get_train_val_test_data(args)
    args.eod_token = get_train_val_test_data(args)
  File "pretrain_gpt2.py", line 619, in get_train_val_test_data
  File "pretrain_gpt2.py", line 619, in get_train_val_test_data
  File "pretrain_gpt2.py", line 619, in get_train_val_test_data
    group=mpu.get_model_parallel_group())
    group=mpu.get_model_parallel_group())
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
    group=mpu.get_model_parallel_group())
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 810, in broadcast
    work = group.broadcast([tensor], opts)
    work = group.broadcast([tensor], opts)
    work = group.broadcast([tensor], opts)
RuntimeError: Socket Timeout
RuntimeError: Socket Timeout
RuntimeError: Socket Timeout

[tjruwase]

Hi @g-karthik.

I am surprised by the FileExistsError. The expected path for webtext data is data/webtext/data.json. Did you modify the code to match your data path of data/webtext/openwebtext.lazy?

Also, can you share the model-parallel (MP) and data-parallel (DP) degrees of your run? Are you able to run with MP=DP=1, which should fit your .3B parameter model?

[g-karthik]

Hi @tjruwase, the .lazy file wasn't created by me, it was created automatically when I ran the code.

And yeah I'd modified the expected path to the webtext data in that line you pointed to, to contain the path to my openwebtext.json file. And I also changed the text_key from "text" to "content" in that same class since my JSON has the text in that key. Other than that, I didn't make any code changes as such.

Yeah the MP degree is 2 and DP degree is 8 when I get this FileExistsError followed by Socket Timeout. See command below:

GPUS_PER_NODE=8
# Change for multinode config
MASTER_ADDR=localhost
MASTER_PORT=6000
NNODES=1
NODE_RANK=0
WORLD_SIZE=$(($GPUS_PER_NODE*$NNODES))

DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT"

python -m torch.distributed.launch $DISTRIBUTED_ARGS \
       pretrain_gpt2.py \
       --model-parallel-size 2 \
       --num-layers 48 \
       --hidden-size 1024 \
       --num-attention-heads 32 \
       --batch-size 8 \
       --seq-length 1024 \
       --max-position-embeddings 1024 \
       --train-iters 320000 \
       --save checkpoints/gpt2_xl_mp2 \
       --load checkpoints/gpt2_xl_mp2 \
       --resume-dataloader \
       --train-data webtext \
       --lazy-loader \
       --tokenizer-type GPT2BPETokenizer \
       --split 949,50,1 \
       --distributed-backend nccl \
       --lr 0.00015 \
       --no-load-optim \
       --lr-decay-style cosine \
       --weight-decay 1e-2 \
       --clip-grad 1.0 \
       --warmup .01 \
       --checkpoint-activations \
       --fp16 \
       --deepspeed \
       --deepspeed_config scripts/ds_zero2_config.json

I haven't tried the MP=DP=1 case with this script yet. I just kicked off the above once again and got the FileExistsError again, but the processes continue to run. I'm waiting to see if I get the Socket Timeout again (I suspect I will).

[tjruwase]

@g-karthik thanks for sharing those details. Another question, why are you using python to launch rather than the deepspeed launcher?

[g-karthik]

@tjruwase oh that's just the contents of the pretrain_gpt2_model_parallel.sh shell script, with my minor modifications to the parameters to be used.

I do use the deepspeed launcher with my integration of DeepSpeed into my own codebase outside of the DeepSpeedExamples repo. Although as a side-note, I prefer to use python with --deepspeed and --deepspeed_mpi to scale up my jobs effectively with mpirun on Kubernetes.

[g-karthik]

@tjruwase just an update on the MP=1, DP=1 case you asked me to run, I set model-parallel-size=1, GPUS_PER_NODE=1. It seems to be running fine, it hasn't thrown a FileExistsError or a Socket Timeout error.

validation loss at iteration 1000 | LM loss: 6.297972E+00 | LM PPL: 5.434685E+02

[tjruwase]

@g-karthik thanks for sharing that update. So it seems we can assume the problem is related to distributed training. Can you enable NCCL debugging information for the distributed case? I believe you can set the NCCL_DEBUG environment variable like:
NCCL_DEBUG=INFO

Also, I will suggest using deepspeed launcher because it handles all the NCCL configuration issues for distributed training. It would be great to find out if it can fix the Socket Timeout error.

Finally, for the distributed case I am curious if all or just some of the processes report the FileExists error. I see that this filename is created here. Although, I have not looked closely at this code before, but I suspect this file is meant to be created only once per node.

[g-karthik]

@tjruwase @samyam

Hope you're doing well! I'm having some trouble reproducing the 1K sequence length setting with Hugging Face's GPT-2 class.

Specifically, I'm using 8 V100 GPUs (32 GB each) and the following deepspeed configuration:

{
  "train_batch_size": 128,
  "gradient_accumulation_steps": 8,
  "gradient_clipping": 1.0,
  "optimizer": {
    "type": "adam",
    "params": {
      "lr": 6.25e-5
    }
  },
  "fp16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 2,
    "cpu_offload": true,
    "contiguous_gradients": true,
    "overlap_comm": false,
    "allgather_bucket_size": 500000000
  },

  "activation_checkpointing": {
    "partition_activations": true,
    "contiguous_memory_optimization": true,
    "cpu_checkpointing": true
  }

}

I'm kicking these jobs off with python using --deepspeed_mpi and mpi4py instead of the regular deepspeed launcher. I'm consistently getting CUDA OOM for a seq length of 1K with even 1.5B parameters. I can certainly scale up my cluster and increase the effective train batch size to eliminate these errors (due to the inverse dependency on DP degree for memory consumption with ZeRO), but it seems like it shouldn't be needed for 1.5B. I also understand there are implementation differences between the Megatron GPT-2 class and the Hugging Face GPT-2 class, but I've trained these models with slightly smaller sequence lengths than 1K before. Any ideas on what I am missing?

[g-karthik]

@tjruwase @jeffra @samyam @ShadenSmith also note that with the above config, I'm able to train with a sequence length of 512 just fine.

I vaguely remember seeing somewhere in the DeepSpeed docs (or the paper) that you use 16 attention heads for 1.5B, whereas Hugging Face's version uses 25 attention heads. Assuming my memory is indeed correct, I wonder if your 16 attention head version was indeed a 1.5B version? Or perhaps you compensated for it by increasing another dimension like hidden size?

[tjruwase]

@g-karthik, I think you should be able to train 1.5B model easily on a single 32GB GPU with cpu-offload. Can you please share a link to the the GPT-2 script? I am curious as to whether you are passing an optimizer into deepspeed.initialize()? I am also interested in repro'ing the OOM.

[g-karthik]

@tjruwase Yeah I'm using this implementation of the GPT2LMHeadModel by Hugging Face: https://github.com/huggingface/transformers/blob/v3.1.0/src/transformers/modeling_gpt2.py#L652

No, I'm not passing an optimizer into deepspeed.initialize().

Although technically my deepspeed_config.json above has "activation_checkpointing" set up, it would go unused if using the above GPT2LMHeadModel as-is. I've now configured deepspeed.checkpointing and added explicit deepspeed.checkpointing.checkpoint() calls within the underlying base Transformer decoder, I haven't tested it out yet. However, would you say that a 1K sequence length should fit easily for a 1.5B configuration even without deepspeed's activation checkpointing set up?

[g-karthik]

@tjruwase @jeffra @samyam

I am unable to reproduce the claim in Section 10.4 of the ZeRO paper: "Fig. 4 shows that ZeRO-100B can train models with up to 13B parameters without MP on 128 GPUs, achieving throughput over 40 TFlops per GPU on average."

I am using Hugging Face's GPT2LMHeadModel with an 8 billion parameter configuration and when I run the new Flops Profiler provided with DeepSpeed, I hit about 2.7 TFLOPS (which I presume is per GPU) as printed by the profiler. I just use 8 GPUs total.

I am also unable to fit 1024 sequence length in my micro-batches, due to presumable deficiencies in deepspeed's activation checkpointing, see relevant discussions #598 (comment) and #598 (comment). It would be great if these could be addressed in a subsequent PR.

Could you please help identify how I could hit 40 TFLOPS per GPU with the Hugging Face implementation, just as you seem to have hit that with the Megatron implementation with MP=1?