Using Huggingface library with DeepSpeed

[exelents]

I'm not completely sure if it's this library problem, but maybe you could help.
Trying to run T5-large from huggingface's library with DeepSpeed library I got a strange result. When I change mode to fp16 training loss is going to be NaN value, as well as some of tensors in model's features output. I'm not sure, can it be a Transformers library fault? Original example that I used utilizes pytorch_pretrained_bert, and it works well.
Training with FP32 does not result any NaN troubles.

I have some code, made out of DeepSpeedExamples code:
https://github.com/exelents/try_t5
If somebody would like to help and try to run it, here is compiled binary dataset:
https://drive.google.com/file/d/1oxCxYCuCWebmaUQ_s9il7EDBkisL7x-_/view?usp=sharing
https://drive.google.com/file/d/1WCzxAnp2bEllbQ0_2d_6hoq5tQjxBFXh/view?usp=sharing

[flozi00]

There is an open PR by @patil-suraj for T5 FP16 #9487

And here is an open PR for deepspeed Integration by @stas00 #9211

[exelents]

Thank you!

[stas00]

Moving my answers from #9487 as they were irrelevant to the PR itself:

Context: @exelents struggles with making rtx-3090 work with pytorch and getting:

nvcc fatal : Unsupported gpu architecture 'compute_86

I explained how I made it to work.

---

So I just did:

pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html -U

Then inside deepspeed github clone:

rm -rf build
TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 pip install --no-cache -v --disable-pip-version-check -e .

8.6 corresponds to rtx-3090 arch.

You can remove 6.1, this is just my 2nd 1070 card's arch.

And you can remove -e if you don't want the develop install.

You can install it normally from pypi too:

pip install deepspeed

it'll use PTX/JIT - I tested it to work just fine - the explicit way from source repo just builds the most optimal specific version for my hardware and pre-compiles all features, which takes much longer to build.

Now inside the deepspeed PR brach #9211, I run against a small t5 model:

export BS=20; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0 deepspeed --num_gpus=2 ./finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 100 --n_val 100 --n_test 100 --deepspeed ds_config.json --fp16 --save_steps 1

All works. t5-base works too.

Follow my steps and see if you can use your rtx-3090 card first. Then compare to what you are doing differently.

[stas00]

I have been using pt-nightly w/ rtx-3090 for the last 2 months, so yes it works. pt-1.7.1 doesn't work.

For building modules that build pytorch extensions like deepspeed, and apex and fairscale I use cuda-11.1. Let me know what you're trying to build and I will tell you how to do it.

I'm still on 11.1 for building extensions, while I know 11.2 is out since I'm not sure 11.1 is compatible with 11.2. 11.0 is compatible with 11.1 so one can use it to build against pt-nightly w/ 11.0 after hacking the build script.

[stas00]

I change current cuda in my system to 11.0 version (cuda-toolkit-11-0 package in Ubuntu)
Then I install latest pytorch nighty by a command which you propose.

I think the difference is that you need cuda-11.1 and not cuda-11.0 system-wide. This is where our setups diverge I think.

Careful though, there is 11.2 out there. I'm on ubuntu, I'm not at all sure it'd work w/ pt-nightly, that's why I'm not upgrading mine.

for rtx-3090 to work

• tf requires cuda-11.1
• pt works with cuda-11.0
• pt extensions need cuda-11.1: apex, fairscale, deepspeed, The first 2 require hacking their build script to support 11.1 w/ pt built w/ 11.0. deepspeed works out of box.

note: If someone reads this at a later time this will probably become incorrect once pt-nightly builds w/ cuda-11.2 - then you should be able to install 11.2 system-wide and hopefully the extensions will just work.

[exelents]

Moving my posts from PR #9487 due to they are irrelevant.

I don't know what is my problem. I even tried solution made by @stas00 in #9211 but I still have the same problem.
Maybe problem is I built Pytorch from source and forgot some option? I did it because pypi's version don't support Cuda 11.2 and supported cuda 11.0 don't support my gpu (rtx 3090)?
Maybe I need install something to enable fp16 support?

[exelents]

I use pytorch-nightly w/ cuda-11.0 which works with rtx-3090:

pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html -U

pt-nightly w/ cuda-11.2 should be released really soon now. You can track it here pytorch/pytorch#50232

It doesn't work. Pytorch nighty gets an error:
nvcc fatal : Unsupported gpu architecture 'compute_86
That's mean that Cuda 11.0 doesn't support RTX 3090

[exelents]

I have been using it for the last 2 months, so yes it works. pt-1.7.1 doesn't work.

Chances are that you have more than one pytorch installed and you have the non-nightly version loaded, check your:

print(torch.__version__)

here is installed version: 1.8.0.dev20210109+cu110

For building extensions like deepspeed, and apex and fairscale I use cuda-11.1
Okay, I'll try cuda 11.1, maybe it'll help.

[exelents]

Excellent. so what exactly do you do when you get that error?

I change current cuda in my system to 11.0 version (cuda-toolkit-11-0 package in Ubuntu)
Then I install latest pytorch nighty by a command which you propose.
Latest, I run deepspeed training script proposed in #9211 issue. But in parameters there I change model name to t5-large and remove language parameters from parameters fed to model

This code in utils.py:

        if data_args.src_lang is not None:
            self.dataset_kwargs["src_lang"] = data_args.src_lang
        if data_args.tgt_lang is not None:
            self.dataset_kwargs["tgt_lang"] = data_args.tgt_lang

This way I get a minimal training code that should run T5-large. It don't take my dataset like in two examples I have shown before, but it should work. What I see is only error "platform not supported" on pt nighty build, or NaNs in output tensors in version which I installed from source.

Here is training script:
https://gist.github.com/exelents/9dd3e6dec64dc0d640b85a7e0cfa53e9

[stas00]

Thank you for making this extra effort, @exelents! We got out of the PR's way now.

Please let me know if you had success with: #9490 (comment) taking into account this: #9490 (comment)

[exelents]

Okay, @stas00
I have installed cuda-toolkit-11-1, installed Pytorch nighty version:
pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html -U
Then I compiled from source DeepSpeed with Cuda 11.1 and Nvidia 8.6 computing platform:
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_OPS=1 pip install --no-cache -v --disable-pip-version-check -e .
Now everything works well, on the loss scale 256.0

upd: Also, I have installed Huggingface Transformers library from source, on brach from PR #9211 merged with branch from PR #9487

It seems that maybe DeepSpeed doesn't support new Cuda 11.2, or because I compiled PyTorch and deepspeed on Cuda 11.2 with TORCH_CUDA_ARCH_LIST=8.0 instead of 8.6.

[stas00]

Glad to hear it works now!

TORCH_CUDA_ARCH_LIST=8.0

Most likely this!

When you build from source you need to add +PTX 8.0+PTX for this newer arch to work if you don't specify 8.6 explicitly. It basically tells the cuda compiler to allow newer archs to be supported as well and will compile the extension during first run-time via JIT and cache and re-use it.

This is how pytorch nightly is built (i.e. it includes +PTX).

The whole PTX has been recently documented in https://pytorch.org/docs/master/cpp_extension.html#torch.utils.cpp_extension.CUDAExtension