Skip to content

fix failure on restart after round 1 train and no eval #3

Closed
stas00 wants to merge 19 commits intodeepspeedai:mainfrom
bigscience-workshop:train-no-eval-restart
Closed

fix failure on restart after round 1 train and no eval #3
stas00 wants to merge 19 commits intodeepspeedai:mainfrom
bigscience-workshop:train-no-eval-restart

Conversation

@stas00
Copy link
Copy Markdown

@stas00 stas00 commented Jul 23, 2021

Was getting:

 File "/mnt/nvme1/code/huggingface/Megatron-DeepSpeed-master/megatron/training.py", line 132, in pretrain
    = build_train_valid_test_data_iterators(
  File "/mnt/nvme1/code/huggingface/Megatron-DeepSpeed-master/megatron/training.py", line 842, in build_train_valid_test_data_iterators
    = build_train_valid_test_data_iterators(
  File "/mnt/nvme1/code/huggingface/Megatron-DeepSpeed-master/megatron/training.py", line 842, in build_train_valid_test_data_iterators
    assert args.train_samples is None, \
AssertionError: only backward compatiblity support for iteration-based training

on restart after a very short first run.

The fix is partial, more work needs to be done for the general solution, but this is enough for our needs.

stas00 and others added 19 commits July 2, 2021 10:46
@stas00
explain what this repo is about.
fb1a120
@stas00
add the explanation about the forking problems.
5502865
@sbmaruf @TevenLeScao
Hf tok pipeline (#2)
b1a30de 
* hf tokenizer pipeline

* unused arg removed

* Changing Hf to HF

* Changing Hf to HF

Co-authored-by: Teven <teven.lescao@gmail.com>
@TevenLeScao
Create README for preprocessing
ebb6a07
@TevenLeScao
Update README.md
a0ab8f4
@TevenLeScao
Update README.md
ca4ea15
@TevenLeScao
HF tokenizers (on the training side) (#5)
0b2f0df
@TevenLeScao
Tiny gpt training script for local testing
d61ca5c
@stas00
update to deepspeed usage; rm irrelevant info
c5e0b24
@stas00
install instructions; consistent indent
7d880b7
@stas00
consistency
8cf27f2
@stas00
update docs
12503bd
@stas00
add example script for one node
2048ca5
@stas00
adjust the script
5ab4fad
@TevenLeScao
Fixed single node test scripts to use DS
1556d7e
@TevenLeScao
Fixed single node test scripts to use DS
a7a534a
@stas00
train_samples check fix
d13f5c6
@stas00
Revert "train_samples check fix"
0e78387 
This reverts commit d13f5c6.
@stas00
fix bug when restarting with no eval in round 1
66b4a6c
@stas00 stas00 closed this Jul 23, 2021
jeffra pushed a commit that referenced this pull request Dec 10, 2021
@awan-10
Merge pull request #3 from awan-10/amawa/inference-moe
3599afd 
Amawa/inference moe
hyoo pushed a commit to hyoo/Megatron-DeepSpeed that referenced this pull request Sep 21, 2022
@ngoyal2707
dont initialize pipeline process groups when not using pipeline paral…
c7ed0b2 
…lelism (deepspeedai#3)

Co-authored-by: Your Name <you@example.com>
@wangguojim wangguojim mentioned this pull request Aug 11, 2023
Open
yueming-yuan added a commit to Supercomputing-System-AI-Lab/Megatron-DeepSpeed that referenced this pull request Sep 14, 2025
@yueming-yuan
Merge pull request deepspeedai#3 from yueming-yuan/cleanup
745777d 
clean up
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@stas00 @sbmaruf @TevenLeScao