Skip to content

init commit#11

Merged
akoumpa merged 70 commits intomainfrom
akoumparouli/distributed
May 28, 2025
Merged

init commit#11
akoumpa merged 70 commits intomainfrom
akoumparouli/distributed

Conversation

@akoumpa
Copy link
Copy Markdown
Contributor

@akoumpa akoumpa commented May 27, 2025 •
edited
Loading

Adds initial support for LLMs, in particular:

  • hellaswag dataset
  • FSDP2/DDP
  • Finetune LLM recipe

@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 27, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

hemildesai and others added 13 commits May 27, 2025 11:36
@hemildesai @akoumpa
Initial commit
74ad624 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
@hemildesai @akoumpa
Add Llama 3.2 1b hellaswag finetuning example
4b0b423 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
@akoumpa
[automodel] misc fixes (#1)
a9097dd 
* move examples to recipes

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* move everything under automodel

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* baby steps

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* import from NeMo 1f511fd & bfbd333

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* simplify

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* add get method with fallback

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* add ranked param

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* update resolve target

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* renmame

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* add rng.py

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* cleanup

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* move files

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* add __contains__

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* minor fix

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* special handle for _fn keys

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* move utils to file

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* move hellaswag to SFTSingleTurnPreprocessor

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* fix for _fn

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* add base recipe

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* fix

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
cleaup (#3)
394b0dd 
* cleaup

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* cleaup

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* move DistInfo

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* move num_epochs to StepScheduler

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* change recipe name to FinetuneRecipeForNextTokenPrediction

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* change recipe name to FinetuneRecipeForNextTokenPrediction

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* change recipe name to FinetuneRecipeForNextTokenPrediction

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
restructure (#2)
b114107 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
step
f3ffb88 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
step
95390fd 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move files
35765be 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move file
5a68b34 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move file
571a3ed 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
f8be415 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
step
7bd5be1 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
state
72afda1 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa force-pushed the akoumparouli/distributed branch from 917a662 to 72afda1 Compare May 27, 2025 18:37
akoumpa added 15 commits May 27, 2025 11:40
@akoumpa
update docsring
692f9af 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
d13fa24 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
186bc3c 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
7c99178 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
change args
2cbb420 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
90bf49c 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
d35d252 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
remove unused imports
b171a02 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
ddp fix
36ad069 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
ddp fix
53a1fb6 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
lint fix
94d583c 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
0bdbd02 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
remove checkpoint.py
4b41124 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix import
6b7e249 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
c5fd160 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
akoumpa added 5 commits May 27, 2025 12:39
@akoumpa
remove unused
ec7cadb 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
remove unused wandb
2b937f4 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move datasets to llm
c2b73b4 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
cfe5c64 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
remove config_utils
72b8df4 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa changed the title Akoumparouli/distributed init commit May 27, 2025
akoumpa added 21 commits May 28, 2025 00:36
@akoumpa
pass num_grad_acc_steps
390f408 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
0ab756f 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix defaults
21fbc73 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
add assert
4451801 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move StepScheduler to base_recipe.py
8d2a12c 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move StepScheduler to base_recipe.py
a9605bf 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
cleanup
e1ce0f9 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
move base_recipe into training
293989f 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fx
fa3bde6 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
daeecbd 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix import path
b361c0c 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix default args
e1c991a 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
add assert
ab76086 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
inherit from Stateful
f9fd6b7 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix
7b874c4 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
fix imports
c332e4d 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
91e2226 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
add epochs propery
3b53630 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
minor fix
e410a80 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
minor update
3ae550b 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
docstring
5d20660 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa requested a review from BoxiangW May 28, 2025 16:08
@akoumpa akoumpa merged commit e655b1f into main May 28, 2025
3 checks passed
@ko3n1g ko3n1g deleted the akoumparouli/distributed branch June 16, 2025 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BoxiangW BoxiangW Awaiting requested review from BoxiangW

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@akoumpa @hemildesai