feat: pretrain dfm automodel by akoumpa · Pull Request #36 · NVIDIA-NeMo/DFM

akoumpa · 2025-11-13T00:04:35Z

git clone -b akoumparouli/feat_pretrain_dfm_automodel git@github.com:NVIDIA-NeMo/DFM.git
git submodule update --init --recursive 3rdparty/

Single GPU

uv run --group automodel --group torch-cu124 --with . python3 examples/automodel/pretrain/pretrain.py -c examples/automodel/pretrain/wan2_1_t2v_flow.yaml

Multi GPU

uv run --group automodel --group torch-cu124 --with . torchrun --nproc-per-node=2 examples/automodel/pretrain/pretrain.py -c examples/automodel/pretrain/wan2_1_t2v_flow.yaml

copy-pr-bot · 2025-11-13T00:04:38Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

linnanwang

These looks good to me, thank you Alex

akoumpa · 2025-11-13T07:19:02Z

/ok to test 3c64e65

copy-pr-bot · 2025-11-13T07:19:05Z

/ok to test 3c64e65

@akoumpa, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

akoumpa · 2025-11-13T07:19:45Z

/ok to test fa1b851

linnanwang

LGTM

akoumpa · 2025-11-13T18:13:30Z

/ok to test a867798

akoumpa · 2025-11-13T21:54:00Z

/ok to test c3ea450

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa · 2025-11-17T18:28:00Z

/ok to test a0c5367

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa · 2025-11-17T18:50:15Z

/ok to test 7b108d1

linnanwang

tested and everthing looks good.

linnanwang · 2025-11-17T22:10:39Z

/ok to test 2d48619

copy-pr-bot · 2025-11-17T22:10:42Z

/ok to test 2d48619

@linnanwang, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

linnanwang · 2025-11-17T22:13:10Z

/ok to test 7b108d1

* init Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/amx Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/max Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename fientune.py to train.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add from_config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass scheduler and model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * introduce NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add mode Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update build_model_and_optimizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move examples Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * more lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix 3rdparty & pyproject Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add torch Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * revert 3rdparty Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* init Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/amx Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/max Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename fientune.py to train.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add from_config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass scheduler and model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * introduce NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add mode Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update build_model_and_optimizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move examples Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * more lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix 3rdparty & pyproject Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add torch Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * revert 3rdparty Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>

* init Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/amx Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add sigma_min/max Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename fientune.py to train.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add from_config Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * pass scheduler and model Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * introduce NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add mode Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update build_model_and_optimizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update NeMoWanPipeline Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move examples Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * more lint Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix 3rdparty & pyproject Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add torch Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * revert 3rdparty Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update uv.lock Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa changed the title ~~Akoumparouli/feat pretrain dfm automodel~~ feat: pretrain dfm automodel Nov 13, 2025

akoumpa marked this pull request as draft November 13, 2025 00:04

linnanwang self-requested a review November 13, 2025 00:12

linnanwang self-assigned this Nov 13, 2025

linnanwang previously approved these changes Nov 13, 2025

View reviewed changes

akoumpa dismissed linnanwang’s stale review via b3702da November 13, 2025 03:56

akoumpa force-pushed the akoumparouli/feat_pretrain_dfm_automodel branch from efea72b to 0ba1055 Compare November 13, 2025 07:04

copy-pr-bot bot temporarily deployed to test November 13, 2025 07:19 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 13, 2025 07:20 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 13, 2025 07:54 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 13, 2025 08:13 Inactive

linnanwang previously approved these changes Nov 13, 2025

View reviewed changes

akoumpa marked this pull request as ready for review November 13, 2025 18:12

copy-pr-bot bot temporarily deployed to test November 13, 2025 18:13 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 13, 2025 18:14 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 13, 2025 18:16 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 13, 2025 18:36 Inactive

akoumpa dismissed linnanwang’s stale review via cf8ae22 November 13, 2025 20:41

akoumpa force-pushed the akoumparouli/feat_pretrain_dfm_automodel branch 2 times, most recently from cf8ae22 to aa8779a Compare November 13, 2025 20:42

akoumpa closed this Nov 13, 2025

akoumpa reopened this Nov 13, 2025

akoumpa added 6 commits November 17, 2025 10:25

update uv.lock

c286199

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

fix

6c6e35d

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

update

3f8b64f

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

fix

494c3fb

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

revert 3rdparty

e3290e0

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

update uv.lock

90f9bbc

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa force-pushed the akoumparouli/feat_pretrain_dfm_automodel branch from 0668fe9 to 90f9bbc Compare November 17, 2025 18:25

fix

a0c5367

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

copy-pr-bot bot temporarily deployed to test November 17, 2025 18:28 Inactive

copy-pr-bot bot had a problem deploying to nemo-ci November 17, 2025 18:28 Failure

update uv.lock

7b108d1

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

copy-pr-bot bot temporarily deployed to test November 17, 2025 18:50 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 17, 2025 18:50 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 17, 2025 19:28 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 17, 2025 19:44 Inactive

akoumpa enabled auto-merge (squash) November 17, 2025 21:48

akoumpa requested a review from linnanwang November 17, 2025 21:48

linnanwang approved these changes Nov 17, 2025

View reviewed changes

pablo-garay approved these changes Nov 17, 2025

View reviewed changes

akoumpa merged commit 19753e8 into main Nov 17, 2025
16 checks passed

Conversation

akoumpa commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Nov 13, 2025

Uh oh!

linnanwang left a comment

Choose a reason for hiding this comment

Uh oh!

akoumpa commented Nov 13, 2025

Uh oh!

copy-pr-bot bot commented Nov 13, 2025

Uh oh!

akoumpa commented Nov 13, 2025

Uh oh!

linnanwang left a comment

Choose a reason for hiding this comment

Uh oh!

akoumpa commented Nov 13, 2025

Uh oh!

akoumpa commented Nov 13, 2025

Uh oh!

akoumpa commented Nov 17, 2025

Uh oh!

akoumpa commented Nov 17, 2025

Uh oh!

linnanwang left a comment

Choose a reason for hiding this comment

Uh oh!

linnanwang commented Nov 17, 2025

Uh oh!

copy-pr-bot bot commented Nov 17, 2025

Uh oh!

linnanwang commented Nov 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

akoumpa commented Nov 13, 2025 •

edited

Loading