Adithyare/sft ultra v3 feb2026 by arendu · Pull Request #3272 · NVIDIA/Megatron-LM

arendu · 2026-02-05T21:15:47Z

What does this PR do ?

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

flowchart LR
    A[Pre-checks] --> B[PR Tests]
    subgraph Code Review/Approval
        C1[Expert Review] --> C2[Final Review]
    end
    B --> C1
    C2 --> D[Merge]

Pre-checks

I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

(Step 1): Add PR label `Expert Review`

(Step 2): Collect the expert reviewers reviews

Attach the Expert Review label when your PR is ready for review.
GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

Add Final Review label
GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

copy-pr-bot · 2026-02-05T21:15:51Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Phlip79 · 2026-02-17T23:52:29Z

@arendu please fix conflicts

megatron/core/tokenizers/utils/build_tokenizer.py

Signed-off-by: adithyare <adithyare@nvidia.com>

Signed-off-by: Soumye Singhal <soumyes@cw-dfw-cs-001-dc-01.cm.cluster>

Cherry-picked from b3684ec4 "added fix to fix crash"

Signed-off-by: adithyare <adithyare@nvidia.com>

…t-ultra-v3-feb2026

megatron/core/models/common/model_chunk_schedule_plan.py

jaredcasper · 2026-02-18T23:16:13Z

Is it possible to split this into distinct PRs? Or at the least describe each change. It's hard to review without knowing the whats and whys.

megatron/core/transformer/multi_token_prediction.py

Signed-off-by: adithyare <adithyare@nvidia.com>

megatron/training/datasets/sft_dataset.py

megatron/core/tokenizers/text/libraries/sft_tokenizer.py

arendu · 2026-02-20T21:20:09Z

@jaredcasper @Phlip79 I've merged with a more recent main. The changes should only be related to tokenizer building and a logic update regarding EOD tokens.

jaredcasper · 2026-02-20T23:03:32Z

megatron/core/tokenizers/utils/build_tokenizer.py

        _set_padded_vocab_size(args, tokenizer)

        return tokenizer
+    elif args.tokenizer_type == 'SFTTokenizer':


How will this ever run with the case on line 66 present?

jaredcasper · 2026-02-20T23:11:52Z

scripts/bindpcie.sh

@@ -0,0 +1,212 @@
+#! /bin/bash
+# Copyright (c) 2018-2022, NVIDIA CORPORATION.  All rights reserved.
+#


Megatron is not Apache licensed and we don't put the license in each file.

jaredcasper · 2026-02-20T23:12:24Z

scripts/ultra_sft_64node_bindpcie.sh

Why are we putting these scripts in the repo? Are the generally applicable to any platform?

…rl binding

arendu · 2026-02-21T00:58:10Z

closing in favor of #3526

arendu marked this pull request as ready for review February 6, 2026 18:58

arendu requested review from a team as code owners February 6, 2026 18:58

ko3n1g requested a review from a team February 6, 2026 18:58

arendu requested a review from a team as a code owner February 13, 2026 19:04

jaredcasper added the Expert Review Apply this label to indicate that your PR is ready for expert review. label Feb 17, 2026

Phlip79 added the complexity: low label Feb 17, 2026

ananthsub reviewed Feb 18, 2026

View reviewed changes

megatron/core/tokenizers/utils/build_tokenizer.py Show resolved Hide resolved

arendu and others added 8 commits February 18, 2026 10:26

rebased main

fa016fd

Signed-off-by: adithyare <adithyare@nvidia.com>

identity template and tool role mask

e039525

revert assert

ae4717d

Signed-off-by: Soumye Singhal <soumyes@cw-dfw-cs-001-dc-01.cm.cluster>

Skip empty sequences and chunks in MTP tensor roll

44c5c24

Cherry-picked from b3684ec4 "added fix to fix crash"

add option for SFTTokenizer to build_tokenizers

8ed4905

Signed-off-by: adithyare <adithyare@nvidia.com>

resolve conflicts between args.sft and args.hybrid_context_parallel

30f1f32

Signed-off-by: adithyare <adithyare@nvidia.com>

do not check for and add any EOD

72a2740

Signed-off-by: adithyare <adithyare@nvidia.com>

removed dup identity template

f82168a

Signed-off-by: adithyare <adithyare@nvidia.com>

arendu force-pushed the adithyare/sft-ultra-v3-feb2026 branch from 9ddfa5d to f82168a Compare February 18, 2026 18:46

arendu requested a review from a team as a code owner February 18, 2026 18:46

arendu added 4 commits February 18, 2026 10:48

added from NVIDIA#2363

68514e8

Signed-off-by: adithyare <adithyare@nvidia.com>

use core sft_tokenizer

d10fa92

Signed-off-by: adithyare <adithyare@nvidia.com>

fmt

edd864c

Signed-off-by: adithyare <adithyare@nvidia.com>

Merge remote-tracking branch 'github-upstream/main' into adithyare/sf…

7bd7bfd

…t-ultra-v3-feb2026

jaredcasper reviewed Feb 18, 2026

View reviewed changes

megatron/core/models/common/model_chunk_schedule_plan.py Outdated Show resolved Hide resolved

jaredcasper reviewed Feb 18, 2026

View reviewed changes

megatron/core/transformer/multi_token_prediction.py Outdated Show resolved Hide resolved

arendu added 2 commits February 19, 2026 16:37

commit from main

f71ad30

Signed-off-by: adithyare <adithyare@nvidia.com>

checkout main

f3c6356

Signed-off-by: adithyare <adithyare@nvidia.com>

jaredcasper reviewed Feb 20, 2026

View reviewed changes

megatron/training/datasets/sft_dataset.py Show resolved Hide resolved

jaredcasper reviewed Feb 20, 2026

View reviewed changes

megatron/core/tokenizers/text/libraries/sft_tokenizer.py Outdated Show resolved Hide resolved

arendu requested review from ananthsub and jaredcasper February 20, 2026 21:20

jaredcasper reviewed Feb 20, 2026

View reviewed changes

guyueh1 self-requested a review February 20, 2026 23:17

seonjinn force-pushed the adithyare/sft-ultra-v3-feb2026 branch from 9a2630e to 2fa3e6f Compare February 20, 2026 23:21

feat: update scripts for perf. optimizations such as HybridEP, Numact…

ee5e581

…rl binding

seonjinn force-pushed the adithyare/sft-ultra-v3-feb2026 branch from 2fa3e6f to ee5e581 Compare February 20, 2026 23:35

arendu closed this Feb 21, 2026

jaredcasper mentioned this pull request Feb 21, 2026

do not add EoD #3526

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Adithyare/sft ultra v3 feb2026#3272

Adithyare/sft ultra v3 feb2026#3272
arendu wants to merge 15 commits intoNVIDIA:mainfrom
arendu:adithyare/sft-ultra-v3-feb2026

arendu commented Feb 5, 2026

Uh oh!

copy-pr-bot bot commented Feb 5, 2026

Uh oh!

Phlip79 commented Feb 17, 2026

Uh oh!

Uh oh!

Uh oh!

jaredcasper commented Feb 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arendu commented Feb 20, 2026

Uh oh!

jaredcasper Feb 20, 2026

Uh oh!

jaredcasper Feb 20, 2026

Uh oh!

jaredcasper Feb 20, 2026

Uh oh!

arendu commented Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Comments

Conversation

arendu commented Feb 5, 2026

What does this PR do ?

Contribution process

Pre-checks

Code review

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

(Step 3): Final Review

(Optional Step 4): Cherry-pick into release branch

Merging your PR

Uh oh!

copy-pr-bot bot commented Feb 5, 2026

Uh oh!

Phlip79 commented Feb 17, 2026

Uh oh!

Uh oh!

Uh oh!

jaredcasper commented Feb 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arendu commented Feb 20, 2026

Uh oh!

jaredcasper Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

jaredcasper Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

jaredcasper Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

arendu commented Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

(Step 1): Add PR label `Expert Review`